Tools

Free tools for AI crawler access

Each tool does one thing, needs no account and stores nothing you submit. They report what a file declares, whether it is well-formed, or what a page returns. They do not predict rankings, citations or traffic.

• AI crawler robots.txt checker and generator

  Reads a robots.txt and shows what it declares for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and the other AI crawlers. Also writes the rules for you to copy.

• llms.txt validator

  Checks an llms.txt against the llmstxt.org format and against Lighthouse's llms.txt audit, and tests that its links respond.

• Sitemap checker

  Finds the sitemap in robots.txt or at /sitemap.xml and reports its type, URL count, lastmod coverage, size, gzip and the status of the first child sitemaps.

• What a crawler without JavaScript sees

  Requests one page as a plain HTTP client and lists its title, H1, description, canonical, JSON-LD types and visible text length, and whether a Markdown version is served.

When you need more than one file

The tools above read one file or one page each. The free scan looks at a page as a whole: crawler access, indexability, sitemap, structured data and how the content reads in raw HTML. The methodology lists every check and what it does not measure.