Skip to content
metryn

Tools

Which AI crawlers does your robots.txt allow?

Enter a site, or paste a robots.txt, and see what the file declares for each AI crawler: whether it is there to search, to train a model, or to fetch a page on a user’s request. Then build the rules you want and copy them.

Generate robots.txt rules

Pick a preset or set each bot yourself. The text below is meant to be added to your existing robots.txt. It does not replace anything you already have, so check it does not conflict with your current groups.

Search
Training
User-requested fetch

The operator says robots.txt may not apply to this one.

The operator says robots.txt may not apply to this one.

The operator says robots.txt may not apply to this one.

robots.txt rules

# AI search crawlers: allowed
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# AI training crawlers: opted out
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: meta-externalagent
Disallow: /

Not generated on purpose: a rule for Googlebot (that is a Search decision), Crawl-delay (Google ignores it) and IP blocking (a blocked bot cannot read your robots.txt).

What this checks

  • The robots.txt at the origin you enter (https://yourdomain.com/robots.txt), or the text you paste, matched against each crawler’s user-agent token the way RFC 9309 describes: the most specific group wins, then the longest matching rule.
  • A missing file (404) means no restrictions. A file that cannot be read (403, 5xx, timeout) is shown as unknown and never as allowed.
  • For a few bots the operator says robots.txt may not apply (ChatGPT-User, Perplexity-User, meta-externalfetcher). Those rows carry that note, because a Disallow there is a statement of intent, not a control.

What it does not measure

  • What any crawler actually did. This is the declared policy, not a log.
  • Whether a firewall or CDN blocks a bot even though robots.txt allows it. The free scan covers more of the page, though it cannot see your server logs either.
  • Rankings, citations or traffic. Allowing a crawler does not mean you will appear, and blocking one does not remove you from other sources.
  • Googlebot and bingbot are listed for reference. Blocking them is a Search decision, so the generator does not write rules for them.

Sources

Bot names and behaviour were read on 2026-10-13 from the operators’ own pages. They change, so check the page before you rely on a detail.

robots.txt is one of several things that decide whether a page is reachable and indexable. The free scan checks the rest in a few seconds.

Run the free scan