How AI crawlers interact with robots.txt
What robots.txt controls
robots.txt is the Robots Exclusion Protocol, standardized as RFC 9309. It lives at a site's root (/robots.txt) and states, per user-agent, which paths a crawler may or may not request. It's a voluntary convention, not an access-control mechanism — a crawler that ignores it isn't breaking anything technical, just an agreed-upon norm that most major operators say they honor.
AI crawlers, and whether robots.txt reliably governs them
| Crawler | Purpose | Governed by robots.txt? |
|---|---|---|
OAI-SearchBot | Search discovery | Yes, per the operator's own documentation |
GPTBot | Model training | Yes, per the operator's own documentation |
ChatGPT-User | User-requested fetch | Not guaranteed — the operator states robots.txt rules may not apply to this crawler |
ClaudeBot | Model training | Yes, per the operator's own documentation |
Claude-User | User-requested fetch | Yes, per the operator's own documentation |
Claude-SearchBot | Search discovery | Yes, per the operator's own documentation |
PerplexityBot | Search discovery | Yes, per the operator's own documentation |
Google-Extended | Model training (Gemini / Vertex AI) | Yes, per the operator's own documentation |
Googlebot | Search indexing | Yes, per the operator's own documentation |
bingbot | Search indexing (also feeds Copilot) | Yes, per the operator's own documentation |
The one exception in the table above, ChatGPT-User, is a user-initiated fetch: OpenAI's own documentation states that because the request was triggered by a person asking ChatGPT to open a specific page, robots.txt rules may not apply the same way they do for its automated crawlers.
A minimal explicit-allow example
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
Sitemap: https://example.com/sitemap.xml/llms.txt is a separate, optional file and not part of robots.txt. Metryn reports whether it is present at 0 points and does not score it. Chrome's Lighthouse has an experimental audit for it, which treats a missing file as not applicable; the details are in https://getmetryn.com/guides/how-to-write-an-llms-txt-file
See /methodology for how a blocked or partially-blocked crawler affects a Metryn score, and /scanner-info for how Metryn's own scanner identifies itself in your logs.