Skip to content
metryn

Can ChatGPT and Perplexity reach your site?

The crawlers OpenAI and Perplexity document, what robots.txt does and doesn't control for each, and how firewalls and bot rules can still block them.

By Metryn · Updated 16 October 2026 · 4 min read

Two things have to be true for an AI search system to read your page. Your robots.txt must allow its crawler, and the request must actually reach your server. The first is easy to check. The second is where sites get caught.

The crawlers, as their operators describe them

CrawlerOperatorWhat the operator says it's forrobots.txt
OAI-SearchBotOpenAISurfaces websites in ChatGPT's search featuresControlled by robots.txt
GPTBotOpenAICrawls content that may be used to train foundation modelsControlled by robots.txt
ChatGPT-UserOpenAIUser-initiated actions in ChatGPT; not automatic crawlingOpenAI says rules may not apply
PerplexityBotPerplexitySurfaces and links websites in Perplexity search resultsPerplexity recommends allowing it
Perplexity-UserPerplexityVisits a page when a user asks a questionPerplexity says it generally ignores robots.txt

The practical reading: if you want to be findable in ChatGPT search, OAI-SearchBot is the one that matters, and OpenAI points to it, not ChatGPT-User, for search opt-outs. Blocking GPTBot is a training decision and doesn't by itself remove you from search. OpenAI says that if both are allowed it may use a single crawl for both purposes.

Step one: read your robots.txt

Open /robots.txt on every hostname you serve and look for Disallow rules that apply to these user-agents, directly or through User-agent: *. robots.txt is a convention defined in RFC 9309. It declares what you want and doesn't enforce anything.

Step two: the part robots.txt can't tell you

A web application firewall, CDN rule or bot-management setting can reject a request before your application ever sees it, and robots.txt has no way to reveal that. Perplexity's documentation recommends matching both the user-agent string and its published IP ranges when you configure a WAF, and refreshing those IP lists regularly. OpenAI publishes IP lists per crawler too, for example for OAI-SearchBot and GPTBot.

The only way to know what really arrives is your own server or CDN logs, filtered by those user-agent strings. If you see the crawler requesting /robots.txt and then your pages with 200 responses, access works. If you see 403 or challenge pages, a firewall rule is the first place to look.

After you change something

Both operators say changes aren't instant. OpenAI says it can take around 24 hours for its systems to adjust to a robots.txt update, and Perplexity says up to 24 hours too. Make the change, wait a day, then check the logs.

One more reminder from Google's AI features documentation, because it's the same principle: for Google's own AI features, being crawlable and indexable is the requirement, and no special file or markup is needed on top of it.

Questions people ask

What if I want AI assistants to find me but not train on my content?

OpenAI's documentation describes separate settings for that: allow OAI-SearchBot for search, and disallow GPTBot if you don't want content used for training. Perplexity states that PerplexityBot isn't used to crawl content for foundation models.

Why do some assistant requests ignore robots.txt?

OpenAI says ChatGPT-User is for user-initiated actions and that robots.txt rules may not apply, and Perplexity says Perplexity-User generally ignores robots.txt because a person asked for the page. These aren't automatic crawlers.

Do I need to allow their IP ranges?

Both recommend it if you run a firewall. OpenAI publishes a list per crawler, and Perplexity recommends matching the user-agent and IP ranges together and refreshing them regularly.

How do I know it works?

Your own logs. Filter by the user-agent names, then look for requests to /robots.txt followed by page requests with a 200 response.

What to do this week

  • Read your robots.txt for rules that apply to the five crawler names above.
  • Check your CDN or firewall for rules that challenge unknown bots.
  • Add the published IP ranges to an allow rule if you use a WAF.
  • Check your logs after 24 hours.

Check your own site

Paste your address below. The free scan looks at robots.txt, your sitemap, access for the main AI crawlers and structured data, and shows what it found in seconds. It tells you what a machine can fetch and read. It doesn't predict rankings or citations.

Free scan · No account needed · Results in seconds

Sources

Keep reading

Want to check one file yourself? Try the free AI crawler robots.txt checker or the llms.txt validator, or read how the score is built.