Docs

OAI-SearchBot and PerplexityBot: checking access

Why these two specifically

OAI-SearchBot and PerplexityBot are both search-discovery crawlers — their purpose is surfacing your pages in answers, not training a model. That's a different question from whether you allow a training crawler like GPTBot: blocking a training crawler is a content-licensing choice, blocking a search crawler is a visibility one.

Their real HTTP identity

Checking access yourself

Fetch /robots.txt and confirm no Disallow rule applies to either crawler for the pages you want discovered — that tells you the declared policy. It does not tell you whether a request from that crawler actually reaches your server: a WAF, CDN, or bot-management rule can block a request before your application ever sees it, and robots.txt has no way to detect that. Checking real delivery requires your own server-side access logs, filtered for the User-Agent strings above.

What Metryn checks for you

Every scan checks the declared robots.txt policy for both crawlers. An experimental, explicitly-labeled simulated access probe exists separately — when enabled, it makes a real request using each crawler's own User-Agent string, but that request originates from Metryn's infrastructure, never the operator's, so it's reported as simulated, not as proof of what the genuine crawler would experience. See /methodology for the full confidence-level breakdown.