Skip to content
metryn

AI search readiness for a SaaS site: where to start

A checklist for SaaS sites: crawler access, pages that render without JavaScript, sitemaps and structured data, per Google, OpenAI and Perplexity docs.

By Metryn · Updated 16 October 2026 · 5 min read

Most SaaS sites lose visibility for boring reasons. A pricing page that only exists after JavaScript runs, a docs subdomain with its own robots.txt that nobody looked at, a sitemap that stopped updating a year ago. None of that needs a new strategy. It needs someone to check.

Start with the crawlers that decide whether you can appear at all

OpenAI documents separate crawlers for separate jobs. OAI-SearchBot is the one used to surface sites in ChatGPT's search features, and OpenAI says sites that opt out of it won't appear in ChatGPT search answers, though they can still show up as navigational links. GPTBot is a different question: it's tied to training, and each of these robots.txt settings is independent, so you can allow one and disallow the other.

Perplexity draws the same line. PerplexityBot is what surfaces and links sites in its search results, and Perplexity says it isn't used to crawl content for foundation models. Perplexity-User fetches pages when someone asks a question, and Perplexity says it generally ignores robots.txt for that reason. Both OpenAI and Perplexity mention that changes to robots.txt can take roughly a day to be picked up, so don't judge a change five minutes after you make it.

Check this on every host you run, not only the marketing site. Docs, status pages and app subdomains each serve their own robots.txt, and a blanket Disallow: / copied from a staging setup is a classic way to hide the part of the product people most want to read.

Make sure the important pages exist in the HTML

Google's documentation is plain about this: it renders JavaScript, but still recommends server-side rendering or pre-rendering because, in its words, not all bots can run JavaScript. Pricing, features, integrations and docs are the pages other systems most need to read. If they're an empty shell until a bundle loads, you're relying on every crawler to execute your app.

A quick test: fetch the page with a plain HTTP client and look for the pricing numbers in the response. If they're not there, that's your answer. Also make sure internal navigation uses real <a href> links rather than click handlers or #/ routes, and that a missing page returns a real 404 instead of a friendly page with a 200 status.

Give them a sitemap, but don't expect it to do the work

Google describes a sitemap as a hint. It says it doesn't guarantee everything listed will be crawled or indexed, and that small sites with solid internal linking may not need one. For a SaaS site with a blog, docs and changelog it's still worth having, as long as it lists the canonical URLs you actually want found and gets regenerated when you publish.

Structured data: be careful with the rich result

Google supports SoftwareApplication markup, with WebApplication as a subtype for web apps. Its documentation lists name, offers.price and either aggregateRating or review as required for the rich result. That last one matters: an app with no genuine rating or review isn't eligible, and inventing one is the wrong fix. If you have real reviews, mark them up. If you don't, the markup is still harmless context, but don't expect a rich result from it.

Skip the magic files

On Google's own AI features (AI Overviews and AI Mode), the documentation says there are no additional requirements beyond being indexed and eligible to show a snippet, and that no special optimization, new machine-readable file or AI-specific markup is needed. The fundamentals carry most of the weight: crawlable, linked, text content, structured data that matches what's on the page.

Questions people ask

Should the docs live on a subdomain or a subfolder?

The documentation I read doesn't say one is better. What it does say is that each hostname serves its own robots.txt, so a subdomain needs its own check. Whichever you pick, make sure the docs are crawlable, linked from the main site and in a sitemap.

Is gating content behind a login a problem?

A crawler can't log in. Google's documentation treats login-protected pages as 401 responses that are not indexed. Anything you want found, such as pricing, features and integration pages, shouldn't sit behind a sign-in.

Do changelog and status pages matter?

They're ordinary pages, and crawlers read them like any other. They're useful as long as they're linked and accurate. A status page that only works with scripts will be as empty to a crawler as any other client-rendered page.

What about an app that's all client-rendered?

Keep the marketing and documentation pages server-rendered or pre-rendered, and let the app itself be whatever it needs to be. The pages people search for are almost never the logged-in screens.

What to do this week

  • Fetch your pricing page with curl and confirm the plan names and prices are in the response.
  • Open the robots.txt of your marketing site, docs and any app subdomain.
  • Check that the sitemap includes your docs and your feature pages.
  • Replace any 'pricing on request' page that hides the numbers with at least a range, if you can.

Check your own site

Paste your address below. The free scan looks at robots.txt, your sitemap, access for the main AI crawlers and structured data, and shows what it found in seconds. It tells you what a machine can fetch and read. It doesn't predict rankings or citations.

Free scan · No account needed · Results in seconds

Sources

Keep reading

Want to check one file yourself? Try the free AI crawler robots.txt checker or the llms.txt validator, or read how the score is built.