Skip to content
metryn

Password pages and firewalls that block crawlers

The ways a store ends up closed to crawlers without anyone deciding it: private mode, firewall bot rules, an edited robots.txt, and how to find which one it is.

By Metryn · Updated 17 October 2026 · 5 min read

Most stores that can't be read by crawlers didn't choose it. Someone switched something on for a good reason and forgot, or a setting changed in an update. This guide lists the three usual places, in the order I'd look.

1. Private mode and the password page

Shopify's help page says that when private mode is active, all of your online store's pages are hidden from visitors and search engines, and the password page is what visitors see. That's what you want while building the store. It's a problem when it stays on after launch, or comes back on after maintenance.

The test is quick and takes no tools. Open your store in a private browser window, where you aren't logged in. If you see a password box, so does every crawler. The same goes for other platforms that have a coming-soon or maintenance mode: check it from a logged-out window.

2. The firewall in front of the store

This one is easy to miss, because robots.txt can say yes while a firewall says no. Bot-management rules work on the request itself, often by user agent, by network or by a score, and they run before your pages are served.

Take Cloudflare, since many stores use it. Its documentation describes a Block AI bots setting that blocks verified bots classified as crawling for AI training, plus some unverified bots that behave similarly, and says it excludes mixed-purpose bots used for both training and search. The same page says the setting was deprecated on 15 September 2026 and points to newer AI bot policies, so the label in your dashboard may differ from what I've described. Look for any option that blocks AI crawlers or bots by category, and read what it says it covers.

If you want to let a particular crawler through, Perplexity's documentation gives an example for Cloudflare: a custom WAF rule that allows requests when the user agent contains PerplexityBot or Perplexity-User and the source IP is in Perplexity's published ranges. It recommends combining the user agent with the IP check, since a user agent alone is easy to fake. OpenAI publishes its ranges too, at the addresses listed on its crawlers page.

A caution from me, not from the documentation: don't loosen a firewall rule just to make a scanner happy. Allow the specific crawlers you've decided to allow, with the IP check, and leave the rest as it was.

3. A robots.txt someone edited

On Shopify, robots.txt is generated, and Shopify says all stores have a default file that's optimal for SEO and that works for most stores. It also lets you customize the file with a robots.txt.liquid template, and calls that an unsupported customization that Shopify Support can't help with, warning that incorrect use can result in loss of all traffic. If you want the default back, Shopify says you can delete the custom template.

Open yourstore.com/robots.txt and read it as a stranger would. A Disallow: / under User-agent: * shuts everyone out. A short list of named bots under Disallow: / shuts out those bots. If you don't recognise a rule, find out who added it, because it's often an SEO app or a developer you hired a year ago.

4. An app or theme change

This is more suspicion than documented fact, so I'll label it that way: apps that handle SEO, speed, security, geolocation redirects or age checks can all change what a crawler receives. I can't point to an official list. What I can say is that the cheapest way to find out is to compare. Test a page before you install an app and after, using the checks below.

How to tell which one it is

  • Private window shows a password box: it's private mode.
  • robots.txt has a blanket Disallow or a named-bot rule: it's robots.txt.
  • robots.txt looks fine but a crawler-like request gets a 403, a challenge page or a different page: it's probably the firewall or a bot rule. Google's documentation lists HTTP status codes and what each means for crawling.
  • The page loads but the product details are missing from view-source: it's JavaScript rendering, not blocking.

Our free scan fetches the page from our servers, which can show a block that your own browser never meets. It isn't proof of what OpenAI's or Perplexity's networks see, because their requests come from their own address ranges.

Questions people ask

I fixed it. How long until crawlers notice?

OpenAI and Perplexity both say robots.txt changes can take up to about 24 hours to be reflected. Firewall changes may take a while to propagate too, so look again the next day.

Can I allow ChatGPT-User and Perplexity-User through robots.txt?

Both operators say user-initiated fetches may not follow robots.txt, so the rules there aren't the whole story. What stops them is more likely to be a firewall.

Is it safe to allow every AI crawler?

That's a business decision. Some stores want to appear in AI search but not in training sets. OpenAI separates those two with different crawlers, which makes it possible.

What to do this week

  • Open the store in a private window and confirm there's no password page.
  • Read your robots.txt line by line.
  • Open your CDN or firewall settings and look for rules about bots or AI crawlers.
  • Write down what you found and the date, so changes are easier to spot next month.

Check your own site

Paste your address below. The free scan looks at robots.txt, your sitemap, access for the main AI crawlers and structured data, and shows what it found in seconds. It tells you what a machine can fetch and read. It doesn't predict rankings or citations.

Free scan · No account needed · Results in seconds

Sources

Keep reading

Running a store? See Metryn for online stores for what the scan checks and the other guides written for store owners.

Want to check one file yourself? Try the free AI crawler robots.txt checker or the llms.txt validator, or read how the score is built.