Store filters and faceted navigation for crawlers

Filters are good for shoppers and hard on crawlers. Colour, size, brand, price band and sort order are each harmless alone. Combined, they generate more URLs than you have products, and a crawler has to fetch each one before it knows it's a near-copy.

First decide which filtered pages deserve to exist in search

Most don't. A page for red + size 42 + sort by price has no search demand worth chasing. A page for a brand inside a category might. Google separates the two cases cleanly, and your choice drives everything else.

If they shouldn't be indexed

If some should be indexed

Use & as the parameter separator, since commas, semicolons and brackets are hard for crawlers to recognize. In path-based filters, keep the order of filters consistent and stop duplicate filters from being valid. And return a 404 for combinations with no results, including nonsensical combinations and pagination that doesn't exist, rather than a soft page that says no results with a 200 status.

What a parameter block can look like

Google's example blocks filter parameters while leaving item pages and an unfiltered listing crawlable. In robots.txt terms that means disallowing URLs that carry the filter parameter and leaving everything else alone. The names below are placeholders. Use the parameters your platform really produces, and test before you publish, because a pattern that's too broad can hide real product pages.

User-agent: *
Disallow: /*?*color=
Disallow: /*?*size=
Disallow: /*?*sort=

Pagination and sorting deserve the same thought as filters. Google advises giving each page in a paginated series its own URL, using one consistent form (always or never ?page=1), and not linking to temporary values such as session IDs or the current time, since those create endless URLs that look different but say the same thing.

Things to check on your own store

• Does a sort parameter change the URL, and is that URL linked from category pages?
• Can you reach the same filtered set by two orders of the same filters?
• Does your sitemap list filtered URLs? Sitemap inclusion is a hint about what you consider canonical, so it shouldn't list pages you've asked crawlers to avoid.
• Do internal links use the clean URL for each product, or a version with tracking parameters?

Questions people ask

Will blocking filters in robots.txt hurt my rankings?

Google's own guidance is that if filtered URLs don't need to appear in search, disallowing the filter parameters is the first option, while keeping item pages and an unfiltered listing crawlable. The risk is a pattern that's too broad and catches real pages, so test it.

Is nofollow enough?

Google describes nofollow as less effective long-term, because every link to the URL has to carry it. It's a supplement, not a replacement.

What if I want some filtered pages to rank?

Then give them clean, stable URLs and keep them consistent. Google asks for & as the separator, a consistent filter order and a 404 for combinations that return nothing.

Do URL fragments help?

Google says it generally ignores fragments for crawling and indexing, so filters after a # neither help nor hurt crawling.

What to do this week

• List every parameter your store adds to category URLs.
• Decide which ones should never be crawled.
• Add those to robots.txt and test with a few URLs before publishing.
• Check that filter combinations with no results return a 404.

Sources

• Google Search Central: Managing crawling of faceted navigation URLs: https://developers.google.com/search/docs/crawling-indexing/crawling-managing-faceted-navigation
• Google Search Central: Designing a URL structure for ecommerce sites: https://developers.google.com/search/docs/specialty/ecommerce/designing-a-url-structure-for-ecommerce-sites
• Google Search Central: Consolidate duplicate URLs: https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls
• Google Search Central: Sitemaps overview: https://developers.google.com/search/docs/crawling-indexing/sitemaps/overview