Thousands of URLs with filter parameters in Search Console? It usually means Google’s crawlers are wasting time going through one filter combination after another. There is no penalty for that. You can sort it out with a few technical decisions. Faceted navigation makes shopping easier for customers, but left unchecked it produces an almost infinite number of pages. Below I answer three questions: which filters to block, which to keep and which Google tools to use for it.
Table of contents
What is faceted navigation and where do thousands of parameter URLs come from?
Faceted navigation is a set of filters that change the view of a product list, for example by category, color or size. Most often it works through parameters appended to the URL, such as ?color=black&size=m. And this is where the trouble starts. Every selected filter, and every combination of filters, is a separate URL. A few attributes with a dozen or so values each are enough for the number of possible URLs to snowball. Google calls this an infinite URL space. Most of these pages show essentially the same products, so the store creates its own duplicate content problem.
Are filter URLs in Search Console a problem for a store’s SEO?
There will be no penalty, but the crawl budget goes down the drain. Google’s guidelines on faceted navigation explain it: the crawler does not know whether a new URL will be useful until it visits it. So it goes through a mass of variants before it decides they are useless. The industry calls this overcrawling, meaning excessive crawling. What does that mean in practice?
- new products and categories are discovered more slowly, because the crawler is stuck in the filters,
- the server renders thousands of versions of the same list and spends computing power on it,
- reports swell with URLs that do nothing for the store’s visibility.
Indexing product filters: when to block and when to keep them?
If filtered pages do not need to be in the search results, block them from being crawled. And if they are supposed to show up there? Then they have to follow Google’s best practices. Before you decide, answer three questions honestly:
- Does anyone actually search for this combination, for example a specific color in a given category?
- Does the filtered page have unique value, or does it only rearrange the same products?
- Can the same list already be seen at a URL without filters?
Google’s documentation recommends keeping individual product pages and one list of all products without filters open to the crawler. That’s it. Every extra variant you leave open is more load on the server. In my opinion it is better to start with the minimum and add only what really makes sense.
How to block filters in robots.txt and when are URL fragments enough?
Google points to two methods that work best in the long run: disallow rules in robots.txt for filter parameters, or filters based on URL fragments (the part of the address after the # sign). Blocking in robots.txt takes four steps:
- List the filter parameters that appear in Search Console reports.
- Add disallow rules for each of them.
- Keep an allow rule for the list of all products.
- Make sure product and category pages are still accessible to the crawler.
An example based on the documentation:
user-agent: Googlebot
disallow: /*?*products=
disallow: /*?*color=
disallow: /*?*size=
allow: /*?products=all$URL fragments such as #products=fish&color=green have no effect on crawling at all. Neither positive nor negative. If you are designing new filters or rebuilding old ones, this is a good moment to consider them.
Canonical in an online store and nofollow: supporting signals
The rel=”canonical” and rel=”nofollow” attributes hint to Google which URLs are preferred. But according to the documentation, in the long run they are less effective than robots.txt and URL fragments. A canonical on the filtered version, pointing to the list without filters, can reduce crawling of non-canonical URLs over time. Over time. Not right away. And the crawler still has to visit the page to read the canonical at all. Nofollow on links to filters? Good as an extra signal that supports the rest, just don’t make it your only safeguard when there are many parameters.
Where to start cleaning up filters in your store?
First comes the list of parameters and the decision on which filters should be indexed. Only then do you touch robots.txt and canonicals. The implementation itself depends on the platform, because every engine builds filter URLs in its own way. Stores built on WooCommerce are configured differently from PrestaShop store implementations. Still planning your store? Then think about filters when you choose the engine (it costs more later). Our guide on choosing an online store platform will come in handy. And when there are tens of thousands of URLs, or you are not sure what can be blocked safely, order a technical SEO store audit.
Well-configured faceted navigation does two things at once: shoppers find the product quickly, and crawlers don’t get bogged down in filters.
Frequently asked questions
Do store filters hurt rankings in Google?
The filters themselves do not. The trouble is the huge number of URLs, which eats up the crawler’s time and puts load on the server. As a result, new products may be discovered later.
Is it enough to set a canonical on filter pages?
A canonical helps, but according to Google it is less effective than robots.txt in the long run. With a large number of parameter URLs it is better to block the parameters and treat the canonical as an extra signal.
Can I block all filters in robots.txt?
Yes, if you do not need filtered pages in the search results. One condition: product pages and the list of all products without filters must stay accessible to the crawler. After the change, look into Search Console and check that you have not blocked something important.
