Faceted navigation: which filter URLs deserve to be indexed

Two people on the same e-commerce team usually end up arguing about filter pages, and both of them are right. The developer points at crawl budget going into URLs that will never rank. The marketer points at long-tail traffic that only a filtered page can win. Neither of them has run the numbers on their own catalog, so the argument never resolves.

The numbers are not close. A single category with six filters can generate more URLs than the store has products several hundred times over. Google has documented the problem for years and describes faceted navigation as an "infinite URL space." Most of those URLs have no search demand behind them. A small number do, and those deserve real pages. The work is separating the two groups, then making sure the controls you apply actually do what you think they do.

The URL math nobody runs before launch

Take one category with six filters: 12 colours, 10 sizes, 20 brands, 5 price bands, 6 materials and 4 sort orders. Allow a shopper to pick one value per filter and that category alone produces about 630,000 combinations. Let two colours be selected at once, or let the same filters appear in any order in the query string, and the count stops being something you can hold in your head.

A crawler cannot tell a useless combination from a valuable one until it fetches the page. That is the whole mechanism behind overcrawling. Googlebot finds thousands of URLs that look new, spends requests confirming they are near-duplicates of the category, and reaches your new product pages later than it should. The second-order effect is worse than the first: discovery slows for everything, including the pages that pay the bills.

Sort orders and page-size parameters make this worse in a way that is easy to miss. Sorting by price low to high is not a facet a shopper searches for, but it multiplies every category by the number of sort options. The same holds for grid versus list view, and for pagination past the point where anyone clicks.

Crawling and indexing are two separate decisions

robots.txt decides whether Googlebot fetches a URL. Canonical tags and noindex decide whether a fetched URL can appear in results. Those are different questions with different answers, and most of the broken setups we audit have the right tools combined in the wrong order.

The order matters because a blocked URL is never read. If you disallow a filter parameter in robots.txt and also put noindex on the page, the noindex never fires. Google never sees it. The page leaves the crawl queue, which is what you wanted, but it can still appear in the index without a snippet if other sites link to that URL.

Three checks that tell you whether this is your problem

Google's crawl budget guidance is written for sites with more than a million pages, or ten thousand pages that change daily, or a large pile of URLs sitting in "Discovered, currently not indexed." A store with a few thousand products can land in that third bucket purely because its filters multiply the URL count past the catalog size, which means you do not need a million products to have a crawl budget problem.

• Crawl stats. In Search Console, open Settings and then Crawl stats. Compare daily requests against the number of products and categories you actually sell, then read the example URLs under each response type. Filter parameters showing up in the examples is the tell.
• Page indexing. A large and growing "Discovered, currently not indexed" count, populated mostly by filter URLs, means Google keeps postponing the crawl. Google is deferring them rather than rejecting them outright.
• Server logs. Group Googlebot requests by URL pattern. If filter and sort URLs take a bigger share of requests than product pages, crawling is going to the wrong place and you can stop guessing.

The symptom shoppers notice is slower indexing. New products take longer to appear, price and stock changes take longer to register, and a seasonal launch reaches search two weeks after it went live on site.

Where the long-tail demand actually lives

Filter demand drops off fast as filters stack. Here is the US search volume ladder for one footwear category, pulled from Ahrefs in October 2026:

• ankle boots: 27,000 searches a month
• black ankle boots: 5,400
• black leather ankle boots: 1,200
• black leather ankle boots size 8: 0

The pattern repeats elsewhere. "Mens linen shirts" gets about 8,200 searches a month and "blue linen shirt men" gets around 300. "Green velvet sofa" gets 3,300. One attribute on top of a category often carries real demand. A second one sometimes does. Size, price and availability on top of that almost never do, because the person searching for a size is already on your category page with a filter applied.

Five tests for a filter page that deserves an index

Run every candidate combination through these in order. The first two decide most cases.

1. Search demand. Check the phrase a shopper would type in a keyword tool for your main market. Pick one threshold for the whole catalog, say 50 searches a month, and apply it identically to every candidate. Changing the threshold per category is how you end up with 400 "indexable" pages and no way to explain them.
2. Different intent from the parent. Someone searching "green velvet sofa" wants green velvet sofas and is disappointed by the full sofa category. Someone searching "sofas under $1,000" is usually happy on the category page with a price filter applied. Colour, material, style, brand and use case tend to pass this test. Size, price and availability tend to fail it.
3. Enough products, consistently. The page needs to fill at least the first screen and stay in a range that does not empty out every few weeks. A filter page that drops to zero products becomes a soft 404, and soft 404s cost more than the page ever earned.
4. One or two filters deep. A category plus one attribute, or plus two at most, like brand and type. Three or more filters almost never pass the demand test, so treat depth as a shortcut for rejection.
5. A page you can own. One clean URL, its own title, H1 and short intro copy, and links from the parent category. If your platform can only produce the combination as a parameter URL whose order changes, build the combination as a real page instead of trying to optimise the parameter URL.

Everything that fails lands in one of two groups. Single filters with no demand can stay crawlable with a canonical pointing at the category, provided the set stays small. Deeper combinations, sort orders, page-size options and price ranges get blocked from crawling.

What each control actually does

• robots.txt disallow. Stops crawling and saves server resources. It does not reliably stop indexing, because a URL linked from elsewhere can be indexed without content. It passes no signals to the category.
• Canonical to the category. Passes ranking signals and, over time, reduces how often the filtered versions are crawled. Google treats it as a suggestion rather than a command, and says robots.txt is the more effective long-term control. Use it for a small set of single-filter URLs, not for thousands.
• Meta robots noindex. Stops indexing and saves no crawling at all. Google still has to request the page to read the tag, then drops it from the index. For a catalog with a hundred thousand filter URLs, this is the wrong tool.
• nofollow on filter links. Only works if every anchor pointing at the URL carries it. Treat it as a support measure, never as the main control.
• 404 for empty combinations. The correct response for a filter that returns no products. It should be a real 404 at that URL, not a redirect to a generic error page.
• Filters that never change the URL. Fragment-based or JavaScript-state filters are invisible to crawlers, which makes them the cheapest option for facets you never want as pages. The category stays crawlable and the combinations never exist as URLs.

Four combinations that cancel each other out

A broken faceted setup usually has the right tools applied in ways that contradict each other.

• robots.txt block plus noindex. Google cannot crawl the page, so it never reads the directive. The noindex is decoration.
• noindex plus a canonical to the category. One tag says drop this page, the other says merge it into the category. Google advises against using noindex for canonicalisation, so pick the canonical and leave the page crawlable.
• A canonical to the category on a URL that sits in your sitemap or main menu. Your internal links point at the filter URL while the tag points away from it. When the signals conflict, Google may ignore the tag and index the filtered version anyway.
• Blocking URLs that are already indexed. They can sit in the index without content for months. The clean sequence is to allow crawling with noindex until they drop out, then add the robots.txt rule.

Building a filter page that can rank

A combination that passed all five tests has to behave like a category page, or Google will treat it as a near-duplicate of the parent and keep the parent in the index instead.

• A clean, stable path such as /boots/black-leather-ankle-boots. If parameters are unavoidable, keep one fixed order and use the standard ampersand separator.
• A self-referencing canonical. Never point an indexable page's canonical at its parent.
• Its own title, H1 and meta description built on the phrase people search.
• A short intro that helps someone choose between the products shown, covering material, fit or care rather than restating the category name.
• Internal links from the parent category, plus breadcrumbs back to it.
• A place in the XML sitemap next to the regular categories.
• A plan for when stock runs out. Keep the page live with related products while the range is seasonal, and return a 404 or redirect to the parent if it is gone permanently.

The links matter as much as the copy. A sitemap entry is a weak signal on its own, so a filter page that only the sitemap points at gives Google little reason to prefer it over the category it came from.

Platform notes: Magento and Shopify

Magento and Adobe Commerce. Filters are appended as query parameters with numeric option IDs, so a filtered page looks like /women/boots.html?color=49&material=148. That is easy to block and hard to turn into a page worth indexing. Enable "Use Canonical Link Meta Tag For Categories" under Stores, then Configuration, then Catalog, then Catalog, then Search Engine Optimization. Add robots.txt rules for each filterable attribute plus the toolbar parameters product_list_order, product_list_dir, product_list_limit and product_list_mode. For combinations that passed the five tests, build a real subcategory with its own URL key, metadata and description, then assign products to it. Visual Merchandiser can populate it by rule, though the rule runs when the category is saved, so products that stop matching only drop out on the next save.

Shopify. Storefront filters from the admin or the Search & Discovery app add parameters like ?filter.v.option.color=Black or ?filter.p.vendor=Nike, and a store can run up to 25 filters. Shopify's default robots.txt already blocks a set of these paths, and the platform handles the canonical case for simple filters. It does not automatically canonicalise multi-filter combinations. That responsibility sits with your theme or app layer, which is the gap most Shopify stores fall into once merchandising starts adding filters.

Write the list down and give it an owner

The output of this work is a short document: the combinations allowed to be indexed, the parameters that are blocked, and the person who decides when either list changes. Without it, every new filter the merchandising team adds becomes a crawl and index decision that nobody made. Six months later the list has grown to include three-way combinations with no demand, and the crawl stats look the way they did at the start.

The cost of skipping this is not dramatic, which is why it goes unfixed. New products take a few days longer to be discovered. Price changes lag the same way. Category pages compete with near-identical twins for the same query and neither wins convincingly. None of that shows up as an outage, so it never gets prioritised, and the catalog keeps growing on top of a URL architecture that was never designed for it.