REI cut 34 million URLs down to 300,000

Faceted navigation is a well-known source of wasted crawling on large sites, and retail category pages are where the damage compounds fastest: every color, size, price band and sort order a shopper can combine has the potential to mint its own crawlable URL. REI’s technical SEO team put a number on that damage at Botify’s Crawl2Convert customer summit in October 2019, and Botify’s own recap of the talk is the source for what follows.

What REI found and did#

Ryan Ricketts, REI’s Technical SEO Manager, told the summit that his team’s site had roughly 34 million URLs. Botify’s recap does not break that figure down. It says bots tend to waste crawl budget on unnecessary and duplicative pages and faceted navigation pages, and lists cutting down URLs crawled from faceted navigation as what Ryan’s team did. Filter and sort combinations on category pages compound multiplicatively: a handful of facets, each with a handful of values, produces a URL count that has nothing to do with how many distinct products a shopper can actually reach.

The team’s response was to close that gap down hard: from 34 million crawlable URLs to 300,000. Botify’s account does not enumerate REI’s exact mechanism, canonical tags, robots.txt disallow rules, or noindex on low-value combinations, the tools our own faceted navigation parameter analyzer is built to help sort between. What the recap does say is that the team then slowly reintroduced significant URLs back into the mix.

What the source does not say#

This is a crawl-budget story, not a traffic story, and it is worth being precise about that gap. Botify’s recap says the cleanup “did wonders for his crawl budget,” a qualitative claim it does not attach a percentage, an indexed-page count, or an organic traffic figure to. No ranking data, no revenue figure and no timeframe for the reintroduction phase appear in the source either. Treat the URL count itself, 34 million down to 300,000, as the one hard number this account supports, and treat everything else in it as directional description rather than a measured result.

Why we care#

REI’s number is a useful gut check for how large a retailer’s crawl surface can get before anyone deliberately intervenes: 34 million URLs, cut to 300,000. The two-step shape of REI’s fix, cut hard first, then slowly reintroduce the significant URLs, is an order of operations worth considering before touching a single filter on a live category page. The recap does not say how REI chose which URLs came back, so use your own search-demand data to decide what earns its way back to a crawlable URL.