--- title: "REI cut 34 million URLs down to 300,000" url: https://surfil.com/blog/rei-faceted-navigation-crawl-budget-botify section: studies language: en published: 2026-09-19T00:00:00.000Z modified: 2026-09-19T00:00:00.000Z author: Adam Hafez topics: ["Technical SEO", "Ecommerce SEO"] --- # REI cut 34 million URLs down to 300,000 ## The short answer REI's technical SEO team disclosed that its site had 34 million URLs before a cleanup cut that figure to 300,000, including URLs crawled from faceted navigation. The team then slowly reintroduced significant URLs. Botify published the account from its own customer conference and gives no traffic or ranking figures. ## Key takeaways - REI's technical SEO team found the retailer's site had roughly 34 million URLs, and Botify's recap names faceted navigation among the URL types that waste crawl budget. - The team cut that figure down to 300,000 crawlable URLs partly by cutting down the URLs crawled from faceted navigation. - REI reintroduced a smaller set of URLs afterward, described only as significant URLs. - Ryan Ricketts, REI's Technical SEO Manager, presented the account at Botify's Crawl2Convert customer summit rather than in a written case study document. - Botify's own recap of the same story does not publish organic traffic, ranking or revenue figures alongside the URL count, only the crawl-budget outcome. Faceted navigation is a well-known source of wasted crawling on large sites, and retail category pages are where the damage compounds fastest: every color, size, price band and sort order a shopper can combine has the potential to mint its own crawlable URL. REI's technical SEO team put a number on that damage at Botify's Crawl2Convert customer summit in October 2019, and Botify's own recap of the talk is the source for what follows. ## What REI found and did Ryan Ricketts, REI's Technical SEO Manager, told the summit that his team's site had roughly 34 million URLs. Botify's recap does not break that figure down. It says bots tend to waste crawl budget on unnecessary and duplicative pages and faceted navigation pages, and lists cutting down URLs crawled from faceted navigation as what Ryan's team did. Filter and sort combinations on category pages compound multiplicatively: a handful of facets, each with a handful of values, produces a URL count that has nothing to do with how many distinct products a shopper can actually reach. The team's response was to close that gap down hard: from 34 million crawlable URLs to 300,000. Botify's account does not enumerate REI's exact mechanism, canonical tags, `robots.txt` disallow rules, or `noindex` on low-value combinations, the tools our own [faceted navigation parameter analyzer](/tools/faceted-navigation-parameter-analyzer) is built to help sort between. What the recap does say is that the team then slowly reintroduced significant URLs back into the mix. ## What the source does not say This is a crawl-budget story, not a traffic story, and it is worth being precise about that gap. Botify's recap says the cleanup "did wonders for his crawl budget," a qualitative claim it does not attach a percentage, an indexed-page count, or an organic traffic figure to. No ranking data, no revenue figure and no timeframe for the reintroduction phase appear in the source either. Treat the URL count itself, 34 million down to 300,000, as the one hard number this account supports, and treat everything else in it as directional description rather than a measured result. ## Why we care REI's number is a useful gut check for how large a retailer's crawl surface can get before anyone deliberately intervenes: 34 million URLs, cut to 300,000. The two-step shape of REI's fix, cut hard first, then slowly reintroduce the significant URLs, is an order of operations worth considering before touching a single filter on a live category page. The recap does not say how REI chose which URLs came back, so use your own search-demand data to decide what earns its way back to a crawlable URL. ## Common questions ### Did REI publish its own case study with these numbers? No. The 34 million to 300,000 figure comes from a talk REI's Ryan Ricketts gave at Botify's Crawl2Convert customer summit in October 2019, which Botify then recapped on its own blog. There is no separate REI-authored case study document behind it. ### Does the source give a traffic or ranking result for REI's cleanup? No. Botify's recap states the URL-count outcome and says the cleanup did wonders for crawl budget in general terms, but it does not publish an organic traffic, ranking or revenue number tied to this specific project. ### What actually drove REI's site to 34 million crawlable URLs? The recap does not break down the 34 million. It lists faceted navigation among the URL types that waste crawl budget and says Ryan's team cut down URLs crawled from faceted navigation, without naming which facets REI blocked or canonicalized. ### Did REI block every facet permanently? No. After the initial cut to 300,000 URLs, the team reintroduced a further set of URLs, described as significant URLs, rather than leaving the site at the reduced count indefinitely. ## Sources 1. [Top 10 Takeaways From Crawl2Convert: Botify's Inaugural Customer Summit](https://www.botify.com/blog/crawl2convert-botifys-inaugural-customer-summit) - Botify (primary)