--- title: "Log analysis cut a jewelry site from 16M to 730K crawlable pages" url: https://surfil.com/blog/jetoctopus-log-analysis-parameter-crawl-waste-case-study section: studies language: en published: 2026-09-18T00:00:00.000Z modified: 2026-09-25T00:00:00.000Z author: Adam Hafez topics: ["Technical SEO", "Indexing"] --- # Log analysis cut a jewelry site from 16M to 730K crawlable pages ## The short answer JetOctopus's case study describes an unnamed jewelry shop whose category, filter, pagination and sort combinations produced about 16 million crawlable pages while relevant pages stayed unindexed for months. After log analysis, the agency recommended closing those combinations, and the site shrank to 730,000 pages, giving Googlebot far fewer URLs to spend its crawl on. ## Key takeaways - JetOctopus's case study reports a jewelry online shop with over 16 million pages but only 342,000 monthly visits, an extreme mismatch. - Profitable pages on that site were not indexed for months while Googlebot spent its resources on canonical-tagged filter and pagination URLs. - The recommendation was to close category, filter, pagination and sort combinations, which cut the site to 730,000 pages. - The same article shows a real estate site where 23.5 million of 38 million monthly Googlebot requests went to AJAX content. - The client is unnamed and JetOctopus sells the log analyzer used, so treat this as the vendor's own account, not an audited result. Server logs record what Googlebot actually requested, not what a crawler simulation guesses. JetOctopus, a log and crawl analysis vendor, published three short case studies from that data in one article by Serge Bezborodov, dated May 27, 2020. This piece covers the one with the largest measured change, and notes what the article does not say. ## What did the logs show? The site had more than 16 million pages and 342,000 monthly visits. One part of it ranked at the top of the results, while another part was not indexed for months, and those were relevant, profitable pages rather than outdated ones. JetOctopus crawled 2 million pages, found almost 900 internal links on each, and estimated around 16 million pages in total, at which point it paused the crawl to work out why the site was so large. It then analyzed the logs and found Googlebot spending resources on pages carrying rel=canonical tags, because the site had placed a canonical or self-canonical on every item in every category, filter and paginated page. The lesson the article draws is that a canonical tag does not stop crawling. Googlebot still has to fetch a URL to read the tag, so millions of canonicalized filter and sort variants still cost crawl requests. ## What changed, with what result? The recommendation was to close the combinations of category, filter, pagination and sort that together generated the bulk of those pages. The source's stated result is that the site's size decreased to 730,000 pages, so the bot could crawl the most profitable pages. That is a cut of about 95% in page count. The article gives no crawl-request counts, indexed-page counts or traffic figures for the site after the change, and no timeframe. We therefore report only the page-count change. ## Was robots.txt the fix? Not for this site. The recommendation says "close" the combinations without naming the mechanism. The article's takeaway does discuss trade-offs: Google recommends noindex over blocking, but on very large e-commerce sites the bot still spends crawl budget fetching a page to read that directive, whereas robots.txt can stop crawling of some sections entirely. ## What Google's docs say now Google's current crawl budget guide is written for large sites of 1 million or more unique pages that change about weekly, medium sites of 10,000 or more pages that change daily, and sites with many URLs stuck as "Discovered - currently not indexed". A 16 million page catalog is squarely in scope. On the noindex question the guide has since become explicit. It says not to use noindex for this, because Google will still request the page and then drop it, wasting crawling time. It recommends consolidating duplicate content and blocking crawling of URLs you do not want crawled with robots.txt, and it adds that robots.txt should not be used to temporarily reallocate crawl budget. Our research note on [noindex and crawl budget](/blog/noindex-crawl-budget-reality) covers that guidance in detail. ## What the other cases show On a marketplace with 226,000 pages and 4 million monthly visits, filter URLs were disallowed with a `Disallow: /*?` rule, yet duplicates stayed in Google's index. The cause was a conflicting `Allow: /product` rule that matched the same URLs. Google's robots.txt documentation explains the logic: crawlers use the most specific rule by path length, and when rules conflict, Google uses the least restrictive one. JetOctopus's advice was to test directives with a validator before shipping them, since each search engine's rules can differ. On a real estate site with over 1 million pages and 4.5 million monthly visits, logs showed 38 million Googlebot requests in a month, of which 23.5 million went to AJAX content, about 62% of the total. The article notes that when Google renders content loaded by XHR POST requests, each sub-request counts against crawl budget. The recommended fix was blocking valueless AJAX in robots.txt, with a warning that careless blocking can strip content a page needs. Neither case reports an after figure. ## What to check in your logs - Share of Googlebot requests going to parameter, filter, sort and paginated URLs versus indexable pages. - Whether canonicalized variants are still fetched heavily, which means the tag is not saving crawl. - Any robots.txt allow rule that overlaps a disallow pattern; test sample URLs with a [robots.txt tester](/tools/robots-txt-tester). - Requests to AJAX or API endpoints that return fragments rather than pages. The vendor sells the log analyzer used here and names no client, so treat all three cases as JetOctopus's own account. ## Sources 1. [How Googlebot Crawls Your Pages. Logs Insights](https://jetoctopus.com/cases/a56-how-googlebot-crawls-your-pages-logs-insights.html) - JetOctopus (primary)