AI crawler robots.txt checker for GPTBot, ClaudeBot and more
Paste a robots.txt to see which AI crawlers it blocks: GPTBot, ClaudeBot, Google-Extended, PerplexityBot and more. Free, runs in your browser.
SEO-friendly URL checker: test a URL's separators, parameters, length, depth and opaque IDs against Google's URL guidance. Free, runs in your browser.
Runs entirely in your browser. Nothing you enter is uploaded, logged or stored.
Paste one full URL and this checker parses it with the browser’s own URL constructor, then runs a fixed set of checks against the parts. Each result carries a label: “Google documented” when it restates Google’s guidance, “Heuristic” when it is this tool’s own rule of thumb.
| Check | Label | Warns when |
|---|---|---|
| HTTPS | Google documented | the scheme is not https |
| Total length | Heuristic | never, it is a count only |
| Path depth | Heuristic | more than 4 path segments |
| Word separator | Google documented | underscores, or a mix of hyphens and underscores |
| Query parameters | Google documented | more than 1 parameter |
| Opaque id segments | Heuristic | a segment looks like an ID |
| Path readability | Heuristic | under 60% of segments look like words |
The Google-documented checks come from two pages. The URL structure guide recommends readable words over long ID numbers, hyphens over underscores, and trimming parameters that do not change the content. The canonicalization guide says Google prefers HTTPS pages over equivalent HTTP pages as canonical.
The URL structure guide gives no character limit and no maximum directory depth. Those numbers are often repeated as fact, so this tool keeps both as heuristics.
Each result shows its value in the success color when good, the warning color for a warning, or muted grey when neutral. Below the value is a note saying where the check comes from.
https://.The tool loads with this URL:
https://www.example.com/shop/mens_running-shoes/product/8f14e45fceea167a5a36dedd4bea2543/index.php?utm_source=newsletter&utm_medium=email&sessionid=93a8e2
| Check | Result |
|---|---|
| HTTPS | Yes |
| Total length | 154 characters |
| Path depth | 5 segments, warning |
| Word separator | Mixed (both used), warning |
| Query parameters | 3: utm_source, utm_medium, sessionid, warning |
| Opaque id segments | 1: 8f14e45fceea167a5a36dedd4bea2543, warning |
| Path readability | Reads as encoded or opaque, warning |
Only 2 of the 5 segments, shop and product, count as word-shaped. The underscore, the hex ID
and the .php extension fail the pattern. A cleaner version of the same page is
https://www.example.com/shop/mens-running-shoes, which passes every check.
format_date style joining
as the reason underscores are less suited./wiki/Aviation against a long topic=42
style ID string.Google's URL structure guide recommends readable words rather than long ID numbers, hyphens rather than underscores between words, trimming parameters that do not change the content, and avoiding session IDs in URLs. It also says not to use fragments to change page content.
Google's URL structure documentation gives no character limit. This tool reports the length as a plain count, marked Heuristic, with no pass or fail attached.
Google publishes no maximum depth. This tool marks more than four path segments as a warning, but that cutoff is its own heuristic, labelled as such, not a Google rule.
Google recommends hyphens to separate words and notes that underscores are used to join words into one concept, as in programming names like format_date. The tool warns on underscores alone and on a mix of both.
No. It reports on the URL as typed. Case matters to Google, which treats /APPLE and /apple as distinct URLs, so pick one casing and redirect the other if your server serves both.
Paste a robots.txt to see which AI crawlers it blocks: GPTBot, ClaudeBot, Google-Extended, PerplexityBot and more. Free, runs in your browser.
RSS feed validator: paste feed XML to check RSS 2.0 and Atom 1.0 for parse errors, required elements, dates, guids, enclosures and URLs. Runs in-browser.
humans.txt generator: build a humans.txt file with TEAM, THANKS and SITE sections from the humanstxt.org template. Free, runs in your browser.
Image lazy loading checker: paste HTML to test every img tag for width and height, lazy loading on the likely LCP image. Free, runs in your browser.