AI crawler robots.txt checker for GPTBot, ClaudeBot and more
Paste a robots.txt to see which AI crawlers it blocks: GPTBot, ClaudeBot, Google-Extended, PerplexityBot and more. Free, runs in your browser.
Pagination canonical tag checker: catch later pages canonicalized to page 1, repeated URLs and page numbers after a #. Free, runs in your browser.
Runs entirely in your browser. Nothing you enter is uploaded, logged or stored.
Enter every page of a paginated series with the canonical its template outputs, and this pagination canonical tag checker tests the set against Google’s own pagination documentation. Everything runs against the URLs you type, in your browser; nothing is sent anywhere.
Each check maps to one line of Google’s pagination documentation:
?page=n parameter, because URLs in a paginated sequence are treated as separate pages. The tool
warns if the same URL appears in two rows.#, so the tool warns
on any page URL that contains one.Each check returns Pass, Warning or Info. A Warning on the page 1 check is the one to act on first, because it tells Google that every later page is a copy of page 1. The self-reference count is Info rather than a Warning when it falls short, because a canonical pointing outside the sequence is not something this tool can judge.
Rows with an empty URL or canonical are ignored. With fewer than two complete rows the tool shows “Not enough pages” instead of running the checks, since one URL cannot conflict with anything.
The tool opens with a real template bug: three rows for https://example.com/shoes?page=1,
?page=2 and ?page=3, and all three declare https://example.com/shoes?page=1 as canonical.
The results are:
Change the canonical in rows 2 and 3 to ?page=2 and ?page=3, and the page 1 check turns to
Pass with “Each later page keeps its own canonical”, while the self-reference count reads “3 of 3”
and passes.
/shoes#page=2 look distinct to a person, but
Google ignores the fragment. They pass the unique URL check as strings and fail the fragment
check./shoes and
/shoes?page=1 produces duplicate rows once you enter what Google actually fetches.The tool reads only the rows you enter. It cannot tell whether a page is missing from the series,
whether pages link to each other with real <a href> links (which Google recommends so Googlebot
can find the next page), or whether filtered and sorted variants of the listing are kept out of the
index with noindex or robots.txt, as Google also recommends. Check those in the rendered page.
No. Google's pagination documentation says: "Don't use the first page of a paginated sequence as the canonical page. Instead, give each page its own canonical URL." Page 2 should point at page 2, page 3 at page 3, and so on.
Google's pagination documentation does not ask for that. It treats each page in a sequence as a separate page with its own URL, and recommends noindex or robots.txt for filtered or re-sorted variations of a list, not for the numbered pages themselves.
Not for Google. Its documentation says Google no longer uses these tags, although other search engines may still use them. Leaving them in does no harm; tick the box in the tool and it reports them as information, not as a problem.
No. Google's documentation says not to use URL fragment identifiers for page numbers, because Google ignores fragment identifiers. Use a query parameter such as ?page=2 or a path segment instead.
The tool compares URLs and canonicals as exact strings after trimming spaces. It does not normalize case, trailing slashes or http versus https, so a canonical that differs from its page URL in any of those ways is reported as not self-referencing, which is usually worth fixing anyway.
Paste a robots.txt to see which AI crawlers it blocks: GPTBot, ClaudeBot, Google-Extended, PerplexityBot and more. Free, runs in your browser.
RSS feed validator: paste feed XML to check RSS 2.0 and Atom 1.0 for parse errors, required elements, dates, guids, enclosures and URLs. Runs in-browser.
humans.txt generator: build a humans.txt file with TEAM, THANKS and SITE sections from the humanstxt.org template. Free, runs in your browser.
Image lazy loading checker: paste HTML to test every img tag for width and height, lazy loading on the likely LCP image. Free, runs in your browser.