AI crawler robots.txt checker for GPTBot, ClaudeBot and more
Paste a robots.txt to see which AI crawlers it blocks: GPTBot, ClaudeBot, Google-Extended, PerplexityBot and more. Free, runs in your browser.
Meta robots tag generator: build noindex, nofollow, max-snippet and other rules as a meta tag or X-Robots-Tag header. Free, runs in your browser.
Runs entirely in your browser. Nothing you enter is uploaded, logged or stored.
Tick the rules a page needs and the builder writes the finished line: a <meta name="robots">
tag for the page head, or an X-Robots-Tag: header line for your server or CDN. It runs in your
browser and sends nothing anywhere.
Every rule and value comes from Google’s robots meta tag specification.
Not covered: indexifembedded, the default all value, the data-nosnippet HTML attribute and crawler-specific names such as googlebot. Add those by hand if you need them.
The output box shows one line. With nothing ticked, it says the page indexes and is followed with no restrictions, which matches Google’s default of all.
A warning appears when max-snippet, max-image-preview or max-video-preview is ticked alongside noindex or none. The combination is valid, but the snippet rules are inert: a page that is not indexed has no snippet to shape. Google also says that when rules conflict, the more restrictive one applies, so noindex wins.
The builder opens with noindex ticked and outputs:
<meta name="robots" content="noindex">
Tick max-snippet and max-image-preview, leaving their defaults of -1 and standard. The output becomes:
<meta name="robots" content="noindex, max-snippet:-1, max-image-preview:standard">
and the warning reads that max-snippet and max-image-preview control how a snippet looks for an indexed page, and combined with noindex have no effect. Switch the format to HTTP header and the same rules come out as:
X-Robots-Tag: noindex, max-snippet:-1, max-image-preview:standard
Untick noindex and the warning clears, leaving a valid tag for an indexed page with a full-length snippet and standard image previews.
max-image-preview:large, so a typo cannot slip in.The meta tag sits in an HTML page's head. X-Robots-Tag is an HTTP response header, so it also works for PDFs, images and other non-HTML files. Google says any rule you can use in the meta tag can also be used in X-Robots-Tag.
Google says -1 lets it choose the snippet length it believes is most effective. max-snippet:0 means no snippet is shown, the same as nosnippet. Any other number is a character limit.
Not for Google. Google says noarchive is no longer used by Google Search, because the cached link feature it controlled no longer exists. The tool keeps it, marked as retired, because other crawlers may still read it.
Google accepts widely adopted formats including RFC 822, RFC 850 and ISO 8601. This tool writes the date as ISO 8601, for example unavailable_after 2026-12-31, using the date picker.
The most common reason is a robots.txt disallow on the same URL. Google says that if a page is disallowed from crawling, its indexing rules are never found and are ignored. Allow crawling so the noindex can be read.
Google supports a user agent name in place of robots, such as meta name googlebot. This builder always writes name robots; change the name attribute by hand if you want a Google-only rule.
Paste a robots.txt to see which AI crawlers it blocks: GPTBot, ClaudeBot, Google-Extended, PerplexityBot and more. Free, runs in your browser.
RSS feed validator: paste feed XML to check RSS 2.0 and Atom 1.0 for parse errors, required elements, dates, guids, enclosures and URLs. Runs in-browser.
humans.txt generator: build a humans.txt file with TEAM, THANKS and SITE sections from the humanstxt.org template. Free, runs in your browser.
Image lazy loading checker: paste HTML to test every img tag for width and height, lazy loading on the likely LCP image. Free, runs in your browser.