Skip to content
Surfilby Adam Hafez
Free tool

Robots.txt tester: check if a URL is blocked for a bot

Robots.txt tester: check if a URL path is blocked for Googlebot, GPTBot or any bot, and see the exact rule that decides it. Free, runs in your browser.

Runs entirely in your browser. Nothing you enter is uploaded, logged or stored.

Nothing you paste here is sent anywhere; it is parsed in your browser only.

    Key takeaways

    • A crawler follows only the most specific user-agent group that names it, and falls back to the * group only when no group matches.
    • Within that group, the longest matching Allow or Disallow path wins, wherever it sits in the file.
    • When an Allow and a Disallow match with the same length, RFC 9309 says the Allow rule should be used.
    • A path with no matching rule, or a group with no rules at all, means the URL is allowed.
    • Robots.txt controls crawling, not access or indexing, and RFC 9309 says it is not a form of access authorization.

    What it takes

    • Existing robots.txt content
    • User-agent to test
    • URL path to test

    More tools

    All of them free, all of them browser-only.

    Browse the tools

    How to use it

    1. 1Paste the robots.txt you want to checkUse the file already live on your site, or a draft you are about to publish. The prefilled example has a wildcard group and a Googlebot group so both matching paths are visible at once.
    2. 2Enter the crawler's user-agentUse the product token a crawler announces, such as Googlebot or GPTBot, not its full user-agent string.
    3. 3Enter the URL path to testA path starting with a slash, such as /private/page.html. Query strings matter too, since a rule can target them directly.
    4. 4Read the verdict and the trace beneath itThe tool names the group that applied and the exact line that decided the outcome, plus any shorter rule that also matched but lost.

    Publishing a robots.txt is easy. Knowing what it actually does to one crawler on one URL is the part most people get wrong, because the rule that wins is often not the one they expect. This tester runs the file forward for a single case and shows its working.

    What this robots.txt tester checks

    It applies the matching rules from RFC 9309, which Google’s own robots.txt documentation follows on each point:

    • Groups. One or more consecutive User-agent lines start a group, and the Allow and Disallow lines after them belong to it.
    • Group selection. The user-agent is matched case-insensitively against each group’s product token, and the most specific match wins. Groups naming the same token are merged. The * group applies only when no named group matches.
    • Longest match. Within the chosen group, the rule with the longest path that matches decides. RFC 9309 says the most specific match, the one with the most octets, must be used.
    • Ties. An Allow and a Disallow of equal length resolve to Allow.
    • Wildcards. A * in a path matches any run of characters, and a $ at the end anchors the rule to the end of the URL.
    • Defaults. No matching rule means allowed, and an empty Disallow blocks nothing.

    How to read the results

    The verdict line reads Allowed or Disallowed for the crawler and path you entered. Under it, the trace explains why, in order:

    1. Which group applied, and whether that was a named group or the * fallback.
    2. The line number and text of the rule that decided the outcome, with its length in characters.
    3. Any other rule that also matched but lost because it was shorter.

    If no rule in the group matches, the trace says so and the verdict is Allowed. A path that does not start with a slash is rejected before anything is evaluated.

    Worked example

    The prefilled file has a * group and a Googlebot group. With user-agent Googlebot and path /private/pricing.html, the tool reports:

    • Googlebot matches the most specific group, declared for Googlebot.
    • Line 11, Allow: /private/pricing.html, is the longest matching rule at 21 characters, so it decides the outcome.
    • Line 10, Disallow: /private/, also matches but is shorter at 9 characters, so it loses.
    • Verdict: Allowed.

    Change the path to /admin and the verdict is still Allowed for Googlebot. The * group disallows /admin, but Googlebot has its own group, and that group has no rule for /admin. Switch the user-agent to Bingbot and the same path is Disallowed by line 3, because Bingbot has no named group and falls back to *. Try Bingbot with /report.pdf and line 6, Disallow: /*.pdf$, beats the one-character Allow: /.

    Common mistakes it catches

    • Expecting * rules to apply to a named crawler. Once a crawler has its own group, it reads nothing else. This is the most common surprise the tester surfaces.
    • Trusting line order. A broad Disallow near the top does not beat a longer Allow further down. Only length counts.
    • A $ where a prefix was meant, or the reverse. Disallow: /*.pdf$ blocks /report.pdf but not /report.pdf?download=1; the tester shows the difference directly.
    • Blocking a page to remove it from search. Google cannot see a noindex on a page it is not allowed to crawl, so a Disallow can keep a page’s noindex from ever being read.

    Remember that robots.txt is a request a well-behaved crawler chooses to honour. RFC 9309 states it is not a form of access authorization, and listing a path in the file makes that path public. A Disallowed verdict describes what the file says, not what every bot will do.

    Sources

    1. 1.RFC 9309: Robots Exclusion Protocol - IETFPrimary
    2. 2.How Google interprets the robots.txt specification - Google Search CentralPrimary
    3. 3.Robots meta tag, data-nosnippet, and X-Robots-Tag specifications - Google Search Central

    Common questions

    How do I check if a URL is blocked by robots.txt?

    Paste the site's robots.txt into the tester, enter the crawler's user-agent token and the URL path starting with a slash. The verdict says Allowed or Disallowed, and the trace names the group that applied and the line that decided it.

    Why does Googlebot ignore my User-agent * rules?

    Because a crawler follows only the most specific group that matches it. If the file has a Googlebot group, Googlebot uses that group and none of the * rules, so any rule you want it to obey must be repeated in its own group.

    Which rule wins when Allow and Disallow both match?

    The one with the longer path, regardless of order in the file. RFC 9309 calls it the match with the most octets. If both have the same length, the Allow rule should be used, and Google documents the same least restrictive tie-break.

    Does a blocked URL stay out of Google's index?

    Not necessarily. Robots.txt stops crawling, not indexing. Google's robots meta tag documentation says that if a page is disallowed in robots.txt, any noindex on it will not be found. To keep a page out of the index, let it be crawled and use noindex.

    How is this different from the robots.txt validator?

    The validator lints the whole file for broken syntax, unsupported fields and misplaced rules. This tester assumes the file parses and answers one case at a time, for one crawler and one path.

    How long until Google sees a robots.txt change?

    Google says it generally caches the contents of a robots.txt file for up to 24 hours, so a change can take up to a day to apply.

    Tools

    humans.txt generator

    humans.txt generator: build a humans.txt file with TEAM, THANKS and SITE sections from the humanstxt.org template. Free, runs in your browser.