Skip to content
Surfilby Adam Hafez
Free tool

Sitemap vs robots.txt checker: find blocked sitemap URLs

Sitemap vs robots.txt checker: find sitemap URLs that a robots.txt Disallow rule blocks for Googlebot or any user-agent. Free, runs in your browser.

Runs entirely in your browser. Nothing you enter is uploaded, logged or stored.

Nothing you paste here is sent anywhere; both files are parsed in your browser only.

    Key takeaways

    • A URL can be listed in a sitemap while a robots.txt Disallow rule blocks the same crawler from fetching it.
    • This checker applies RFC 9309's rule that the most specific match wins, with Allow winning an equal-length tie.
    • Google says to list the URLs you want in its search results, so a blocked sitemap URL sends a contradictory signal.
    • Google states robots.txt is not a way to keep a page out of search, and a blocked URL can still be indexed if linked.
    • Each conflict is resolved one of two ways, by removing the URL from the sitemap or by narrowing the Disallow rule.

    What it takes

    • Pasted sitemap XML content
    • Pasted robots.txt content
    • User-agent to test, defaulting to the wildcard *

    More tools

    All of them free, all of them browser-only.

    Browse the tools

    How to use it

    1. 1Paste the sitemap XMLCopy the raw contents of the sitemap file, not a link to it. Like the site's sitemap validator, this tool only parses what you paste and never fetches a URL.
    2. 2Paste the robots.txt contentUse the file already live on the site, or a draft before publishing it, the same input the site's robots.txt tester accepts.
    3. 3Set the user-agent to testDefaults to the wildcard * group; enter a named crawler such as Googlebot to test its own, more specific group instead.
    4. 4Read the conflict listEvery sitemap URL is marked allowed or flagged as a conflict, with the exact Disallow rule that blocks it named beside the flag.
    5. 5Fix a flagged conflictEither remove the URL from the sitemap or narrow the Disallow rule so it no longer matches that path, then recheck.

    A sitemap lists the URLs you want a search engine to crawl. A robots.txt Disallow rule tells the same crawler not to fetch a path. Both files can be valid on their own and still contradict each other about one URL. This checker reads both, applies the real robots.txt matching rules, and lists every sitemap URL that lands on the blocked side.

    What the checker checks

    • Every loc in the sitemap. The sitemap is parsed as XML. Its root must be urlset, and every absolute loc inside a url entry is collected. A parse error or a missing urlset stops the check with the reason shown.
    • The right robots.txt group. Groups are read as RFC 9309 defines them: consecutive User-agent lines followed by their Allow and Disallow rules. The checker picks the group matching the user-agent you entered, and falls back to the * group when none matches, which RFC 9309 requires. Groups that match the same crawler are combined, as the RFC also requires.
    • The winning rule for each URL. Each URL’s path and query string are tested against the group’s rules. The most specific match wins, meaning the longest matching path, and an Allow beats a Disallow of equal length. * matches any run of characters and $ anchors the end of the path. A URL that matches no rule is allowed. The checker skips an empty Disallow: line.

    How to read the results

    The summary shows three counts: URLs checked, Allowed and Conflicts. Below it, each URL gets a line. “Allowed” means no rule blocks it for that user-agent. “Conflict” means the sitemap lists it while robots.txt disallows it, and the exact robots.txt line responsible is quoted, so you know which rule to change. The last line names the group used, either the wildcard group or the group declared for a specific crawler.

    A conflict needs a decision. Google’s sitemap guidance says to include the URLs you want to see in its search results, so a URL you have also blocked is sending two opposite messages. If the block is intended, remove the URL from the sitemap. If the page should be crawled, narrow or delete the Disallow rule. Blocking alone does not hide the URL: Google states a disallowed page can still be indexed if other sites link to it.

    Worked example

    The tool loads with this example. The sitemap lists three URLs: https://example.com/, https://example.com/blog/an-article and https://example.com/private/quarterly-report. The robots.txt reads:

    User-agent: *
Allow: /
Disallow: /private/
Sitemap: https://example.com/sitemap.xml

    With the user-agent left as *, the summary reads 3 URLs checked, 2 Allowed, 1 Conflict. The home page and the article only match Allow: /, so both are allowed. The report path matches both Allow: / (1 character) and Disallow: /private/ (9 characters). The longer Disallow wins, so it is flagged as a conflict blocked by Disallow: /private/. The Sitemap: line is not a group rule and is ignored.

    Common mistakes it catches

    • A section blocked after the sitemap was built. A staging path, internal search results or a private folder is disallowed, but the sitemap generator keeps listing every page under it.
    • A broad rule that swallows real pages. Disallow: /p blocks /products/ and /pricing as well as /p/, because rules match by prefix. The conflict list shows each URL it caught.
    • A crawler-specific group you forgot about. A User-agent: Googlebot group replaces the wildcard group for Googlebot entirely. Test Googlebot as well as * to see what it is actually told.
    • Blocking used to deindex. Google states robots.txt is not a way to keep a page out of search, so a URL you want gone needs noindex or password protection instead.

    Sources

    1. 1.Robots.txt introduction and guide - Google Search CentralPrimary
    2. 2.Build and submit a sitemap - Google Search Central
    3. 3.RFC 9309: Robots Exclusion Protocol - IETF

    Common questions

    Why would a URL be in my sitemap and blocked by robots.txt at the same time?

    The two files are usually maintained separately. A CMS or build step regenerates the sitemap from every page that exists, while robots.txt is edited by hand. When a new Disallow rule is added, nothing tells the sitemap generator, so it keeps listing the blocked URLs.

    Does blocking a URL in robots.txt keep it out of Google?

    No. Google states that robots.txt is not a mechanism for keeping a page out of Google, and that a disallowed URL can still be indexed if other sites link to it, usually shown without a description. Use noindex or password protection to keep a page out of results.

    Which robots.txt group does the checker use?

    It looks for a group whose user-agent matches the name you enter, preferring the longest matching token, and combines groups that match equally. If none matches, it falls back to the * group, as RFC 9309 requires. The group it used is named under the results.

    How does it decide between an Allow and a Disallow rule?

    It follows RFC 9309. The rule with the longest matching path wins, and if an Allow and a Disallow rule match with equal length, Allow wins. The * wildcard and the $ end anchor are supported. A URL that matches no rule is allowed.

    Does it fetch my live sitemap or robots.txt?

    No. Both files are parsed in your browser from what you paste, and nothing is sent anywhere. That also means it cannot see a robots.txt or sitemap you only have a link to, so copy the contents in.

    Tools

    humans.txt generator

    humans.txt generator: build a humans.txt file with TEAM, THANKS and SITE sections from the humanstxt.org template. Free, runs in your browser.