Skip to content
Surfilby Adam Hafez
Free tool

Robots.txt validator: whole-file syntax checker

Robots.txt validator: lint every line for syntax errors, unsupported fields, bad paths and relative sitemap URLs. Free, runs in your browser.

Runs entirely in your browser. Nothing you enter is uploaded, logged or stored.

Nothing you paste here is sent anywhere; it is parsed line by line in your browser only.

    Key takeaways

    • A valid robots.txt line is a field, a colon and a value, and Google ignores invalid lines rather than rejecting the file.
    • Google supports four fields: user-agent, allow, disallow and sitemap, and states that crawl-delay is not supported.
    • An allow or disallow path must start with a slash, and a sitemap value must be a fully qualified URL.
    • Google enforces a 500 KiB size limit on robots.txt and ignores content after that point.
    • This validator checks structure only; whether a rule blocks a given URL for a given bot is the job of the robots.txt tester.

    What it takes

    • The full contents of a robots.txt file

    More tools

    All of them free, all of them browser-only.

    Browse the tools

    How to use it

    1. 1Paste the whole robots.txt fileInclude every line, not just the group you are focused on, since structural errors like a misplaced rule depend on what comes before it.
    2. 2Read the error and warning countAn error is a line that breaks the documented syntax outright; a warning is a line that parses but that Google does not support or that affects the whole file.
    3. 3Check any "no colon found" errors firstA missing colon is the most common copy-paste mistake, usually from an "Allow /path" line that dropped its colon.
    4. 4Check for rules before any user-agent lineAn allow or disallow line with no user-agent line above it has no group to belong to, per RFC 9309's group structure.
    5. 5Check sitemap lines are absolute URLsA relative sitemap path, such as sitemap.xml instead of a full https:// URL, fails Google's documented requirement for a fully qualified URL.
    6. 6Check for fields Google does not supportCrawl-delay and similar directives are real for some other crawlers but are flagged here because Google's documentation states it does not use them.

    Paste a full robots.txt file, and this validator checks whether it is well-formed: each line’s syntax, each field name, each path and URL value, and whether every rule has a user-agent to belong to. It runs in your browser, line by line, and nothing you paste is sent anywhere.

    What this robots.txt validator checks

    RFC 9309 defines a robots.txt line as a field, a colon and a value, with rules grouped under the user-agent lines before them. Google’s documentation adds the specifics. The validator applies these checks:

    • Missing colon. A line with no colon is an error, because it is not a field and value pair.
    • Space in a field name. “Dis allow: /x” is an error; the field will not be recognised.
    • Unsupported field. Google supports user-agent, allow, disallow and sitemap only. Any other field is a warning, with a specific note for crawl-delay, host, clean-param, request-rate, visit-time and noindex, which other crawlers use but Google does not.
    • Empty user-agent. A user-agent line with no value is an error.
    • Rule before any user-agent. An allow or disallow line above the first user-agent line has no group, so it is an error.
    • Path without a leading slash. Google says the path value must start with a slash to designate the root. A lone * is accepted.
    • Relative sitemap. A sitemap value must be a fully qualified URL with protocol and host.
    • Whole-file checks. A file with no user-agent line at all is a warning, and so is a file over 500 KiB, Google’s documented size limit.

    Comments after a # and blank lines are always valid.

    How to read the results

    The summary line gives a count. Errors come first: they are lines a crawler will ignore, so the rule you meant to set does not exist. Warnings are lines that parse but do nothing for Google, or problems with the file as a whole. If there is neither, the summary reads “No issues found”.

    Each flagged line shows its line number, the line itself and the reason. Lines that parsed cleanly are not listed, so a short list is a good sign.

    Worked example

    The prefilled file contains three deliberate mistakes. The validator reports “2 errors, 1 warning”:

    • Line 4: Allow /admin/public has no colon, so it is an error. As written, /admin/public stays blocked by the Disallow above it.
    • Line 5: Crawl-delay: 10 is a warning. Google’s documentation states it does not support it.
    • Line 10: Sitemap: sitemap.xml is an error. It needs the full URL, such as https://example.com/sitemap.xml.

    Line 7, Disallow: /search, is not flagged even though a blank line separates it from the group above. Groups are set by user-agent lines, not blank lines, so it still belongs to the * group.

    Common mistakes it catches

    • A dropped colon after a copy and paste. Google ignores the line and gives no error, so the rule silently does not exist.
    • Crawl-delay written for Google. It has no effect on Googlebot.
    • Paths written without the slash, such as Disallow: private/, which does not follow Google’s documented format.
    • A relative sitemap line, which Google’s documentation says must be a full URL.
    • Rules pasted above the first user-agent line, which belong to no group.

    What it does not check

    Syntax is only half the question. The prefilled file ends with a Googlebot group that holds no allow or disallow rules. That is valid syntax, so the validator passes it, but it means Googlebot follows an empty group and ignores every * rule above it, which RFC 9309 treats as allowed for every URL. To see what a file actually permits for one crawler and one path, use the robots.txt tester.

    Sources

    1. 1.RFC 9309: Robots Exclusion Protocol - IETFPrimary
    2. 2.How Google interprets the robots.txt specification - Google Search CentralPrimary

    Common questions

    How do I validate my robots.txt file?

    Open yoursite.com/robots.txt in a browser, copy the whole file and paste it into the validator. It lists every line that breaks the syntax or uses a field Google does not support, with the line number and the reason.

    Does Google support crawl-delay in robots.txt?

    No. Google's robots.txt documentation supports only user-agent, allow, disallow and sitemap, and says other fields such as crawl-delay are not supported. The validator flags crawl-delay, host, clean-param, request-rate, visit-time and noindex lines as warnings for that reason.

    What happens to an invalid line in robots.txt?

    It is skipped. RFC 9309 says crawlers must try to parse each line and must use the parseable rules, and Google says it ignores invalid lines. The rest of the file still works, which is why a broken rule can go unnoticed for months.

    Is there a size limit for robots.txt?

    Google enforces a 500 KiB limit and ignores content after it. RFC 9309 requires crawlers to parse at least 500 kibibytes. The validator warns when a pasted file is larger than 500 KiB.

    Are robots.txt paths case-sensitive?

    Google treats field names such as Disallow as case-insensitive, but path values are case-sensitive, so Disallow /Admin does not block /admin.

    How is this different from the robots.txt tester?

    The validator checks that every line of the file is well-formed. The tester takes one crawler and one URL path and tells you whether that path is allowed or blocked, and by which line.

    Tools

    humans.txt generator

    humans.txt generator: build a humans.txt file with TEAM, THANKS and SITE sections from the humanstxt.org template. Free, runs in your browser.