Robots.txt tester

Reads the one file that can remove an entire site from search and tells you what it is really doing: what it blocks, what it lets through, and what it forgot to say.

This is one of the checks inside Growth Score

Growth Score runs it with the rest on your live site, scores the result and shows what to fix first.

Test your whole site Open in your dashboard

Why it matters

Why this is worth checking

One line in this file can remove an entire site from search, and the mistake is invisible: the site keeps working perfectly for everybody except the crawler.

Blocking the CSS and JavaScript is the subtler version, because the page still gets crawled and then gets judged on how it looks without them.

What it reads

What it looks at

Rules blocking the site or key parts
Whether page assets are reachable
Whether a sitemap is declared
Rules with no crawler to apply to

Reading a result

What a finding means

Each finding says what was seen. This is what it tells you about the page.

A Disallow on the rootCrawlers are asked to skip the whole site, which is rarely intended.

Blocked scripts or stylesPages are crawled, then judged without the files that render them.

No Sitemap lineNothing wrong, but crawlers lose the shortest path to your list of pages.

Common fixes

What to change first

The fixes that come up most, in the order they usually pay off.

Remove a blanket blockDelete the root Disallow, or limit it to paths that should stay private.

Unblock page assetsAllow the script and style folders so search can render your pages.

Declare the sitemapAdd one Sitemap line with the full https address of your sitemap file.

In Growth Score

Feeds Findability

Whether search engines can find and then list your pages. txt and sitemap, your page titles and descriptions, and how your pages are addressed.

It reads the rules as written. Whether a crawler obeys them is its own decision, and the ones that ignore robots.txt were never going to read it.

Glossary

Terms behind this check

robots.txtA text file at the root of a site that tells crawlers which paths they may or may not fetch.CrawlabilityHow easily a search engine's bots can reach and read the pages of your site.XML sitemapA file that lists the URLs you want search engines to know about, with optional update dates.

Go deeper

Guides that cover it

Search for a new siteWhat moves a new domain, in what order, and what to skip so early effort is not wasted.Get pages found in searchSix checks in the order search meets them, from robots.txt through to indexing.

FAQ

Questions

Does robots.txt hide a page?

Not reliably. It asks crawlers not to fetch a page, but a blocked address can still be listed if other sites link to it. Use noindex to keep a page out.

Would I notice a mistake here?

No. The site works for every visitor, which is why a wrong line can sit unnoticed until traffic falls.

Does the check change my site?

No. It only reads. Nothing is installed, edited or sent to your site by a check.

The other 11

The rest of the checks

Run it with the rest

Growth Score reads your whole site, scores it against sites like yours and ranks what to fix.

Test your whole site All the checks

Your Growth Score is 100

Your next fix is waiting in your dashboard.

Open your Growth Score Open dashboard