← All terms

robots.txtWhat it means for you.

A text file at the root of a site that tells crawlers which paths they may or may not fetch.

Growth Score · Acquisition

Definition

What it means

txt file lives at the top of your domain and lists rules for bots. It controls crawling, not indexing, so a blocked page can still appear in results if other sites link to it.

A single wrong line can hide an entire site from search, so it is worth testing after every change.

In Growth Score

Part of your score

Test it

Checks that read it on your site

Robots.txt testerReads the one file that can remove an entire site from search and tells you what it is really doing: what it blocks, what it lets through, and what it forgot to say.

Go deeper

Guides that cover it

Get pages found in searchSix checks in the order search meets them, from robots.txt through to indexing.Search for a new siteWhat moves a new domain, in what order, and what to skip so early effort is not wasted.Your first 100 customersFour zero-spend channels, run one at a time, each read in two weeks.

Acquisition

More in Acquisition

FAQ

Quick answers

What is robots.txt?

A text file at the root of a site that tells crawlers which paths they may or may not fetch.

Where does it sit in Growth Score?

It belongs to the Acquisition pillar, one of the 12 pillars your score is built from.

What should I read next?

Start with Crawlability and XML sitemap, then browse the other terms in Acquisition.

Browse

Nearby in the A to Z

PreviousRisk reversalNextSample size

How does your site score?

Growth Score reads your site and scores its Acquisition pillar against sites like yours.

Test your siteAll terms

Your Growth Score is 100

Your next fix is waiting in your dashboard.

Open your Growth ScoreOpen dashboard