AI crawler access checker

Reads your robots.txt for the AI crawlers specifically: which ones can train on your writing, which can actually open a page to answer or cite it, and which rule is quietly blocking each one.

This is one of the checks inside Growth Score

Growth Score runs it with the rest on your live site, scores the result and shows what to fix first.

Test your whole site Open in your dashboard

Why it matters

Why this is worth checking

txt was written for search engines, and almost nobody has reread it since ChatGPT, Claude and Perplexity started fetching pages too.

A rule copied from an old template can block an AI answer engine nobody meant to name, while a training crawler nobody named either keeps reading everything by default.

What it reads

What it looks at

Whether AI assistants can read you
What Google-Extended blocking does
Training crawlers named or defaulted
Which rule decides each crawler

Reading a result

What a finding means

Each finding says what was seen. This is what it tells you about the page.

An assistant is blockedThat AI system is asked not to read your pages, so it cannot quote you.

A crawler is only defaultedNo rule names it, so the general rule decides. Decide on purpose instead.

Google-Extended blockedIt limits use for Google's AI products, not ordinary search ranking.

Common fixes

What to change first

The fixes that come up most, in the order they usually pay off.

Name the assistants you wantAdd an explicit Allow for each AI crawler that should quote your pages.

Decide on training separatelyTraining and answer crawlers are different agents. Set a rule for each.

Remove old template rulesDelete template rules that block crawlers you never meant to stop.

In Growth Score

Feeds Answerability

Whether AI answers can read, quote and cite your content. txt exists and whether your pages are marked up to be quoted.

It reads the rules as written. Some AI crawlers have a documented history of crawling past the limits set for them, so a block here is a request, not a lock.

Glossary

Terms behind this check

AI crawlerA bot operated by an AI company that reads web pages to train models or to answer queries.AI visibilityHow often and how favorably your brand appears in answers produced by AI systems.Answer engine optimizationStructuring pages so they can be used as the direct answer to a question.

Go deeper

Guides that cover it

Search for a new siteWhat moves a new domain, in what order, and what to skip so early effort is not wasted.Get cited in AI answersLet the right crawlers in, make pages easy to quote, then check visits.

FAQ

Questions

Is a block here a lock?

No. It is a request that well behaved crawlers follow. Some have been documented crawling past the limits set for them.

Should I allow every AI crawler?

That is your call. Allow the ones whose answers you want to appear in, and decide on training use separately.

Does the check change my site?

No. It only reads. Nothing is installed, edited or sent to your site by a check.

The other 11

The rest of the checks

Run it with the rest

Growth Score reads your whole site, scores it against sites like yours and ranks what to fix.

Test your whole site All the checks

Your Growth Score is 100

Your next fix is waiting in your dashboard.

Open your Growth Score Open dashboard