← All terms

AI crawlerWhat it means for you.

A bot operated by an AI company that reads web pages to train models or to answer queries.

Growth Score · Acquisition

Definition

What it means

AI crawlers fetch pages for different purposes: building training sets, indexing for AI search, or fetching a page live for a user. txt. Blocking some of them can remove you from AI answers.

In Growth Score

Part of your score

Test it

Checks that read it on your site

AI crawler access checkerReads your robots.txt for the AI crawlers specifically: which ones can train on your writing, which can actually open a page to answer or cite it, and which rule is quietly blocking each one.

Go deeper

Guides that cover it

Get cited in AI answersLet the right crawlers in, make pages easy to quote, then check visits.Search for a new siteWhat moves a new domain, in what order, and what to skip so early effort is not wasted.Your first 100 customersFour zero-spend channels, run one at a time, each read in two weeks.

Acquisition

More in Acquisition

FAQ

Quick answers

What is AI crawler?

A bot operated by an AI company that reads web pages to train models or to answer queries.

Where does it sit in Growth Score?

It belongs to the Acquisition pillar, one of the 12 pillars your score is built from.

What should I read next?

Start with robots.txt and llms.txt, then browse the other terms in Acquisition.

Browse

Nearby in the A to Z

PreviousAI citationNextAI overview

How does your site score?

Growth Score reads your site and scores its Acquisition pillar against sites like yours.

Test your siteAll terms

Your Growth Score is 100

Your next fix is waiting in your dashboard.

Open your Growth ScoreOpen dashboard