Skip to content
Surfilby Adam Hafez
Free tool

robots.txt and llms.txt generator for AI crawlers

robots.txt and llms.txt generator for AI crawlers: allow or block GPTBot, ClaudeBot, Google-Extended and 11 more by purpose. Free, runs in your browser.

Runs entirely in your browser. Nothing you enter is uploaded, logged or stored.

Who may take your content

Blocking a training crawler does not remove you from the answer engine that shares its name, and allowing a search crawler does not consent to training. They are separate agents, which is why they are listed separately.

robots.txt

llms.txt
Key takeaways
  • A single company usually runs several crawlers with different purposes, and refusing one of them says nothing about the others.
  • Blocking a training crawler does not remove you from the answer engine that shares its name, because that engine reads with a different agent.
  • Google-Extended controls Gemini training only, and refusing it has no effect on how the page ranks in Google Search.
  • robots.txt states intent, so a bot rule at your CDN can silently overrule everything the file says.
What it takes
  • Your site URL
  • Paths you want kept out of results
  • A yes or no per crawler
More tools
All of them free, all of them browser-only.
Browse the tools
How to use it
  1. 1Set the site URL and the paths to excludeSearch pages and anything behind a query string are the usual candidates, because they generate near-infinite low-value URLs.
  2. 2Choose a starting postureThree presets: allow everything, allow search but refuse training, or refuse the AI crawlers entirely. Then adjust individual agents.
  3. 3Read the purpose label on each agentSearch, training and live fetch are genuinely different exchanges. Deciding them as one group is how sites end up invisible in answers they wanted to appear in.
  4. 4Publish both files at the rootrobots.txt goes at /robots.txt, llms.txt at /llms.txt, and then verify the policy with a real request rather than assuming.
Most AI crawler policies are decided with one checkbox, and that is the mistake. This robots.txt
and llms.txt generator lists 14 agents by what they do, lets you allow or refuse each one, and
writes both files from that decision in your browser.


What the generator does


Each agent is labelled with its purpose:


    

  • Search: indexes pages so they can be cited in an AI answer. OAI-SearchBot, Claude-SearchBot,
PerplexityBot, Amazonbot and DuckAssistBot.
    • 

  • Training: collects content that may be used to train models. GPTBot, ClaudeBot,
Google-Extended, Applebot-Extended, meta-externalagent and Bytespider.
    • 

  • Live fetch: retrieves a page because a person asked about it. ChatGPT-User, Claude-User and
Perplexity-User.
    • 



The distinction is documented, not assumed. OpenAI
describes three separate agents and says each setting is independent of the others. Google says
Google-Extended
controls Gemini training and grounding, does not affect inclusion or ranking in Google Search, and
has no user agent of its own: it is a robots.txt token only.


Three presets set a starting point: allow everything, allow search but refuse training (which
refuses every agent labelled training and allows the rest), or block all 14. Then change any agent
by hand. You also enter the site URL, used for the Sitemap line and the llms.txt links, and a
comma-separated list of paths to disallow.


How to read the output


The robots.txt starts with a User-agent: * group that allows everything except your listed
paths. Allowed AI agents follow in one group with Allow: /, refused ones in another with
Disallow: /, each group under a comment line, and the file ends with a Sitemap line.


One consequence matters. RFC 9309 says a crawler
obeys the group that names its user agent and uses the * group only when no group matches. So
the paths you exclude bind every crawler not named in the file, but not the AI agents you allowed
by name. If /search should stay closed to them too, copy those Disallow lines into their group.


The llms.txt follows the structure proposed at llmstxt.org: an H1 with
your host name, a blockquote summary, H2 sections listing links, and an Optional section for links
an agent can skip. The summary sentence and the three links are placeholders. Replace them with
your real pages before you publish.


Worked example


With the defaults, https://example.com and the paths /search, /cart, /*?, the preset “Search,
not training” produces this robots.txt (comment lines left out):


User-agent: *
Allow: /
Disallow: /search
Disallow: /cart
Disallow: /*?

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Amazonbot
User-agent: DuckAssistBot
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Bytespider
Disallow: /

Sitemap: https://example.com/sitemap.xml


Googlebot is not named, so it follows the first group: everything except the three excluded
paths. The eight named search and fetch agents may crawl everything, including /search.


Common mistakes it prevents


    

  • Blocking the search agent with the training one. Refusing everything from one company also
removes you from the answers that company’s search agent builds.
    • 

  • Refusing Google-Extended to protect rankings. Google documents that it has no effect on
Search, in either direction.
    • 

  • Expecting robots.txt to enforce anything. RFC 9309 says the rules are not access
authorization. Well-behaved crawlers comply; your CDN or firewall can still override the file.
    • 



Verify the result with requests rather than by reading the file:


for ua in GPTBot OAI-SearchBot ClaudeBot PerplexityBot; do
  curl -sA "$ua" -o /dev/null -w "$ua %{http_code}\n" https://example.com/
done


A 403 means something other than robots.txt is making the decision. Crawlers also cache the file:
RFC 9309 says they should not use a cached copy for more than 24 hours.



Sources
  1. 1.RFC 9309: Robots Exclusion Protocol - IETFPrimary
  2. 2.Overview of Google crawlers and fetchers - Google Search CentralPrimary
  3. 3.OpenAI bots and how to control them - OpenAIPrimary
  4. 4.The llms.txt proposal - llmstxt.org
  5. 5.Google's common crawlers - Google Search CentralPrimary
Common questions
If I block GPTBot, do I disappear from ChatGPT?
No. OpenAI says GPTBot crawls content that may be used in training, OAI-SearchBot surfaces sites in ChatGPT search, and ChatGPT-User visits pages for user actions. It states each setting is independent, so you can allow OAI-SearchBot while disallowing GPTBot.
Does refusing Google-Extended hurt my rankings?
No. Google documents that Google-Extended controls whether crawled content may be used for training future Gemini models and for grounding, and that it does not affect a site's inclusion in Google Search nor act as a ranking signal. It has no user agent of its own; it is a robots.txt token only.
Why do the AI crawlers I allowed ignore my Disallow paths?
RFC 9309 says a crawler obeys the group that names it and falls back to the * group only if none does. This tool gives each allowed agent its own group with Allow: /, so your excluded paths bind only unnamed crawlers. Copy the Disallow lines into that group if they should apply to AI crawlers too.
Does ChatGPT-User obey robots.txt?
Not necessarily. OpenAI says that because ChatGPT-User actions are initiated by a user, robots.txt rules may not apply. OpenAI also says it can take about 24 hours for its systems to reflect a robots.txt change.
Is llms.txt a standard that engines obey?
It is a proposal published at llmstxt.org, not a standard, and no engine is obliged to read it. The format is simple: an H1 with the site name, which is the only required part, a short blockquote summary, and H2 sections listing links, with an Optional section for links an agent can skip.
My robots.txt allows these bots but they still get blocked. Why?
RFC 9309 says robots.txt rules are not a form of access authorization; they state intent. A bot-management, firewall or CDN rule can block a request before robots.txt matters. Test with a real request that sends the agent's user-agent string.