robots.txt for AI bots: every user agent and what it does
AI companies run three kinds of bots, and robots.txt can allow or block each one by name: training crawlers that collect data for future models (GPTBot, ClaudeBot, Google-Extended), search crawlers that index pages so assistants can cite them (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and user agents that fetch a page because a person asked (ChatGPT-User, Claude-User, Perplexity-User).
From our data: As of 3 Nov 2026, 13 of the 214 software vendors in our rankings disallow at least one AI bot in robots.txt, and 0 block a user-triggered agent such as ChatGPT-User.
The user agents
What each vendor says its bots do, from their own docs as of 3 Nov 2026. Training opt-outs and user-triggered agents are separate switches: blocking one doesn't block the other.
GPTBot (OpenAI)
Collects training data. Honours robots.txt.
OAI-SearchBot (OpenAI)
Indexes pages for ChatGPT search results. Honours robots.txt; OpenAI says changes take about a day to apply.
ChatGPT-User (OpenAI)
Fetches a page when a person asks ChatGPT to. OpenAI says robots.txt rules may not apply, because a person started the request.
ClaudeBot (Anthropic)
Collects training data. Honours robots.txt, including Crawl-delay.
Claude-SearchBot (Anthropic)
Indexes pages to improve Claude's search answers. Honours robots.txt.
Claude-User (Anthropic)
Fetches a page when a person asks Claude to. Anthropic says it honours robots.txt.
PerplexityBot (Perplexity)
Indexes pages for Perplexity's answers. Perplexity says it isn't used for training.
Perplexity-User (Perplexity)
Fetches a page when a person asks. Perplexity says it generally ignores robots.txt.
Google-Extended (Google)
Not a crawler: a robots.txt token. Disallowing it opts your content out of Gemini training and grounding. It doesn't affect Google Search.
Applebot-Extended (Apple)
Also a token, not a crawler. Opts your content out of Apple's model training; pages stay in Siri, Spotlight and Safari.
meta-externalagent, meta-webindexer and meta-externalfetcher (Meta)
Training and indexing, Meta AI search citations, and user-requested fetches. Meta says the fetcher may bypass robots.txt.
CCBot (Common Crawl)
Builds the open Common Crawl dataset, which many models are trained on.
A robots.txt that welcomes assistants and opts out of training
Groups can list several User-agent lines before their rules. This lets people's assistants and answer engines in, and asks training crawlers to stay out. Drop the second group if you're happy to be used for training.
# People's assistants and answer engines User-agent: ChatGPT-User User-agent: Claude-User User-agent: Perplexity-User User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot Allow: / # Model training (optional opt-out) User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot Disallow: / User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml
Vendors whose robots.txt blocks AI bots
13 of the vendors in our rankings, from our scan on 3 Nov 2026.
| Vendor | Category | Disallowed |
|---|---|---|
| FreshBooksRe-check live | accounting software | Bytespider |
| Together AIRe-check live | AI model API | Google-Extended |
| AffinityRe-check live | CRM | Bytespider |
| AttioRe-check live | CRM | CCBot, Bytespider |
| folkRe-check live | CRM | Bytespider |
| OneSpanRe-check live | e-signature tool | GPTBot |
| TelnyxRe-check live | email or SMS API | Bytespider |
| SAPRe-check live | ERP | anthropic-ai, Applebot-Extended, meta-externalagent, Bytespider |
| RazorpayRe-check live | payments platform | Bytespider |
| AmplitudeRe-check live | product analytics tool | Bytespider |
| NotionRe-check live | project management tool | meta-externalagent |
| CalendlyRe-check live | scheduling tool | CCBot |
| HostingerRe-check live | website builder | CCBot, Bytespider |
Check it yourself
Shows every robots.txt group that names an AI bot, with the rules under it. robots.txt is only half the story: a firewall can still block a bot you allow, so also run the free check.
curl -s https://example.com/robots.txt | grep -iE -B1 -A3 "gptbot|oai-searchbot|chatgpt-user|claude|perplexity|google-extended|ccbot"
Questions
Does robots.txt actually stop AI bots?
Training and search crawlers from the major AI companies say they honour it. Several user-triggered fetchers (ChatGPT-User, Perplexity-User, meta-externalfetcher) say robots.txt may not apply, because a person asked. And it can't stop a bot that ignores it.
What's the difference between GPTBot and ChatGPT-User?
GPTBot crawls for model training. ChatGPT-User fetches a page when a person in ChatGPT asks about it. Blocking GPTBot doesn't block ChatGPT-User.
Should I use User-agent: * Disallow: / to block AI?
No. That blocks every crawler, including search engines. Name the AI bots you want to block instead.