Is your site blocking ChatGPT and other AI agents?
Your site blocks ChatGPT if its robots.txt disallows OpenAI's bots (ChatGPT-User, OAI-SearchBot or GPTBot), or if a firewall or bot-protection rule answers AI user-agents with a challenge page or a 403 instead of your content. The second is more common and harder to spot, because the site still looks fine in your own browser.
From our data: As of 3 Nov 2026, 19 of the 214 software vendors in our rankings served a bot wall to an AI user-agent, and 13 disallow at least one AI crawler in robots.txt. Across our 1,846-company benchmark (11 Oct 2026), 11% blocked AI agents with a bot wall.
The two ways sites block AI agents
robots.txt rules
A Disallow for GPTBot, ChatGPT-User, ClaudeBot or PerplexityBot. Well-behaved bots read it before crawling. It's a request, not an enforcement, and it's easy to check.
Bot walls
A CDN or firewall rule (Cloudflare, Akamai, DataDome, Imperva and others) that serves AI user-agents a JavaScript challenge, a CAPTCHA, or a 403. The agent never sees your page, and nothing in robots.txt tells you it's happening.
How to fix it
Treat user-triggered agents like browsers
ChatGPT-User, Claude-User and Perplexity-User fetch a page because a person asked a question. Allow them in your bot rules and in robots.txt.
Allow answer-engine crawlers if you want to be cited
OAI-SearchBot, Claude-SearchBot and PerplexityBot index pages for AI answers. Blocking them makes you harder to cite.
Decide on training crawlers separately
GPTBot, ClaudeBot and Google-Extended are about model training. Blocking them doesn't stop the user-triggered agents above.
Check your CDN's AI bot settings
In Cloudflare: Security, then Settings, filter by Bot traffic, open AI Crawler and Agent access policies, and set Search and Agents to allow. Cloudflare's earlier one-click AI block (on by default for new domains from July 2025) covered all of these at once, so older sites may still have it on. Other CDNs and bot tools have similar switches.
Vendors that block AI agents
19 of the vendors in our rankings, from our scan on 3 Nov 2026.
Check it yourself
The first command shows any robots.txt rules for AI bots. The next two fetch your homepage as Claude and ChatGPT would: anything other than 200 means a bot wall. Compare with a normal browser request if you're unsure.
curl -s https://example.com/robots.txt | grep -iE -A2 "gptbot|chatgpt|oai-searchbot|claude|perplexity"
curl -s -o /dev/null -w "%{http_code}\n" \
-A "Mozilla/5.0 (compatible; Claude-User/1.0)" https://example.com/
curl -s -o /dev/null -w "%{http_code}\n" \
-A "Mozilla/5.0 (compatible; ChatGPT-User/1.0; +https://openai.com/bot)" https://example.com/Questions
Does blocking GPTBot stop ChatGPT from citing my site?
Not on its own. GPTBot collects training data. ChatGPT search uses OAI-SearchBot, and when a person asks ChatGPT to read a page it fetches it as ChatGPT-User. Each is controlled separately.
Does Cloudflare block ChatGPT by default?
Not on new domains any more. As of Cloudflare's docs on 2 Nov 2026, new domains default to allowing AI search, agent and training traffic. Domains set up while the old default block was on (from July 2025) may still block, so check Security, then Settings, then AI Crawler and Agent access policies.
How do I know if a firewall is blocking AI bots on my site?
Fetch your homepage with an AI user-agent (the commands above). A 403, or a page titled "Just a moment...", means a rule is challenging it.
Should I block AI crawlers?
Block training crawlers if you don't want your content used to train models. Blocking the user-triggered agents mostly hurts you: they act for a real person who asked about you.