How to test if ChatGPT agent and Claude can use your site
Test in three layers. First, fetch your pages with AI user-agents (ChatGPT-User, Claude-User, Perplexity-User) from the command line to catch bot walls and robots.txt blocks. Second, read what came back to make sure prices, contact details and other facts are in the HTML. Third, give a real agent, such as ChatGPT agent or Claude with web access, the task you care about and watch where it stops. The first two take a minute. Only the third proves the job works, because firewalls also judge agents by IP address and signatures, which a curl from your laptop can't copy.
From our data: As of 3 Nov 2026, 19 of the 214 software vendors in our rankings served a bot wall to an AI user-agent on their homepage. Across our 1,846-company benchmark (11 Oct 2026), 11% did.
1. Fetch as the agents do
Compare each AI user-agent with a normal browser. A 200 for the browser and a 403, 429 or 503 for an agent means a rule is singling agents out. A 200 whose body says "Just a moment" or "Verify you are human" is a challenge page, not your content.
for ua in "ChatGPT-User/1.0; +https://openai.com/bot" "Claude-User/1.0" "Perplexity-User/1.0" \
"OAI-SearchBot/1.4; +https://openai.com/searchbot" "Claude-SearchBot/1.0" "Google-Agent"; do
printf '%-50s ' "$ua"
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; $ua)" https://example.com/
done
curl -s -o /dev/null -w "browser: %{http_code}\n" \
-A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0 Safari/537.36" https://example.com/2. Check what they can read
The facts are in the HTML
Strip the tags from the agent's response and look for your product name, prices, contact details and policies. If they only load with JavaScript, a fetch-based agent can't quote them.
The pages behind the homepage open too
Repeat the fetch for /pricing, /cart, your sign-up page and your docs. Bot rules are often stricter on checkout and sign-up paths.
robots.txt doesn't shut them out
Look for Disallow rules under ChatGPT-User, Claude-User, OAI-SearchBot or Claude-SearchBot, and for a blanket User-agent: * Disallow: /.
3. Give a real agent the job
User-agent tests can't reproduce everything. Bot managers also look at IP ranges, TLS fingerprints and, increasingly, signed requests: Cloudflare classifies ChatGPT agent as a signed agent because it signs requests with Web Bot Auth. So finish with a real run, with your own test account and no real payment details.
Be recommended
Ask: "What does example.com do, who is it for, how much does it cost, and how do I contact them?" Check the answer against your site, and note which pages it cites.
Buy
Ask an agent to find a specific product on your store, report its price and stock, and get to the checkout page without paying. Note every pop-up, challenge or dead end.
Sign up
Ask it to create a free account with a test email address you control. Watch for CAPTCHAs, phone verification and "book a demo" detours.
Get an API key
Point a coding agent at your docs and ask it to get a key and make one successful call. Count the steps where it has to ask you for help.
4. Check your logs
Search your CDN or server logs for ChatGPT-User, Claude-User, Perplexity-User and Google-Agent, and look at the status codes they got. OpenAI and Anthropic publish their IP ranges (openai.com/chatgpt-user.json, claude.com/crawling/bots.json), and Google publishes user-triggered-agents.json for Google-Agent, so you can tell real agents from impostors.
Check it yourself
The quickest version of test 1. Then run the free check: it fetches your homepage, robots.txt, cart or sign-up page and docs as Claude-User, scores the result out of 100, and can send a real agent to try the job.
curl -s -o /dev/null -w "%{http_code}\n" \
-A "Mozilla/5.0 (compatible; Claude-User/1.0)" https://example.com/
curl -s -o /dev/null -w "%{http_code}\n" \
-A "Mozilla/5.0 (compatible; ChatGPT-User/1.0; +https://openai.com/bot)" https://example.com/Questions
Why does curl return 200 but ChatGPT says it can't open my site?
Your bot protection is probably judging more than the user-agent: ChatGPT's requests come from OpenAI's IP ranges and, for ChatGPT agent, carry a signature. Check your CDN's security events for the time of the failed request.
Is faking the user-agent a fair test?
It's a good first test because many bot rules key on the user-agent. It can miss blocks based on IPs or signatures, and it can also pass where the real agent fails. That's why the last step is a real agent run.
What does a "Just a moment..." page mean?
It's a JavaScript challenge from a bot-protection service. Agents that fetch pages can't solve it, so they see that page instead of yours.
How often should I test?
After any change to your CDN, firewall or bot-protection settings, and monthly otherwise. Bot vendors change defaults, and a rule meant for scrapers can start catching assistants.