What is Web Bot Auth?
Web Bot Auth is a proposed IETF standard that lets bots and AI agents prove who they are by cryptographically signing each HTTP request, using HTTP Message Signatures (RFC 9421) and a public key directory the operator hosts at /.well-known/http-message-signatures-directory. A site or CDN checks the signature instead of trusting the User-Agent header, which anyone can fake. Cloudflare uses it to recognise signed agents such as ChatGPT agent, and Google says it is experimenting with it for Google-Agent.
From our data: Why it matters: as of 3 Nov 2026, 19 of the 214 software vendors in our rankings served a bot wall to an AI user-agent, and across our 1,846-company benchmark (11 Oct 2026) 11% did. Signed requests give a firewall a reliable way to tell a real assistant from a scraper borrowing its name.
How it works
The operator publishes keys
The agent's operator generates an Ed25519 key pair and serves the public key as a JSON Web Key Set at /.well-known/http-message-signatures-directory, with the content type application/http-message-signatures-directory+json.
Every request is signed
The agent adds a Signature-Input header (which parts are signed, plus created, expires, keyid and tag="web-bot-auth"), a Signature header, and a Signature-Agent header pointing at its key directory.
The site verifies
Your CDN or bot manager fetches the operator's keys and checks the signature. A valid one proves the request came from that operator, whatever IP it came from.
GET /pricing HTTP/1.1
Host: example.com
User-Agent: Mozilla/5.0 (compatible; ExampleAgent/1.0)
Signature-Agent: "https://agent.example.net"
Signature-Input: sig1=("@authority" "signature-agent");created=1762250000;expires=1762250060;keyid="<JWK thumbprint>";tag="web-bot-auth"
Signature: sig1=:<base64 signature>:Who signs today
ChatGPT agent
In Cloudflare's first cohort of signed agents (announced 28 Aug 2025), alongside Goose from Block, Browserbase and Anchor Browser.
Google-Agent
Google's user-triggered fetcher for agents on its infrastructure. Google says it is experimenting with Web Bot Auth using the identity https://agent.bot.goog.
Paying crawlers
Cloudflare's pay per crawl requires crawlers to use Web Bot Auth, with the payment headers included in the signed components.
What site owners should do
Stop allowlisting by user-agent alone
A User-Agent header proves nothing. Prefer signatures, then the operator's published IP ranges (OpenAI, Anthropic and Google all publish them).
Let verified and signed agents through
On Cloudflare, a verified signature sets the cf.bot_management.verified_bot field you can use in rules. Other bot managers have similar verified-bot lists; check that assistants you want are on them and allowed.
Keep proxies from stripping headers
If a proxy or load balancer sits in front of your bot manager, make sure Signature, Signature-Input and Signature-Agent reach it unchanged.
Still block bad behaviour
A signature proves identity, not good intent. Keep rate limits and abuse rules; apply them per operator instead of blocking every agent.
If you run an agent
Generate a key, host the directory, sign every request with a short expiry, and register: Cloudflare takes submissions through the Bot Submission Form in its dashboard (verification method: Request Signature). The protocol is still a draft. The architecture draft's last revision (-05, 2 Mar 2026) was replaced by draft-meunier-webbotauth-httpsig-protocol, so follow the current draft and your CDN's docs for exact header formats.
Check it yourself
The first command fetches an agent operator's public keys (swap in the operator's host). The second looks for signed and AI agent requests in your own logs; log the Signature-Agent header if you don't already. Then run the free check to see if AI user-agents get past your bot rules.
curl -s -D - https://agent.example.net/.well-known/http-message-signatures-directory | head -20 grep -iE "signature-agent|chatgpt-user|claude-user|google-agent" /var/log/nginx/access.log | tail -20
Questions
Is Web Bot Auth an official standard?
Not yet. It's an IETF Internet-Draft built on RFC 9421, HTTP Message Signatures, which is a published standard. Cloudflare, OpenAI and Google already use or test it.
Does Web Bot Auth replace robots.txt?
No. robots.txt says what a bot may crawl; Web Bot Auth proves which bot is asking. You can use the signature to apply your robots and access rules reliably.
Do I need to do anything if I use Cloudflare?
Cloudflare verifies signatures for you. Check that your bot and AI crawler settings allow the signed agents and verified bots you want, and that no custom rule challenges them anyway.
Can a signed agent still be blocked?
Yes. A signature tells you who it is; you still decide what it may do. Most sites should let assistants that act for a person read pages, and rate-limit rather than block.