How to make your website ready for AI agents
A website is ready for AI agents when an assistant like ChatGPT, Claude or Perplexity can fetch its pages without hitting a bot wall, read the facts in plain HTML, and then finish the job a person asked for: answer a question about you, buy something, sign someone up or connect through your API. Most of it is plain web hygiene. The rest depends on which of those four jobs you want agents to do.
From our data: In our benchmark of 1,846 reachable companies (11 Oct 2026), the average readiness score was 41/100. 11% blocked AI agents with a bot wall, 50% published llms.txt and 20% ran a remote MCP server for their product.
1. Let agents in
Nothing else matters if the agent gets a challenge page. Check your firewall and bot rules first, then robots.txt.
Agents get the real page, not a bot wall
Serve AI user-agents (Claude-User, ChatGPT-User) the same HTML as browsers: no challenge, no 403. Check your WAF and bot rules.
robots.txt lets user-triggered agents in
Don't Disallow ChatGPT-User, Claude-User or Perplexity-User. They fetch pages because a person asked.
Answer-engine crawlers are allowed
Allow OAI-SearchBot, Claude-SearchBot, PerplexityBot and meta-webindexer in robots.txt so you can be cited.
2. Make every page readable
Agents read the raw HTML, usually without running your JavaScript. Put the facts in the markup and describe them with structured data.
Content is in the raw HTML
Server-render the page. If the copy isn't in the response to a plain request, most AI crawlers can't see it.
Semantic HTML (main, nav, header)
Use <header>, <nav>, <main> and <footer>, and real <button> and <a href> elements instead of clickable <div>s.
Icon buttons and links have names
Give every icon-only button or link an aria-label, like aria-label="Open menu".
Form fields have labels
Give every input a <label for> (or aria-label). A placeholder is not a label.
Images have alt text
Add descriptive alt text to every meaningful image (alt="" for decorative ones), and never put key facts only in an image.
The page has an H1
Give every page one H1 that says what it is, then H2s that read like a table of contents.
Structured data (JSON-LD)
Add JSON-LD: Organization (or Person on a personal site), plus Product, SoftwareApplication, LocalBusiness or Article depending on the page.
Meta description and Open Graph tags
Add a one-sentence meta description that defines what you are, plus og:title and og:description.
sitemap.xml
Publish sitemap.xml with honest lastmod dates and reference it from robots.txt.
llms.txt
Publish /llms.txt: a one-paragraph definition of what you do and a markdown list of your key pages.
Dates show the content is fresh
Show published and updated dates (and dateModified in JSON-LD). Put "as of" dates on key facts like pricing.
3. Make the job work
Pick the job you want agents to finish on your site. Each one has its own checks; a shop needs a cart agents can open, a SaaS product needs a sign-up page and public pricing, a developer platform needs an API key without a sales call.
Be recommended: Can an agent explain what you do, who it's for, what it costs and how to reach you?
Biggest checks: organization or localbusiness data; contact details in text; an about page agents can find.
Buy: Can an agent find a product, its price and stock, and get to checkout?
Biggest checks: product and offer data on product pages; cart and checkout open to agents; a machine-readable catalog.
Sign up: Can an agent get a new user from your homepage to a working account?
Biggest checks: public pricing page; a self-serve sign-up link; sign-up page opens for agents.
Get an API key: Can a coding agent get a working API key?
Biggest checks: remote mcp server; openapi spec; self-serve api keys.
Check it yourself
Fetch your homepage the way an AI agent does. 200 means it gets the page. 403, 429, 503, or a page that says "Just a moment" means a bot wall is in the way. Then run the free check for the full 100-point score.
curl -s -o /dev/null -w "%{http_code}\n" \
-A "Mozilla/5.0 (compatible; Claude-User/1.0)" https://example.com/
curl -s -o /dev/null -w "%{http_code}\n" \
-A "Mozilla/5.0 (compatible; ChatGPT-User/1.0; +https://openai.com/bot)" https://example.com/Questions
What does agent-ready mean?
That an AI agent can reach your site, understand it and complete a task on it for a person without getting stuck. We score it out of 100: 60 points every site needs and 40 for the job you want agents to do.
Is this the same as SEO?
It overlaps. Server-rendered pages, structured data and sitemaps help both. Agent readiness adds things search engines don't need: not blocking AI user-agents, llms.txt, markdown docs, and an action path (checkout, sign-up or API key) an agent can finish.
Do I need an MCP server?
Only if you want agents to act inside your product, such as creating records or running reports. A shop or a content site gets more from fixing bot walls, structured data and checkout first.
How do I check my site?
Run the free check on agentsready.dev. It probes your site as an AI agent, scores it and lists the fixes in order, then can send a real agent to try the job.