NewAgent-ready skill for Claude Code, Cursor and Codex →
AgentsReady
Guides · As of

How do AI agents browse websites?

AI agents read websites in three ways. Fetchers like ChatGPT-User and Claude-User request the page over HTTP and turn the HTML (or markdown, if you serve it) into text, usually without running JavaScript. Browser agents, like ChatGPT agent and the agents Google runs as Google-Agent, load the page in a real browser and read the accessibility tree: every button, link and field with its role, name and state. Both can also take screenshots to understand layout. Most agents combine them, so a site works best when all three tell the same story.

From our data: As of 3 Nov 2026, of the 214 software vendors in our rankings, 198 had their homepage copy in the raw HTML, 140 used semantic landmarks (main plus nav or header), 165 gave names to (nearly) all icon buttons, and 171 labelled at least 90% of form fields.

The three ways agents read a page

  1. A plain fetch

    One HTTP request, then the HTML is stripped to text. It's fast and cheap, which is why chat assistants and coding agents use it to read pages. Content that only appears after JavaScript runs isn't there. Some clients ask for markdown with an Accept: text/markdown header: Cloudflare cites a February 2026 Checkly test in which 3 of 7 agents (Claude Code, OpenCode and Cursor) did so by default.

  2. The accessibility tree

    Browser agents read the same structure screen readers use: a list of interactive elements with roles (button, link, textbox), names ("Add to cart") and states (checked, expanded). web.dev's guide calls it the agent's view of what can be done on a page. A <div> with a click handler and no role doesn't appear as a button at all.

  3. Screenshots

    A vision model looks at the rendered page to judge layout and importance. It is slower and costs more tokens, so agents tend to use it to confirm what the DOM and accessibility tree already said.

What breaks each one

  1. Client-side rendering

    If the HTML response is an empty app shell, a plain fetch sees nothing. Server-render or prerender the pages agents need.

  2. Clickable divs

    Use <button> and <a href>. If you can't, add role="button" and tabindex="0", as web.dev recommends.

  3. Unnamed icon buttons

    A magnifying-glass icon with no text is an unnamed button in the accessibility tree. Add aria-label="Search".

  4. Placeholder-only fields

    Placeholders disappear and aren't reliable names. Link a <label for> to every input.

  5. Ghost overlays and shifting layouts

    A transparent layer over a button can hide it from visual analysis, and an "Add to cart" button that moves between pages confuses screenshot-based agents. Both are called out in web.dev's guide.

  6. Pop-ups over the content

    Cookie walls, newsletter modals and app-install banners get in the way of every kind of agent. Keep them dismissible with a real, named button.

A control agents can and can't use

<!-- Invisible to the accessibility tree as a button, and unnamed -->
<div class="btn" onclick="addToCart()"><svg>...</svg></div>

<!-- What agents (and screen readers) need -->
<button type="button" aria-label="Add to cart" onclick="addToCart()">
  <svg aria-hidden="true">...</svg>
</button>

<label for="email">Work email</label>
<input id="email" name="email" type="email" autocomplete="email">

See your site the way an agent does

  1. Read the raw HTML

    Run the command below. If your headline, prices and key facts aren't in the output, fetch-based agents can't see them.

  2. Inspect the accessibility tree

    In Chrome DevTools, open Elements, then the Accessibility pane (or turn on the full-page accessibility tree). Every control you'd click should have a role and a name.

  3. Ask for markdown

    Request the page with Accept: text/markdown. If you get markdown back, agents that ask for it save most of their tokens.

  4. Run the free check

    It fetches your site as Claude-User, scores semantic HTML, named controls, labelled fields and alt text, and can send a real agent to try the job.

Check it yourself

The first command shows the text a fetch-based agent gets from your homepage. The second asks for markdown: content-type: text/markdown plus vary: Accept means you serve it; text/html means agents get the full HTML.

curl -s -A "Mozilla/5.0 (compatible; Claude-User/1.0)" https://example.com/ \
  | sed -e 's/<script[^>]*>.*<\/script>//g' -e 's/<[^>]*>//g' | tr -s ' \n' | head -c 1500
curl -s -D - -o /dev/null -H "Accept: text/markdown" https://example.com/ \
  | grep -iE "^(content-type|vary|link):"
Run the free check on your siteGet the full checklist

Questions

Do AI agents run JavaScript?

Fetch-based agents (the ones that read a page mid-chat) usually don't. Browser agents do, because they run a real browser. Server-render key content and you're covered either way.

What is the accessibility tree?

A browser-built summary of a page's interactive elements with their roles, names and states, made for assistive technology. Browser agents use it as a map of what they can click and type into.

Do AI agents look at screenshots?

Some do, usually to check layout alongside the DOM and accessibility tree. It's the slowest and most expensive way for them to read a page, so clean HTML matters more.

Does making a site agent-friendly hurt human users?

No. It's mostly accessibility work: semantic elements, labels, alt text and stable layouts. web.dev's guide notes the same changes improve the site for people.