How to serve markdown to AI agents
Serving markdown for agents means returning a clean markdown version of a page when a client asks for it with an Accept: text/markdown header, from the same URL, with Content-Type: text/markdown and Vary: Accept so caches keep the two versions apart. Agents read markdown with far fewer tokens than HTML: Vercel measured one of its pages at about 500KB of HTML and 3KB of markdown. Cloudflare can convert pages at the edge, and frameworks like Next.js can do it with a rewrite.
From our data: As of 3 Nov 2026, 101 of the 214 software vendors in our rankings served their docs as markdown, either through Accept: text/markdown or .md links in llms.txt. Cloudflare's Radar scan of the top 200,000 domains (published 17 Apr 2026) found 3.9% passed its markdown negotiation check.
The request and the response
GET /pricing HTTP/1.1 Host: example.com Accept: text/markdown, text/html;q=0.9 HTTP/1.1 200 OK Content-Type: text/markdown; charset=utf-8 Vary: Accept Link: <https://example.com/pricing>; rel="canonical" --- title: Pricing description: Plans for Example Co, as of Nov 2026. --- # Pricing - Free: $0, 3 projects - Team: $12 per user a month
Three ways to ship it
Cloudflare: turn it on
Markdown for Agents converts HTML at the edge for requests that ask for markdown, on Pro, Business and Enterprise plans and SSL for SaaS at no extra cost. Enable it in AI Crawl Control, through the API (the content_converter zone setting), or for specific paths with a Configuration Rule. It adds YAML frontmatter from your meta tags and keeps your JSON-LD.
Your framework: a rewrite
In Next.js, Vercel rewrites requests whose Accept header includes text/markdown to a route handler that returns the markdown with Content-Type: text/markdown. Any server can do the same with content negotiation.
.md URLs
Serve page.md next to each page and advertise it with <link rel="alternate" type="text/markdown" href="...">, as the llms.txt proposal suggests. GitBook and Mintlify do this for docs automatically.
// next.config.ts: send markdown requests for /blog/* to a route handler
export default {
async rewrites() {
return {
beforeFiles: [{
source: "/blog/:slug",
has: [{ type: "header", key: "accept", value: "(.*)text/markdown(.*)" }],
destination: "/blog/md/:slug",
}],
};
},
};Get the details right
Always send Vary: Accept
Without it, a cache can serve markdown to a browser or HTML to an agent.
Keep the content identical
Same facts, prices and dates as the HTML. Generate both from one source so they can't drift.
Advertise it
Many agents don't send the header. Cloudflare cites a February 2026 Checkly test in which only Claude Code, OpenCode and Cursor did by default, so also add the alternate link in your HTML head or a Link header.
Fall back to HTML
If a page has no markdown version, return the HTML with its normal content type, not an error.
Keep a markdown sitemap
Vercel serves /blog/sitemap.md and /docs/sitemap.md so agents can list pages without parsing XML.
Vendors whose docs serve markdown
101 of the vendors in our rankings, from our scan on 3 Nov 2026.
Check it yourself
You want content-type: text/markdown and vary: Accept. If you get text/html back, agents that ask for markdown get your full HTML instead. Try a docs or blog URL too, since many sites only convert those.
curl -s -D - -o /dev/null -H "Accept: text/markdown" https://example.com/ \ | grep -iE "^(content-type|vary|link):"
Questions
Is serving markdown to agents cloaking?
Not when the content is the same. It's the same page in another format, chosen by the client's Accept header, at the same URL with Vary: Accept. Keep facts identical and don't serve markdown to browsers.
Is this the same as llms.txt?
No. llms.txt is one index file listing your key pages. Markdown for agents gives every page a markdown version. They work together: link the .md versions from llms.txt.
Does Google use markdown versions?
Google's AI optimization guide says Search doesn't use markdown files. The benefit is for agents that read pages directly, which load them faster and with far fewer tokens.
Do I need a paid Cloudflare plan?
Yes, for the edge conversion: Cloudflare lists Pro, Business and Enterprise plans and SSL for SaaS. On any plan or host you can serve markdown yourself from your app.