Making Next.js agent-readable

The test

Run this against one of your docs pages:

curl -sL https://maciejwiatr.com/blog/making-nextjs-agent-readable -H "accept: text/markdown" | wc -c

This page answers with 9.1 KB of markdown. The same URL without the header returns 25 KB of HTML. If your site returns the same HTML both times, an agent reads the whole page, including the navbar. You can run the same test here:

Check this page with the agent-readable checker

What is in a 500 KB page

When an agent fetches a docs page, it gets what a browser gets: the navigation, the cookie banner, a script tag for every bundle, font preloads, analytics and a footer with forty links. It pays tokens for all of it and then has to find the article inside. Vercel measured one of its own pages at about 500 KB of HTML against 3 KB of markdown, a 99.37% cut. Its functions docs page, as I write this, is 975,833 bytes of HTML and 11,807 bytes of markdown.

My pages are smaller. Here are three of them, measured both ways:

This page loads no analytics or cookie banner, so the biggest share of its extra HTML is syntax highlighting. The code blocks below take about 10 KB of HTML, because each token sits in its own styled <span> and there are more than 200 of them. The markdown carries the same code as plain text with a language marker. The rest is the head metadata, the nav, the footer and the checker above.

The homepage is a hydrated React page with canvas scenes. I generate its markdown at build time from the pre-rendered HTML, and it contains only the page’s text.

The mechanism

Content negotiation is a standard part of HTTP. The client lists the formats it accepts in the Accept header and the server picks one representation of the same resource. A browser asks for text/html, and an agent that wants markdown asks for text/markdown at the same URL.

This works better than a separate .md route because the agent doesn’t need to know anything about your site. An agent that sends the header gets markdown from every site that supports it, and the URL it shows its user is one a person can open in a browser.

The implementation

This site is a static Astro build on Vercel, so the version I ship is mostly configuration. Each post is built twice from one markdown file, as HTML and as .md, and vercel.json picks one per request:

{
  "rewrites": [
    {
      "source": "/blog/:slug([^/.]+)",
      "has": [{ "type": "header", "key": "accept", "value": "(.*)text/markdown(.*)" }],
      "destination": "/blog/:slug.md"
    },
    { "source": "/blog/:slug([^/.]+)", "destination": "/html/blog/:slug" }
  ],
  "headers": [
    { "source": "/blog(/.*)?", "headers": [{ "key": "Vary", "value": "Accept" }] }
  ]
}

The HTML lives under /html/ because Vercel serves a file that matches the URL before any rewrite runs. If the HTML were at /blog/making-nextjs-agent-readable.html, the markdown rule would never fire. The homepage has the same problem. Vercel treats any index.* file as the directory index before rewrites run, so the homepage is built to _index.html and reached through a rewrite as well. Next.js also checks files before rewrites, and a beforeFiles rewrite runs ahead of that check.

On a Next.js site the same idea is a beforeFiles rewrite on the header and a route handler that returns the markdown. I ran both files in a scratch Next.js 16.4 app before putting them here:

import type { NextConfig } from "next";

const nextConfig: NextConfig = {
  async rewrites() {
    return {
      beforeFiles: [
        {
          source: "/blog/:slug",
          has: [{ type: "header", key: "accept", value: "(.*)text/markdown(.*)" }],
          destination: "/blog/md/:slug",
        },
      ],
    };
  },
};

export default nextConfig;

import { getPostMarkdown } from "@/lib/posts";

export async function GET(_req: Request, { params }: { params: Promise<{ slug: string }> }) {
  const { slug } = await params;
  const markdown = await getPostMarkdown(slug);
  if (!markdown) return new Response("Not found", { status: 404 });
  return new Response(markdown, {
    headers: { "Content-Type": "text/markdown; charset=utf-8", Vary: "Accept" },
  });
}

My posts are already markdown files, so I don’t need a conversion step. The markdown version is the source file without its frontmatter, with a title line added on top and site links made absolute. Code fences keep their language markers. A CMS-backed site has to turn rich text into markdown in that handler. I would build that markdown from the same content in the same deploy as the HTML, so an edit can’t reach one version and miss the other.

What I’d review in the PR

Vary: Accept. One URL now has two bodies, so every cache between the server and the client has to key on the Accept header. Without Vary: Accept, a CDN can cache whichever version was requested first and serve it to everyone after, so browsers can get markdown and agents can get HTML. On this site vercel.json sets it on both representations under /blog. In Next.js I only managed to set it on one of the two responses. In my scratch app under next start, the route handler’s Vary: Accept came through, but the prerendered HTML page did not get it from a headers() rule or from a proxy.ts, because Next sends its own Vary list on those pages. Vercel’s docs show the same split. When I checked them on 25 October 2026, the markdown response carried Vary: Accept and the HTML response did not. Curl both representations after you deploy and look at what your CDN keys on. If your site sits behind Cloudflare, its cache keys on the URL and a few fixed headers. Since July 2026 it can vary on Accept, but only for headers you add to a Cache Rule.

Agents that send */*. I checked what common HTTP clients send by default. curl 8.7, Node’s fetch and Python’s requests send Accept: */*. Python’s urllib sends no Accept header. None of those requests match the markdown rule, so all four clients get HTML. For those clients every post has a <link rel="alternate" type="text/markdown"> in its head, and /llms.txt lists the markdown URLs. I point the alternate link at the .md file because a client that followed the page URL while sending */* would get the HTML again.

Regex header matching. The rewrite looks for text/markdown anywhere in the header and ignores quality values. A client that sends text/html, text/markdown;q=0.1 prefers HTML but still gets markdown. No browser sends that header, so I leave it as is. If it ever matters, I’ll switch to a route handler that parses Accept and its quality values.

A markdown sitemap. An XML sitemap is a flat list of URLs for crawlers. /blog/sitemap.md gives an agent titles, dates and one-line summaries, newest first, which is enough for it to decide what to read. /blog serves it when asked for markdown:

export const GET: APIRoute = async () =>
  new Response(blogSitemap(await getPosts()), {
    headers: { "Content-Type": "text/markdown; charset=utf-8" },
  });

Both versions from one file. The HTML page and the markdown come from the same file in the same build, so an edit to the content always changes both. The test suite checks that every post’s markdown contains its source body unchanged.

Try it on your site

Paste one of your docs URLs into the checker, or run the curl from the top of this page against it.

Agents are rarely part of a docs site’s requirements yet, so on most teams nobody owns this. The rewrite and the route handler above fit in a single PR.

Sources

• Vercel, Making agent-friendly pages with content negotiation, 3 February 2026
• Next.js docs, rewrites (beforeFiles, has)
• Next.js docs, Route Handlers
• Vercel docs, rewrites in vercel.json
• Cloudflare docs, Vary
• RFC 9110, section 12.5.1, Accept
• llms.txt