Programmatic SEO AI agent fetches: what Grok reads from your pages

Grok’s live fetcher reads one thing from your programmatic page: the first raw HTML response your server sends. Anything a client-side script adds afterwards does not reach the answer, even when Grok’s fetcher runs the script. For a programmatic SEO AI agent visit, the unique data on each templated page (fields, tables, prices, counts) has to be in that HTML, and your bot protection has to let a fast, disguised burst of requests through. This post covers six things: what Grok sees, which template blocks to move to the server, how to test a page yourself, how WAF rules break it, and when the work isn’t worth doing.

Which Grok this is about

“Grok” now refers to two products, and they fetch pages differently.

• The chat fetcher. This runs when someone pastes a URL into Grok or asks about it. It sends spoofed user agents in parallel bursts. This post is about this one.
• Grok Bot. It launched in beta on 11 August 2026. xAI’s overview says each Bot gets “a browser, filesystem, and terminal” on a cloud computer. A full browser presumably renders JavaScript, although xAI doesn’t say so. When a site blocks automation or needs a human step, the Bot hands that step back to the user.

The rest of this post treats the chat fetcher as a non-rendering client, because that is how it behaves in the tests below.

Programmatic SEO AI agent fetches: what does Grok see?

Grok sees the HTML your server returns for the first request. The best evidence is Andre Alpar’s June 2026 test for Search Engine World. He gave 12 assistants their own secret, unlinked URLs. Each page’s raw HTML contained a decoy “reference number”, and an external script replaced it with the real one from a second endpoint.

Grok reported the decoy. ChatGPT, Claude, Gemini, Perplexity, Meta AI and Copilot also reported the decoy. DeepSeek, ERNIE, Qwen, Kimi and Mistral rendered the script and reported the real number.

On a directory page, the decoy is your loading skeleton. If the listings table, the price range or the “124 providers in Kraków” count arrives from a client-side fetch, Grok answers from the empty shell.

Does Grok render JavaScript when it fetches a URL?

Partly, but the rendering doesn’t affect the answer. In Alpar’s test, “One node of its worldwide proxy swarm actually executed the JavaScript”, and the follow-up call showed up in his logs. Grok still answered with the decoy. His summary: “The plumbing and the model were not talking to each other.”

The brief I started from said Grok behaves like the non-Google fetchers in the Vercel/MERJ study. That study (December 2024) didn’t test xAI. It found that OAI-SearchBot, ChatGPT-User, GPTBot, ClaudeBot, PerplexityBot and others don’t render JavaScript. ChatGPT and Claude download JS files (11.50% and 23.84% of their fetches) without running them. Only Gemini, through Googlebot, and AppleBot rendered.

The accurate statement for Grok is this: its answers come from the raw HTML even when its fetcher runs your JS. All of these tests are single-site or single-prompt, so treat them as strong hints, not as specifications.

What user agent does Grok use, and does it respect robots.txt?

You won’t see a Grok token. In StackFox’s February 2026 test, the tokens it calls documented (GrokBot/1.0, xAI-Grok/1.0, Grok-DeepSearch/1.0) never appeared in the logs. These did:

Go-http-client/1.1
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36
an iPhone Safari string (iOS 18)

StackFox points out that Chrome 139 dates from mid-2025, while stable Chrome was at 146 when it wrote the post. A stale Chrome version is one of the few tells.

StackFox saw 30 requests in under one second against a news site. One datacenter IP (M247 Europe) made 19 requests in about a second: the homepage, all 7 categories, then 9 articles. A second IP (Datacamp Limited) made 10 requests in about 0.3 seconds. DataDome research, as reported by The Weather Report, saw one prompt trigger 16 requests from 12 IPs described as residential and mobile-carrier addresses, at about 7 per second. So the IP type probably varies.

On robots.txt, StackFox lists xAI as neither checking robots.txt nor identifying itself. Claude-User, by contrast, fetches /robots.txt first. Bot directories such as Centinel Analytica note that xAI publishes no crawler documentation, IP ranges or reverse-DNS hostnames. A User-agent: GrokBot rule therefore matches nothing.

Which parts of a programmatic page template must be in the server-rendered HTML?

Vercel’s recommendation is to server-render “main content, metadata, navigation”. It says client-side rendering is fine for view counters, chat widgets and social feeds. Glenn Gabe’s October 2026 tests add a warning: a static header and footer around JS-rendered main content still failed for ChatGPT, Claude and Perplexity. Shipping the page chrome doesn’t count.

Applied to a programmatic template (my mapping, not a tested list):

Two caveats. First, I found no controlled test showing that Grok, ChatGPT or Claude use JSON-LD from a live fetch. Vercel only says JSON in the initial HTML “may” be indexed. Second, the line between shared blocks and unique data is the same one that matters for Google. I covered it in shared blocks vs unique page data. If you run Next.js, Making Next.js agent-readable shows where data fetching tends to slip to the client.

How do I test what Grok actually retrieved from one of my templated pages?

Copy Alpar’s decoy design and StackFox’s log matching. This sequence shows the steps:

sequenceDiagram
    participant You
    participant Grok
    participant Server
    You->>Server: Publish unlinked canary URL
    You->>Grok: Prompt with the URL
    Grok->>Server: Burst from proxy IPs
    Server-->>Grok: Raw HTML with decoy
    Grok-->>You: Reported value
    You->>Server: curl same URL
    You->>You: Compare logs, curl, answer

1. Make a copy of one real template page at an unlinked path. Put a unique canary string in the HTML. Add a decoy value in the raw HTML and load the real value with JS.
2. Ask Grok about that URL and note the time.
3. Look for a burst seconds later:

grep "/canary-7f3a" access.log | grep -E 'Go-http-client|Chrome/139|iPhone OS 18'

4. Fetch the page the way a non-rendering client does, with the same user agent:

curl -s -D headers.txt \
  -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36" \
  https://example.com/canary-7f3a > raw.html
grep -c "DECOY-VALUE" raw.html
grep -i "cf-mitigated" headers.txt

5. Compare. If Grok quotes the decoy, it read the raw HTML. If your log shows a 403 or the curl response is a challenge page, Grok read nothing useful at all.

Because the path is unlinked, any hit on it came from your prompt. That makes the burst easy to match, even when the user agent looks like a normal browser.

Will rate limiting or bot protection block Grok’s burst of requests to my directory pages?

It can, and it fails silently. A May 2026 write-up describes Cloudflare serving AI agents a challenge instead of content. The challenge is an HTTP 403 of about 1 KB, mostly JavaScript, with <meta name="robots" content="noindex,nofollow"> and a cf-mitigated: challenge header. A non-rendering agent can’t solve it. An auditing tool scored that site an F, with zero schema, zero words and no H1, because it was grading the challenge page.

The fixes it suggests, in order:

1. Don’t challenge Verified Bots.
2. Allow retrieval crawlers in Cloudflare’s AI Audit.
3. Add a WAF Skip rule by user agent. The author calls this the weakest option because user agents can be spoofed.

Grok’s chat fetcher makes that third option close to useless: a skip rule for Chrome 139 also lets through anyone else using that string. Cloudflare’s Verified Bots program requires transparent identity, which the chat fetcher doesn’t offer. One vendor claims Grok Bot is listed as a verified agent with Web Bot Auth, but I couldn’t verify that.

Rate limits are the other risk. 19 requests in a second from one IP, as StackFox saw, could trip a strict per-IP rule. That is my inference, not a tested result. DataDome’s pattern of 12 IPs at about 7 requests per second would pass per-IP limits but may trip bot scoring instead. Cloudflare’s rate-limiting rules can count by IP, IP with NAT support, ASN, headers or path. My recommendation is a separate, looser rule scoped to your template paths, then step 4 of the test above to confirm you get real HTML back.

When is optimizing programmatic templates for Grok not worth the engineering time?

Often. Statcounter’s August 2026 chatbot referral share, as compiled by fatjoe, lists ChatGPT at 79.4% and Gemini at 10.9%, and doesn’t list Grok at all. Dog on the Table puts Grok at about 2.4% of chatbot web visits per Similarweb, with no measurable referral share. Next to Google, that is close to zero for most directories.

Skip Grok-specific work when:

• your templates already server-render their data (run the curl check and you’re done)
• Grok sends no referrals you can see in your measurement setup
• loosening bot rules for spoofed traffic would expose you to scrapers you can’t afford

Server rendering is different, because it isn’t Grok-specific. The same raw HTML is what ChatGPT, Claude and Perplexity read. If your listing tables load client-side, fix that for all of them. Write Grok-specific WAF exceptions only if Grok turns up in your referral data.