Programmatic SEO and AI agents: is Grok Bot hitting your pages?

Tell Grok's undeclared fetchers apart from the Grok Bot browser agent in your logs, then decide which programmatic templates to allow, rate-limit or block.

Maciej Wiatr

7 min read

Read as markdown

“Grok” now names two kinds of traffic that reach a programmatic site. One is xAI’s chat and search fetching, which has been seen in logs. The other is Grok Bot, a computer-use agent that drives a real browser and can buy things. If you run programmatic SEO pages, the useful questions are: which of the two is hitting your templates, can you tell, and do you want it? This post separates the two, shows the log signals that identify them, and sets out when to allow, rate-limit or block.

Programmatic SEO and AI agents: do agents like Grok Bot use your pages?

Very likely, but nobody has published logs of Grok Bot on a programmatic site yet. Here is what the sources do show.

xAI launched Grok Bot in beta on August 11, 2026. Bots “have their own computer” in the cloud and can “sign in and work across apps, tools, and websites, including platforms with no clean API or MCP.” Digital Applied describes a persistent cloud computer with a browser, filesystem and terminal. Each Bot runs one computer-use task at a time.

The agent shops too. RuntimeWire reported on August 28 that Bots can buy through Stripe Link single-use cards, with each charge approved by a human. On October 10, Musk posted “I just ordered LEGO and Grok bought it from the LEGO website.”

The fetch side has been logged. In a February 6, 2026 test, StackFox watched Grok’s fetches move from the homepage to all 7 categories and then to articles. That is the hub-to-leaf order a directory is built around.

What is the difference between GrokBot, xAI-Grok, Grok-DeepSearch and the Grok Bot agent?

NameWhat it is said to beStatus
GrokBot/1.0General crawlingListed by StackFox, not seen in its logs
xAI-Grok/1.0TrainingBotcrawl calls it “unconfirmed”
Grok-DeepSearch/1.0Deep research fetchesListed by StackFox, not seen in its logs
Grok BotComputer-use agent on a cloud VMOfficial product, no documented user agent

StackFox says xAI documented the three tokens. NoHacks says xAI publishes no crawler docs, no IP file and no robots.txt contract, and puts it in the “undeclared” group with Bytespider. I couldn’t find an official xAI page that lists these tokens. Treat them as rumoured.

Does Grok respect robots.txt, and can you block it by user agent?

The evidence says no to both. In StackFox’s test, “Grok” and “xAI” never appeared in any user agent. The requests came as Go-http-client/1.1, a macOS Chrome/139.0.0.0 string and an iPhone OS 18_0 Safari string. Chrome 139 was about 7 months old at the time, with stable at 146. The test also saw no robots.txt fetch, so the rules there had nothing to apply to. An @grok post on X says it sends an iPhone UA “to avoid website blocks.” Self-reports from an LLM are weak evidence, and StackFox says so too.

Grok Bot is less clear still. Neither the launch page nor the security docs say what UA it sends, whether it signs requests or whether it reads robots.txt. NoHacks notes that browser-driving agents like Claude for Chrome and Comet send the normal browser UA. My inference is that Grok Bot will look like a current Chromium browser. No source confirms it.

Other agents have opted in to identification. ChatGPT agent signs requests with Web Bot Auth. Google-Agent publishes IP ranges and signs a subset of requests as https://agent.bot.goog. Grok offers neither, so a UA rule in your robots.txt or WAF will only match traffic that has chosen to be matched.

How do you spot undeclared AI agent traffic in server logs?

Two log studies give you the pattern. They are not proof of Grok Bot behaviour: StackFox’s test ran six months before the agent launched, and the second study only links to xAI “probably.”

Kitsunemimi’s December 2025 notes logged more than 10,000 requests in about a month carrying a Trailer/93.3.8652.5 token on spoofed Chrome 107 to 134 and Safari UAs. The top ASNs were Datacamp AS212238 (2,517 requests) and M247 AS9009 (2,210). Over 80% of URLs were requested only once, and 91% of IPs made a single request. StackFox saw the same two ASNs. Its “30 requests in under one second” adds up to 29 in its own table: 19 from one IP in about a second and 10 from another in about 0.3 seconds.

Look for these signals together:

A first pass on a combined access log:

# Requests per second per IP, busiest first
awk '{print $4, $1}' access.log | sort | uniq -c | sort -rn | head -50

# Fetcher-style UAs worth a closer look
grep -E 'Go-http-client|Trailer/|Chrome/1[0-3][0-9]\.' access.log | awk '{print $1}' | sort | uniq -c | sort -rn

Then resolve the top IPs to ASNs. A single signal means little, but four of them on the same IPs is a pattern.

Should a directory, marketplace or SaaS comparison site allow, rate-limit or block Grok’s traffic?

None of the sources gives a framework for this, so the one below is mine. It rests on two facts.

The first is that agents now complete purchases. Grok Bot bought from lego.com. Amazon went the other way: a user posted an Amazon warning about an unauthorized AI agent, and Musk replied “Maybe we should start a competitor to Amazon.” Amazon didn’t confirm a Grok-specific block.

The second is that bot traffic costs money. Read the Docs cut daily traffic from 800GB to 200GB by blocking AI crawlers and saved about $1,500 a month. A Vercel Hobby app with two users went over its limits on about 882K ClaudeBot requests. If your leaves are rendered on demand, every first visit from a bot is a render you pay for. My Next.js ISR guide covers where that cost sits.

flowchart TD
    A[Agent traffic on template] --> B{Can agent act here?}
    B -->|Yes| C[Allow and fix template]
    B -->|No| D{Expensive render per hit?}
    D -->|Yes| E[Rate-limit at WAF]
    D -->|No| F{Data easily scraped?}
    F -->|Yes| G[Challenge or block]
    F -->|No| H[Allow, monitor logs]

One detail helps with the middle ground. xAI’s docs say a CAPTCHA hands control back to the human (“Take over”, then “I’m done”). So a challenge on a form step pauses Grok Bot and its user can finish. It doesn’t kill the sale.

Geography is a trade-off too. One user found sites geo-blocking their Grok Bot because the VM’s IP sat in an unexpected country. Country rules can block a buyer you wanted.

How should programmatic templates change so a browsing agent can complete tasks?

A CHI 2026 study cited by SEJ ran Claude Sonnet 4.5 on 60 real tasks. It succeeded 78.33% of the time in standard conditions, 41.67% with keyboard only and 28.33% with a magnified viewport. The failures came from missing ARIA state, lost task state, and drag-and-drop or keyboard shortcuts.

What I would check in a template review:

Start with semantic HTML before ARIA. Adrian Roselli warns against ARIA keyword stuffing. I wrote up the Next.js side in making Next.js agent-readable.

Is SEO dead now with AI?

No. Google’s AI features page says “There are no additional requirements to appear in AI Overviews or AI Mode,” and those features draw on the Search index. Clicks are a separate question. Ahrefs reports 58% lower CTR for #1 results when an AI Overview appears, while Seer saw AIO-query CTR recover from 1.3% to 2.4% between December 2025 and February 2026. Both figures come via secondary summaries.

Keyword tools like Ahrefs or LowFruits tell you which templates to build. They tell you nothing about who loads those pages afterwards. That’s what your logs are for.

My recommendation

Split your templates by what an agent can do on them. Leave action pages open and make them easy to complete. Put per-path rate limits on expensive leaves. Challenge the routes where the data itself is the product. Don’t spend time on Grok UA rules or robots.txt lines, because there is no documented token for them to match. Review logs monthly for the combined signals above.

This approach has costs. Rate limits will slow some real users on shared proxies. Challenges add a human step for buyers using agents. Allowing agents through means paying for renders you can’t attribute. If you can’t see the traffic in your logs, measure the problem before you build anything to handle it.