Programmatic SEO AI agent rules: letting Grok crawl safely

Should a programmatic SEO AI agent like Grok be allowed onto 50,000 templated pages? Usually yes, but only onto the pages you would want quoted, and only with limits you enforce yourself. This post covers which Grok tokens to name in robots.txt, how to open entity pages and hubs while keeping facets closed, why robots.txt is not enough for Grok, how to rate-limit agents without hurting Googlebot, and how to confirm in your logs that the setup works. Earlier posts covered detecting Grok and what its fetcher reads, so I won’t repeat that here.

Should a programmatic SEO AI agent like Grok crawl your templated pages?

For canonical entity pages and hubs, I’d let it in. The upside is small, though. Similarweb put Grok at about 3.4% of gen-AI website traffic in early January 2026. A Similarweb-based breakdown found “no measurable” referral share for Grok, and one B2B referral panel called Grok’s outbound referrals “effectively zero.”

The cost is server load. In a StackFox test, one Grok fetch produced about 30 requests in about 1 second. On a site with facets, that kind of burst lands on thousands of URLs nobody needs fetched.

So the default I recommend is: open the pages that answer questions, close everything combinatorial, and enforce the split at the edge.

Is SEO dead now with AI?

No. Google’s AI features documentation says AI Overviews and AI Mode need no special optimization. Pages must be indexed and eligible for snippets. Clicks are falling, though. In Pew’s March 2025 browsing data, as reported by AI Weekly, people clicked a traditional result 8% of the time when an AI summary appeared, against 15% without one. They clicked a source inside the summary 1% of the time. Google disputed the methodology.

Classic indexing still decides what gets surfaced. I covered what Google’s 2026 updates removed in Is programmatic SEO dead?.

Which user-agent tokens does Grok use, and which respect robots.txt?

xAI publishes no crawler documentation that anyone could fetch. NoHacks says xAI makes “no robots.txt contract” and has “no JSON IP range file.” Here is what the third-party sources report:

Token	Reported by	Seen in real logs?	robots.txt
GrokBot	ai.robots.txt, StackFox	No tester reports it	“Unclear at this time”
xAI-Grok	ai.robots.txt, Botcrawl	No	“Unknown”
Grok-DeepSearch	Botcrawl, StackFox	No	“Unknown”
GrokAgent	Cloudflare Radar, via AICrawlerCheck	Yes, GrokAgent/1.0 (u:…)	Undocumented

StackFox says xAI documents the first three, but also says none of them appeared in any log its author examined. Members of a WebmasterWorld thread found no “grok” or “xai” in raw logs at all, and nobody saw Grok fetch robots.txt.

GrokAgent is the only token with a verifiable mechanism. RuntimeWire captured GrokAgent/1.0 (u:<16 hex characters>) from Grok Bot’s cloud Chrome browser, where the 16 characters are an account-derived ID. AICrawlerCheck says it first appeared on October 11, 2026, and that it visits only when a person asks Grok to open a page. Its Web Bot Auth key directory sits on cursorusercontent.com, not on an xAI domain. Nobody has explained that arrangement. Of 5,583 valid robots.txt files from the Tranco top 10,000, AICrawlerCheck found none that named GrokAgent.

Treat a Grok robots.txt group as a statement of intent that compliant agents may read. It does not control access.

How do you write robots.txt rules for Grok?

The rule that trips people up: a crawler obeys only its most specific matching group. Google’s robots.txt spec says “user agent specific groups and global groups (*) are not combined.” If you add a Grok group with only Allow: /, you have just opened your facets to it. Repeat every Disallow inside the named group.

User-agent: *
Disallow: /search/
Disallow: /filter/
Disallow: /*?*sort=
Disallow: /*?*color=

User-agent: GrokBot
User-agent: xAI-Grok
User-agent: Grok-DeepSearch
User-agent: GrokAgent
Disallow: /search/
Disallow: /filter/
Disallow: /*?*sort=
Disallow: /*?*color=
Allow: /

Sitemap: https://example.com/sitemaps/index.xml

Under Google’s precedence rules the longest path wins, so /filter/ beats Allow: /. Google supports * and $. Nobody knows whether Grok agents parse wildcards, so for non-Google groups I’d rely on plain path prefixes like /filter/ wherever your URL design allows. That is an argument for putting facets under a path rather than in query strings.

Other details from the spec:

• sitemap lines apply to every agent, so list your hub and entity sitemaps once.
• The file limit is 500 KiB, and robots.txt is cached for up to 24 hours.
• Google does not support crawl-delay.

Google’s faceted navigation page adds that filter combinations with no results should return 404. That fixes the problem for every crawler at once. Near-empty entity pages are better fixed or removed than hidden. I covered that choice in prune, merge or rebuild.

Why isn’t robots.txt enough for Grok?

Some Grok retrieval traffic reportedly doesn’t identify itself. In a DataDome test reported by PPC Land, one Grok fetch prompt produced 16 requests from 12 IPs, none identifying as xAI or Grok. The user agents were spoofed Chrome on macOS and Safari on iPhone. Seven near-simultaneous requests used Go-http-client/1.1. The IPs came from hosting providers, residential ISPs and mobile networks.

StackFox saw two datacenter or proxy IPs sending Chrome 139 when the stable version was 146. One IP fetched the homepage, 7 categories and 9 articles in about a second. WebmasterWorld members dispute the iPhone attribution, and both DataDome and StackFox sell bot protection. Even so, a rule keyed on a user-agent string can’t catch traffic that doesn’t send one.

There is no public IP list. AICrawlerCheck reports that xAI says Grok Bot uses shared static egress ranges “available from its account team.”

How to enforce access at the edge without an IP list

Cloudflare verifies bots by Web Bot Auth signature, published IPs with a stable UA, or reverse DNS. It exposes the result as cf.bot_management.verified_bot. Web Bot Auth checks Signature, Signature-Input and Signature-Agent headers against keys at /.well-known/http-message-signatures-directory. ModPageSpeed warns that a published directory “does not show that every request is signed,” and recommends pinning trusted directory URLs.

This flowchart shows the order I’d evaluate requests in:

flowchart TD
    A[Request arrives] --> B{Verified search bot?}
    B -->|Yes| C[Serve, no rate limit]
    B -->|No| D{Signed, pinned directory?}
    D -->|Yes| E[Apply agent rate limit]
    D -->|No| F{Burst or odd fingerprint?}
    F -->|Yes| G[Return 429 Retry-After]
    F -->|No| H[Serve normally]

The “odd fingerprint” signals from the tests above are outdated Chrome versions, Go-http-client, many requests per second from one IP, and hits on honeypot links. IPs rotate, so treat any IP block as temporary.

How do you rate-limit AI agents without hurting Googlebot?

Exempt verified Googlebot first. Google lets you verify requests by reverse DNS plus forward DNS, or against files like common-crawlers.json. Cloudflare itself warns that rate-limiting verified bots may affect SEO.

Then limit everything else, within your plan:

Source: Cloudflare rate limiting rules. JA3/JA4 counting needs Bot Management.

Throttle with 429 and Retry-After. Google says not to use 401 or 403 to limit crawling.

Google’s crawl budget page says crawl capacity “is shared across all crawlers,” but it means Google’s own crawlers. My reading is that AI bots don’t spend Googlebot’s budget directly. They hurt it by slowing the server or causing 5xx and 429 responses, which lowers Googlebot’s capacity limit. Caching and 304s make agent bursts cheap. If a 429 rule ever catches Googlebot, Google’s crawl rate guidance says not to keep it up longer than 1 to 2 days.

Decision table: allow, limit or block

These are my recommendations, based on the sources above:

Programmatic SEO software and keyword tools like LowFruits help you decide which pages to build. They don’t decide who gets to fetch them.

How do you check in server logs that your Grok config works?

Start with the declared traffic:

# Declared Grok tokens, including the signed agent pattern
grep -Ei 'grokbot|xai-grok|grok-deepsearch|GrokAgent/1\.0 \(u:' access.log

# Declared Grok hits on paths you disallowed (should be empty)
grep -Ei 'grokbot|xai-grok|grok-deepsearch|grokagent' access.log \
  | grep -E '/(search|filter)/|\?.*(sort|color)='

# Canary: every hit on a unique, unlinked URL you asked Grok to open
grep '/canary-7f3a9c' access.log

The canary repeats the DataDome and StackFox method. Note the IPs, ASNs, user agents and timing of every hit, then confirm your edge rule throttled the unsigned ones and let signed requests through. Watch honeypot URLs for hits. Finally, check Googlebot’s crawl stats and response times in Search Console for regressions after any rule change.

If you serve pages from Next.js, Making Next.js agent-readable covers what the agent should find once it’s allowed in.