# Programmatic SEO: Indexing

> Getting thousands of URLs crawled and indexed: sitemaps, canonicals, crawl budget, and pages Google decides not to keep.

Part of the Programmatic SEO hub: https://maciejwiatr.com/programmatic-seo

- [Programmatic SEO AI agent rules: letting Grok crawl safely](https://maciejwiatr.com/blog/grok-robots-txt-programmatic-seo-pages) (2026-10-31): Which Grok user-agent tokens to name in robots.txt, how to open entity pages but keep facets closed, and how to rate-limit agents without hurting Googlebot.
- [Programmatic SEO deindexing recovery: prune, merge or rebuild](https://maciejwiatr.com/blog/programmatic-seo-deindexing-recovery) (2026-10-30): How to diagnose crawled-but-not-indexed programmatic pages by template, decide what to prune, merge or improve, and measure recovery per template.
- [Programmatic SEO in Next.js: ISR, 404s and sitemaps at scale](https://maciejwiatr.com/blog/programmatic-seo-nextjs-app-router-guide) (2026-10-30): How to build 50,000+ programmatic pages in Next.js 15 and 16: what to prerender, ISR for the long tail, real 404 status codes, and sharded sitemaps Google trusts.