# How to noindex programmatic pages with Search Console data

> A template-level audit for programmatic sites: count indexing per template, pick what Google's new guidance says to noindex, ship it and track recrawl.

Maciej Wiatr, 2026-11-02. https://maciejwiatr.com/blog/noindex-programmatic-pages-search-console-audit

On October 26, 2026, Google told sites that let users create shareable pages, generated outputs or tool results to keep those pages out of the index by default. On a programmatic site, that rule covers more templates than most teams expect. You can't get the full answer from the Page indexing report alone, so the way to noindex programmatic pages with Search Console data is a template-level audit. Count indexing per template, compare indexed pages with those marked crawled but not indexed, decide which templates were never meant for Search, ship noindex, and track the counts until Google recrawls. Below is the audit as I'd run it on a directory like mine, which has about 200,000 indexed pages, with the code to get each number.

## What does "Only index content meant for Search" mean for programmatic pages?

Google added the section to [Prevent user-generated spam on your site and platform](https://developers.google.com/search/docs/monitor-debug/prevent-abuse), which was last updated 2026-10-26 when I checked it in November 2026. The guidance asks you to "consider whether that content is meant to be found through search engines or only accessed directly by people with a link." For pages made "for direct sharing, collaboration, or utility rather than public search discovery", it says to apply noindex with a robots meta tag or an X-Robots-Tag header, or to block crawling in robots.txt, by default.

The [changelog](https://developers.google.com/search/updates) gives the reason: to keep user-created pages "that aren't meant for Search out of the index and detect spam earlier."

Some things are not documented as of November 2026:

- Google gives no numeric thresholds for the spikes you should watch for, as [PPC Land](https://ppc.land/google-tells-platforms-to-noindex-link-shared-pages-by-default/) also noted.
- There is no enforcement date. [SEJ](https://www.searchenginejournal.com/google-signals-a-wider-crackdown-on-ugc-seo-spam/594679/) says the update "does not establish that a crackdown is imminent."
- There is no new Search Console feature. The guidance and Search Console are both free.

On a programmatic site, the templates in scope are calculator result URLs, saved comparisons, shared lists and per-query generated summaries. The incidents behind the guidance are similar. PPC Land cites more than 370,000 Grok conversations indexed in 2025. Duck.ai share pages launched in October 2026 with noindex and a 48-hour expiry.

## Should I noindex or block programmatic pages in robots.txt?

Use noindex for anything that is indexed or could be. Google's [noindex documentation](https://developers.google.com/search/docs/crawling-indexing/block-indexing) says that if robots.txt blocks a page, "the crawler will never see the noindex rule." John Mueller [put it more briefly](https://www.searchenginejournal.com/why-google-indexes-blocked-web-pages/526257/): "if we can't crawl the page, we can't see the noindex."

The new section lists noindex and robots.txt as equal defaults, and [Elsop](https://www.elsop.com/google-shared-pages-noindex-default) points out that Google hasn't said which reading it means. My reading, which is an inference and not a Google statement, works like this:

1. For URL patterns that were never indexed, robots.txt is fine. [Google's crawl budget guide](https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget) recommends it for pages you never want crawled, on sites of roughly 1M+ pages.
2. For URLs that are already indexed, apply noindex first and wait until they drop out. Only then consider a robots.txt block.

Noindex has a cost. Google still requests the page. Mueller said noindexed URLs "end up in the Search Console report for crawled/not indexed", and added that "neither of these statuses cause issues to the rest of the site."

## The audit, step by step

The flowchart shows the five steps and the decision in the middle.

```mermaid
flowchart TD
    A[Sitemaps split by template] --> B[Counts per template]
    B --> C[Sample with Inspection API]
    C --> D{Meant for Search?}
    D -->|No| E[Noindex, crawl allowed]
    D -->|Yes but weak| F[Merge or improve]
    D -->|Yes and earning| G[Keep indexed]
    E --> H[Track before and after]
```

### Step 1: Export the Page indexing report past the 1,000-row cap

The [Page indexing report](https://support.google.com/webmasters/answer/7440203) has two limits. The example table for each status "is limited to 1,000 items, and isn't guaranteed to show all URLs." You also can't filter it by URL. Bulk export to BigQuery won't help here, because [it covers Performance data](https://developers.google.com/search/blog/2023/02/bulk-data-export) and I found no source saying coverage data is included.

So you build the export from three workarounds:

- **One sitemap set per template.** Each sitemap holds at most 50,000 URLs or 50 MB uncompressed. If you filter the Page indexing report by one sitemap, every count you see belongs to a single template. The [Sitemaps report](https://support.google.com/webmasters/answer/7451001) handles up to 1,000 sitemaps, and the Sitemaps API returns the same data.
- **A URL-prefix property per directory.** Each property gets its own 1,000 example rows and its own Inspection quota. [SEJ](https://www.searchenginejournal.com/how-to-make-search-console-work-harder-for-you/560697/) notes that an account can hold up to 1,000 properties.
- **URL Inspection API samples.** The [quota](https://developers.google.com/webmaster-tools/limits) is 2,000 queries a day and 600 a minute per property. Inspecting all 200,000 URLs under one property would take 100 days, so sample a few hundred URLs per template instead.

### Step 2: Group URLs by template

Your own database or sitemaps give the full URL list. Search Console statuses come from the per-sitemap counts and the Inspection samples. Join the two and assign each URL to a template:

```python
import csv, re, sys
from collections import Counter, defaultdict

TEMPLATES = [
    ("listing",    re.compile(r"^/[a-z-]+/[a-z-]+/?$")),
    ("city_hub",   re.compile(r"^/in/[a-z-]+/?$")),
    ("compare",    re.compile(r"^/compare/.+-vs-.+")),
    ("tool_result",re.compile(r"^/tools/.+/result/")),
    ("shared_list",re.compile(r"^/lists/[A-Za-z0-9]+$")),
]

def template_of(path):
    for name, rx in TEMPLATES:
        if rx.match(path):
            return name
    return "unmatched"

counts = defaultdict(Counter)
with open(sys.argv[1]) as f:            # columns: path,coverage_state
    for row in csv.DictReader(f):
        counts[template_of(row["path"])][row["coverage_state"]] += 1

for tpl, c in counts.items():
    total = sum(c.values())
    cni = c["Crawled - currently not indexed"]
    print(f"{tpl:12} n={total:6} indexed={c['Indexed']:6} cni_share={cni/total:.2f}")
```

Look at "unmatched" first. It usually turns up a template nobody remembers shipping.

### Step 3: Ratios per template, then the decision

Build one row per template:

| Template | Submitted | Indexed | Crawled, not indexed (CNI) | CNI share | Clicks per indexed page |
|---|---|---|---|---|---|
| from sitemap | Sitemaps API | per-sitemap filter | per-sitemap filter | CNI / submitted | Performance export / indexed |

The second axis comes from Glenn Gabe's [large-site method](https://www.gsqi.com/marketing-blog/revisiting-quality-indexation-large-scale-sites/). In one audit, a directory with 25.3K indexed pages earned 28 clicks, while another with 3.04M earned 5.79K. Neither Google nor any practitioner I found publishes a ratio that means "noindex this", so I rank templates against each other and don't use a fixed cut-off. My rules:

- **High CNI share, few clicks, built for sharing or utility:** noindex. These are the pages Google's section describes.
- **High CNI share, but meant for Search:** merge or improve. Noindex would only hide the problem. I cover that path in [prune, merge or rebuild](https://maciejwiatr.com/blog/programmatic-seo-deindexing-recovery).
- **Low CNI share, earning clicks:** keep.

Keep URLs that are discovered but not indexed separate. That status means Google hasn't crawled the URL yet, usually for crawl-capacity reasons, so it doesn't tell you anything about quality.

### Step 4: Roll out noindex

Use either form, and leave the path open in robots.txt:

```html
<meta name="robots" content="noindex">
```

```nginx
location /tools/ {
    location ~ /result/ {
        add_header X-Robots-Tag "noindex" always;
    }
}
```

Put the tag in the server HTML. [Google's JavaScript guide](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics) says it "may skip rendering" when it finds noindex, and that using JavaScript to change or remove the tag "may not work as expected."

Sources disagree about sitemaps. Mueller [reportedly](https://www.seoinc.com/seo-blog/fastest-way-to-deindex-pages/) advised keeping noindexed URLs listed, with an updated lastmod, so they get recrawled. Some vendor blogs say to remove them. I'd keep them listed until the counts settle.

### Step 5: Track before and after counts

Record a baseline per template sitemap before you deploy: indexed, crawled but not indexed, and `Excluded by 'noindex' tag`. After the change, the noindex count should rise while that sitemap's indexed count falls. Google warns "it may take months for Googlebot to revisit a page." A Validate Fix run "typically takes up to about two weeks."

If you need pages gone faster, the [Removals tool](https://support.google.com/webmasters/answer/9689846) can hide a whole prefix, usually within a day, for about six months. It doesn't stop crawling, so the noindex still has to be there.

Add the 2026 report glitches to your timeline so you don't mistake them for results:

- From June 11, the data was stale for about three weeks, [reportedly fixed July 3](https://almcorp.com/news/google-search-console-indexing-report-fixed-july-2026/).
- On September 8, indexed pages showed as crawled but not indexed for about an hour.
- On September 11, a block of June data disappeared from the chart for all properties.

The last two come from [a secondary log](https://keywordseverywhere.com/news/google-algorithm-updates/search-console-changes/).

## Why are pages crawled but not indexed when there's no noindex or robots.txt block?

Google crawled the page and decided not to index it. The page "may or may not be indexed in the future", and resubmitting it doesn't help. When a large share of a site has this status and there's no technical cause, Mueller [says](https://seroundtable.com/google-crawled-not-indexed-quality-ai-content-41701.html) "you almost need to take a step back and think about the quality overall." On a programmatic site, read it as a verdict on the template, not on individual URLs.

## Why does Search Console say noindex when the page is set to index?

[Common causes](https://www.searchenginejournal.com/google-on-search-console-noindex-detected-errors/540829/):

- The noindex is in the initial HTML and JavaScript removes it later.
- A CDN or edge rule adds an X-Robots-Tag header.
- The data comes from an old crawl.
- Caching, or a different response served to Googlebot.

To check, compare the crawled page in URL Inspection with the live test, then look at your server logs to see what Googlebot actually received.

## How do I set alerts for spikes in new indexable URLs?

Search Console has no custom threshold alerts. Google's advice is to track "platform-level" creation and indexing metrics yourself, so run a query like this against your own database:

```sql
-- daily new indexable pages, and the top creators' share
SELECT created_at::date AS day,
       COUNT(*) AS new_indexable,
       MAX(per_user) * 1.0 / COUNT(*) AS top_user_share
FROM (
  SELECT created_at, user_id,
         COUNT(*) OVER (PARTITION BY created_at::date, user_id) AS per_user
  FROM pages
  WHERE is_indexable
) t
GROUP BY 1 ORDER BY 1 DESC;
```

For impression and query spikes, run alerts on the Performance bulk export in BigQuery. Use custom annotations to mark the noindex rollout. [Reports say](https://searchengineland.com/google-search-console-rolls-out-custom-annotations-for-performance-reports-464701) you get up to 200 per property, and they are deleted after 500 days.

## When not to noindex

- **The template earns clicks.** [Orbit Media](https://www.orbitmedia.com/blog/why-the-noindex-tag-is-good-for-seo/) found that noindexing taxonomy pages on their own cut traffic by about 20% on one site.
- **You're trying to save crawl budget on URLs that were never indexed.** Robots.txt fits that case better.
- **You expect a sitewide lift.** I found no Google statement that noindexing weak pages helps the rest of the site.
- **The report is in a glitch window.** Wait until the data is reliable before you judge the results.

## What this changes for programmatic SEO at scale

- **Generating:** decide indexability when the page is generated. Make tool outputs and user-created pages noindex by default.
- **Indexing:** sitemaps split by template become your measuring tool, not only a discovery file.
- **Linking:** I'd point hub links only at templates that are meant to rank.
- **Measuring:** you need per-template counts and your own spike alerts, because Google provides neither.

More on that setup is in the [indexing](https://maciejwiatr.com/programmatic-seo/indexing) hub.

Most Reddit threads and older guides from 2022 deal with one stuck URL at a time. At 200,000 pages the useful unit is the template, so audit it, decide whether it's meant for Search, and track the counts until Google catches up.