How to noindex programmatic pages with Search Console data

A template-level audit for programmatic sites: count indexing per template, pick what Google's new guidance says to noindex, ship it and track recrawl.

Maciej Wiatr

8 min read

Read as markdown

On October 26, 2026, Google told sites that let users create shareable pages, generated outputs or tool results to keep those pages out of the index by default. On a programmatic site, that rule covers more templates than most teams expect. You can’t get the full answer from the Page indexing report alone, so the way to noindex programmatic pages with Search Console data is a template-level audit. Count indexing per template, compare indexed pages with those marked crawled but not indexed, decide which templates were never meant for Search, ship noindex, and track the counts until Google recrawls. Below is the audit as I’d run it on a directory like mine, which has about 200,000 indexed pages, with the code to get each number.

What does “Only index content meant for Search” mean for programmatic pages?

Google added the section to Prevent user-generated spam on your site and platform, which was last updated 2026-10-26 when I checked it in November 2026. The guidance asks you to “consider whether that content is meant to be found through search engines or only accessed directly by people with a link.” For pages made “for direct sharing, collaboration, or utility rather than public search discovery”, it says to apply noindex with a robots meta tag or an X-Robots-Tag header, or to block crawling in robots.txt, by default.

The changelog gives the reason: to keep user-created pages “that aren’t meant for Search out of the index and detect spam earlier.”

Some things are not documented as of November 2026:

On a programmatic site, the templates in scope are calculator result URLs, saved comparisons, shared lists and per-query generated summaries. The incidents behind the guidance are similar. PPC Land cites more than 370,000 Grok conversations indexed in 2025. Duck.ai share pages launched in October 2026 with noindex and a 48-hour expiry.

Should I noindex or block programmatic pages in robots.txt?

Use noindex for anything that is indexed or could be. Google’s noindex documentation says that if robots.txt blocks a page, “the crawler will never see the noindex rule.” John Mueller put it more briefly: “if we can’t crawl the page, we can’t see the noindex.”

The new section lists noindex and robots.txt as equal defaults, and Elsop points out that Google hasn’t said which reading it means. My reading, which is an inference and not a Google statement, works like this:

  1. For URL patterns that were never indexed, robots.txt is fine. Google’s crawl budget guide recommends it for pages you never want crawled, on sites of roughly 1M+ pages.
  2. For URLs that are already indexed, apply noindex first and wait until they drop out. Only then consider a robots.txt block.

Noindex has a cost. Google still requests the page. Mueller said noindexed URLs “end up in the Search Console report for crawled/not indexed”, and added that “neither of these statuses cause issues to the rest of the site.”

The audit, step by step

The flowchart shows the five steps and the decision in the middle.

flowchart TD
    A[Sitemaps split by template] --> B[Counts per template]
    B --> C[Sample with Inspection API]
    C --> D{Meant for Search?}
    D -->|No| E[Noindex, crawl allowed]
    D -->|Yes but weak| F[Merge or improve]
    D -->|Yes and earning| G[Keep indexed]
    E --> H[Track before and after]

Step 1: Export the Page indexing report past the 1,000-row cap

The Page indexing report has two limits. The example table for each status “is limited to 1,000 items, and isn’t guaranteed to show all URLs.” You also can’t filter it by URL. Bulk export to BigQuery won’t help here, because it covers Performance data and I found no source saying coverage data is included.

So you build the export from three workarounds:

Step 2: Group URLs by template

Your own database or sitemaps give the full URL list. Search Console statuses come from the per-sitemap counts and the Inspection samples. Join the two and assign each URL to a template:

import csv, re, sys
from collections import Counter, defaultdict

TEMPLATES = [
    ("listing",    re.compile(r"^/[a-z-]+/[a-z-]+/?$")),
    ("city_hub",   re.compile(r"^/in/[a-z-]+/?$")),
    ("compare",    re.compile(r"^/compare/.+-vs-.+")),
    ("tool_result",re.compile(r"^/tools/.+/result/")),
    ("shared_list",re.compile(r"^/lists/[A-Za-z0-9]+$")),
]

def template_of(path):
    for name, rx in TEMPLATES:
        if rx.match(path):
            return name
    return "unmatched"

counts = defaultdict(Counter)
with open(sys.argv[1]) as f:            # columns: path,coverage_state
    for row in csv.DictReader(f):
        counts[template_of(row["path"])][row["coverage_state"]] += 1

for tpl, c in counts.items():
    total = sum(c.values())
    cni = c["Crawled - currently not indexed"]
    print(f"{tpl:12} n={total:6} indexed={c['Indexed']:6} cni_share={cni/total:.2f}")

Look at “unmatched” first. It usually turns up a template nobody remembers shipping.

Step 3: Ratios per template, then the decision

Build one row per template:

TemplateSubmittedIndexedCrawled, not indexed (CNI)CNI shareClicks per indexed page
from sitemapSitemaps APIper-sitemap filterper-sitemap filterCNI / submittedPerformance export / indexed

The second axis comes from Glenn Gabe’s large-site method. In one audit, a directory with 25.3K indexed pages earned 28 clicks, while another with 3.04M earned 5.79K. Neither Google nor any practitioner I found publishes a ratio that means “noindex this”, so I rank templates against each other and don’t use a fixed cut-off. My rules:

Keep URLs that are discovered but not indexed separate. That status means Google hasn’t crawled the URL yet, usually for crawl-capacity reasons, so it doesn’t tell you anything about quality.

Step 4: Roll out noindex

Use either form, and leave the path open in robots.txt:

<meta name="robots" content="noindex">
location /tools/ {
    location ~ /result/ {
        add_header X-Robots-Tag "noindex" always;
    }
}

Put the tag in the server HTML. Google’s JavaScript guide says it “may skip rendering” when it finds noindex, and that using JavaScript to change or remove the tag “may not work as expected.”

Sources disagree about sitemaps. Mueller reportedly advised keeping noindexed URLs listed, with an updated lastmod, so they get recrawled. Some vendor blogs say to remove them. I’d keep them listed until the counts settle.

Step 5: Track before and after counts

Record a baseline per template sitemap before you deploy: indexed, crawled but not indexed, and Excluded by 'noindex' tag. After the change, the noindex count should rise while that sitemap’s indexed count falls. Google warns “it may take months for Googlebot to revisit a page.” A Validate Fix run “typically takes up to about two weeks.”

If you need pages gone faster, the Removals tool can hide a whole prefix, usually within a day, for about six months. It doesn’t stop crawling, so the noindex still has to be there.

Add the 2026 report glitches to your timeline so you don’t mistake them for results:

The last two come from a secondary log.

Why are pages crawled but not indexed when there’s no noindex or robots.txt block?

Google crawled the page and decided not to index it. The page “may or may not be indexed in the future”, and resubmitting it doesn’t help. When a large share of a site has this status and there’s no technical cause, Mueller says “you almost need to take a step back and think about the quality overall.” On a programmatic site, read it as a verdict on the template, not on individual URLs.

Why does Search Console say noindex when the page is set to index?

Common causes:

To check, compare the crawled page in URL Inspection with the live test, then look at your server logs to see what Googlebot actually received.

How do I set alerts for spikes in new indexable URLs?

Search Console has no custom threshold alerts. Google’s advice is to track “platform-level” creation and indexing metrics yourself, so run a query like this against your own database:

-- daily new indexable pages, and the top creators' share
SELECT created_at::date AS day,
       COUNT(*) AS new_indexable,
       MAX(per_user) * 1.0 / COUNT(*) AS top_user_share
FROM (
  SELECT created_at, user_id,
         COUNT(*) OVER (PARTITION BY created_at::date, user_id) AS per_user
  FROM pages
  WHERE is_indexable
) t
GROUP BY 1 ORDER BY 1 DESC;

For impression and query spikes, run alerts on the Performance bulk export in BigQuery. Use custom annotations to mark the noindex rollout. Reports say you get up to 200 per property, and they are deleted after 500 days.

When not to noindex

What this changes for programmatic SEO at scale

More on that setup is in the indexing hub.

Most Reddit threads and older guides from 2022 deal with one stuck URL at a time. At 200,000 pages the useful unit is the template, so audit it, decide whether it’s meant for Search, and track the counts until Google catches up.