# Programmatic SEO duplicate content: find and fix clusters

> Learn why Google clusters templated pages, how to find duplicate clusters on 10,000+ URL sites, measure similarity, and choose canonical, noindex or removal.

Maciej Wiatr, 2026-11-03. https://maciejwiatr.com/blog/programmatic-seo-duplicate-content-clusters

When thousands of pages share one template, the programmatic SEO duplicate content problem rarely shows up as a penalty. It shows up as quiet statuses in Search Console: "Duplicate, Google chose different canonical than user", "Duplicate without user-selected canonical", or a long list of URLs that were crawled but are currently not indexed. This post covers why Google groups templated pages, how to find the clusters on a site with 10,000+ URLs, how to measure how alike sibling pages really are, and how to pick a fix per cluster. It also covers when the right answer is to stop generating a page pattern.

## Does programmatic SEO create duplicate content?

Not by itself. It does when the template is the main content and only a variable such as a city or tool name changes.

Google's [canonicalization docs](https://developers.google.com/search/docs/crawling-indexing/canonicalization) say that when it finds pages that "seem to be the same or the primary content very similar, it clusters them together." It then picks the version it judges most complete and useful as the canonical. Canonical pages get crawled more often, and duplicates less.

The same doc says that pages are still duplicates when only the header, footer or other non-critical text differs and the body stays the same. That is the closest official description of "a template plus a swapped variable."

Real data changes the picture. [Ahrefs](https://ahrefs.com/blog/canonicalization/) notes that pages whose price or currency differs end up in different clusters. Fields that change from page to page do the work. Words that wrap a variable do not.

## Why does Google say "Duplicate, Google chose different canonical than user"?

The [Page indexing help page](https://support.google.com/webmasters/answer/7440203) defines it: "This page is marked as canonical for a set of pages, but Google thinks another URL makes a better canonical." Your page says it is the canonical, and Google picked a sibling instead.

"Duplicate without user-selected canonical" is the same situation on a page that declares no canonical, so Google chooses one and doesn't serve this page.

On templated pages, three causes come up most:

- **Thin centerpiece content.** Gary Illyes [describes](https://www.searchenginejournal.com/how-google-chooses-canonical-pages/513157/) Google reducing content "into a hash or checksum and then compar[ing] the checksums." Pages that are "fairly similar, or at least a little bit similar" share a dupe cluster, and the cluster is built around the main content, which he calls the "centerpiece."
- **A JavaScript app shell.** Ahrefs [points out](https://ahrefs.com/blog/duplicate-google-chose-different-canonical-than-user/) that client-rendered pages can return near-identical initial HTML and get grouped before rendering. Its fix is SSR or pre-rendering.
- **Stronger signals on a sibling.** Ahrefs cites Google's Allan Scott on "~40 different canonical selection signals". Its other article says about 20, so treat the count loosely. These include internal and external links, PageRank, HTTPS and shorter URLs. Only pages returning HTTP 200 can be clustered.

A canonical tag doesn't settle the matter. Google calls a canonical preference "a hint, not a rule," and its [consolidation guide](https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls) also calls `rel="canonical"` "a strong signal." Both are true: Google weighs the tag, and other signals can outweigh it.

## How similar can templated pages be before Google treats them as duplicates?

There is no published number. Asked about a percentage in 2022, John Mueller [said](https://www.searchenginejournal.com/google-on-percentage-that-represents-duplicate-content/465885/): "There is no number (also how do you measure it anyway?)." Vendor posts quote figures like "70% unique," but I couldn't trace them to a primary source.

You can still measure your own templates in a way that matches how Google describes the process. Compare main content only, with navigation, footer and shared blocks stripped out. Some summaries say Google excludes nav and footer before hashing. That claim is unverified, but measuring the centerpiece is the sensible choice anyway, because that is where your pages either differ or don't.

A shingle comparison on a sample of sibling pages from one template is enough to rank templates by risk:

```python
import itertools, re, statistics

def shingles(text, k=5):
    words = re.findall(r"\w+", text.lower())
    return {" ".join(words[i:i + k]) for i in range(len(words) - k + 1)}

def jaccard(a, b):
    union = a | b
    return len(a & b) / len(union) if union else 1.0

def template_similarity(main_texts):
    """main_texts: main-content strings for one template, boilerplate removed."""
    sets = [shingles(t) for t in main_texts]
    scores = [jaccard(a, b) for a, b in itertools.combinations(sets, 2)]
    return {"median": statistics.median(scores), "max": max(scores)}
```

Any cutoff you set is your own heuristic, not Google's. What I'd use the scores for is ordering. Templates with the highest median similarity should be the ones you check first in Search Console. I wrote more about splitting shared blocks from unique fields in [shared blocks vs unique page data](https://maciejwiatr.com/blog/programmatic-seo-template-unique-content).

## How do I find duplicate clusters across 10,000+ URLs?

The Page indexing report won't show them to you directly. Its examples table "is limited to 1,000 rows." Google suggests submitting a sitemap of your important pages and filtering the report by it. On a large site, my inference is that per-template sitemaps are the practical way to do this, because then every count belongs to one template.

The workflow combines four sources:

1. **Per-template sitemaps.** Filter the Page indexing report by each one to get status counts per template.
2. **URL Inspection API samples.** The [quota](https://developers.google.com/webmaster-tools/limits) is 2,000 queries a day and 600 a minute per property. Inspect a sample from each status and template, and record the Google-selected canonical next to the one you declared.
3. **A crawl.** Screaming Frog or Ahrefs Site Audit groups exact and near-duplicates on your side.
4. **A join.** Merge crawl groups and inspection results by URL, then group by template and Google-chosen canonical.

The ranking pages on this topic don't describe this join, so here is the shape I'd use:

```python
import pandas as pd

crawl = pd.read_csv("crawl.csv")          # url, template, near_dup_group
inspect = pd.read_csv("inspection.csv")   # url, status, google_canonical, declared_canonical

df = crawl.merge(inspect, on="url", how="inner")
df["overridden"] = df["google_canonical"] != df["declared_canonical"]

clusters = (
    df.groupby(["template", "google_canonical"])
      .agg(pages=("url", "count"), overridden=("overridden", "sum"))
      .sort_values("pages", ascending=False)
)
print(clusters.head(20))
```

A Google-chosen canonical that collects many of your URLs is a cluster. If that URL is a parent page, Google has already merged the variants for you.

## Should I use canonical tags or noindex for near-duplicate pages?

It depends on whether the pages are actually duplicates. This split is the part most guides skip.

**Canonical is for real duplicates.** If two pages are interchangeable for the user, point the weaker one at the parent. If the variant doesn't need to exist, a 301 redirect is the stronger signal. Google ranks redirects and `rel="canonical"` as strong and sitemaps as weak. It warns against mixed signals, against robots.txt for canonicalization, and against noindex for choosing a canonical, because noindex "will completely block the page from Search."

**Noindex is for distinct pages you don't want indexed.** Glenn Gabe [documented](https://www.gsqi.com/marketing-blog/google-ignore-rel-canonical-different-content/) a site that mass-canonicalized pages to URLs with significantly different content. Google ignored the tags. About 1,360 of those URLs still appeared in a site: search, and Search Console showed 2,867 canonicalized URLs still earning impressions and clicks. Gabe's fix was `noindex, follow` for those pages, with canonical kept for real duplicates.

The flowchart shows how I'd pick a fix for each cluster.

```mermaid
flowchart TD
    A[Cluster from join] --> B{Unique data fields?}
    B -->|Yes| C[Keep, surface data]
    B -->|No| D{Same intent as parent?}
    D -->|Yes| E[301 or canonical]
    D -->|No| F{Useful to users?}
    F -->|Yes| G[Noindex, follow]
    F -->|No| H[Stop generating pattern]
```

| Cluster looks like | Fix | Why |
|---|---|---|
| Siblings differ in real fields (price, items, specs) | Keep, move the data into the main content | Differing data can separate clusters |
| Variants are the parent with a swapped word | 301 to parent, or canonical if the URL must stay | Strong consolidation signals |
| Distinct pages, useful on site, weak for Search | `noindex, follow` | Canonicals between non-equivalent pages get ignored |
| Keyword + city pages with nothing local | Remove the pattern | Doorway risk |

For the mechanics of shipping noindex by template, see [how to noindex programmatic pages with Search Console data](https://maciejwiatr.com/blog/noindex-programmatic-pages-search-console-audit).

## When should I merge thin variant pages into one parent page?

I'd merge city + service or similar variants in three situations:

- The variants have no unique data and send users to the same place. That matches Google's [doorway definition](https://developers.google.com/search/docs/essentials/spam-policies): "pages targeted at specific regions or cities that funnel users to one page."
- Google already picks the parent as canonical for most of the cluster.
- The variants sit in bulk under the crawled but not indexed status. Google's help says those pages "may or may not be indexed in the future." One unverified analysis claims many of them were indexed earlier and dropped.

Use 301s and expect it to take a while. In a [Search Engine Land case study](https://searchengineland.com/long-takes-deindex-low-quality-thin-content-published-accident-case-study-282793), thin URLs took about three months to leave the index.

A cheaper fix is to stop the pages before they exist. One developer [noindexes category pages with fewer than three items](https://dev.to/radarmakers/programmatic-seo-pages-and-thin-content-noindex-until-there-are-three-items-3o8b) and reports no results. A [vendor guide](https://www.thesharpdigital.com/blog/programmatic-seo-case-studies-the-complete-2026-guide) says a team stopped generating comparison pairs with fewer than 5 comparable features and saw its indexing rate reach 94%. That figure is unverified, but the pattern is sound: gate page creation on data, not on keyword lists.

## When a page pattern shouldn't exist

Sometimes no tag will help. In a [DEV postmortem](https://dev.to/_9848c5582063b42abecb7/i-built-8000-programmatic-seo-pages-google-indexed-zero-a-gsc-postmortem-569i), a Seoul cafe map shipped about 8,000 SSR pages across four templates. Of 8,653 submitted URLs, 0 were indexed. Impressions peaked at 209 a day and fell to about 0. The author estimated about 89% of pages were thin near-duplicates and cut the sitemap from about 8,600 URLs to about 560. No result had been reported yet.

In 2019, Mueller [warned](https://localsearchforum.com/threads/google-city-landing-pages-can-be-doorway-pages-against-guidelines.55438/) an SEO that about 1,300 "keyword + city" landing pages would be doorway pages. If a template's honest description is "the parent page with a different noun," I'd delete the pattern. Recovery options are in [prune, merge or rebuild](https://maciejwiatr.com/blog/programmatic-seo-deindexing-recovery).

## Is programmatic SEO bad for a site if many pages are flagged as duplicates?

Duplicates alone aren't punished. Ahrefs and [Discovered Labs](https://discoveredlabs.com/blog/content-deduplication-and-canonicalization-in-programmatic-seo) both say Google has no duplicate content penalty. It filters duplicates instead.

The risk sits next to that. The scaled content abuse policy covers pages "generated for the primary purpose of manipulating search rankings and not helping users… no matter how it's created." Google [says](https://blog.google/products/search/google-search-update-march-2024/) its March 2024 update cut low-quality, unoriginal content in results by 45%. Gabe also argues that ignored canonicals can leave many low-value pages indexed and affect how the whole site is judged.

In review, I'd check four things per template: median main-content similarity, the share of sampled URLs where Google overrode the canonical, whether the page renders its data in the initial HTML, and whether the template has a data threshold below which no page is generated. If a template fails all four, merging or deleting it will help more than any canonical tag.