Crawl budget on large catalogs: where deep product pages go to hide
Large catalogs waste crawl budget on filter combinations and duplicate variants — leaving deep product pages undiscovered. Here's how to find and fix it.
Every year the same buyer costs more to rent. How organic visibility compounds into an asset you own, where the compounding comes from, and the order of operations that gets you there.
The short version
- Paid traffic is rented: the price rises, and the traffic stops when the budget does.
- Organic visibility is an asset — it keeps working after the campaign ends.
- The compounding comes from three places, and they have to be built in order.
- Measure the shift in revenue that survives a budget pause, not in rankings.
Most store owners can describe the feeling before they can name it. Spend holds steady,
the campaigns look healthy, and yet the cost of getting one more customer creeps up
quarter after quarter. Nothing broke. The auction simply got more expensive, and the
buyer you reached cheaply last year now costs more to reach at all.
That creep is what we call the ad platform tax. It isn’t a
penalty for doing something wrong — it’s the natural result of renting access to an
audience in a market where more bidders arrive every year and the tracking that once
made bidding efficient keeps getting narrower.
Why renting gets more expensive #
Three forces push the same direction at once. More advertisers compete for the same
finite attention, so the clearing price of a click rises. Privacy changes reduce the
signal that made targeting precise, so more spend is needed to find the same buyer.
And every competitor discovering the same channel bids the cost up further.
None of these is a problem you can fix inside the ad account. You can be excellent at
paid media — most of our clients already are — and still watch acquisition cost climb,
because the mechanism causing the climb sits outside the account entirely.
- Cost to rent the same customer
- Revenue from the organic baseline
Illustrative — the shape of the two curves, not measured data. What matters is what
each line is: one is a cost that recurs at a rising price, the other is an
asset that keeps paying after the work stops.
The second you turn down your daily budget, your sales numbers tank. That is the definition of renting.
Key term
Owned demand
Visits that arrive because a page you control answers something a buyer searched for
— with no per-click cost attached, and no dependency on a campaign staying switched
on. You still pay to build and maintain it. You stop paying per visit.
Owning traffic doesn’t mean abandoning paid. It means building a source of demand that
belongs to your brand — pages that rank because they genuinely answer what buyers are
searching for, on a site search engines can crawl and understand.
The distinction matters commercially, not philosophically. Rented traffic is an expense
that recurs at an increasing rate. Owned traffic is closer to an asset: you invest in
it once, maintain it, and it keeps returning visitors after the work is finished. Over
a long enough horizon, one of those lines bends up and the other bends down.
What owning looks like
- A collection page that ranks for how people actually describe the product
- Buying guides that answer the question before the comparison starts
- A catalog search engines can crawl end to end without getting lost
- Coverage on sites your buyers already read
What it isn’t
- Ranking first for your own brand name and calling it organic growth
- Publishing weekly because a calendar says to
- Chasing keyword volume nobody buys on
- Buying links and hoping the invoice counts as authority
Where the compounding comes from #
“Compounding” gets used loosely in marketing. In organic search it has a specific,
mechanical meaning, and it comes from three places that reinforce each other.
A technical foundation that lets your pages be found
Large catalogs waste crawl on filter combinations, parameter URLs and duplicate variants.
Every unit of crawl spent on a page that will never rank is a unit not spent on a
collection page that could. Fixing this doesn’t create demand — it stops you from
hiding the pages that capture it.
Coverage of real buying intent
A store ranks for what it deserves to rank for. Mapping transactional and commercial
queries cleanly onto collection and product pages — one page per intent, no internal
competition — is what turns a catalog into a set of answers.
Authority that accrues
Structure and intent get you eligible. Authority decides where you land among the
eligible. Contextual coverage from publications your buyers already read is the part
that takes longest and lasts longest.
Every unit of crawl spent on a filter combination is a unit not spent on a page that sells.
Adlux growth team
The technical layer, in practice #
This is the least glamorous section of any SEO engagement and usually the one that
moves first, because it doesn’t require anybody to write anything. The pages already
exist. They’re just competing with thousands of near-identical URLs the store
generates automatically.
Faceted navigation is the usual culprit. Three filters with six options each can
produce hundreds of crawlable combinations from a single collection, none of which
anybody searched for. A first pass usually looks something like this:
robots.txt
# Keep crawl out of filter combinations nobody searches for
User-agent: *
Disallow: /*?*filter.
Disallow: /*?*color=
Disallow: /*?*size=
Disallow: /*?*sort_by=
# Pagination stays crawlable — deep products live behind it
Allow: /*?*page=
Sitemap: https://example-store.com/sitemap.xmlThe rest of the technical pass is a short list of patterns that recur on almost every
large store. None of them are exotic; all of them quietly eat the crawl your revenue
pages need.
| Pattern | How it shows up | The usual fix |
|---|---|---|
| Faceted URLs | Hundreds of filter combinations indexed from one collection | Disallow the filter parameters, keep pagination crawlable |
| Variant duplication | One product available at a dozen colour and size URLs | Canonical every variant to the parent product page |
| Internal redirect chains | Old category links hopping through two or three redirects | Repoint the internal links at the final destination |
| Orphaned products | Live products no internal link points to | Surface them through collections and related-product blocks |
| Thin collections | Auto-generated tag pages with one or two products | Merge into a parent collection, or noindex them |
Illustrative — proportions shown to make the point, not measured from an account.
Nothing new was published between these two bars; the same crawl budget was simply
pointed somewhere useful.
The order of operations #
When we take on a store, the sequence is deliberately unglamorous. It front-loads the
work that removes waste, because that work pays for itself soonest.
- Crawl and fix. Find the duplicate content, redirect loops and faceted
navigation traps quietly consuming the crawl your revenue pages need. - Map intent to pages. Align the catalog with how buyers actually
search, and stop your own variants from competing with each other. - Clean up and mark up. Canonical rules, schema across the catalog, and
internal links that push authority to the pages that sell. - Expand. Buying guides that capture earlier-stage demand, and outreach
that earns links from publications with real audiences.
What to measure instead of rankings #
Rankings are a means, not a result. A store can climb for terms nobody buys on and feel
like it’s winning. The measures that actually track the shift from renting to owning are
the ones tied to revenue.
- Non-branded organic revenue — demand you earned, not demand that already knew your name.
- Blended customer acquisition cost — the number that should fall as the organic baseline grows.
- Share of revenue that survives a budget pause — the most honest test of whether you own anything.
That last one is uncomfortable and worth running deliberately. If turning the budget down
for a week erases the month, the asset isn’t built yet.
The quarterly review, in one page
- Non-branded organic revenue, split from branded — one line, quarter over quarter
- Blended CAC across every paid channel, not per-platform ROAS
- Indexable pages that earned at least one click, against total indexable pages
- Crawl requests going to collection and product URLs versus parameters
- Collections ranking on page one for the term they were built for
- New referring domains that sit on sites your buyers actually read
Where paid still earns its place #
None of this is an argument against paid media. Paid is the fastest way to test demand,
to launch a product, to capture a competitor’s customers at the moment they’re shopping.
It’s an excellent accelerator and a poor foundation.
The stores that get this right run both, and judge them differently: paid on how quickly
it finds profitable demand, organic on whether the baseline underneath keeps rising. When
the baseline rises, paid stops being the thing holding the business up and starts being
the thing that makes a good month great.
Frequently asked questions #
Technical fixes move first, because they release pages that already exist and are
already crawled. Intent coverage and authority are measured in quarters, not weeks.
We don’t quote a timeline before crawling the site — it depends entirely on how much
of your catalog is currently indexable and how competitive your category is.
No. Cutting paid before the organic baseline exists just removes revenue and makes
the SEO work look like the cause. Run both. Reduce paid dependence deliberately, as
the baseline proves itself — not in advance of it.
Yes, but the order shifts. Crawl waste is mostly a large-catalog problem. Smaller
stores usually fail on intent coverage instead — too few pages aimed at how buyers
actually search — and on authority, which no amount of technical work substitutes for.
The foundations are the same ones described here: pages that can be crawled, clean
structured data, and content that answers a question directly rather than circling
it. What changes is the payoff — being the source an answer engine quotes, instead
of the tenth blue link underneath it.
Often, yes — if one person owns it. The usual failure isn’t skill, it’s scheduling:
the technical fixes need developer time, and developer time competes with the product
roadmap. Whoever runs this needs the standing to get a crawl fix shipped in the same
sprint it was found.
The takeaway #
The ad platform tax isn’t going away, and no amount of account optimisation repeals it.
What you can change is how much of your revenue depends on paying it. That starts with a
crawl, not a content calendar — and it’s measured by what still sells when the budget
goes quiet.
If you only take one action from this piece, run the pause test. Whatever survives is the
part of the business you actually own.
Written by the Adlux team from live account work. If you want this thinking applied to
your catalog, the audit is free and the findings come in writing.