Crawl traps and crawl waste

Recognise an expanding URL space

A calendar that links indefinitely into the future is a straightforward trap. Other patterns are less obvious: filters applied repeatedly, session identifiers added to every link, sorting parameters combined in different orders, and relative links that grow a directory path after each request. Each address can look new while returning substantially the same information.

Start by describing the expansion rule. For example, a listing permits the same colour filter twice, then offers another link that appends it again. Record the shortest path from an ordinary landing page to that repeating state. A reproducible sequence gives a developer something to fix; a spreadsheet containing thousands of long addresses does not explain the mechanism.

Large archives and legitimate pagination are not automatically traps. They may contain distinct records and terminate correctly. Establish what the site is supposed to publish before applying exclusions. The faceted navigation guide helps separate useful filtered landing pages from arbitrary combinations that have no intended search role.

Build an evidence checklist before blocking anything

Preserve raw URL strings before normalising them. Parameter order, repeated keys, encoded separators and case differences can reveal the defect. Store an analysis key separately so deduplication does not erase evidence of how the route expands.

• Capture seed URLs, crawl settings, discovery depth and the source link for each suspicious family.
• Sample response status, final destination, canonical annotation, robots directives and visible results.
• Count distinct addresses and requests separately, grouped by route and parameter signature.
• Compare application inventory with the records actually displayed by those addresses.
• Check public CDN logs as well as origin logs, including dates, cache outcomes and response times.
• Verify requests attributed to Google using its published verification methods; a user-agent string alone is insufficient.

Record the observation window and crawler identity alongside each finding. An internal audit crawler can expose a theoretical trap that Google has never entered. Conversely, Google may know obsolete variants that current navigation no longer exposes. The intervention should reflect both generation risk and observed requests, with those two evidence types clearly distinguished.

Test competing explanations for apparent crawl waste

One hypothesis is genuine catalogue growth. Test it against published record counts and whether newly discovered URLs contain distinct products or documents. A second is duplicate-state expansion: many addresses resolve to the same result set. Compare representative content and the ordered sequence of filters, rather than relying only on page titles.

A third explanation is historical discovery. Current links may be clean while old external links or previous sitemaps continue feeding requests. Check first-seen dates in your own records and replay the current navigation from a clean session. A fourth is a measurement error, such as counting unverified bots or treating every redirected request as a separate content page.

Write down what would disprove the preferred explanation. If the suspected calendar has an enforced final month and every page exposes different events, investigate another source. If the audit tool appends a tracking parameter itself, correct its configuration before changing production. Use server log analysis to establish which suspicious families are receiving real crawler traffic and whether important routes are affected at the same time.

Stop the generator at a defensible boundary

Define allowed states at the routing and navigation layers. A filter should not append an already selected value. Equivalent combinations should use a consistent order. Pagination should stop at the actual final page, and malformed page values should not silently produce another successful copy of page one.

For calendars, agree a useful historical and future range with the content owner. Do not select an arbitrary date merely to make a crawl finish. Historical events may still serve a clear purpose. The boundary should follow available content and the intended user journey, including what happens when the next event is published.

Google's faceted navigation documentation recommends a 404 response for nonsensical combinations, empty filter results and nonexistent pagination pages. Apply that guidance to those invalid states; do not extend it blindly to a useful category that is temporarily out of stock. Keep the requested address when serving an error rather than redirecting every invalid state to a shared error URL. Test both the HTTP response and the page users see.

Choose crawl and indexing controls separately

Removing links that manufacture unwanted states addresses their internal source. A narrowly scoped robots.txt rule can prevent compliant crawlers from fetching an unwanted URL family. It does not reliably remove an address already known to search, and it is not access control for confidential information.

A noindex directive serves a different purpose: the crawler must fetch the page to see it. Canonical annotations express a preferred representative for duplicate or very similar content; they do not stop the first request. Work through robots.txt versus noindex before combining controls that prevent each other from being read.

Separate any removal objective from the long-term crawl policy. Already indexed unwanted pages may need a crawlable removal signal before the final restriction is introduced. Keep wanted destinations and required rendering resources accessible. Avoid blocking every query string merely because one parameter is defective: pagination, localisation or useful filters may depend on parameters too. The policy needs explicit exceptions based on real routes, not an assumption that all dynamic addresses are disposable.

Illustrative worked example: repeated catalogue filters

Illustrative example: an outdoor catalogue has 6,000 product records. A controlled audit discovers 48,000 listing addresses before its safety limit stops the run. These figures describe a hypothetical investigation, not a client result or an expected improvement.

The first suspicious sequence is /boots/?colour=black, followed by /boots/?colour=black&colour=black. The filter component carries the current query into every generated link and appends another copy. A competing hypothesis is that legitimate size and brand combinations explain the growth. Sampling result sets shows the repeated colour keys add no distinction.

The proposed correction generates each filter URL from structured state, permits one value per single-select key, and uses a stable key order. Invalid duplicate-key requests receive the agreed error response. Curated category pages remain accessible through ordinary links; product discovery does not depend on traversing filtered duplicates.

For review, compare the same seed set and crawl settings before and after the patch. Acceptance means repeated selection cannot create new addresses, valid filters still work and product reachability is preserved. A smaller crawl alone is insufficient: accidentally removing all listing links would also reduce the count while damaging navigation.

Set implementation acceptance criteria

Create a route matrix with valid, duplicate, empty, encoded and out-of-range examples. Include parameter permutations and mixed valid-invalid combinations. For each case, specify the expected response, generated links and search eligibility. Keep the fixture set small enough to review but broad enough to exercise the expansion rule.

Require a crawl from the same starting pages to terminate within the agreed legitimate inventory envelope. Check the last pagination page, the calendar boundary and a filter selection followed by deselection. Inspect server-rendered and rendered links when JavaScript changes navigation. Useful links should remain real anchors with resolvable href values.

Where robots rules are part of the change, evaluate the actual deployed file against wanted and unwanted examples, including exceptions. Confirm that a product URL, useful category and required resource remain allowed. Review the first production responses through the public hostname. Give the routing owner responsibility for preventing recurrence and the SEO owner responsibility for checking intended search pages; neither should treat the other's checks as a substitute.

Plan rollback around the failure mode

The main rollback risk is restoring unlimited generation when reversing an overbroad restriction. Keep navigation changes and robots changes independently reversible where practical. If a rule blocks an important category, remove or narrow that rule while retaining the safe URL generator, provided the components can operate independently.

Preserve the previous route behaviour, rule file and representative responses before deployment. Define stop conditions in advance: wanted URLs becoming blocked, valid filters returning errors, broken pagination or a sudden loss of product reachability should trigger immediate investigation. A temporary rise in expected 404 responses for malformed variants does not carry the same meaning.

Watch old requests separately from newly generated addresses. Search engines can revisit historical URLs after the source has been fixed, so continued requests do not alone prove failure. Use the crawl-budget investigation when there is evidence that unwanted fetching coincides with delayed processing of important content. Removing a trap cannot guarantee that released capacity is spent on a particular page.