Crawl budget optimisation

Establish whether crawling is the actual constraint

A missing search result does not establish a crawl-budget problem. First determine whether the intended URL has been fetched since publication or its latest significant update. If it has, investigate processing, content and canonical selection rather than assuming that another request is the missing ingredient.

Google's advanced guidance primarily concerns very large or rapidly changing sites. A small, stable website whose important pages are fetched promptly usually needs dependable discovery and monitoring rather than a dedicated budget project. Avoid treating a published page-count guideline as an eligibility threshold or a reason to buy more infrastructure.

Define an operational question that can be measured: are newly published product pages being fetched late enough to miss their useful selling period? Establish publication timestamps and a stable cohort, then compare observed requests. The Search Console indexing workflow helps determine which stage needs investigation. Keep the objective tied to useful content reaching the crawler, rather than an arbitrary increase in daily requests.

Separate host capacity from demand for content

Google describes crawl budget through capacity and demand. Capacity concerns what can be fetched without overloading a host; demand concerns what the crawler wants to revisit or discover. A healthy server does not automatically create demand, and persistent delivery errors can constrain otherwise worthwhile fetching.

Use this distinction to frame competing hypotheses. A latency increase and repeated server failures suggest investigating delivery capacity. Healthy delivery combined with weak discovery of a new section suggests examining links and publication signals. Extensive requests for generated variants point towards inventory control. These are investigation paths, not diagnoses obtainable from one graph.

Keep hostname boundaries visible. A fast marketing hostname does not demonstrate that the catalogue hostname is healthy, even when both share a brand and Search Console property. Map each relevant host to its delivery owner and logging source. Separate crawler types in analysis as well: a change in requests from another Google product should not be presented as evidence that Googlebot is revisiting important HTML pages more promptly.

Build a baseline that survives comparison

Choose a period that includes ordinary publishing activity and any known disruption. Keep a note of promotions, migrations and infrastructure releases. The baseline should describe what happened, including gaps in observation, rather than imply that one week represents normal behaviour indefinitely.

• Export the intended URL inventory with template, hostname and publication or revision time.
• Collect public edge and origin request records with response status, duration and cache outcome.
• Verify Google requests using published IP ranges or the documented DNS procedure.
• Preserve raw URLs while grouping a separate analysis key by path and parameter pattern.
• Record host availability, deployment events and relevant robots changes.
• Capture report scope, filters and extraction dates alongside Search Console observations.

Use log-file analysis for the request-level work. Origin logs alone can miss responses served from a CDN cache. Retention gaps also matter: the first request visible in a short export is not necessarily the first request ever made. Describe it as the first observed request unless the evidence supports a stronger statement.

Measure freshness and useful coverage, not only volume

Count requests, distinct requested URLs and intended pages reached as different measures. For new publication, calculate elapsed time from release to the first observed verified fetch. For substantial edits, identify the first observed fetch after the revision. Keep unfetched pages in a separate outstanding group rather than dropping them from an average.

Segment the analysis by template and publishing cohort. A busy homepage and frequently fetched category pages can make total activity look healthy while new detail pages remain untouched. Report both the number of eligible pages and the share observed receiving a relevant request during the window.

Search Console Crawl Stats includes resources and repeated requests; its examples are not exhaustive. Its average response time therefore should not be read as the HTML latency of one template. Compare like-for-like request groups in logs and reconcile differences in scope. Treat your freshness measure as an operational indicator with documented observation limits, not a Google ranking metric or proof that fetched content entered the index.

Reduce avoidable URL expansion at the source

Rank unwanted URL families by observed requests, generation rate and risk to useful discovery. A route that manufactures new variants continuously deserves a different intervention from a few old addresses still being revisited. Reproduce the generating link or application behaviour before assigning a fix.

Follow the crawl-trap workflow for repeated filters, calendars and pagination without boundaries. Establish the intended searchable states with the content owner, then prevent the application from emitting duplicate combinations. Review internal links and the sitemap so they use the agreed preferred routes.

Do not use noindex as a request-saving mechanism: it needs a fetch to be read. Equally, a robots block is not a promise to transfer the avoided requests to selected pages. Where crawling an entire family is unnecessary, evaluate a narrow restriction with positive exceptions. Where duplicate consolidation is the objective, assess canonical signals instead. Avoid a blanket rule against parameters that also support legitimate pagination or distinct content.

Improve delivery without falsifying freshness

Investigate slow templates using cache outcomes, upstream response times and failing dependencies. Compare a cache hit with a miss and an ordinary request with one during a publishing batch. A global average can hide expensive routes or a brief overload that affects newly released content.

HTTP validators can reduce unnecessary body transfer. With a matching validator, a conditional request can receive a 304 Not Modified response without a response body. The validator must accurately represent the selected resource. Returning 304 after meaningful content changes risks leaving the requester with stale information.

Ask developers to test an unchanged resource and a revised one through the actual CDN path. Confirm that a changed representation is returned when the old validator no longer matches. Do not reset modification dates merely to invite more attention, or keep them fixed merely to improve cache statistics. Maintain sitemap dates from defensible revision events using sitemap publishing rules. Any delivery improvement still needs content-integrity checks, including correct availability and indexing directives.

Illustrative worked example: a fast-changing spare-parts catalogue

Illustrative example: a catalogue publishes 400 replacement-part pages during a fortnight. A hypothetical log review finds verified fetching of sort variants alongside a group of new pages with no observed fetch. These figures are an invented teaching scenario, not a measured business outcome.

Three explanations remain plausible: the crawler is spending requests on variants, new pages lack discoverable links, or cache misses cause delivery failures. The team checks each independently. The sort component emits repeated keys; the new pages are linked from valid category pagination; and affected cache misses coincide with upstream errors during import jobs.

The proposed release removes repeated-key links and isolates the expensive import work from public requests. It retains valid category pagination and preferred product routes. A before-and-after plan records request mix, upstream failures and the same publication-cohort measure, while keeping later publication separate.

This example deliberately includes two defects. If they ship together, the team cannot attribute any later freshness change to one correction alone. It can still establish whether both defects were removed and whether important pages remain reachable. A causal claim would need a more controlled comparison and enough comparable observation.

Give the release measurable acceptance criteria

Specify checks for every changed layer. The URL generator must not create repeated filter states; wanted pagination and product links must remain reachable from the agreed seeds. Public responses must preserve the expected status, canonical and indexing directives. Cache validators must distinguish changed content from unchanged content.

For infrastructure work, agree the measurement method and local service objective with the delivery team. Compare the same routes, cache states and representative load. Require the failing dependency or overload behaviour to be reproduced before the change and absent under the agreed test afterwards. Avoid inventing a universal response-time threshold for crawling.

Verify deployed behaviour at the edge, not only in an application test environment. Check a sample of high-value pages, unpopular routes and exceptions. Separate immediate release checks from later monitoring: developers can demonstrate correct links and reliable responses, while the subsequent crawler schedule remains external. Keep an owner for unexplained observation gaps so the project does not close solely because a request-total chart has fallen.

Control rollback risk and interpret later changes cautiously

Prepare rollback separately for routing, cache and crawl rules where the architecture permits. An overbroad restriction may need immediate reversal, while a safe generator correction can remain. Restoring old cache settings may protect content freshness but increase origin load, so name the delivery owner who will monitor that trade-off.

Set triggers for action: useful routes becoming blocked, meaningful updates receiving stale responses, rising delivery failures or lost navigation paths. Do not interpret every reduction in requests as a regression. Removing redundant variants can lower total volume while useful coverage stays stable. Conversely, rising volume can reflect an error loop rather than better discovery.

Compare the follow-up period with its publishing mix and release context. Old unwanted URLs can remain known, and report updates are not instantaneous. Revisit the original hypotheses if important pages are still not fetched, then investigate selection separately when they are. The outcome is a more controlled inventory and dependable delivery; it cannot promise indexing, rankings or a particular allocation of Google's future requests.