Search Console indexing diagnosis

Define the intended search inventory first

Begin with the pages the organisation intends to make searchable. Export published records from the CMS or another maintained inventory, retaining content type, preferred URL, publication date and business importance. Exclude private records and drafts from that target set, but keep a separate record of deliberate exclusions.

The Page indexing report includes known URLs that may legitimately be absent from the index. An alternate address, a removed page and a newly published article should not all trigger the same response. Google also limits the examples shown in report tables; those examples are not a complete inventory for reconciliation.

Choose the property carefully. Record whether it is a Domain property or a URL-prefix property, together with the active report filter and export date. A hostname or protocol mismatch can make an apparent gap a scope problem. Use the crawling and indexing framework to distinguish discovery, fetching and selection before assigning work. The operational goal is to investigate missing intended pages, rather than make every known address appear indexed.

Collect an evidence packet for each affected cohort

Group affected URLs by template, release date, language or routing behaviour. Select important examples and a comparison group that still behaves as expected. Keep the sample method explicit; the first ten rows in an export may all belong to one unusual directory.

• Record the exact requested URL, intended canonical and publishing state.
• Save the reported reason, recorded crawl time and available inspection details.
• Capture current response status, redirect chain, robots access and indexing directives.
• Inspect visible content, rendered content and required resource availability.
• Record the user-declared canonical and Google's selected canonical when available.
• Check internal discovery, sitemap membership and relevant deployment timestamps.

Keep historical observations separate from current tests. An engineer should be able to tell whether a noindex directive was present during the recorded crawl or appeared only in today's response. Preserve representative headers and HTML with timestamps in the team's evidence system. This packet makes escalation useful: it identifies an affected component and a reproducible discrepancy instead of asking developers to fix an unexplained report count.

Investigate access and exclusion states at their source

For blocked or excluded URLs, compare the result with publishing intent. An intentional noindex on an account page needs no correction. The same directive on a public service page requires tracing its source through templates, response headers and deployment configuration. Check both the HTML and X-Robots-Tag response header.

A robots.txt block prevents fetching; it does not prove an address cannot appear in search. If a wanted page is blocked, examine the matching rule and its scope. If an unwanted page needs noindex processing, ensure the crawler can access that directive. Follow robots.txt and noindex diagnosis before combining these controls.

Investigate server failures using the recorded time and delivery path. A healthy origin now does not disprove an earlier CDN challenge or outage. For genuine missing content, verify the intended 404 or 410 response rather than redirecting every retired page to the homepage. A soft 404 warrants examining the content delivered as well as the status: an empty template or error message served successfully can still fail the user's task.

Separate discovered and crawled URLs without inventing causes

Discovered - currently not indexed and Crawled - currently not indexed describe different stages. Neither label, by itself, identifies one universal defect. Treat the wording as evidence of the reported processing state and build competing explanations around the actual page family.

For discovered URLs, test weak discovery paths, very recent publication, excessive variant generation and delivery constraints. Check whether ordinary crawlable links reach the pages and whether their sitemap entries match the intended addresses. If server capacity is suspected, require logs or availability evidence rather than inferring it from the label alone.

For crawled URLs, compare substantive content with nearby indexed pages, inspect rendering and investigate duplication. A missing product description caused by a failed data request is a different problem from fifty near-identical regional pages. Avoid treating the status as a confirmed quality penalty. Use the page-level indexing investigation to structure those checks. Document which observation supports each explanation and which test could reject it before recommending a rewrite, infrastructure upgrade or canonical change.

Compare canonical candidates as complete pages

When Google chooses a different canonical, inspect both addresses. Compare their purpose, main content, status, annotations, internal links and sitemap signals. A correct-looking canonical element on one page is not enough evidence that the wider implementation is coherent.

Competing explanations include an accidental cross-template canonical, substantially overlapping content, inconsistent preferred hosts and internal links pointing mainly to the alternate version. Build a short comparison table for the engineering ticket. Include the exact output observed at each address so the discussion is about delivered behaviour rather than the CMS setting alone.

Canonical annotations express a preference for duplicate or very similar pages. They are not a command to merge unrelated resources. If two pages genuinely answer different needs, investigate what makes that distinction visible instead of assigning both to the parent directory. The canonical-tag guide covers signal alignment. Keep intentional alternate URLs in a separate monitoring group so successful consolidation does not continue to generate false alarms about missing pages.

Read stored inspection and live tests on separate timelines

Initial URL Inspection results describe Google's stored information. A live test checks the current response through the inspection tool. A live success does not establish current index inclusion or predict the selected canonical, and it does not test every condition affecting search eligibility.

Record three times: the last reported crawl, the deployment and the live test. If a fix happened between the crawl and test, different results are plausible. If the reported crawl happened after the fix, investigate uneven deployment, cached responses or a rule that applies only to particular routes. Avoid dismissing every discrepancy as reporting delay.

Inspect the rendered output when the page depends on JavaScript. Compare the main content and links with what the publishing system intended to serve. Keep performance investigations separate unless evidence connects them to failed delivery. A visually quick browser visit cannot establish what a crawler received earlier. Where access differs, compare edge logs and security rules before changing application content that is already correct.

Illustrative worked example: a documentation release

Illustrative example: a team publishes 120 documentation pages. Search Console shows a group marked noindex, while other pages from the same release appear normally. These numbers are hypothetical and do not represent a client engagement or forecast.

The first hypothesis is a shared staging directive copied into production. The second is historical reporting from before a deliberate launch change. The third is a route-specific response header. Inspecting an affected URL finds no noindex in its HTML, but its current response includes X-Robots-Tag: noindex. An unaffected comparison page lacks that header.

Deployment records show that the affected pages sit beneath a directory matched by an obsolete CDN rule. The proposed change narrows that rule to the private preview path. Acceptance requires public documentation to remain accessible without the exclusion header while preview URLs retain their intended protection.

After deployment, repeat header and rendered-content checks through the public hostname, then inspect selected URLs live. Later, compare stored inspection and the affected cohort again. The technical correction can be demonstrated immediately; Google's subsequent indexing decision must be observed separately. Repeatedly requesting indexing would not have removed the header.

Define acceptance criteria and a controlled rollback

Write the expected behaviour per URL family: final response, crawl permission, indexing directive, canonical destination and visible content. Include both positive examples and pages that should remain excluded. A global removal of noindex may make the public sample pass while exposing account or preview content.

Require the affected templates to pass checks on the public delivery path after deployment. Validate links and sitemap membership where discovery was the issue. For a canonical correction, compare the served annotations immediately and monitor Google's later selection separately. Do not make release acceptance depend on an immediate report change the developer cannot control.

Preserve the previous configuration and define the rollback trigger. Exposure of private routes, broken redirects or missing content calls for prompt containment. Restoring an old global exclusion can also remove the intended fix, so document a narrower containment option where possible. Assign someone to check related templates after the change; a shared header rule or layout component may affect routes outside the initial examples.

Validate the cause, then monitor the intended cohort

Use fix validation when the relevant report issue supports it and the affected implementation has been corrected. For a few important changed URLs, an indexing request can ask Google to revisit them. For broader publication, maintain the sitemap and useful internal discovery paths. Neither route supplies an indexing guarantee or a reliable completion deadline.

Track the original cohort against stable URL keys and keep new publication separate. Otherwise a growing catalogue can hide persistent failures, or newly discovered duplicates can make a successful correction appear unsuccessful. Review aggregate changes with representative inspections and current response checks.

If reported counts change unexpectedly, check filters, property scope, releases and known reporting anomalies before assigning a new engineering incident. Keep a clear distinction between a verified defect, a plausible hypothesis and an unresolved selection outcome. The technical SEO monitoring workflow can turn recurring checks into owned alerts, with enough context to avoid reopening intentional exclusions as faults.