XML sitemap best practices

Generate from publishing state, not every known address

Define an eligible sitemap record before writing XML. It should represent a published page you intend to make searchable, using its preferred absolute URL. Exclude drafts, private resources, removed pages and obvious duplicate variants. Redirecting addresses should normally be replaced with their final intended destinations.

Use the CMS or a reliable publishing inventory as the source of truth. A crawler-generated sitemap can reproduce gaps in internal navigation because the crawler cannot include pages it never finds. Conversely, a raw database export can include records that have never been publicly published. Document which state transitions add and remove entries.

Join the proposed inventory to response and canonical checks. A page listed in the sitemap but marked noindex expresses conflicting intentions. A URL that canonicalises elsewhere needs review before inclusion. Read canonical-tag guidance when the preferred address is unclear. The generator should encode the site's publishing decisions, not guess those decisions from whatever response happened to appear during one crawl.

Validate XML, encoding and file boundaries

Use a supported sitemap format and validate the actual output, not just the template that produces it. For XML, include the correct sitemap namespace and well-formed elements. Entity-escape values so an ampersand in a URL does not break parsing. Preserve the distinction between URL encoding and XML escaping.

Google's documented limit for a single sitemap is 50,000 URLs or 50 MB uncompressed. Split larger inventories and use a sitemap index where appropriate. Compression reduces transfer size but does not remove the uncompressed size limit. Keep generated files comfortably within boundaries so routine growth does not silently invalidate the next release.

Use fully qualified URLs with the intended scheme and hostname. Check that the deployment does not replace production addresses with staging values. If the file lives below the root, review its effective scope and submission method rather than assuming any path is equivalent. For cross-site arrangements, follow the supported verification or robots declaration method and keep ownership explicit. Do not turn a small sitemap into an unnecessarily complex distribution system.

Make lastmod a trustworthy publishing signal

A lastmod value should reflect a significant change to the represented page. Tie it to a meaningful content or metadata revision that the publishing system can track. Regenerating the sitemap, serving a request or changing a footer copyright date should not make every article appear newly revised.

Agree which events update the value: a substantive article correction, an important product specification change or an altered set of relevant links may qualify. Keep this rule consistent across templates. If the system cannot reliably distinguish meaningful changes from background jobs, omit the value rather than fabricating precision.

Audit dates by sampling unchanged pages across two sitemap generations and comparing genuinely edited pages with their revision history. Watch for timezone conversion errors and timestamps in the future. Google ignores sitemap priority and changefreq values, so tuning them is not a substitute for accurate inventory and dates. A useful sitemap tells a consistent story about publication; it does not need every optional element merely because the generator supports it.

Split files to answer operational questions

Partition large inventories around stable content types or publishing boundaries: products, categories, documentation and articles are often useful groups. A segment should help an owner investigate coverage or generator faults. Arbitrary tiny files make maintenance harder without improving the underlying discovery signals.

Keep membership stable enough to compare cohorts over time. If every generation reshuffles URLs between files, a change in segment coverage becomes difficult to interpret. For extremely large sets, deterministic bucketing can support stable comparisons while respecting size limits. Document how records are assigned and when files become obsolete.

Separate active files from historical migration artefacts. Submitting old and new inventories together can leave teams monitoring redirects as if they were current pages. For multilingual sites, coordinate the base inventory with multilingual sitemap requirements rather than duplicating contradictory exports. Use segment reporting to identify where investigation is needed, then inspect actual URLs. A successfully processed file does not prove each listed page was fetched or indexed.

Distinguish file-fetch failures from page-selection problems

Start with the sitemap's public response: status, final address, content type, body and any access challenge. Check the index and each child file independently. A valid child sitemap cannot help if the index points to an outdated filename or a protected endpoint.

Then validate parsing and inventory. Count entries, identify duplicate locations, compare hosts and check whether sample URLs meet publishing rules. Preserve the generation timestamp and source inventory count. This evidence separates a broken export from a genuine reduction in published content.

If Search Console reports successful processing but important pages are missing, examine their individual crawl and index status. Competing explanations include weak internal discovery, accidental exclusions, duplicate canonical selection and delayed processing. The sitemap's success message cannot distinguish them. Use Search Console diagnosis for that next step. If the report says the file could not be fetched while it works now, investigate the recorded timing, transient delivery faults and bot-facing configuration before rewriting valid XML.

Illustrative example: a catalogue export after a rebuild

Illustrative example: a catalogue has 18,000 published products, but a rebuilt sitemap includes 27,000 entries. The first explanation is newly published stock. A second is that the export now includes archived products and parameter variants. Comparing record IDs with publishing state reveals the latter.

The proposed eligibility rule includes public products with a current detail page and uses the canonical route from the routing service. Archived records with useful retained pages require a separate business decision; they are not removed solely because they are old. Tracking variants are excluded because they do not represent separate products.

During testing, a small set of products disappears from the export after a supplier update. Investigation finds that a feed timestamp was incorrectly treated as an archive flag. This demonstrates why inventory reconciliation must precede deployment.

The acceptance target is a coherent export matching the agreed eligible records, not the illustrative number itself. Preserve the prior file and inventory snapshot so a generator regression can be reversed. Later coverage checks examine the intended cohort without claiming that sitemap correction guarantees indexing.

Give developers measurable generator requirements

Require valid XML, absolute preferred URLs, no duplicate entries and compliance with file limits. Require each published eligible record to appear once in the correct segment, with excluded states absent. Test records near pagination and batching boundaries so a generator that skips the final batch cannot pass on small samples.

For dates, require an unchanged page to retain its lastmod across regeneration and a significant edit to update it correctly. Include characters needing escaping and a range of legitimate URL structures. Verify files through the public hostname after deployment, not only from build output.

Design publication atomically. A sitemap index should not reference files that have not yet been uploaded, and removed files should not disappear while the active index still needs them. Rollback may require restoring both the index and its matching children. If a failed job leaves the last valid set available, monitor its age and alert on staleness. Silent success with an empty but syntactically valid sitemap is a particularly important failure case to prevent.

Use sitemaps alongside links and realistic monitoring

Sitemaps supplement discovery; they do not supply the contextual relationships provided by internal links. If a valuable page appears only in the export, investigate whether it belongs in navigation or related editorial links. The orphan-page workflow helps distinguish deliberate standalone resources from broken architecture.

Do not use the number of submitted entries as an index count. Known URLs, eligible records and indexed canonical pages are different sets. Compare them using exact URL keys and preserve normalisation rules. A change in sitemap membership can alter report filters without any change in Google's treatment of the page.

Monitor generation health, file availability, membership drift and selected page outcomes separately. Keep reasonable expectations about reporting delay and crawler scheduling. If a page is already fetched and excluded as a duplicate, resubmitting the same sitemap is unlikely to address the cause. Fix the contradictory signal or page overlap identified by the evidence. The sitemap is successful when it accurately represents publishing intent and remains dependable through routine content changes.