The Index File Under Review: A Quarterly XML Sitemap Audit for UK Business Sites

A file that makes promises

Almost every UK business website carries a sitemap. Far fewer carry one that is still true. The file is typically generated at launch, connected to a Search Console property and a Bing Webmaster Tools account, and then left alone while the site beneath it is rebuilt, re-platformed, pruned and redirected. Three years later it is still submitting a list of URLs that no longer reflects the business it belongs to.

That matters because a sitemap is not a passive inventory. It is a declaration: these pages exist, they are worth crawling, and some of them have changed recently. Search engines treat the declaration as a hint to be tested against what they actually find. When the two disagree often enough, the hint loses value — and the pages that genuinely deserve prompt attention are buried among the ones that do not.

This is a technical housekeeping task, not a growth strategy. But it is inexpensive, it is within reach of any SME with a developer or a competent agency, and it removes a class of quiet friction that rarely appears in a dashboard.

What the file is genuinely for

It is worth being precise about the purpose before auditing against it. A sitemap serves two functions: it helps crawlers discover URLs they might not otherwise reach, and it can flag that a known URL has been updated. It is not a request for indexing, and inclusion carries no ranking benefit whatsoever. Search engines retain discretion over what they crawl and index, and a sitemap submitted in good faith does not override a noindex directive, a canonical pointing elsewhere, or a thin-content assessment.

The corollary is that a sitemap full of contradictions is worse than a smaller, accurate one. Precision beats coverage.

Fault line one: URLs that ask not to be indexed

Start with the most straightforward conflict. A URL listed in the sitemap while carrying a noindex meta tag, an X-Robots-Tag: noindex response header, or a block in robots.txt is sending two opposite instructions from the same site.

In practice this usually arises from ordinary commercial decisions: a thank-you page added to the template, a set of tag archives switched off after an audit, a PDF price list retired in favour of a gated download, a staging environment promoted to production with its directives intact. None of these are unreasonable in isolation. Together they turn the sitemap into a list of pages the site has explicitly disowned.

Detection is mechanical. Export every URL in the sitemap, crawl them, and reconcile status codes and indexability directives against the export. Any URL appearing in both the sitemap and the blocked set should be removed from the sitemap or unblocked deliberately — but not left in limbo.

Fault line two: redirects and vanished pages

A sitemap URL that returns a 301, a 302 or a chain of either is a small admission that the site's own records are out of date. A sitemap URL that returns a 404 or 410 is a larger one.

UK sites tend to accumulate these in predictable places. Ecommerce ranges turn over seasonally, so discontinued product URLs linger in generated sitemaps for months. Professional services firms restructure their team pages and forget that the old solicitor or consultant profiles were submitted. Multi-site groups consolidate regional pages after an acquisition and leave the legacy URLs in the index file.

The correct action is not always deletion. Where the destination page is a genuine successor, replace the old URL with the new one so the sitemap reflects the current structure. Where the page is gone with no successor, remove the entry. What should not happen is a sitemap that routes crawlers through a sequence of redirects to arrive somewhere that was never the point.

Soft 404s deserve a mention here too. A page that returns HTTP 200 but renders an apology or an empty template will not be caught by a status-code check alone; it needs either a manual sample or a crawl configuration that flags thin responses.

Fault line three: lastmod used as decoration

Of the fields in the sitemap protocol, lastmod is the one that still carries practical weight — which is precisely why careless values are damaging. Search engines have been explicit that the signal is discounted when it proves unreliable. A template that stamps the current timestamp onto every URL at build time, or rewrites lastmod on each deployment regardless of whether any content changed, produces exactly that outcome.

A defensible approach is to derive the value from the content record itself: the date the page's substantive copy, pricing or structure last changed, stored in the CMS and exposed honestly. Test it on a sample each quarter. Pick ten URLs, compare the submitted date with the actual edit history, and note the discrepancy rate. A site where the two never agree has effectively opted out of the signal.

changefreq and priority are widely ignored by the major search engines and can be removed without loss. Leaving them in is harmless; relying on them is not.

Fault line four: orphaned content

A sitemap URL that no internal link points to is a page the site has quietly disowned while still advertising it. Crawlers may find it, but with no internal context and no pathway for users, it will underperform — and it tends to be the kind of page that accumulates outdated claims about services, prices or regulatory status.

The check requires two datasets: the sitemap inventory and a crawl of the internal link graph. Any URL that appears in the first and not the second needs a decision. Either it earns a place in the navigation or a contextual link from a relevant page, or it should be unpublished and removed from the sitemap. There is rarely a third option worth defending.

Fault line five: an index that has outgrown its purpose

The file itself needs scrutiny. Sitemap indexes that reference dozens of child sitemaps — some empty, some pointing to removed sections, some exceeding the 50,000-URL or 50MB uncompressed limits set out in the protocol — are common on sites that have changed platform without changing configuration. Compressed delivery is standard and cheap; leaving large files uncompressed simply slows retrieval.

Look also for child sitemaps generated for sections that no longer exist, and for duplicate entries across children. An index that takes longer to read than the site takes to crawl is not earning its place.

A quarterly routine

The following sequence takes a competent developer or technical marketer roughly half a day on a typical SME site.

1. Export the full sitemap inventory, resolving any index structure into a single list of URLs.
2. Crawl the site and reconcile the sitemap list against the live crawl.
3. Flag every sitemap URL returning a non-200 status, and every URL whose directives contradict inclusion.
4. Sample lastmod values against the CMS edit history and record the agreement rate.
5. Compare the sitemap list against the internal link graph to surface orphans.
6. Inspect the index file for stale, empty or oversized children.

Record the counts each quarter. The trend is as informative as the absolute numbers: a URL count that falls by fifteen per cent without a corresponding reduction in the site is a signal that generation has broken, not that the site has been tidied.

Ranking the fixes

Not everything needs correcting in the same week. Treat contradictions as the priority: noindex and blocked URLs in the sitemap, broken or empty child sitemaps, and 404s in the index file all undermine trust in the file itself and are usually quick to resolve. Second-tier work covers lastmod integrity and orphaned pages, both of which require configuration changes or editorial decisions. Housekeeping — removing deprecated fields, compressing files, consolidating children — can wait for the next release cycle.

The point of the exercise

None of this will rescue a site with weak content or a confused information architecture. What it will do is remove a persistent, low-grade contradiction between what a UK business tells search engines and what its own website actually does. That contradiction costs little to fix and, left alone, quietly erodes the value of every other technical signal the site emits. A sitemap that can be trusted is a modest asset. A sitemap that cannot is a liability with a submission deadline.