Sitemap best practices: the rules that affect indexing

A sitemap does not make anything rank and does not guarantee indexing. What it does is tell search engines which URLs you consider canonical, and unlock the single most useful diagnostic you have: the gap between submitted and indexed.

The four inclusion rules

A URL belongs in your sitemap only if it is all four:

1. Canonical - and its own canonical tag agrees.
2. Indexable - 200, no noindex, not blocked in robots.txt.
3. Worth ranking - a real destination, not a utility route.
4. Parameter-free - no session IDs, no tracking, no filter combinations.

Including URLs that redirect, 404 or carry noindex sends contradictory signals and erodes how much weight the file is given.

lastmod, or nothing

Google uses lastmod - but only when it is consistently accurate.

A build process that stamps today's date on every URL every deploy tells Google the entire site changed, which is false, and the field gets ignored from then on. That is worse than omitting it.

Set it from real modification data: updated_at for database rows, frontmatter for content, git commit date at a push. If you cannot do that reliably, leave the field out.

changefreq and priority are ignored entirely. Including them is harmless and pointless.

Splitting

The hard limits are 50,000 URLs or 50MB uncompressed per file. Split long before either.

Not for size - for diagnosis. One sitemap tells you 58 URLs are unindexed. Separate sitemaps for pages, blog, tools and products tell you which type is failing, which is the difference between a mystery and a fix.

<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap><loc>https://example.com/sitemap/pages.xml</loc></sitemap>
  <sitemap><loc>https://example.com/sitemap/blog.xml</loc></sitemap>
</sitemapindex>

Discovery

Both mechanisms, always:

• A Sitemap: line in robots.txt - works for every engine, costs one line.
• Submitted in Search Console and Bing Webmaster Tools - this is what gives you the coverage report.

Reading coverage

The indexed-to-submitted ratio is your content quality score:

• Over 90% - healthy. Keep publishing.
• 70–90% - some thin or duplicative pages. Find which type.
• Under 70% - a real problem. Publishing more will make it worse.

The exclusion reasons, in order of how often they appear:

"Crawled - currently not indexed." Google looked and decided it was not worth indexing. Thin, or duplicative of something else on your site. The most common and most actionable.

"Discovered - currently not indexed." Known but not yet crawled. Usually a crawl budget or internal linking problem.

"Duplicate, Google chose a different canonical." Your canonical and Google's judgement disagree - almost always real duplication.

"Page with redirect." You put a redirecting URL in the sitemap. Remove it.

Programmatic sitemaps

Generate the sitemap from the same predicate that decides whether a page should exist. If a page is too thin to publish, it must be impossible for it to appear in the sitemap - not a separate check someone has to remember.