Three mechanisms, constantly confused, and one combination that produces the exact opposite of what people intend.
What each one does
robots.txt controls crawling. It says "do not fetch this URL". It says nothing about indexing.
Meta robots controls indexing for HTML pages. It says "you may fetch this, but do not put it in the index".
<meta name="robots" content="noindex, follow" />
X-Robots-Tag does the same as meta robots, delivered as an HTTP header - which means it works for files that have no HTML head.
X-Robots-Tag: noindex, follow
The combination that backfires
# robots.txt
Disallow: /private-page
<!-- /private-page -->
<meta name="robots" content="noindex" />
This keeps the page indexed.
robots.txt stops the crawler fetching the page. Because it never fetches the page, it never sees the noindex. If anything links to the URL, Google knows it exists, cannot see what is on it, and may index the URL with no snippet - the "no information is available for this page" result.
To remove a page from the index: allow crawling and serve noindex. Once it has dropped out, you can block it in robots.txt if you also want to save crawl budget. Order matters.
Free toolMeta Tags CheckerCheck any page's title, meta description, canonical, robots and heading structure, and see the exact issues that hurt its rankings.Which to use when
| Case | Mechanism |
|---|---|
| HTML page you want out of the index | Meta robots noindex |
| PDF, image, CSV, or any non-HTML file | X-Robots-Tag header |
| Whole directory of generated files | X-Robots-Tag via server config |
| Saving crawl budget on infinite URL spaces | robots.txt Disallow |
| Page already deindexed, now save crawl budget | robots.txt, after it has dropped |
The directives worth knowing
noindex- keep out of the index.nofollow- do not follow links on this page. Rarely what you want; it strands whatever the page links to.noindex, follow- the usual choice. Keep the page out, but let equity flow through its links. This is right for paginated series, filtered views and internal search results.noarchive- no cached copy.nosnippet- no snippet in results. Also suppresses you from AI overviews that quote snippets, so use deliberately.max-snippet:-1,max-image-preview:large- the opposite: explicitly allow full snippets and large previews. Worth setting on content you want quoted.
X-Robots-Tag in practice
For a directory of PDFs in nginx:
location ~* \.pdf$ {
add_header X-Robots-Tag "noindex, follow";
}
The header applies to any file type, which is the whole reason it exists. A PDF has no <head> to put a meta tag in.
Checking it worked
Read the rendered HTML, not the template. A noindex injected by client-side JavaScript is unreliable - Google may index the page before it runs.
Check the response headers too. A X-Robots-Tag: noindex set at the CDN will silently override what your HTML says, and it is invisible in view-source.
Then wait. Deindexing takes days to weeks. Use the URL removal tool in Search Console if you need it gone immediately, remembering that it is a temporary suppression, not a removal.
The one-line rule: robots.txt controls whether a crawler may look; meta robots and X-Robots-Tag control what it does after looking. Blocking a page you want deindexed prevents the instruction from ever being read.
Launch it where the numbers are checked
RankCert ranks products on domain control we verify ourselves. Listing is free and the link stays dofollow whether or not you display the badge.