# noindex three ways: meta robots, X-Robots-Tag and robots.txt
> Use a meta robots noindex for HTML pages, the X-Robots-Tag header for PDFs and other non-HTML files, and robots.txt only to control crawling. Never block a page in robots.txt that carries a noindex - the crawler cannot read the directive it is blocked from fetching.
Source: https://rankcert.com/blog/meta-robots-vs-x-robots
Published: 2026-08-29 ยท Updated: 2026-08-29
---
Three mechanisms, constantly confused, and one combination that produces the exact opposite of what people intend.
## What each one does
**robots.txt** controls **crawling**. It says "do not fetch this URL". It says nothing about indexing.
**Meta robots** controls **indexing** for HTML pages. It says "you may fetch this, but do not put it in the index".
```html
```
**X-Robots-Tag** does the same as meta robots, delivered as an HTTP header - which means it works for files that have no HTML head.
```
X-Robots-Tag: noindex, follow
```
## The combination that backfires
```
# robots.txt
Disallow: /private-page
```
```html
```
This keeps the page indexed.
robots.txt stops the crawler fetching the page. Because it never fetches the page, it never sees the `noindex`. If anything links to the URL, Google knows it exists, cannot see what is on it, and may index the URL with no snippet - the "no information is available for this page" result.
**To remove a page from the index: allow crawling and serve noindex.** Once it has dropped out, you can block it in robots.txt if you also want to save crawl budget. Order matters.
Free tool: [Meta Tags Checker](https://rankcert.com/tools/meta-tags-checker) - Check any page's title, meta description, canonical, robots and heading structure, and see the exact issues that hurt its rankings.
## Which to use when
| Case | Mechanism |
|---|---|
| HTML page you want out of the index | Meta robots `noindex` |
| PDF, image, CSV, or any non-HTML file | `X-Robots-Tag` header |
| Whole directory of generated files | `X-Robots-Tag` via server config |
| Saving crawl budget on infinite URL spaces | robots.txt `Disallow` |
| Page already deindexed, now save crawl budget | robots.txt, **after** it has dropped |
## The directives worth knowing
- **`noindex`** - keep out of the index.
- **`nofollow`** - do not follow links on this page. Rarely what you want; it strands whatever the page links to.
- **`noindex, follow`** - the usual choice. Keep the page out, but let equity flow through its links. This is right for paginated series, filtered views and internal search results.
- **`noarchive`** - no cached copy.
- **`nosnippet`** - no snippet in results. Also suppresses you from AI overviews that quote snippets, so use deliberately.
- **`max-snippet:-1`, `max-image-preview:large`** - the opposite: explicitly allow full snippets and large previews. Worth setting on content you want quoted.
## X-Robots-Tag in practice
For a directory of PDFs in nginx:
```
location ~* \.pdf$ {
add_header X-Robots-Tag "noindex, follow";
}
```
The header applies to any file type, which is the whole reason it exists. A PDF has no `` to put a meta tag in.
## Checking it worked
**Read the rendered HTML, not the template.** A `noindex` injected by client-side JavaScript is unreliable - Google may index the page before it runs.
**Check the response headers too.** A `X-Robots-Tag: noindex` set at the CDN will silently override what your HTML says, and it is invisible in view-source.
**Then wait.** Deindexing takes days to weeks. Use the URL removal tool in Search Console if you need it gone immediately, remembering that it is a temporary suppression, not a removal.
The one-line rule: robots.txt controls whether a crawler may look; meta robots and X-Robots-Tag control what it does after looking. Blocking a page you want deindexed prevents the instruction from ever being read.