# robots.txt: what it does, and the AI crawler question

> robots.txt controls crawling, not indexing. A blocked URL can still appear in Google with no snippet if other sites link to it. To keep a page out of the index, allow crawling and serve a noindex meta tag - blocking it prevents Google from ever reading that directive.

Source: https://rankcert.com/blog/robots-txt-guide
Published: 2026-08-29 · Updated: 2026-09-01

---


robots.txt is the oldest and most misunderstood file in SEO. The single most important thing about it:

**robots.txt controls crawling, not indexing.**

A URL you block in robots.txt can still appear in Google results. If other sites link to it, Google knows the URL exists, cannot fetch it to see what is there, and may index it anyway with no snippet - the "no information is available for this page" result.

To keep a page out of the index you must let Google crawl it and serve a `noindex` meta tag. Blocking it in robots.txt actively prevents that, because Google can never read the noindex.

## The syntax

```
User-agent: *
Disallow: /admin
Disallow: /api

Sitemap: https://example.com/sitemap.xml
```

- **`User-agent`** - which crawler the block applies to. `*` is everyone.
- **`Disallow`** - a path prefix to skip. `Disallow: /admin` blocks `/admin`, `/admin/users`, and `/administrator` - it is a prefix, not a directory.
- **`Allow`** - an exception carving a path out of a broader Disallow.
- **`Sitemap`** - absolute URL, and it applies globally regardless of which user-agent block it sits in.

Rules people get wrong:

- **Most specific rule wins**, not first or last. `Disallow: /blog` plus `Allow: /blog/public` permits the second.
- **`Disallow:` with nothing after it means allow everything.** `Disallow: /` means block everything. One character apart, opposite meanings.
- **The file must be at the domain root**, served as `text/plain`. `example.com/pages/robots.txt` is ignored, and every subdomain needs its own.
- **`Crawl-delay` is ignored by Google.** Bing and Yandex honour it.
- **Case matters in paths**, not in directives.

Free tool: [robots.txt Generator](https://rankcert.com/tools/robots-txt-generator) - Build a valid robots.txt with your disallow rules, sitemap reference and optional AI crawler blocks, ready to paste at the root of your site.

## What to block on a small site

Very little. The default should be blocking nothing.

Worth blocking:

- **Admin and account areas** - no search value, and crawling them wastes budget.
- **Internal search results** - `/search?q=` generates infinite URLs from crawler exploration.
- **Faceted filter parameters** that produce combinatorial URLs.

Not worth blocking:

- **CSS and JavaScript.** Google renders pages; blocking assets means it sees a broken page. This was common advice in 2010 and is actively harmful now.
- **Duplicate content.** Use canonicals. Blocking prevents Google from seeing the canonical.
- **Pages you want removed.** Use `noindex`, as above.

## The AI crawler question

You can block AI crawlers by user-agent:

```
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /
```

Other named agents include `OAI-SearchBot`, `PerplexityBot`, `Google-Extended` and `CCBot`.

**For most products, blocking these is a mistake.** They serve two different purposes and the tradeoff is different for each:

- **Training crawlers** (`GPTBot`, `ClaudeBot`, `CCBot`) collect data for model training. Blocking keeps you out of training sets.
- **Search crawlers** (`OAI-SearchBot`, `PerplexityBot`) fetch pages to answer live questions with citations. Blocking these removes you from AI answers entirely.

If a growing share of your buyers ask an assistant "what's a good tool for X", being absent from that answer is a real cost. Publishers with licensable archives have a genuine reason to block. A small SaaS wanting to be discovered almost never does.

Note that compliance is voluntary. robots.txt is a request, and only well-behaved crawlers honour it. It is not access control.

## Common mistakes that cost real traffic

**Shipping a staging robots.txt.** `Disallow: /` deployed to production deindexes the entire site within days. Guard against it in your deploy pipeline.

**Blocking a URL you also canonicalise to.** Google cannot follow a canonical pointing at a blocked URL, so the signal is discarded.

**Blocking parameters instead of canonicalising.** UTM-tagged URLs blocked in robots.txt can still get indexed, with no way for Google to learn they are duplicates.

**Forgetting the sitemap line.** Free, and it is one of the two ways Google discovers your sitemap.

<Callout>
Before you block anything, ask whether you want the page crawled or want it out of the index. They are different problems with different, mutually exclusive solutions.
</Callout>

<Cta />