robots.txt: what it does, and the AI crawler question

robots.txt is the oldest and most misunderstood file in SEO. The single most important thing about it:

robots.txt controls crawling, not indexing.

A URL you block in robots.txt can still appear in Google results. If other sites link to it, Google knows the URL exists, cannot fetch it to see what is there, and may index it anyway with no snippet - the "no information is available for this page" result.

To keep a page out of the index you must let Google crawl it and serve a noindex meta tag. Blocking it in robots.txt actively prevents that, because Google can never read the noindex.

The syntax

User-agent: *
Disallow: /admin
Disallow: /api

Sitemap: https://example.com/sitemap.xml

• User-agent - which crawler the block applies to. * is everyone.
• Disallow - a path prefix to skip. Disallow: /admin blocks /admin, /admin/users, and /administrator - it is a prefix, not a directory.
• Allow - an exception carving a path out of a broader Disallow.
• Sitemap - absolute URL, and it applies globally regardless of which user-agent block it sits in.

Rules people get wrong:

• Most specific rule wins, not first or last. Disallow: /blog plus Allow: /blog/public permits the second.
• Disallow: with nothing after it means allow everything. Disallow: / means block everything. One character apart, opposite meanings.
• The file must be at the domain root, served as text/plain. example.com/pages/robots.txt is ignored, and every subdomain needs its own.
• Crawl-delay is ignored by Google. Bing and Yandex honour it.
• Case matters in paths, not in directives.

What to block on a small site

Very little. The default should be blocking nothing.

Worth blocking:

• Admin and account areas - no search value, and crawling them wastes budget.
• Internal search results - /search?q= generates infinite URLs from crawler exploration.
• Faceted filter parameters that produce combinatorial URLs.

Not worth blocking:

• CSS and JavaScript. Google renders pages; blocking assets means it sees a broken page. This was common advice in 2010 and is actively harmful now.
• Duplicate content. Use canonicals. Blocking prevents Google from seeing the canonical.
• Pages you want removed. Use noindex, as above.

The AI crawler question

You can block AI crawlers by user-agent:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

Other named agents include OAI-SearchBot, PerplexityBot, Google-Extended and CCBot.

For most products, blocking these is a mistake. They serve two different purposes and the tradeoff is different for each:

• Training crawlers (GPTBot, ClaudeBot, CCBot) collect data for model training. Blocking keeps you out of training sets.
• Search crawlers (OAI-SearchBot, PerplexityBot) fetch pages to answer live questions with citations. Blocking these removes you from AI answers entirely.

If a growing share of your buyers ask an assistant "what's a good tool for X", being absent from that answer is a real cost. Publishers with licensable archives have a genuine reason to block. A small SaaS wanting to be discovered almost never does.

Note that compliance is voluntary. robots.txt is a request, and only well-behaved crawlers honour it. It is not access control.

Common mistakes that cost real traffic

Shipping a staging robots.txt. Disallow: / deployed to production deindexes the entire site within days. Guard against it in your deploy pipeline.

Blocking a URL you also canonicalise to. Google cannot follow a canonical pointing at a blocked URL, so the signal is discarded.

Blocking parameters instead of canonicalising. UTM-tagged URLs blocked in robots.txt can still get indexed, with no way for Google to learn they are duplicates.

Forgetting the sitemap line. Free, and it is one of the two ways Google discovers your sitemap.