# robots.txt Generator

> A robots.txt generator builds the plain-text file at the root of your site that tells crawlers which paths they may fetch. Pick the crawlers and the paths to disallow, and you get a valid file with your sitemap declared, ready to upload.

Source: https://rankcert.com/tools/robots-txt-generator
Free, no signup. Updated: 2026-08-29

## How to use it

1. Add the paths you want kept out of search results.
2. Point it at your sitemap.
3. Save the output as robots.txt at the root of your domain.

## Questions

### Does robots.txt stop a page being indexed?

No. It stops crawling, not indexing. A blocked URL with external links can still appear in results with no snippet. To keep a page out of the index, allow crawling and use a noindex meta tag.

### Should I block AI crawlers?

Usually not. Blocking GPTBot or ClaudeBot removes you from AI answers as well as training data, and for a small product the citations are worth more than the exclusion.

### Does Crawl-delay work?

Google ignores it. Bing and Yandex honour it. Only set it if your server is genuinely struggling.

### Where does the file go?

Exactly at yourdomain.com/robots.txt, served as text/plain. Subdirectories do not work - each subdomain needs its own.

### What should a normal site block?

Admin, checkout, account and internal search result pages. Nothing else. Blocking CSS and JavaScript stops Google rendering your page and is a common self-inflicted wound.

### Does robots.txt need to be at the root?

Yes, exactly at yourdomain.com/robots.txt. A file at /blog/robots.txt is ignored entirely, and each subdomain needs its own.

### How do I keep a page out of Google properly?

Use a noindex meta robots tag and let the page stay crawlable. Blocking it in robots.txt means Google cannot read the noindex and may index the URL anyway from external links.

### What happens if robots.txt returns a 500 error?

Google treats a server error as a full disallow and may stop crawling the site entirely. A 404 is safe - it is read as allow everything.

## Related tools

- [SEO Checker](https://rankcert.com/tools/seo-checker): Run eleven on-page checks against any URL: title, description, canonical, headings, alt text, Open Graph, word count and indexability, with the exact fixes.
- [Redirect Checker](https://rankcert.com/tools/redirect-checker): Follow every hop a URL takes, see the status code at each step, and find the chains, loops and temporary redirects that leak link equity.
- [Broken Link Checker](https://rankcert.com/tools/broken-link-checker): Crawl any page for broken links, see the status code behind every dead URL, and separate the internal ones that cost you link equity from the external rot.
- [Backlink Verifier](https://rankcert.com/tools/backlink-verifier): Check whether a page really links to your domain, what rel attribute the link carries, and whether the page is indexable at all.
- [Outbound Link Checker](https://rankcert.com/tools/outbound-link-checker): List every external link on a page with its rel attribute, count how many pass equity, and catch affiliate links that should be marked sponsored.
- [llms.txt Generator](https://rankcert.com/tools/llms-txt-generator): Generate an llms.txt file that tells AI answer engines what your product is and which pages matter, in the markdown format the proposal specifies.