robots.txt Generator
Build a valid robots.txt with your disallow rules, sitemap reference and optional AI crawler blocks, ready to paste at the root of your site.
A robots.txt generator builds the plain-text file at the root of your site that tells crawlers which paths they may fetch. Pick the crawlers and the paths to disallow, and you get a valid file with your sitemap declared, ready to upload.
Runs entirely in your browser · no signup · nothing is uploaded · updated 2026-08-29
Block AI crawlers
Blocking these removes you from AI answers as well as AI training. Most products want the citations more than they want the exclusion.
How to use it
- 1.Add the paths you want kept out of search results.
- 2.Point it at your sitemap.
- 3.Save the output as robots.txt at the root of your domain.
Frequently asked questions
Does robots.txt stop a page being indexed?
No. It stops crawling, not indexing. A blocked URL with external links can still appear in results with no snippet. To keep a page out of the index, allow crawling and use a noindex meta tag.
Should I block AI crawlers?
Usually not. Blocking GPTBot or ClaudeBot removes you from AI answers as well as training data, and for a small product the citations are worth more than the exclusion.
Does Crawl-delay work?
Google ignores it. Bing and Yandex honour it. Only set it if your server is genuinely struggling.
Where does the file go?
Exactly at yourdomain.com/robots.txt, served as text/plain. Subdirectories do not work - each subdomain needs its own.
What should a normal site block?
Admin, checkout, account and internal search result pages. Nothing else. Blocking CSS and JavaScript stops Google rendering your page and is a common self-inflicted wound.
Does robots.txt need to be at the root?
Yes, exactly at yourdomain.com/robots.txt. A file at /blog/robots.txt is ignored entirely, and each subdomain needs its own.
How do I keep a page out of Google properly?
Use a noindex meta robots tag and let the page stay crawlable. Blocking it in robots.txt means Google cannot read the noindex and may index the URL anyway from external links.
What happens if robots.txt returns a 500 error?
Google treats a server error as a full disallow and may stop crawling the site entirely. A 404 is safe - it is read as allow everything.
The guide behind this tool
3 min readllms.txt: worth adding, and honest about what it does
What the llms.txt proposal specifies, how it differs from robots.txt, whether any AI engine reads it yet, and what actually gets you cited in AI answers.
3 min readnoindex three ways: meta robots, X-Robots-Tag and robots.txt
The three mechanisms for controlling indexing, which one to use for which case, and the combination that accidentally keeps a page in the index forever.
3 min readrobots.txt: what it does, and the AI crawler question
How robots.txt actually works, why blocking a page does not remove it from Google, and whether you should block GPTBot and ClaudeBot.
Tools that pair with this one
SEO & on-pageSEO Checker
Run eleven on-page checks against any URL: title, description, canonical, headings, alt text, Open Graph, word count and indexability, with the exact fixes.
SEO & on-pageRedirect Checker
Follow every hop a URL takes, see the status code at each step, and find the chains, loops and temporary redirects that leak link equity.
SEO & on-pageBroken Link Checker
Crawl any page for broken links, see the status code behind every dead URL, and separate the internal ones that cost you link equity from the external rot.
SEO & on-pageBacklink Verifier
Check whether a page really links to your domain, what rel attribute the link carries, and whether the page is indexable at all.
Built something?
RankCert is a weekly launch board where products rank on domain control we verify ourselves - not upvotes. Listing is free and the link is dofollow whether or not you display our badge.