Skip to content

robots.txt generator with validation

Free · no sign-up · runs in your browser

Build a robots.txt from user-agent groups, allow and disallow rules, crawl-delay and sitemap URLs — with presets for allowing everything, blocking AI crawlers, or locking down a staging site, and warnings for the mistakes that quietly block your whole site.

Start from

User-agent groups

One absolute URL per line.

  • An empty Disallow allows everything — which is valid, and worth being sure you meant.

Copy the file

robots.txt

User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

robots.txt controls crawling, not indexing. A disallowed URL can still be listed if something links to it — use a noindex tag to keep a page out of results.

robots.txt is four directives and a great deal of folklore. The file that blocks your entire site differs from the one that blocks nothing by a single character, the Crawl-delay line most sites ship is ignored by Google and Bing, and the whole file governs crawling rather than indexing — which is why a disallowed page can still turn up in results. This builds the file from a form and flags each of those as you go.

How it works

  1. 01Each user-agent gets its own group. `*` means every crawler that has no group of its own — a crawler obeys the most specific group that matches it, and only that one, so rules are not inherited from the wildcard group.
  2. 02Allow and disallow rules keep the order you write them in. Where two rules match the same URL, the longest path wins, not the first — which is how a broad disallow with a narrow allow carved out of it works.
  3. 03Sitemap lines are emitted last and outside any group, because they are global directives. Putting them inside a group implies a scope they do not have.
  4. 04Crawl-delay is emitted only when it is a positive number, and flagged when it is set for an agent that ignores it — Google and Bing both do.
  5. 05The file is checked as you type: a wildcard group disallowing everything, a sitemap that is not an absolute URL, a path missing its leading slash, or no sitemap at all.
  6. 06Presets give you a starting point: allow everything, block the common AI crawlers while staying open to search engines, or block everything for a staging site.

Use cases

  • Setting up robots.txt for a new site without guessing at the syntax.
  • Opting out of AI training crawlers while staying indexable by search engines.
  • Locking a staging or preview deployment away from every crawler.
  • Keeping an admin, cart or search-results path out of the crawl budget.
  • Adding a sitemap reference to a robots.txt that never had one.
  • Checking an existing file for the one-character mistake that blocks the site.

Frequently asked questions

What is the difference between Disallow: / and Disallow:?

Everything. `Disallow: /` blocks the entire site; `Disallow:` with no value blocks nothing and is the standard way to say 'crawl freely'. One character separates a site that ranks from one that disappears, which is why this tool warns when a wildcard group disallows the root.

Does robots.txt stop a page from being indexed?

No — that is the most consequential misunderstanding about it. robots.txt controls *crawling*. If another site links to a disallowed URL, it can still appear in results, typically without a description, because the crawler was never allowed to fetch the page and read the noindex that would have removed it. To keep a page out of the index, allow crawling and serve a noindex meta tag or header.

How do I block AI crawlers?

Give each one its own group with `Disallow: /`. The block-AI preset above covers GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, PerplexityBot, Google-Extended, CCBot, Bytespider and meta-externalagent, while leaving ordinary search crawlers alone. Note that compliance is voluntary: robots.txt is a request, not an enforcement mechanism, and a crawler that ignores it is not stopped by it.

Does Crawl-delay work?

Not for Google or Bing — both ignore it entirely. Some other crawlers honour it. If Googlebot is hitting your server too hard, the crawl rate setting in Search Console is the lever, or better, a server that responds faster. This tool flags a Crawl-delay set on an agent known to ignore it.

Where does robots.txt have to live?

At the root of the host: https://example.com/robots.txt, served as text/plain. It applies to that host and scheme only, so a subdomain needs its own file and http and https are treated as separate origins. A robots.txt in a subdirectory is ignored.

Which rule wins when two match the same URL?

The most specific one — the longest matching path — not the first one written. So `Disallow: /private/` with `Allow: /private/public.html` leaves that one file crawlable. Where two rules are the same length, the least restrictive wins.

Related reading

More tools