robots.txt generator with validation
Build a robots.txt from user-agent groups, allow and disallow rules, crawl-delay and sitemap URLs — with presets for allowing everything, blocking AI crawlers, or locking down a staging site, and warnings for the mistakes that quietly block your whole site.
Free · no sign-up · runs in your browser
Start from
User-agent groups
One absolute URL per line.
- An empty Disallow allows everything — which is valid, and worth being sure you meant.
Copy the file
robots.txt
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xmlrobots.txt controls crawling, not indexing. A disallowed URL can still be listed if something links to it — use a noindex tag to keep a page out of results.
robots.txt is four directives and a great deal of folklore. The file that blocks your entire site differs from the one that blocks nothing by a single character, the Crawl-delay line most sites ship is ignored by Google and Bing, and the whole file governs crawling rather than indexing — which is why a disallowed page can still turn up in results. This builds the file from a form and flags each of those as you go.
How it works
- 01Each user-agent gets its own group. `*` means every crawler that has no group of its own — a crawler obeys the most specific group that matches it, and only that one, so rules are not inherited from the wildcard group.
- 02Allow and disallow rules keep the order you write them in. Where two rules match the same URL, the longest path wins, not the first — which is how a broad disallow with a narrow allow carved out of it works.
- 03Sitemap lines are emitted last and outside any group, because they are global directives. Putting them inside a group implies a scope they do not have.
- 04Crawl-delay is emitted only when it is a positive number, and flagged when it is set for an agent that ignores it — Google and Bing both do.
- 05The file is checked as you type: a wildcard group disallowing everything, a sitemap that is not an absolute URL, a path missing its leading slash, or no sitemap at all.
- 06Presets give you a starting point: allow everything, block the common AI crawlers while staying open to search engines, or block everything for a staging site.
Use cases
- Setting up robots.txt for a new site without guessing at the syntax.
- Opting out of AI training crawlers while staying indexable by search engines.
- Locking a staging or preview deployment away from every crawler.
- Keeping an admin, cart or search-results path out of the crawl budget.
- Adding a sitemap reference to a robots.txt that never had one.
- Checking an existing file for the one-character mistake that blocks the site.
Frequently asked questions
What is the difference between Disallow: / and Disallow:?
Everything. `Disallow: /` blocks the entire site; `Disallow:` with no value blocks nothing and is the standard way to say 'crawl freely'. One character separates a site that ranks from one that disappears, which is why this tool warns when a wildcard group disallows the root.
Does robots.txt stop a page from being indexed?
No — that is the most consequential misunderstanding about it. robots.txt controls *crawling*. If another site links to a disallowed URL, it can still appear in results, typically without a description, because the crawler was never allowed to fetch the page and read the noindex that would have removed it. To keep a page out of the index, allow crawling and serve a noindex meta tag or header.
How do I block AI crawlers?
Give each one its own group with `Disallow: /`. The block-AI preset above covers GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, PerplexityBot, Google-Extended, CCBot, Bytespider and meta-externalagent, while leaving ordinary search crawlers alone. Note that compliance is voluntary: robots.txt is a request, not an enforcement mechanism, and a crawler that ignores it is not stopped by it.
Does Crawl-delay work?
Not for Google or Bing — both ignore it entirely. Some other crawlers honour it. If Googlebot is hitting your server too hard, the crawl rate setting in Search Console is the lever, or better, a server that responds faster. This tool flags a Crawl-delay set on an agent known to ignore it.
Where does robots.txt have to live?
At the root of the host: https://example.com/robots.txt, served as text/plain. It applies to that host and scheme only, so a subdomain needs its own file and http and https are treated as separate origins. A robots.txt in a subdirectory is ignored.
Which rule wins when two match the same URL?
The most specific one — the longest matching path — not the first one written. So `Disallow: /private/` with `Allow: /private/public.html` leaves that one file crawlable. Where two rules are the same length, the least restrictive wins.
Related resources
Related reading
- Frontend5 Frontend Tools Worth Bookmarking in 2026A short, opinionated list of the frontend tools and references we reach for every week — frameworks, styling, components, and icons.
More tools
Tailwind CSS color palette generator
Open →Turn one hex code into a full Tailwind color scale — 11 OKLCH-tuned shades from 50 to 950, with contrast ratios, ready to paste into Tailwind v4's @theme block or a v3 config. Free, no sign-up, runs in your browser.
CSS clamp() calculator for fluid typography
Open →Generate a responsive CSS clamp() font size from a minimum and maximum size and the viewport range they should grow across — with the slope and intercept shown, a live preview, and the rendered size at every common screen width. Free, no sign-up.
SVG to JSX converter for React components
Open →Paste an SVG and get a React component back — attributes camelCased, class turned into className, style strings converted to objects, xmlns removed, with optional currentColor, a size prop and TypeScript output. Runs entirely in your browser; nothing is uploaded.
Meta tag generator with Google SERP preview
Open →Fill in a title, description, canonical and Open Graph image and get the complete <head> snippet — with a live Google result preview that measures your title and description in pixels, not characters, so you can see exactly where they will be cut.
Favicon generator — .ico, PNG set and manifest
Open →Turn a PNG or SVG, or just a couple of letters, into the full favicon set: 16, 32, 48, 180, 192 and 512 PNGs, a real multi-size favicon.ico, a web app manifest and the <head> tags. Everything runs in your browser — nothing is uploaded.
JSON-LD schema generator and validator
Open →Build valid JSON-LD structured data for Article, FAQPage, Product, Organization and BreadcrumbList from a form, or paste existing markup to have it pretty-printed and checked for the mistakes that stop a rich result appearing. Free, runs in your browser.
UTM link builder and campaign URL generator
Open →Build correctly encoded UTM tracking links from a URL and your source, medium and campaign — with the existing query string and fragment preserved, values properly percent-encoded, and warnings for the capitalisation and spacing that split one campaign across several rows in an analytics report.
Colour contrast checker with WCAG suggestions
Open →Check any two colours against WCAG 2 — normal text, large text and UI components — see the ratio and live samples, and get the nearest passing colour when a pair fails, adjusted in OKLCH so the hue stays yours. Free, runs in your browser.
Image converter — WebP, AVIF, JPEG and PNG
Open →Convert and resize images to WebP, AVIF, JPEG or PNG at the quality you choose, see exactly how many bytes each one saved, and download them one at a time or as a zip. Everything happens in your browser — no image is uploaded.
JSON formatter and validator — free, in your browser
Open →Format, minify and validate JSON in your browser. Broken documents get the exact line and column of the error. Nothing is uploaded, and there is no size limit but your machine.
JSON to TypeScript — generate interfaces from a payload
Open →Paste a JSON payload and get TypeScript interfaces: merged array shapes, optional properties, and real unions instead of any. Runs in your browser — nothing is uploaded.
OG image generator — free 1200×630 social cards
Open →Generate an Open Graph image for any page: type a title, pick a theme, download a 1200×630 PNG. Runs entirely in your browser — no account, no upload, no watermark.