How to Get Your Dev Tool Cited by AI Search: llms.txt, Crawlers, Structured Data

To get a dev tool cited by AI search, let the search crawlers in, get indexed by normal search, and publish clear text pages that answer real questions about your tool. Google says AI Overviews and AI Mode have no extra requirements beyond regular Search, and the AI vendors' own docs mainly give you one lever: robots.txt rules per crawler. An llms.txt file is cheap to add, but none of the major providers documents reading it for search. Several docs platforms generate one for you.

Every claim below comes from the vendor's own documentation, checked on 11 November 2026. Where we are reasoning rather than quoting a vendor, the section says so.

What is documented, and what is not

Topic	Who documents it	What the docs say
AI Overviews / AI Mode	Google Search Central	No extra requirements; page must be indexed and snippet-eligible
llms.txt	llmstxt.org (a proposal)	A Markdown map of your site for agents; no provider commits to reading it
Search crawlers	OpenAI, Anthropic, Perplexity	Separate user agents for search vs training, controlled by robots.txt
Google-Extended	Google	Controls Gemini training and grounding, not Search inclusion
Structured data	Google, schema.org	Helps Google understand a page; no special schema needed for AI features
Being mentioned elsewhere	Nobody	Reasonable guess, not a documented ranking rule

What Google says about AI features

Google's page on AI features and your website is unusually direct: "There are no additional requirements to appear in AI Overviews or AI Mode." A page has to be indexed and eligible to show in Search with a snippet, and even then inclusion is not guaranteed. The same page says you don't need new machine-readable files, AI text files or markup, and that there is no special schema.org structured data to add.

What it does recommend is the usual list: allow crawling in robots.txt and at your CDN, link pages internally so they can be found, keep important content in text, write people-first content, and make structured data match the visible text on the page. Traffic from AI Overviews and AI Mode shows up in Search Console under the "Web" search type, so you measure it in the report you already have.

To limit what appears, Google points to the existing controls: noindex, nosnippet, data-nosnippet and max-snippet.

llms.txt: a proposal, not a standard

llms.txt was proposed by Jeremy Howard on 3 September 2024. It is a Markdown file at /llms.txt with an H1 (the only required part), a short blockquote summary, and H2 sections of links to detailed pages, plus an optional Optional section an agent can skip. The proposal also suggests serving a Markdown version of each page, for example at the same URL with .md appended. Its author expected it to be useful mainly at inference time, when an agent is helping a user, rather than for training.

What the big providers have said, from their own pages:

Google says you don't need "AI text files" to appear in AI features. Its page does not name llms.txt.
OpenAI and Perplexity link an llms.txt on their crawler docs pages, but only as an index of their own documentation. Neither page describes their crawlers reading yours.
Anthropic's crawler help page does not mention llms.txt.

The llmstxt.org page itself notes that the AI labs publish llms.txt files for their developer docs. Publishing one is not the same as reading other sites' files, and we found no provider page that commits to the second.

This site serves both /llms.txt and /llms-full.txt, built from the same config the pages use, so they cannot drift from what the site shows. The -full variant is our own convention, not part of the llmstxt.org proposal.

AI crawlers and robots.txt

Each vendor splits crawling by purpose, so you can block training and still allow search. The user agents below are from each vendor's own docs.

Vendor	Search / index	Training	User-initiated fetch
OpenAI	`OAI-SearchBot`	`GPTBot`	`ChatGPT-User`
Anthropic	`Claude-SearchBot`	`ClaudeBot`	`Claude-User`
Perplexity	`PerplexityBot`	none listed	`Perplexity-User`
Google	`Googlebot` (Search)	`Google-Extended` token	n/a

OpenAI

OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, apart from navigational links. GPTBot is the training crawler, and the two settings are independent. For ChatGPT-User, robots.txt rules may not apply, because a user asked for the fetch. OpenAI notes that robots.txt changes can take about 24 hours to reach search.

Anthropic

ClaudeBot collects content that may be used for training; Claude-SearchBot indexes for search quality; Claude-User fetches pages when a user asks. Anthropic says blocking either of the last two may reduce your visibility in Claude's search results. Its bots honour robots.txt and the non-standard Crawl-delay, and it advises against blocking its IP addresses, since that can stop the bots from reading your robots.txt at all.

Perplexity

Perplexity says PerplexityBot surfaces and links sites in its search results and is not used to crawl content for foundation models. Perplexity-User handles user-requested fetches and generally ignores robots.txt. Changes may take up to 24 hours.

Google-Extended

Google-Extended is a robots.txt token with no HTTP user agent of its own. It controls whether content is used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. Google states it does not affect inclusion in Google Search and is not a ranking signal. AI Overviews are part of Search, so Googlebot rules are what count there.

A starting robots.txt

Allow search, block training, keep your sitemap line:

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Drop the three Disallow blocks if you are happy for your docs to be used in training; for a dev tool that wants to be known, many teams will be. The robots.txt generator in our toolbox builds the file and warns about rules that block everything.

Structured data

Google describes structured data as a standardized format for describing a page, recommends JSON-LD, and says valid markup "can enable" richer results rather than guaranteeing them. For AI features it adds no special type; it only asks that markup match the visible text.

For a tool, the relevant vocabulary is schema.org SoftwareApplication (plus Organization and WebSite on the home page, and Article on posts). Google's software app guide requires name, offers.price (use 0 for free) and either aggregateRating or review for the rich result. Do not invent ratings to qualify: the guidelines allow manual actions for markup that misrepresents the page. Without real reviews you can still add the markup; you just won't get that rich result.

Generate and validate blocks with the JSON-LD generator, then check the live page with Google's Rich Results Test.

Be present where models read

This section is our reasoning, not vendor documentation. AI search answers are built from pages a crawler fetched or a search index holds, so the more accurate, crawlable pages describe your tool, the more chances an answer has to cite one of them.

Your own docs. Plain-text pages that answer "what is X", "X vs Y", "how do I install X" and "is X free". Put the answer in the first paragraph, not behind a tab or a script.
Your GitHub README. One sentence on what the tool does, install command, a short example, licence, link to the docs. Keep it consistent with your site.
Directories and launch sites. Listings on curated directories and launch sites give crawlers another description of the tool. Our guide to where to launch your side project covers which ones are worth the time.
Consistency. Use the same name, one-line description and pricing everywhere. A model that finds three different prices may quote the wrong one.

How to choose what to do first

Launching this week: robots.txt, sitemap, indexing, and the launch-day SEO checklist. Google says AI features run on normal Search eligibility, so this comes first.
Have docs: add an llms.txt and Markdown versions of key pages for agents.
Care about training use: block GPTBot, ClaudeBot and Google-Extended, and leave the search bots allowed.
Have real reviews: add SoftwareApplication with ratings; otherwise keep the markup minimal and accurate.

Checklist

Step	Done when
Indexed in Google and Bing	Sitemap submitted, key pages show as indexed
Search crawlers allowed	`OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot` not disallowed
CDN not blocking bots	Bot protection does not challenge the crawlers you allow
Training choice made	`GPTBot` / `ClaudeBot` / `Google-Extended` set on purpose
Answer pages in text	"What is", "vs", install and pricing pages answer in the first paragraph
Structured data	JSON-LD matches visible text and passes the Rich Results Test
llms.txt (optional)	H1, summary and links to your main docs pages
README and listings	Same name, description and price as the site

FAQ

Do I need llms.txt to appear in AI Overviews?

No. Google says you don't need AI text files or special markup for AI Overviews or AI Mode; a page has to be indexed and eligible for a snippet.

Does blocking GPTBot remove me from ChatGPT search?

Not according to OpenAI. GPTBot is the training crawler; ChatGPT search uses OAI-SearchBot, and the two settings are independent.

Does Google-Extended affect Google Search?

No. Google says it does not affect inclusion in Search and is not a ranking signal. It only covers Gemini training and grounding in Gemini Apps and Vertex AI.

Can I stop user-triggered fetches with robots.txt?

Not reliably. OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it, because a person asked for the page.

Launching soon? Start with the launch-day SEO checklist, build your robots.txt and JSON-LD in the toolbox, and pick your launch sites from where to launch your side project.