# llms.txt and AI Crawlers: What to Allow in robots.txt (2026 Guide)

What llms.txt is, whether it helps AI visibility, and how to configure robots.txt for GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot and Google-Extended, with a crawler table and test steps.

_Source: https://www.gtm8020.com/blog/llms-txt-ai-crawlers-robots-txt_

- **Author:** [Jimmy Pal](https://www.gtm8020.com/experts/jimmy-pal), Scaling Organic Growth For Ambitious Brands
- **Published:** 2026-10-24

llms.txt is a proposed Markdown file, placed at your site root, that gives AI agents a short curated map of your content. It is not a ranking factor, and Google says you do not need AI text files to appear in AI Overviews or AI Mode. The control that matters is robots.txt: allow the AI search and user bots, then decide separately about the bots that collect training data.

This guide explains what llms.txt is, what it can and cannot do, which AI crawlers exist, and how to set up robots.txt so your B2B site stays visible in AI answers without handing over more than you intend.

## Key takeaways

-   llms.txt was proposed by Jeremy Howard in September 2024. Its only required element is an H1 with the site name, and agents follow the links inside it to find clean content.
-   Google states there are no additional requirements, new machine readable files or AI text files needed to appear in AI Overviews or AI Mode.
-   OpenAI, Anthropic and Perplexity each run separate bots for search, user-initiated fetches and training, so one robots.txt rule rarely covers all three jobs.
-   Blocking a training bot does not remove you from AI search. OpenAI says a site can allow OAI-SearchBot and disallow GPTBot at the same time.
-   Check your CDN and firewall too. A robots.txt that allows a bot does nothing if your host blocks it.

## What is llms.txt?

llms.txt is a Markdown file at the root of a website, such as example.com/llms.txt, that offers brief background information, guidance and links to detailed Markdown files. It was proposed by Jeremy Howard on September 3, 2024, so agents can read a small curated overview instead of parsing a whole website.

According to the llmstxt.org specification, a valid file contains, in order, an H1 with the name of the project or site (the only required section), a blockquote summary, optional detail sections, and H2 sections holding lists of links with optional notes. A section named "Optional" marks secondary links an agent can skip. The proposal also suggests publishing a clean Markdown version of each important page at the same URL with `.md` appended.

```
# Your Company

> One-sentence summary of what you do and who you serve.

## Key pages

- [Services](https://example.com/services.md): What we offer and pricing approach
- [Case studies](https://example.com/case-studies.md): Results by customer type

## Optional

- [Blog index](https://example.com/blog.md)
```

For a live example, GTM 8020's own file at gtm8020.com/llms.txt opens with an H1, a one-paragraph summary, and a note that every page is available as Markdown by appending `.md` to the URL.

## Does llms.txt help you rank or get cited by AI?

There is no confirmed evidence that llms.txt raises rankings or citations. Google's Search Central guidance says you do not need new machine readable files, AI text files or special markup to appear in AI Overviews or AI Mode. Treat llms.txt as low-cost housekeeping for agents, not a visibility lever.

The llmstxt.org page says the file is used most heavily for software documentation, where coding agents follow it to find API references, and that thousands of sites now publish one. It also notes that Chrome's Lighthouse checks for one as part of its agentic browsing audits. Those are adoption facts, not proof of an effect on citations.

Our advice is practical. If you publish documentation or product pages that agents may fetch on a buyer's behalf, an llms.txt takes an hour and does no harm. If you have not fixed crawl access, page speed, and answer-first content, do that first. Content structure drives citations, and the [generative engine optimization statistics](/blog/generative-engine-optimization-statistics) show which tactics have measured effects.

## Which AI crawlers should you allow in robots.txt?

Allow the bots that power AI search and user questions: OAI-SearchBot, Claude-SearchBot, PerplexityBot and the user-initiated agents. Make a separate business decision on training bots such as GPTBot and ClaudeBot. Googlebot remains the control for Google Search, including AI Overviews and AI Mode.

| User agent | Operator | What it does (per the operator) | If you block it |
| --- | --- | --- | --- |
| OAI-SearchBot | OpenAI | Surfaces websites in ChatGPT's search features | Site is not shown in ChatGPT search answers, though it can still appear as a navigational link |
| GPTBot | OpenAI | Crawls content that may be used to train generative AI models | Signals content should not be used for training |
| ChatGPT-User | OpenAI | Visits pages for certain user actions in ChatGPT and Custom GPTs | OpenAI says robots.txt rules may not apply; it is not used to decide Search inclusion |
| Claude-SearchBot | Anthropic | Improves the quality of search results | Content is not indexed for search; visibility and accuracy may drop |
| Claude-User | Anthropic | Fetches pages when users ask Claude questions | Claude cannot retrieve your content for user queries |
| ClaudeBot | Anthropic | Collects content that could contribute to model training | Future content is excluded from training datasets |
| PerplexityBot | Perplexity | Surfaces and links sites in Perplexity results; not used for foundation model training | Site may not appear in Perplexity search results |
| Perplexity-User | Perplexity | Visits pages when users ask questions | Perplexity says this fetcher generally ignores robots.txt |
| Google-Extended | Google | Token that controls use of crawled content for training Gemini models and for grounding in Gemini Apps | Does not affect Google Search inclusion or ranking |

## How do you write robots.txt for AI search and still opt out of training?

Add one group per bot. Allow the search and user agents, and add a Disallow rule for any training agent you want to exclude. Each rule works independently, and OpenAI and Perplexity both say changes can take around 24 hours to take effect. Repeat the rules for every subdomain.

```
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /
```

This example is a choice, not a recommendation for everyone. A B2B company that wants maximum reach may allow every bot, and that is the approach GTM 8020 takes today: its robots.txt explicitly allows, among others, GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended. A publisher with licensed content may block training bots and keep the search bots.

Anthropic warns that blocking by IP address may not reliably work, because it can stop its bots from reading robots.txt in the first place. Use robots.txt rules for opt-outs, not IP blocks.

## How do you check that AI bots can reach your site?

Confirm access at three layers: robots.txt rules, your CDN or firewall, and your server logs. A page can be allowed in robots.txt and still be blocked by a bot-protection rule. Google lists this explicitly, advising that crawling is allowed in robots.txt and by any CDN or hosting infrastructure.

1.  Open yoursite.com/robots.txt and confirm each bot you want has an Allow rule or no matching Disallow.
2.  Review firewall or WAF rules. Perplexity, for example, advises combining user agent matching with its published IP ranges when allowing its bots.
3.  Search server logs for the user agent names in the table above. Google warns that user agent strings can be spoofed, so verify against published IP lists where operators provide them.
4.  Check that important pages return a 200 status and carry no noindex tag. OpenAI notes a crawler must be able to read a noindex tag to honor it.
5.  Track the visits that follow. Our guide to [tracking ChatGPT and AI assistant traffic in GA4](/blog/track-chatgpt-ai-traffic-ga4) shows how to measure what arrives.

Technical access is the base layer. The [technical SEO statistics](/blog/technical-seo-statistics) show how much crawl and indexing problems cost, and if you want a team to audit it, our [SEO and GEO service](/services/seo-geo) covers it. For tools that monitor your AI visibility afterward, see our [comparison of AEO tools](/blog/best-aeo-tools).

## Frequently asked questions

### Do I need an llms.txt file to appear in ChatGPT or Google AI Overviews?

No. Google says you do not need new machine readable files, AI text files or special markup to appear in AI Overviews or AI Mode. OpenAI's guidance for ChatGPT search centers on not blocking OAI-SearchBot. An llms.txt can be a helpful extra, but neither company requires it.

### What is the difference between GPTBot and OAI-SearchBot?

OAI-SearchBot surfaces websites in ChatGPT's search features. GPTBot crawls content that may be used to train OpenAI's generative models. OpenAI says the settings are independent, so a site can allow OAI-SearchBot to appear in search results while disallowing GPTBot to opt out of training.

### Does blocking Google-Extended remove me from Google Search?

No. Google states that Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal. It controls whether crawled content may be used to train future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI.

### Does ChatGPT-User follow robots.txt?

OpenAI says ChatGPT-User handles actions initiated by users, so robots.txt rules may not apply. It is not used to crawl the web automatically, and it is not used to decide whether content appears in search. For search opt-outs, OpenAI points site owners to OAI-SearchBot.

### How long do robots.txt changes take to affect AI search?

OpenAI says it can take about 24 hours from a robots.txt update for its systems to adjust, and Perplexity gives the same window. Google's timing depends on recrawling, which can take days to months. Plan on at least a day before you test a change.

## Sources

-   llmstxt.org, [The /llms.txt file, v2](https://llmstxt.org/) (Jeremy Howard, published September 3, 2024)
-   Google Search Central, [AI features and your website](https://developers.google.com/search/docs/appearance/ai-features)
-   Google Search Central, [Google's common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)
-   OpenAI, [Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots)
-   OpenAI Help Center, [Publishers and Developers FAQ](https://help.openai.com/en/articles/12627856-publishers-and-developers-faq)
-   Claude Help Center, [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
-   Perplexity, [Perplexity crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)

---
_GTM 8020 — https://www.gtm8020.com. This is a Markdown rendering of https://www.gtm8020.com/blog/llms-txt-ai-crawlers-robots-txt for AI and agent readers._