llms.txt Explained — The New Standard for AI Bot Consent (with Generator)

What is llms.txt?

llms.txt is a Markdown-formatted file in the root directory of your domain that machine-readably tells Large Language Model crawlers (GPTBot, PerplexityBot, ClaudeBot, Google-Extended, among others) which content on your site is particularly relevant, in what order it should be prioritized, and how it is structured. The proposal comes from Jeremy Howard (Answer.AI, fast.ai) from September 2024 and has established itself as a de facto standard within a year — as of May 2026, 4 of the 5 major LLM providers officially or informally support the standard.

In practice, an llms.txt looks like this:

# My Domain

> Short description of the domain in one sentence.

## Docs

- [Getting Started](https://example.com/docs/start.md): Get started in 5 minutes
- [API Reference](https://example.com/docs/api.md): Complete API documentation

## Optional

- [Changelog](https://example.com/changelog.md)

The structure is intentionally minimal: H1 with site name, optional blockquote as description, then H2 sections with links and 1-sentence descriptions. An optional section at the end for nice-to-have content that LLMs may skip if necessary.

Difference from robots.txt

The difference is often misunderstood, so let's be explicit:

Aspect	robots.txt	llms.txt
Purpose	Control crawl access (allow/disallow)	Prioritize important content
Recipient	All crawlers (Search + AI)	Specifically LLM crawlers
Format	Plain-text directives	Markdown
Standard since	1994 (RFC 9309 since 2022)	September 2024 (de facto)
Mandatory?	No, but best practice	No, but increasingly mandatory
Position	Root: /robots.txt	Root: /llms.txt

My take: robots.txt is the bouncer ("who can come in?"), llms.txt is the concierge ("where should I start?"). Both are useful, both have their own logic. Those who only have one are missing out on visibility.

Robots.txt tells LLMs whether they can read your site. Llms.txt tells them what to read first.

Syntax + Directives

The llms.txt syntax is intentionally kept small — that's a feature, not a bug. Six elements are enough:

H1 (Mandatory)

The name of your domain or product. Exactly one H1 per file.

# SEOlyze

Blockquote (recommended)

A 1-2 sentence description. LLMs use this as context for citation decisions.

> SaaS platform for SEO + GEO. WDF/IDF, Rank Tracking, AI Overview Monitoring.

H2 Sections

Any number. Convention: Docs, Examples, Optional. Freely selectable.

List Items with Links

One Markdown list per H2. Each item: - [Title](URL): Optional 1-sentence description.

Optional Section at the End

An H2 named "Optional" — content here may be skipped by crawlers when token limits are reached. Useful for changelogs, blog archives, FAQ collections.

Companion Files: .md Variants

Best practice (not mandatory): For each page linked in llms.txt, provide a .md variant that is clean (no sidebar, no cookie banner, no ads). URL convention: example.com/docs/intro → example.com/docs/intro.md. This gives LLMs significantly better parsing results.

Live Generator

Here is a simple generator. Enter your data, click "Generate" — you will get a valid llms.txt to copy into your root directory.

The generator gives you a valid llms.txt. For the bot allow directives, you must also adapt your robots.txt — this does not happen in the llms.txt itself (it has no bot control syntax). Example for robots.txt:

User-agent: GPTBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Best Practices

1. Curate, don't duplicate

Maximum 30 URLs in llms.txt. If you have a sitemap with 5000 posts, pick the 20 most valuable ones. LLM crawlers have token limits — the longer your llms.txt, the higher the chance that it will be cut off at the end.

2. Order by Importance

The first 10 links are the most important. Pillar content, definition pages, main categories first. Tools/calculators in the middle. Blog posts and changelog in "Optional".

3. Provide Markdown Versions

For your top 10 URLs, you should build .md endpoints that render cleanly. This requires setup effort, but LLMs consume Markdown significantly better than HTML with a sidebar/cookie banner/ads.

4. Keep it Current

Include a last-updated date as a comment, refresh every 3–6 months. Dead URLs in llms.txt are a negative signal — you appear to be a poorly maintained source.

5. Validate

There are initial open-source validators (e.g., llmstxt.org). Have your file validated before going live. Most common errors: multiple H1s, missing spaces after -, incorrect URL encoding for umlauts.

The last number is interesting: Less than 1% of the top 10k domains have a valid llms.txt — that's huge whitespace. Those who set it up now will have a competitive advantage that will disappear in 12–18 months.

How to check if your llms.txt is working

Three ways that work in practice:

Check 1: HTTP Test

Trivial requirement: curl https://your-domain.de/llms.txt must return status 200 and valid Markdown. Sounds self-explanatory, but is often the bug: CDN caching sends 404 because the file is not in the CDN path.

Check 2: LLM Direct Test

Ask ChatGPT, Perplexity, and Claude directly: "What is in the llms.txt of example.com?". If the model doesn't find it, it's irrelevant for citations. If it correctly summarizes the content, it's active.

Check 3: Bot Logs

Filter your server logs for user agents like GPTBot, PerplexityBot, ClaudeBot, Google-Extended. If these bots don't retrieve /llms.txt, you haven't reached them. Pattern: grep -i "GPTBot\|PerplexityBot" access.log | grep "llms.txt".

Further reading: Optimize AI Overview — the complete guide for the layer above llms.txt, and GEO Pillar Guide for the overall context.