llms.txt: The File Google Is Already Auditing (But Won't Tell You About)

What is llms.txt?

llms.txt is a Markdown file you place at your website's root — yoursite.com/llms.txt — that tells AI systems what your site is about and where to find your most important content.

It was created by Jeremy Howard (founder of Answer.AI and fast.ai) in September 2024. The specification lives at llmstxt.org and has accumulated over 2,400 stars on GitHub.

Think of it this way: robots.txt tells crawlers what not to access. sitemap.xml listseverything you have. llms.txt curates what matters most — specifically for AI agents that need to understand your site quickly within a limited context window.

The format is intentionally Markdown (not XML or JSON) because that's what LLMs understand best. It includes a title, summary, and categorized links to your key pages — with descriptions that help an AI decide which pages are relevant to a given query.

The Google contradiction

Here's where it gets interesting. Two teams within Google have taken opposite stances:

Both statements make sense if you understand the distinction: Google Search ranks web pages for humans. Lighthouse evaluates whether your site is ready for the agentic web — a world where AI agents (not just search crawlers) visit your site to research, compare, and act on behalf of users.

The Google Search team is correct that llms.txt won't help your traditional SEO rankings or AI Overviews. The Lighthouse team is correct that AI agents are an emerging audience your site should serve. Both things are true simultaneously.

Who is actually using llms.txt?

As of May 2026, 849+ sites are tracked in the llmstxt.cloud directory. A broader study found 5.86% of the top 10,000 domains now have a valid llms.txt file. That's not universal adoption, but it's meaningful — and the list reads like a who's-who of developer infrastructure:

The pattern is clear: companies building for developers and APIs adopted first. This makes sense — their users are the ones most likely to be working with AI coding assistants that consume llms.txt files to understand documentation.

Platform-level support is growing too. Mintlify and Fern (documentation platforms) auto-generate llms.txt for all their customers. WordPress plugins including Rank Math, Yoast, and Squirrly have all added llms.txt generation. VitePress and Docusaurus have community plugins available.

Does it actually work? The honest answer.

Here's where most articles lose credibility — they either hype llms.txt as a magic SEO bullet or dismiss it entirely. The truth is more nuanced, and it depends on what you mean by "work."

Where it does NOT help (yet)

• Google AI Overviews: No evidence that having llms.txt influences whether your content appears in AI-generated search results.
• ChatGPT Search / Perplexity Search: No AI platform has officially announced native consumption of llms.txt files during web search. These systems rely on standard HTML crawling.
• Traditional SEO rankings: llms.txt has zero impact on how Google ranks your pages for human searchers.

Where it DOES help

• AI coding assistants: Claude Code, Cursor, Windsurf, and similar tools actively fetch llms.txt when developers point them at documentation. This is the primary real-world consumption today.
• Custom AI agents and RAG systems: Any company building agentic workflows can use llms.txt as an authoritative source map for what content to ingest from your site.
• Developer experience: When someone uses an AI tool to work with your API or product, llms.txt dramatically improves the quality of AI-generated code and answers about your product.
• Lighthouse compliance: If your team tracks Lighthouse scores, the Agentic Browsing category now factors in llms.txt presence.

The creator, Jeremy Howard, has been clear about this: llms.txt is designed for inference-time context assembly — when a user's AI agent needs to understand your site right now. It was never intended for training crawlers or search indexing.

The agentic web is the real story

llms.txt is part of a larger shift. Lighthouse's Agentic Browsing category doesn't just audit for llms.txt — it checks for nine things in total, including:

• llms.txt — your site's AI-readable index
• agents.json — declares what actions AI agents can take on your site
• agent runbooks — step-by-step instructions for agents performing tasks
• Auto-discovery links — so agents can find related resources without human guidance

This is the future Google is building toward: a web where AI agents aren't just reading your content — they're navigating it, comparing options, and taking actions on behalf of users. Your site either participates in that future or gets skipped.

llms.txt is the simplest entry point. It takes five minutes to create, costs nothing to maintain, and positions your site for a shift that's already underway.

What a well-structured llms.txt looks like

The format is simple Markdown with a specific structure. Here's a real-world example pattern based on the official spec:

# Your Company Name

> One-paragraph summary of what your company does,
> who it's for, and what makes it different.

Additional context that helps an AI understand your
positioning — what problem you solve, for whom.

## Docs

- [Getting Started](https://yoursite.com/docs/start.md): Quick-start guide for new users
- [API Reference](https://yoursite.com/docs/api.md): Full API documentation
- [Authentication](https://yoursite.com/docs/auth.md): How auth works

## Product

- [Features](https://yoursite.com/features.md): What the product does
- [Pricing](https://yoursite.com/pricing.md): Plans and pricing
- [Changelog](https://yoursite.com/changelog.md): Recent updates

## Optional

- [Blog](https://yoursite.com/blog.md): Company blog posts
- [Case Studies](https://yoursite.com/cases.md): Customer stories

Key points about the structure:

• The H1 (# Title) is the only required element
• The blockquote (>) provides essential context an AI needs to understand everything that follows
• H2 sections group links by category — each link includes a description
• The ## Optional section has special meaning — AI agents can skip it when context windows are tight
• Links should point to .md versions of pages when available (clean markdown, not HTML)

Some sites also maintain tiered versions — llms-full.txt with expanded content and llms-small.txt for quick reference. Svelte's documentation uses this approach effectively.

How to implement llms.txt

You have two options:

Option 1: Generate it automatically

If you want a spec-compliant llms.txt generated from your existing site structure, you can use our free llms.txt generator. Paste your URL, and it crawls your site to produce a ready-to-deploy file in about 30 seconds. No signup required.

Option 2: Write it manually

For most sites, manually writing llms.txt produces a better result because you can curate exactly what matters. Follow the structure above, then:

1. 1. Save the file as llms.txt in your site's public root directory
2. 2. Verify it's accessible at yoursite.com/llms.txt
3. 3. Optionally, create Markdown versions of your key pages (e.g., /docs/api.html.md)
4. 4. Run a Lighthouse audit to confirm it passes the Agentic Browsing checks

If you're on a docs platform

Mintlify and Fern generate llms.txt automatically. For Docusaurus, VitePress, and WordPress (via Rank Math or Yoast), plugins are available that generate the file from your existing content structure.

Who should care about this?

Not everyone needs llms.txt today. Here's a realistic framework:

The bottom line

llms.txt won't magically boost your traffic tomorrow. Anyone telling you otherwise is selling something. What it does is position your site for an emerging audience — AI agents that research, compare, and recommend on behalf of real users.

The adoption numbers tell the story: 849+ tracked sites, 5.86% of top domains, Cloudflare and Anthropic and Stripe leading the way. Google's Lighthouse now audits for it. The infrastructure is being laid.

It takes five minutes to create. It costs nothing to maintain. And unlike most "SEO tactics," this one is backed by a real specification, real tooling, and real companies building for the same future.