How to Write an llms.txt File

If you've spent any time thinking about how large language models discover, index, and cite web content, you've probably noticed a gap. Search engines have robots.txt and sitemaps. Social platforms have Open Graph tags. But until recently, there was no standard way to tell an AI system: "here's what my site is actually about, and here are the pages worth reading."

That gap is what llms.txt is designed to close.

Proposed by Jeremy Howard - co-founder of fast.ai and a figure whose work sits squarely at the intersection of practical machine learning and the open web - llms.txt is a simple, markdown-formatted file that lives at the root of your domain and gives AI crawlers a curated, machine-readable map of your most important content. Think of it as a handshake between your site and the models that will inevitably read it, summarise it, and potentially cite it in response to someone's question.

This guide covers everything you need to know: the format, the rationale, what to include, how to deploy it, and how to validate it works.

What Is llms.txt and Why Does It Matter?
How AI Crawlers Use llms.txt
The llms.txt Format Explained
llms-full.txt: The Extended Version
What to Include (and What to Leave Out)
Writing the Site Description Section
Listing Your Key Pages Effectively
Brand Disambiguation: When and How
Keeping Your llms.txt Updated
Deploying llms.txt: Step-by-Step
Testing and Validating Your llms.txt
A Complete Real-World Example
Frequently Asked Questions

What Is llms.txt and Why Does It Matter?

llms.txt is a plain-text file, written in Markdown, served at https://yourdomain.com/llms.txt. It tells AI language models - during training, retrieval-augmented generation (RAG), or live web browsing - which pages on your site matter most, what your site is actually about, and any context a model needs to represent you accurately.

The convention was proposed by Jeremy Howard in September 2024, drawing on the established pattern of robots.txt (which tells crawlers what not to read) and expanding it into something more proactive: a file that tells AI systems what to read and how to understand it.

Why does this matter? Three reasons.

AI systems don't read everything. When a model is pulling context from a website - whether through a training crawl, a retrieval pipeline, or a browsing tool - it faces a signal-to-noise problem. Most pages on most sites contain navigation chrome, cookie banners, footer links, and boilerplate. The actual content is buried. llms.txt cuts through that by surfacing your best, most semantically dense pages in a single, clean file.

Misrepresentation is a real risk. If a model's only exposure to your brand is a press release from three years ago and a cluttered homepage, it will form an incomplete - possibly inaccurate - picture of what you do. llms.txt lets you shape that picture deliberately.

Generative Engine Optimisation (GEO) is replacing SEO as a discipline. As AI answer engines (Perplexity, ChatGPT search, Gemini, Claude) increasingly provide direct answers rather than blue links, being cited by a model matters as much as ranking on a results page. llms.txt is one of the few levers site owners can pull to influence that.

How AI Crawlers Use llms.txt

Different systems use llms.txt in different ways, and it helps to understand the range.

Training crawlers - the ones that ingest the web to build model weights - may use llms.txt as a priority signal. If your most important content is explicitly listed, a well-configured crawler can weight those pages more heavily, fetch the full-text versions, and skip the noise.

RAG pipelines - systems that retrieve live content to augment a model's responses - are probably the most immediately practical consumers. A RAG system that respects llms.txt will know exactly which URLs to index for your domain, saving it from having to crawl and parse your entire site hierarchy.

Browsing tools - like the web-search functionality built into Claude, ChatGPT, and Perplexity - may check for llms.txt when a user asks about a specific company or domain. The file helps the tool quickly understand the site's structure without fetching dozens of pages.

Developers building agents - anyone building an AI tool that needs to know about your product, API, or documentation can point their agent at your llms.txt as the canonical starting point.

Not every AI system honours the file today - adoption is still growing - but the convention is gaining traction quickly, and the cost of creating the file is low enough that there's no good reason to wait.

The llms.txt Format Explained

The format is intentionally simple. It's valid Markdown. A model can read it as plain text; a human can read it without tooling. The structure has three main parts:

A top-level heading - your site or brand name.
A blockquote - a short description of what your site does.
Sections with link lists - each section groups related pages under an ## heading, with each page as a Markdown link followed by an optional inline description.

Here's a minimal but correct example:

# Acme Analytics

> Acme Analytics is a web and product analytics platform for growth teams. We help companies understand user behaviour, run A/B tests, and ship features with confidence.

## Documentation

- [Getting Started](https://acme.io/docs/getting-started): Installation, first event, and your first dashboard in under ten minutes.
- [JavaScript SDK](https://acme.io/docs/sdk/javascript): Full reference for the browser SDK, including autocapture and custom events.
- [REST API](https://acme.io/docs/api): API reference for server-side event ingestion and data export.

## Key Product Pages

- [Features Overview](https://acme.io/features): What Acme Analytics does and how it compares to alternatives.
- [Pricing](https://acme.io/pricing): Plans, usage limits, and enterprise options.
- [Changelog](https://acme.io/changelog): Recent product updates.

## Optional

- [Blog](https://acme.io/blog): Long-form articles on analytics, growth, and experimentation.
- [Case Studies](https://acme.io/customers): How teams like Stripe, Notion, and Linear use Acme.

A few things to note about the format:

The blockquote (>) is the canonical description. Keep it to two or three sentences. This is often the text a model will use when it needs to describe your site in a single paragraph.
Section headings are ## - second level only. Don't use ### for sub-groupings inside sections; keep the hierarchy flat.
Link descriptions (after the colon) are optional but valuable. They give the model context about why a page matters, not just that it exists.
One section - ## Optional - is conventionally used for content that's useful but lower priority. A model with limited context budget can skip ## Optional sections if it needs to.

llms-full.txt: The Extended Version

Alongside llms.txt, the convention supports a companion file: llms-full.txt. Where llms.txt is a map, llms-full.txt is the territory - it contains the actual full text of your most important pages, concatenated into a single file.

This is particularly useful for:

Offline or cached RAG pipelines that want to index your content without making dozens of HTTP requests.
Fine-tuning and evaluation datasets where someone wants clean, authoritative text from your site.
Agents with large context windows that can ingest your entire documentation in a single pass.

llms-full.txt follows the same structural conventions as llms.txt but includes the full Markdown content of each linked page beneath its link entry. Because it can be large - potentially megabytes for a documentation-heavy site - it's served separately, and systems that don't need it simply don't fetch it.

If your site has substantial documentation, shipping both files is best practice.

What to Include (and What to Leave Out)

The temptation is to list every page. Resist it. The whole point of llms.txt is curation - you're helping a model prioritise, not giving it a full sitemap.

Include:

Your core product or service pages - the pages that best explain what you do.
Documentation, API references, and guides - especially if your audience includes developers.
Your pricing page - models are frequently asked "how much does X cost?"
Your changelog or release notes - useful for recency signals.
High-quality long-form content that represents your expertise (key blog posts, whitepapers, case studies).
Your "about" page, if it contains substantive information about your company, team, or mission.

Leave out:

Navigation pages, tag pages, category archives - these are structural, not semantic.
Thin pages: login screens, 404 pages, cookie policy, T&Cs.
Duplicate or near-duplicate content.
Pages behind authentication - a model can't read them anyway.
Internal tools, admin panels, staging URLs.
Anything you'd prefer not to have indexed and cited publicly.

A good llms.txt for most SaaS companies will have between fifteen and forty links, grouped into four to six sections. If you're listing more than sixty URLs, you're almost certainly including noise.

Writing the Site Description Section

The blockquote description is the most important single piece of text in your llms.txt. It's the model's anchor - when it needs to describe your brand in a sentence or cite you in passing, this is often the text it will draw from.

Write it as you would write the opening sentence of a Wikipedia article about your company: factual, specific, and free of marketing language.

Weak:

> We're the leading AI-powered platform for modern teams who want to move faster and smarter.

Strong:

> Acme Analytics is a product analytics platform that helps software teams track user behaviour, run multivariate A/B tests, and measure feature adoption. It supports JavaScript, Python, and server-side SDKs and is used by over 3,000 companies.

The strong version tells a model: what category the product is in, what it specifically does, how developers interact with it, and its scale. Every clause carries information.

Some guidelines:

Name the product explicitly - don't rely on the # heading alone.
State the category clearly (analytics platform, API gateway, payroll software, etc.).
Include one or two specific, verifiable facts (number of customers, SDK languages, integrations).
Avoid superlatives ("best", "leading", "most powerful") - models learn to discount them.
Keep it to two to four sentences. Much longer and it starts to feel like marketing copy, which degrades the signal.

Listing Your Key Pages Effectively

Each link entry should give a model enough context to decide whether to fetch that page, and to understand what it will find there.

The format is:

- [Page Title](https://example.com/page): One sentence describing what this page contains and why it's useful.

The description after the colon is doing real work. It should answer the question: "If a model were trying to answer a question about [your topic], why would it fetch this page?"

Compare:

- [JavaScript SDK](https://acme.io/docs/sdk/javascript)

versus:

- [JavaScript SDK](https://acme.io/docs/sdk/javascript): Full API reference for browser-side event tracking, including autocapture configuration, identity management, and custom properties.

The second version tells the model what kind of content is there, what specific topics it covers, and implicitly which queries it's useful for.

A few practical tips:

Use the actual <title> of the page where possible - it creates consistency between the llms.txt listing and what the model will see if it fetches the page.
Mention specific proper nouns, features, or technologies in the descriptions - these act as retrieval anchors.
Don't pad descriptions to make them look more impressive. Concise and accurate beats verbose and vague.

Brand Disambiguation: When and How

If your brand name is shared with other things - a common word, another company, a person, a place - your llms.txt is a good place to address that directly.

You can add a disambiguation note in your blockquote or as a dedicated paragraph before your link sections:

# Mercury

> Mercury is a banking platform for startups and small businesses, offering business bank accounts, corporate cards, and financial tools. Mercury is not affiliated with the planet, the automobile brand, or the Roman god. The company was founded in 2019 and is headquartered in San Francisco.

## Key Pages

- [How Mercury Works](https://mercury.com/how-it-works): Overview of the Mercury bank account, debit and credit cards, and treasury products.

The "not affiliated with" construction looks slightly awkward in human-facing copy, but it's actually quite effective for AI consumption - it explicitly cancels out the most common alternative interpretations a model might apply.

Disambiguation is most important when:

Your brand name is a common English word (Linear, Notion, Mercury, Arc, Reflect).
There's a much more famous entity with the same name.
You've been confused with a competitor in AI-generated summaries.
Your product name is also the name of a well-known technology (e.g. a company called "Spark" operating outside the Apache ecosystem).

Keeping Your llms.txt Updated

A stale llms.txt is worse than a partial one. If you list a page that's been deleted, a feature that's been deprecated, or pricing that's out of date, you're feeding incorrect information directly to the systems that will cite you.

Build llms.txt maintenance into your existing content workflows:

When you publish a major new page (new product launch, new documentation section, new case study) - add it to llms.txt at the same time.
When you deprecate or delete a page - remove it from llms.txt immediately.
When your pricing changes - update the description on the pricing link to note the change (or at least ensure the page itself is current).
Quarterly audit - once every three months, walk through every link in your llms.txt and verify it still resolves and still contains what the description says it does.

If your site is generated from a CMS or static site generator, consider automating llms.txt generation. Many documentation tools (Docusaurus, Mintlify, GitBook) are already adding llms.txt support, which means your documentation section at least can be generated automatically.

For the product and marketing sections, manual curation tends to produce better results - automation tends to include too much.

Deploying llms.txt: Step-by-Step

Next.js

Place the file in your public/ directory:

public/
  llms.txt
  llms-full.txt

Next.js serves everything in public/ at the root path, so https://yourdomain.com/llms.txt will resolve automatically. No additional configuration needed.
If you want to generate the file dynamically (for instance, pulling links from a CMS), create a route handler instead:

// app/llms.txt/route.ts
import { NextResponse } from 'next/server'

export async function GET() {
  const content = await generateLlmsTxt() // your generation logic
  return new NextResponse(content, {
    headers: {
      'Content-Type': 'text/plain; charset=utf-8',
    },
  })
}

WordPress

Upload llms.txt to the root of your WordPress install (the same directory as wp-config.php). WordPress doesn't intercept requests for files that actually exist on the filesystem, so this works without any plugin or .htaccess modification.
Alternatively, use a plugin like "Custom File Endpoint" to serve the file programmatically, which is useful if you want to generate the content from your post database.

Static Sites (Hugo, Eleventy, Jekyll)

Place llms.txt in your static assets directory - the folder that gets copied verbatim to your build output:

Generator	Location
Hugo	`static/llms.txt`
Eleventy	Add `llms.txt` to your `passthrough` copy configuration, or place it in your input directory with a `permalink` of `/llms.txt`
Jekyll	Place it in the root of your project; Jekyll copies non-underscored files to `_site/`

Nginx

If you're serving a static site or proxying to an app, you can serve llms.txt directly from Nginx:

server {
    listen 443 ssl;
    server_name yourdomain.com;

    # Serve llms.txt directly
    location = /llms.txt {
        root /var/www/static;
        default_type text/plain;
        add_header Content-Type "text/plain; charset=utf-8";
    }

    # Serve llms-full.txt directly
    location = /llms-full.txt {
        root /var/www/static;
        default_type text/plain;
        add_header Content-Type "text/plain; charset=utf-8";
    }

    # Rest of your config...
}

Place the files in /var/www/static/ and reload Nginx.

Content-Type note: Serve llms.txt as text/plain rather than text/markdown. The Markdown formatting is a convention within a plain-text file - you don't need a specialised MIME type, and text/plain is universally handled.

Testing and Validating Your llms.txt

Once deployed, verify the file before assuming it works.

Basic accessibility check:

curl -I https://yourdomain.com/llms.txt

You're looking for HTTP/2 200 and content-type: text/plain. A 404 means the file isn't being served from the right path.

Content check:

curl https://yourdomain.com/llms.txt

Eyeball the output. Does it look like valid Markdown? Is the blockquote present? Are the links correctly formatted?

Link validation:

Use a tool like linkchecker or a simple script to verify every URL in your llms.txt returns a 200:

# Extract all URLs from llms.txt and check them
grep -oP 'https?://[^)]+' llms.txt | while read url; do
  status=$(curl -o /dev/null -s -w "%{http_code}" "$url")
  echo "$status $url"
done

Any 404 or 301 in this output is a link that needs fixing.

Validator tools:

The llms.txt community has produced several online validators - search for "llms.txt validator" to find the current options. These parse your file and flag formatting issues, broken links, and deviations from the spec.

Manual AI test:

The most direct test is to ask an AI assistant that supports web browsing to look up your site. Ask it: "What does [your company] do?" and see whether the response reflects the description in your llms.txt, or whether it's pulling from older, less accurate sources. This isn't a controlled test, but it gives you a qualitative sense of how models currently represent you.

A Complete Real-World Example

Here's a realistic llms.txt for a fictional SaaS company - Harbour, a document workflow platform:

# Harbour

> Harbour is a document workflow platform for legal, finance, and operations teams. It provides contract drafting, e-signature, approval routing, and a searchable document repository - replacing fragmented email and PDF workflows with a single, auditable system. Harbour is not affiliated with Harbour.io (a container registry tool) or Harbour Software (a legacy ERP vendor).

## Product

- [How Harbour Works](https://harbour.app/how-it-works): End-to-end overview of the Harbour document lifecycle, from template to signed archive.
- [Features](https://harbour.app/features): Full feature list including conditional logic in templates, role-based approval chains, audit trails, and third-party integrations.
- [Pricing](https://harbour.app/pricing): Starter, Growth, and Enterprise plans with per-seat pricing and usage limits. Updated June 2025.
- [Security & Compliance](https://harbour.app/security): SOC 2 Type II certification, data residency options, SSO support, and encryption details.
- [Integrations](https://harbour.app/integrations): Native connectors for Salesforce, HubSpot, Notion, Slack, and 40+ other tools via Zapier and webhooks.

## Documentation

- [Getting Started Guide](https://harbour.app/docs/getting-started): Create your first template, invite a signatory, and collect a signature in under fifteen minutes.
- [Template Language Reference](https://harbour.app/docs/templates): Full reference for Harbour's template syntax, including variables, conditional blocks, and repeating sections.
- [API Reference](https://harbour.app/docs/api): REST API for programmatic document creation, sending, and status webhooks. Includes authentication, rate limits, and code samples in Python, Node, and Ruby.
- [Approval Workflows](https://harbour.app/docs/workflows): How to configure sequential and parallel approval chains, escalation rules, and deadline reminders.
- [Audit Trail & Compliance](https://harbour.app/docs/audit): How Harbour logs every action on a document, what the audit export contains, and how to use it for compliance reporting.
- [SSO & SCIM Setup](https://harbour.app/docs/sso): Configuring SAML 2.0 SSO with Okta, Azure AD, and Google Workspace, plus SCIM provisioning.

## Comparison & Alternatives

- [Harbour vs DocuSign](https://harbour.app/vs/docusign): Feature and pricing comparison between Harbour and DocuSign for mid-market teams.
- [Harbour vs PandaDoc](https://harbour.app/vs/pandadoc): How Harbour differs from PandaDoc on workflow complexity and API flexibility.

## Company

- [About Harbour](https://harbour.app/about): Company background, founding story, team, and investors.
- [Customers](https://harbour.app/customers): Case studies from Meridian Law Group, Foxrock Capital, and Tessellate Operations.
- [Changelog](https://harbour.app/changelog): Product updates, new features, and bug fixes. Updated with every release.
- [Blog](https://harbour.app/blog): Articles on contract management, legal ops, and document automation best practices.

## Optional

- [Webinars & Events](https://harbour.app/events): Recorded demos, customer stories, and upcoming live sessions.
- [Partner Programme](https://harbour.app/partners): Referral and reseller programme details for consultants and agencies.
- [Status Page](https://status.harbour.app): Live system status and incident history.

This example demonstrates several best practices: explicit brand disambiguation (the "not affiliated with" note), factual description without marketing language, consistent link descriptions that name specific capabilities, clear section organisation, and an ## Optional section for lower-priority content. The pricing page description includes a date - a simple signal that the information is current.

Frequently Asked Questions

Does Google use llms.txt?

Not in any confirmed, documented way as of mid-2025. Google's crawlers are focused on its own indexing infrastructure, and Google has not announced support for the llms.txt convention. That said, Google is aggressively developing AI-native search features (AI Overviews, etc.) and the convention may become relevant to those systems over time. For now, llms.txt is primarily useful for AI answer engines, RAG pipelines, and agent-based tools - not traditional search ranking.

Do all AI models support llms.txt?

No - adoption is uneven. Perplexity has signalled interest in the convention; Anthropic and OpenAI have not made specific public commitments about crawlers respecting it during training. Where llms.txt is most reliably useful today is in developer-built RAG pipelines, browsing tools, and any system where a developer has explicitly chosen to respect the convention. Adoption is growing, but you shouldn't assume universal support.

Can llms.txt hurt you?

In theory, listing pages in llms.txt could draw attention to content you'd prefer not to be prominently cited - for instance, if your changelog documents a security incident or your pricing page lists a higher price than competitors'. In practice, the solution is simply not to list those pages, or to phrase descriptions carefully. The file only guides models towards what you include; it doesn't hide what you exclude. Content that's publicly accessible can still be found and indexed through other means.

Should llms.txt replace robots.txt?

No - they serve different purposes. robots.txt tells crawlers what they are not permitted to access. llms.txt tells AI systems what they should prioritise. They're complementary. If you want to block AI training crawlers from specific pages, use robots.txt with the appropriate user-agent directives (e.g. User-agent: GPTBot). Use llms.txt to surface your most valuable content to the systems you want to engage with you.

How long should llms.txt be?

For most sites, the file should be readable in under two minutes - roughly four hundred to eight hundred words of content. If yours is substantially longer, you're probably including too much. The file is meant to be a curated shortlist, not a comprehensive inventory. Save comprehensive content for llms-full.txt.

Does llms.txt affect my SEO?

Indirectly, potentially. As AI answer engines drive more traffic and citations, being accurately and prominently represented in AI-generated answers matters commercially. llms.txt is one input into that. It has no direct effect on traditional search ranking - Google's PageRank-style signals are not influenced by the file.

Is there an official standard body behind llms.txt?

No - it's a community convention proposed by Jeremy Howard and adopted organically. There's a reference specification at llmstxt.org, but no standards body governs it. This is both a strength (low barrier to adoption) and a limitation (no guaranteed enforcement or universal compliance). The convention is evolving, and the spec may be refined as adoption grows.

What if my site doesn't have much content to list?

A minimal llms.txt with five or six links is still worthwhile. Even a short file establishes the canonical description of your site and ensures that the few pages you do have are surfaced cleanly. As your site grows, the file can grow with it.

Table of Contents