llms.txt Specification

The plain-text file at /llms.txt that gives AI crawlers a structured, Markdown-formatted index of your site — titles, URLs, and descriptions designed for LLM consumption.

Location and format

The file must be served at {origin}/llms.txt — for example, https://example.com/llms.txt. An optional extended form at {origin}/llms-full.txt provides full page content for deep indexing.

Both files must be served with a text/plain content type. The format is Markdown: an H1 site name, a blockquote description, H2 section headings, and list items in [Title](url): description format.

Why text/plain?

AI crawlers parse llms.txt as raw text. Serving it as text/html or application/octet-stream causes some crawlers to skip or misparse the file.

Concise form (llms.txt)

The concise form provides a lightweight index: one line per resource with a title, URL, and short description. AI crawlers fetch this first for a quick overview of your site.

/llms.txt

# Example Site

> Example Site helps developers build faster with modern tooling and open-source libraries.

## Pages

- [Home](https://example.com/): Welcome to Example Site
- [About](https://example.com/about): Our story, mission, and team
- [Contact](https://example.com/contact): Get in touch with us

## Products

- [Widget Pro](https://example.com/products/widget-pro): Professional-grade widget — $29.99
- [Widget Lite](https://example.com/products/widget-lite): Entry-level widget — $9.99

## Blog

- [Getting Started](https://example.com/blog/getting-started): How to set up Widget Pro in five minutes
- [Advanced Configuration](https://example.com/blog/advanced-config): Fine-tuning Widget Pro for production

Extended form (llms-full.txt)

The extended form includes full content for each resource. For products this means pricing, availability, and variant details. For pages and articles it means the full body text. AI crawlers fetch llms-full.txt when they need richer context — for example, to answer a product question or summarize an article.

/llms-full.txt

# Example Site

> Example Site helps developers build faster with modern tooling and open-source libraries.

## Pages

- [Home](https://example.com/): Welcome to Example Site
- [About](https://example.com/about): Our story, mission, and team

## Products

### Widget Pro

URL: https://example.com/products/widget-pro
Price: $29.99
Available: true
Category: Widgets

Our flagship widget. Built for professionals who need reliability and performance.
Available in three sizes with free shipping on orders over $50.

Variants:
- Small — $24.99 — In stock
- Medium — $29.99 — In stock
- Large — $34.99 — Low stock

## Blog

### Getting Started

URL: https://example.com/blog/getting-started
Author: Jane Smith
Published: 2024-11-01

This guide walks you through setting up Widget Pro in a fresh project. You will need
Node.js 18 or later and a package manager of your choice.

Step 1: Install the package
Step 2: Configure your environment
Step 3: Run the development server

Required and optional fields

The concise form has two required structural elements and one optional extended section.

Element	Syntax	Required	Purpose
Site name	# Site Name	Yes	H1 heading — identifies the site to the AI model
Site description	> Description	Yes	Blockquote — one or two sentences describing the site
Section heading	## Section	Yes (at least one)	H2 heading — groups related resources (Pages, Products, Blog, etc.)
Resource entry	- [Title](url): desc	Yes (at least one)	List item — title, URL, and short description for one resource
Extended resource block	### Resource Title	No	H3 heading in llms-full.txt — introduces full content for a resource

Minimum viable llms.txt

A valid file needs at minimum: an H1 site name, a blockquote description, one H2 section, and one list item. The body must be at least 100 characters and contain informative content (URLs, paths, or a site description) to pass the Analyzer.

Analyzer checks

The checkLlmsTxt function fetches {origin}/llms.txt and evaluates the response against the following conditions:

Condition	Status	Recommendation code
HTTP 200 + `text/plain` content type + body ≥ 100 chars + informative content	pass	—
HTTP 200 + empty body	warn	POPULATE_LLMS_TXT
HTTP 200 + non-text content type	warn	FIX_LLMS_TXT_CONTENT_TYPE
HTTP 200 + body < 100 chars or no informative signals	warn	IMPROVE_LLMS_TXT
HTTP 404	not_found	ADD_LLMS_TXT
Any other HTTP status (403, 500, etc.)	warn	FIX_LLMS_TXT_ACCESS
Network error (DNS failure, timeout, etc.)	unknown	—

Informative content is detected by the presence of at least one of: an absolute URL matching https?://\S+, a path matching /[\w\-./]+, or a brand description pattern matching ^#\s+\S+ or [A-Z][a-z]+ is a .

Common mistakes

Wrong content type. Serving llms.txt as text/html (the default for many static site generators) causes the Analyzer to return warn. Set the content type explicitly to text/plain in your server or CDN configuration.
Thin content. A file with only a site name and description but no resource list is fewer than 100 characters and will return warn. Include at least a few resource entries with URLs and descriptions.
Missing blockquote description. Omitting the > blockquote line means AI models have no site-level context. The Analyzer does not fail on this alone, but it reduces the quality of AI-generated summaries about your site.
Blocking the file in robots.txt. A Disallow: /llms.txt rule prevents AI crawlers from fetching the file even if it exists. The Analyzer fetches directly and will return pass, but real crawlers will not see the file.
Stale content. llms.txt is not automatically updated when you add pages or products. Use a GEO AI package or a build-time generation step to keep the file in sync with your content.

PreviousOverview NextAI Metadata