llms.txt: The 2026 Guide (Format, Spec, Examples, Platform Setup)
llms.txt is a markdown file placed at the root of a website (/llms.txt) that gives large language models a curated overview of the site's content. Proposed by Jeremy Howard (Answer.AI) in September 2024, it's an emerging standard. Anthropic recommends it for Claude docs ingestion; Mintlify auto-generates it. OpenAI, Google, and Perplexity have not formally committed.
Free generator · 30 seconds · no signup
What is llms.txt?
llms.txt is a markdown file placed at the root of a website that gives large language models a curated overview of the site's content. Where robots.txt tells crawlers what they may fetch and sitemap.xml tells them what exists, llms.txt tells LLMs what matters.
Origin
Proposed on September 3, 2024 by Jeremy Howard, co-founder of Answer.AI and fast.ai. Canonical spec at llmstxt.org. First major adoption catalyst: Mintlify's November 2024 rollout across all hosted docs sites.
Location
Root level of the site — yoursite.com/llms.txt. Not on a subdomain, not gated behind auth, not nested in a subdirectory. Must return HTTP 200 with Content-Type: text/plain or text/markdown.
Format
Markdown — not XML like sitemap.xml, not plain-text-directive like robots.txt. The markdown format is deliberate: LLMs already tokenize markdown natively, so no additional parser is needed for a model to consume the file.
Structure
H1 site name → optional summary blockquote → optional details paragraph → H2 sections → bullet list of linked pages with short descriptions. Two optional variants exist: llms-full.txt (inlines full content) and legacy ai.txt (training-data consent — different discipline).
llms.txt is not a Google ranking signal, not a Perplexity retrieval signal, and not (publicly) an OpenAI retrieval signal. It is a curation file — a table of contents you author for LLMs — with the honest 2026 verdict below.
llms.txt example (with format spec)
A concrete example is the fastest way to internalize the format. Below is a spec-conformant llms.txt for TurboAudit, followed by a line-by-line breakdown.
# TurboAudit
> TurboAudit is a per-page AI SEO audit and cross-engine visibility platform for teams optimizing content for ChatGPT, Perplexity, Gemini, and Google AI Overviews.
TurboAudit scores any URL against verified 2026 AI-search ranking factors — semantic concept density, freshness, named-source citations, extraction-friendly structure, entity relationship density, and machine accessibility — and returns a prioritized action plan.
## Core guides
- [LLM SEO](https://turboaudit.ai/llm-seo): 2026 cornerstone on optimizing content for large-language-model answer engines.
- [Perplexity SEO](https://turboaudit.ai/perplexity-seo): Ranking factors, playbook, and monitoring for Perplexity's Sonar surface.
- [ChatGPT SEO](https://turboaudit.ai/chatgpt-seo): Getting cited in ChatGPT — GPTBot, OAI-SearchBot, and Bing-index dependency.
- [Generative Engine Optimization (GEO)](https://turboaudit.ai/generative-engine-optimization): Princeton-rooted GEO discipline.
- [Answer Engine Optimization (AEO)](https://turboaudit.ai/answer-engine-optimization): Answer-first content pattern.
## Free tools
- [llms.txt generator](https://turboaudit.ai/tools/llms-txt-generator): Generate a spec-conformant llms.txt from a URL.
- [AI bot checker](https://turboaudit.ai/tools/ai-bot-checker): Verify robots.txt access for 12+ AI crawlers.
- [Schema generator](https://turboaudit.ai/tools/schema-generator): Article, FAQPage, HowTo, BreadcrumbList JSON-LD.
## Product
- [Pricing](https://turboaudit.ai/pricing): 2026 plans, free tier, per-page audit and monitoring bundles.
- [AI SEO audit](https://turboaudit.ai/ai-seo-audit): Per-URL 6-signal score with a prioritized action plan.
## Optional
- [About](https://turboaudit.ai/about): Team, methodology, editorial standards.
Line-by-line breakdown
- # TurboAudit — H1 with the site name. Exactly one H1; treat it as the LLM-facing title of the file.
- > TurboAudit is … — Blockquote summary. One sentence, compressed context statement, no links. Optional but strongly recommended — it's the LLM equivalent of a meta description for the whole file.
- Details paragraph — Plain markdown paragraph after the blockquote. Optional. Use for a two-to-three-sentence expansion of what the site does and what content it publishes.
- ## Core guides / ## Free tools / ## Product — H2 section headings that describe topical groupings. Use real topic labels; the H2 is read as context for every bullet underneath.
- - [Page title](https://…): short description. — Standard markdown bullet with a linked title and a colon-separated description. Descriptions should be 5–20 words and specific.
- ## Optional — Reserved H2 label the spec uses for lower-priority links. If a section is titled "Optional," consuming LLMs treat those links as lower weight than earlier sections.
Spec at a glance
H1 name (required) → blockquote summary (optional) → details paragraph (optional) → H2 sections with linked-bullet lists (repeat). No inline images. No code blocks. No tables. Absolute URLs preferred. UTF-8. Served at /llms.txt with HTTP 200.
llms.txt vs llms-full.txt vs ai.txt vs robots.txt
Four related root-level files that AI ecosystems interact with. Different jobs, different histories, different adoption levels. The table below disambiguates them so you don't conflate curation (llms.txt), full-content ingestion (llms-full.txt), training-data consent (ai.txt), and crawler access control (robots.txt).
| File | Purpose | Format | Status in 2026 |
|---|---|---|---|
llms.txt | Curated markdown overview of a site's content for LLMs — a table of contents pointing to the most important pages with short descriptions. | Markdown, root-level (/llms.txt) | Proposed Sept 2024 by Jeremy Howard (Answer.AI). Emerging community standard; Anthropic recommends for Claude docs ingestion; Mintlify auto-generates. |
llms-full.txt | Optional companion file containing the full body content of the linked pages concatenated into one file, so an LLM can consume the site in a single fetch. | Markdown, root-level (/llms-full.txt) | Same proposal (llmstxt.org). Adoption thinner than llms.txt; heavier to maintain because the whole content is inlined. |
ai.txt | Earlier proposal (Spawning, 2023) focused on training-data opt-out — declaring which AI training uses are permitted or refused. | Plain text, root-level (/ai.txt) | Distinct discipline (consent, not curation). Never became broadly adopted; largely superseded by robots.txt AI-crawler directives in practice. |
robots.txt | Machine-access instructions to crawlers — which user-agents may (or may not) fetch which URL paths. | Plain text, root-level (/robots.txt) | Universally supported since 1994. In 2026, the primary AI-facing accessibility file — GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, and 12+ others read it. |
The four files are complementary, not substitutes. In 2026 most content-heavy sites publish robots.txt (universal) plus sitemap.xml (universal) and add llms.txt as a low-cost belt-and-braces file — llms-full.txt only if the maintenance burden is manageable and ai.txt almost never (largely superseded).
Do AI engines actually read llms.txt in 2026?
Honest answer: some do, most don't publicly commit. Below is the verified 2026-07-27 adoption status per engine. No engine treats llms.txt as a documented ranking signal; a growing minority consume it for docs ingestion and content discovery.
| Engine / Vendor | Stance | Evidence (2026-07-27) |
|---|---|---|
| Anthropic (Claude) | Recommends | Anthropic's own docs ingestion guidance recommends publishing an llms.txt for structured content discovery by Claude. Adoption partner of the standard since 2024. |
| Mintlify | Auto-generates | Rolled out /llms.txt across all hosted docs sites in November 2024 — the single largest adoption catalyst for the standard. Any Mintlify-hosted docs site now serves llms.txt by default. |
| OpenAI (ChatGPT) | Not publicly committed | OpenAI has not stated that GPTBot, OAI-SearchBot, or ChatGPT-User consume llms.txt as a ranking or retrieval signal. Silence is not confirmation of use. |
| Not publicly committed | Google has made no public statement that its AI Overviews, AI Mode, or Gemini ingestion pipeline reads llms.txt. Google's John Mueller has expressed skepticism of it as a ranking signal. | |
| Perplexity | Not publicly committed | Perplexity's Sonar retrieval documentation does not mention llms.txt. No primary evidence that PerplexityBot or Perplexity-User treat it as a signal. |
| Firecrawl | Deprecated hosted tool (June 2025) | Firecrawl deprecated its hosted llms.txt generator in June 2025 — a realism signal that adoption enthusiasm has cooled since the November 2024 Mintlify wave. |
Skeptic counter-argument (honest)
Kai Spriestersbach published a 2026 Medium piece titled “llms.txt is dead / a dud” making a serious skeptic argument: without documented consumption by the three biggest engines (OpenAI, Google, Perplexity), llms.txt risks being a maintenance burden that produces no measurable citation lift. His critique is worth taking seriously — it's the strongest published counter-position on the standard.
TurboAudit verdict
Cheap to implement, zero downside, upside uncertain but non-zero for content-heavy sites and SaaS docs. Ship llms.txt if the maintenance cost is a one-time 20-minute build plus a quarterly refresh. Do not treat it as a substitute for the content-level signals (concept density, freshness, named-source citations, extraction-friendly structure) that empirically dominate 2026 citation odds. For the ranking-factor evidence audit, see our AI search ranking factors cornerstone.
How to create llms.txt
Two paths: a free generator that crawls your site and outputs the file in ~30 seconds, or a manual four-step build that gives you full editorial control.
Recommended
Use the TurboAudit llms.txt generator (free)
Enter your URL; the generator crawls the site, curates the most important pages by internal linking and content signal, and returns a spec-conformant llms.txt you can copy directly into your web root. No signup for the basic flow, ~30 seconds end-to-end.
Manual — four steps
1. Write the H1 + summary blockquote
One-line H1 with the site name. One-sentence blockquote directly beneath — compressed context, no links. This is what an LLM reads first.
2. Pick 20–50 pages worth curating
Don't dump the sitemap. Choose the pages you want an LLM to prioritize — cornerstone guides, primary docs, key product pages, pricing, contact. Skip low-value archive posts, thin utility pages, and duplicates.
3. Group into H2 sections with real topic labels
"Core guides," "API reference," "Free tools," "Product," "Legal." Use whatever taxonomy fits the site. The H2 label is read as context for every bullet inside.
4. Upload to the web root and verify HTTP 200
Serve at
yoursite.com/llms.txtwithContent-Type: text/plainortext/markdown. Do not gate it behind auth, do not redirect it, do not serve it from a subdomain. Confirm the URL returns 200 in a browser or with curl.
Platform-specific setup
Every platform serves root-level files differently. Below is the 2026 setup path for the five most common stacks the TurboAudit audience runs.
WordPress
Two paths. (1) FTP/SFTP the llms.txt file into the WordPress web root — same directory as /wp-config.php — and confirm the file is served at yoursite.com/llms.txt with an HTTP 200 response. (2) Install a dedicated 2026 WordPress plugin that generates and serves llms.txt from your published pages and posts. Do not upload the file to /wp-content/uploads/ — it must sit at the site root, not inside a subdirectory.
Shopify
Shopify does not natively allow arbitrary root-level files, which is the single biggest platform-adoption blocker for llms.txt in 2026. Three workarounds: (a) serve /llms.txt from a theme template that maps to the root path, (b) use Shopify Plus custom routing to expose the file, (c) install a dedicated Shopify app that publishes and maintains llms.txt for you. Test yoursite.com/llms.txt for a 200 response after any of the three approaches.
Mintlify
Zero configuration. Mintlify rolled out /llms.txt across all hosted docs sites in November 2024 — every Mintlify-hosted docs site serves an auto-generated llms.txt at the root, kept in sync with the docs content. This November 2024 rollout was the single largest adoption catalyst for the standard.
Next.js
Drop the llms.txt file into the /public/ directory of your Next.js project. Next.js serves everything in /public/ at the site root, so /public/llms.txt is served at yoursite.com/llms.txt automatically. For dynamic generation, use a route handler at app/llms.txt/route.ts that returns Content-Type: text/plain.
Webflow, Astro, Docusaurus, Framer
Same pattern across static-site and docs frameworks. Webflow: use the site-settings custom-code area for uploaded files, or map a page to /llms.txt with a raw-markdown template. Astro: place llms.txt in /public/. Docusaurus: place it in /static/. Framer: use the site settings' custom-file upload with the /llms.txt route. All four require the file at the site root — not on a subdomain, not gated behind auth.
llms.txt best practices
Six-item checklist for a well-formed, LLM-useful llms.txt. Applied in order, they map the common failure modes: dumping instead of curating, vague titles, missing H2 grouping, staleness, missing summary, and broken HTTP delivery.
Curate — don't dump
llms.txt is a table of contents, not a sitemap. Include the pages you want an LLM to prioritize; exclude noise, marketing pages, duplicates, and archived content. If you include everything, you've signaled nothing.
Write descriptive link titles
The bullet title is the label an LLM sees. "Pricing — 2026 tiers, per-seat vs per-page, free trial terms" beats "Pricing." Titles are read; brevity + specificity wins.
Group into H2 sections with real topic labels
Use H2 headings that describe the section ("Docs," "API reference," "Guides," "Legal"). LLMs use the H2 as topical context for the bullets underneath.
Keep it fresh
Refresh whenever you ship a major page, retire a URL, or restructure the site. A stale llms.txt pointing to 404s is worse than no file — it signals you don't maintain the site.
Include a summary blockquote at the top
One-sentence quote block under the H1 gives the LLM a compressed context statement it can use verbatim. Treat it like meta-description weighting for the whole file.
Validate — visit /llms.txt and check the HTTP response
The file must return HTTP 200 with Content-Type: text/plain or text/markdown. Do not gate it behind auth, do not redirect it, do not serve it from a subdomain.
FAQ
What is llms.txt?
llms.txt is a markdown file placed at the root of a website (/llms.txt) that gives large language models a curated overview of the site's content. Proposed by Jeremy Howard (Answer.AI) in September 2024, it's an emerging standard. Anthropic recommends it for Claude docs ingestion; Mintlify auto-generates it.
What is llms txt?
llms txt (spelled with a space) refers to the same file as llms.txt — a curated markdown table of contents served at yoursite.com/llms.txt that summarizes site content for large language models. The dot spelling matches the actual filename; the space spelling is common search-query phrasing.
Do AI engines read llms.txt in 2026?
Some do, most do not publicly confirm. Anthropic recommends llms.txt for Claude docs ingestion and Mintlify auto-generates it for hosted docs sites. OpenAI, Google, and Perplexity have not publicly committed to consuming it as a ranking or retrieval signal. Firecrawl deprecated its hosted generator in June 2025.
Is llms.txt the same as robots.txt?
No. robots.txt is a plain-text access-control file that tells crawlers which URLs they may fetch. llms.txt is a markdown curation file that tells LLMs which pages you consider most important. Different formats, different jobs, different histories — robots.txt dates to 1994; llms.txt was proposed in September 2024.
What's the difference between llms.txt and llms-full.txt?
llms.txt is a curated markdown index of your most important pages with short descriptions and links. llms-full.txt is an optional companion that inlines the full body content of those pages into one file so an LLM can consume everything in a single fetch. llms-full.txt is heavier and less commonly adopted.
What is ai.txt vs llms.txt?
ai.txt is an earlier 2023 proposal from Spawning focused on training-data consent — declaring which AI uses of your content are permitted or refused. llms.txt is a 2024 curation proposal from Jeremy Howard focused on helping LLMs understand your site. Different disciplines: consent versus curation. ai.txt never reached broad adoption.
Do I need llms.txt for SEO?
Not for traditional Google SEO. For AI-engine visibility, llms.txt is a low-cost belt-and-braces addition — cheap to publish, zero downside, upside uncertain but non-zero for content-heavy sites and SaaS docs. Ship it if you also ship the content-level signals (concept density, freshness, named sources) that empirically dominate citation odds.
How do I add llms.txt to WordPress?
Two paths. Manual: FTP or SFTP the file into the WordPress web root next to /wp-config.php, and confirm yoursite.com/llms.txt returns 200. Plugin: several 2026 WordPress plugins generate and serve llms.txt automatically from your published content. Do not upload it to /wp-content/uploads/ — it must sit at the web root.
How do I add llms.txt to Shopify?
Shopify does not natively allow arbitrary root-level files. Three workarounds in 2026: (1) serve /llms.txt from a theme template that maps to the root path, (2) use Shopify Plus custom routing to expose the file, (3) install a dedicated Shopify app that publishes and maintains llms.txt for you. Test yoursite.com/llms.txt for a 200 response.
Where can I generate llms.txt for free?
TurboAudit ships a free llms.txt generator that crawls your site, curates the most important pages, and outputs a spec-conformant markdown file you can copy into your web root — no signup required for the basic flow. Mintlify auto-generates for hosted docs customers; several open-source scripts exist for developers who prefer scripting the build.
Generate your llms.txt in 30 seconds — free, no signup
The TurboAudit generator crawls your site, curates the most important pages, and outputs a spec-conformant llms.txt you can copy into your web root. Free, no card, no signup for the basic flow.
Sources
Primary sources verified July 27, 2026. Every claim on this page — origin date, Mintlify rollout timing, Firecrawl deprecation, per-engine adoption stance — traces back to one of the entries below.
- llmstxt.org — canonical llms.txt specification (Jeremy Howard, September 3, 2024)— llmstxt.org
- Answer.AI — original llms.txt announcement (Sept 3, 2024, Jeremy Howard co-founder Answer.AI + fast.ai)— answer.ai
- Mintlify — llms.txt documentation + November 2024 rollout across all hosted docs sites— mintlify.com
- Anthropic Claude — docs ingestion recommends llms.txt— docs.anthropic.com
- Firecrawl — hosted llms.txt tool deprecation notice (June 2025)— firecrawl.dev
- Kai Spriestersbach — "llms.txt is dead" 2026 Medium critique (honest counter-argument)— medium.com
- Semrush — 2026 llms.txt adoption coverage— semrush.com
- Ahrefs — 2026 llms.txt explainer and adoption analysis— ahrefs.com
- Search Engine Land — 2026 llms.txt reporting— searchengineland.com
- Google Search Central — FAQPage rich results retired May 7, 2026 (Ryan Levering)— developers.google.com
- Wix Studio / Kevin Indig — 2026 LLM citation format study (40–60w capsules 72.4%; tables 80.9%)— wixstudio.com
FAQPage rich results retired May 7, 2026. Google's Ryan Levering confirmed FAQPage rich results were removed from Search Console and Search results starting May 7, 2026. FAQ schema is still parsed for content understanding — we ship it here — but the visual rich-result treatment is gone. See our AEO guide for the full retirement timeline.