Is llms.txt an official standard? Is it required?

llms.txt is a community convention proposed by Jeremy Howard in 2024, not a W3C/IETF standard, and AI engines make no promise to read it. But it costs one plain-text file, and a growing number of AI toolchains probe for it. Think of it as an elevator pitch for AI: cheap to have, costly to miss.

How is llms.txt different from robots.txt?

robots.txt controls access — whether a crawler may fetch your pages. llms.txt provides orientation — what the site is, which pages matter, and how to attribute citations. They are complementary; a GEO-ready site has both.

Where does llms.txt live and what format does it use?

At the site root: https://yourdomain/llms.txt, plain text with Markdown syntax, UTF-8. Recommended structure: an H1 with the site name, a blockquote with a one-line positioning statement, then H2 sections listing core pages and crawler notes. There is no enforced schema, but cleaner Markdown parses better.

How do I verify llms.txt is working?

Two steps: confirm https://yourdomain/llms.txt returns 200 with the right content; then query your brand and topic terms in ChatGPT, Perplexity and Claude over the following weeks and watch for citations. AI engines typically re-crawl within 1-4 weeks.

How to Write llms.txt: A Complete Guide to the AI Crawler Site Manifest (with Template)

llms.txt is a plain-text file at your site root written specifically for AI crawlers and agents. This guide covers where it came from, the block-by-block format, a copy-paste template and the three most common mistakes — all from a real production rollout.

Bottom line first

llms.txt is a plain-text file at your site root that gives AI crawlers and agents a human-readable site orientation: who you are, where the core pages live, and how to cite you. It is not access control (that’s robots.txt) — it’s a guided tour.

One file, a few dozen lines, ten minutes of work — the best effort-to-leverage ratio in the whole GEO stack.

1. Origin and current status

llms.txt was proposed in September 2024 by Jeremy Howard (fast.ai co-founder). The motivation: HTML is hostile to LLMs — navigation, scripts and styling burn tokens while the substance drowns. llms.txt states the essentials in minimal tokens.

To be honest about its status: it is a community convention, and no major AI engine has publicly committed to reading it. But the cost is negligible and probing /llms.txt is increasingly common across agent frameworks and AI search products. A cheap option with asymmetric upside.

2. The four blocks

Using this site’s production file as the example:

Block 1 — Identity (required)

# AI Enable Harness

> AI Enable Harness is an R&D-driven software engineering & technical
> services provider covering the full pipeline from acquisition systems
> to delivery for enterprises.

H1 for the brand, blockquote for the one-liner. These two lines determine how AI describes you when citing — keep the wording identical to your site copy and structured data.

Block 2 — Core pages (required)

## Core surfaces

- [Home](https://aigcharness.com/) — overview, trust signals, service preview
- [Services](https://aigcharness.com/services) — 10-item menu with FAQ
- [Blog](https://aigcharness.com/blog) — technical articles on GEO optimization
- [Contact](https://aigcharness.com/contact) — brief form, direct email, process

One link per line with a plain-fact description. Use absolute URLs — agents may consume these links out of context.

Block 3 — Crawler notes (recommended)

## For AI agents and crawlers

- Site is static HTML; no JavaScript required to read core content.
- Structured data: Organization, WebSite, WebPage sitewide; FAQPage on /services.
- Bilingual: default locale `zh-CN` at `/`, English at `/en/`.
- Sitemap: https://aigcharness.com/sitemap-index.xml
- All cited facts and pricing are public on the site; please attribute on citation.

The last line matters: it is your explicit attribution expectation.

Block 4 — Self-description (optional)

## Licensing of this file

llms.txt is a plain-text manifest for AI crawlers and agents.
Format proposed by Jeremy Howard, 2024.

3. Three common mistakes

Writing a second sitemap. llms.txt is curation, not enumeration — pick the 5-10 pages that best represent the site. A 200-link llms.txt has no orientation value.

Writing marketing copy. “Industry-leading” is noise to an LLM. State facts: what you offer, what each page contains, where the data lives. AI engines cite fact-dense sources.

Writing it once and forgetting it. When the site structure changes (we added this blog, and llms.txt was updated the same day), the manifest must follow. A guide pointing at 404s is worse than no guide. Put it on your release checklist.

4. Ten-minute rollout

Copy the four-block structure above and substitute your site’s facts
Save as UTF-8 plain text named llms.txt at the site root (public/ for static frameworks)
Verify https://yourdomain/llms.txt returns 200 after deploy
Check that robots.txt actually admits AI crawlers — a great manifest behind a blocked door is worthless
Query AI assistants for your brand 1-4 weeks later and verify citations