SEO

What Is llms.txt? Making Your Website Readable by AI (2026)

By REM Web Development4 min read

This tutorial is part of the SEO series for web developers. It builds on JSON-LD Schema Markup and Canonical URLs.

In 2026 a growing share of people never see your site in a list of ten blue links. They ask ChatGPT, Claude, Gemini or Perplexity a question, and the assistant reads a few pages and writes the answer for them, ideally citing where it came from. For a web developer that raises a new practical question: how easy is your site for an AI to read? llms.txt is one small, concrete answer, and you can add one in about ten minutes.

What is llms.txt?

llms.txt is a proposed convention (see llmstxt.org): a plain Markdown file at the root of your site, /llms.txt, that gives a language model a short, clean overview of what the site is and where the important content lives. Think of it as a sitemap written for a reader rather than a crawler:

  • robots.txt says what bots may fetch.
  • sitemap.xml lists every URL, with no context.
  • llms.txt explains, in plain language, what the site is about and which pages matter, with a sentence on each.

Web pages are full of navigation, cookie banners, scripts and layout markup. An AI model has a limited context window, so a compact Markdown summary with links to the best pages is much cheaper for it to use than crawling and cleaning your HTML.

The format

The structure is deliberately simple. Only the title is required:

# Site name

> One-paragraph summary of what this site is and who it is for.

Optional extra notes in plain Markdown.

## Section name

- [Page title](https://example.com/page.html): one-line description
- [Another page](https://example.com/other.html): one-line description

## Optional

- [Less important page](https://example.com/archive/): can be skipped if context is short
  • H1: the name of the site or project.
  • Blockquote: a short summary, the most important part.
  • H2 sections: lists of links, each with a short description after a colon.
  • A section called Optional marks links a tool can drop if it needs to save space.

A real llms.txt example

This site has one, and you can open it right now: remwebdevelopment.com/llms.txt. Here is the beginning of it, slightly shortened: 

# REM Web Development

> Free web programming tutorials and courses for beginners (HTML, CSS, JavaScript,
> Vue, npm), plus SEO and hosting guides for developers.

## Courses

- [Introduction to Web Programming](https://www.remwebdevelopment.com/courses/introduction-to-web-programming/): HTML, CSS and JavaScript for people with no programming experience
- [Node Package Manager Crash Course](https://www.remwebdevelopment.com/courses/node-package-manager/): npm, Babel, Gulp and Sass

## SEO for web developers

- [Canonical URLs Explained](https://www.remwebdevelopment.com/blog/seo/canonical-urls-explained.html): when to use a canonical tag instead of a redirect
…

Notice what it doesn't do: it doesn't list all 280 pages. It points to the courses, the topic pages and the most useful articles. That's the editorial judgement a sitemap can't make.

Generate it from your content

A hand-written llms.txt goes stale the first time you publish something new. If your site is built from data, generate the file at build time. On this Astro site it's an endpoint, src/pages/llms.txt.ts, that builds the Markdown from the same post list the blog uses: 

// src/pages/llms.txt.ts
export function GET() {
    const line = (p) => `- [${p.title}](https://www.example.com${p.path}): ${p.summary}`;
    const body = [
        '# Example Site',
        '',
        '> Free tutorials for beginner web developers.',
        '',
        '## Recent posts',
        '',
        ...posts.slice(0, 15).map(line),
    ].join('\n');

    return new Response(body, { headers: { 'Content-Type': 'text/plain; charset=utf-8' } });
}

On WordPress there are plugins for it, and on a hand-coded site a plain text file uploaded next to robots.txt works fine.

Does it actually help?

Be realistic about it. In 2026 llms.txt is still a proposal, not a standard that every AI company has committed to. Some AI tools and coding assistants do read it, especially for documentation sites. Major search engines have said they don't need it for ranking. So treat it as cheap insurance, not a ranking trick. The things that make a page useful to an AI are the same things that make it useful to people:

  1. Server-rendered HTML. Many AI crawlers don't run JavaScript. If your content only appears after a React app loads, they may see an empty page. Static or server-rendered HTML is the safest choice (see Astro SEO).
  2. Clear structure. One h1, sensible h2/h3 headings, short paragraphs and real lists make it easy to pull out a precise answer.
  3. Answer first. Put the direct answer near the top of the section, then the explanation. That's what gets quoted.
  4. Structured data. JSON-LD tells machines what a page is, who wrote it and when.
  5. Let the crawlers in. Check that robots.txt isn't blocking the AI crawlers you want, such as GPTBot, ClaudeBot, PerplexityBot or Google-Extended. A single Disallow: / under User-agent: * blocks them all.
# robots.txt: allow everyone, point to the sitemap
User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap-index.xml

Summary

llms.txt takes ten minutes, costs nothing and can't hurt your search rankings. Add one, generate it from your content so it stays current, and spend the rest of your effort on the fundamentals: fast, server-rendered pages with a clear structure that answer real questions. Those are what AI assistants (and people) end up citing.