How to write an llms.txt file that AI crawlers actually use.

Writing your llms.txt file by hand in a plain text editor gives you complete control over what AI crawlers see and how your site is described. Open any text editor, start with a one-paragraph site description, add Markdown H2 headings for each content section, and list your most important pages as bullet points with absolute URLs and one-sentence descriptions.

The main risk is staleness. A hand-written file that reflects your site as it was six months ago actively misleads crawlers. Set a calendar reminder to review and update it quarterly at minimum — or every time you publish a major new section. For sites that publish frequently or have large catalogues, one of the automated approaches below will serve you better long-term. Pair this with the broader GEO strategies covered at Ryze AI to get the full picture.

Several free generators now exist that crawl your site and produce a draft llms.txt automatically. WordLift’s generator is URL-based and fully free; Ahrefs has integrated llms.txt generation into its site audit toolset. You feed in your domain, the tool crawls your pages, and outputs a structured Markdown file within minutes.

In our tests, generator-produced files had a median of four description errors per 20 links — pages described inaccurately because the generator pulled meta descriptions written for Google rather than for machine comprehension. Treat the output as a first draft: review every description, remove low-priority pages, and add any important pages the crawler missed. For ecommerce sites, also check our guide on AI search visibility for product pages.

Mintlify has built the most advanced llms.txt pipeline we tested. It automatically generates both the curated llms.txt index and the llms-full.txt concatenated file from your documentation source, and it now also provisions MCP (Model Context Protocol) servers that make your docs queryable by AI coding agents like Cursor and Claude.

For a SaaS product with a documentation site, this positions your knowledge base as a first-class resource for AI developer tools — a distribution channel that did not exist two years ago. The limitation is narrow scope: if your site is primarily ecommerce, marketing pages, or a content blog rather than technical documentation, Mintlify is overkill and the fit is awkward.

The WordPress ecosystem has responded quickly to the llms.txt standard. RankMath added llms.txt generation in a 2025 update; several dedicated plugins also exist. Most work by reading your existing sitemap and post/page metadata to build the file, then auto-updating it whenever you publish new content.

Plugin quality varies significantly. In our tests, the best plugins let you exclude page types (archive pages, tag pages, author pages) and customise descriptions per section. The worst plugins included every URL in the sitemap verbatim with no descriptions — producing a file that was technically valid but practically useless for a crawler trying to understand content priority. Audit any plugin’s output before going live.

Static site generators can output an llms.txt file as part of every build, pulling page titles and descriptions from your content metadata automatically. In Next.js, a simple API route at /app/llms.txt/route.ts can read your content directory and generate the file dynamically on each request, or you can write it as a static file during the build step. Hugo and Eleventy have community templates for the same pattern.

This approach scores highest on maintenance burden because there is none — the file updates automatically whenever content changes. The catch is that it requires a developer to set it up correctly, and the auto-generated descriptions are only as good as your page metadata. Invest time in writing accurate meta descriptions; they feed both your llms.txt and your traditional SEO simultaneously. For more on structuring Next.js sites for AI visibility, see our post on technical GEO for modern web apps.

Some site owners manage AI crawler access entirely through robots.txt, adding named User-agent blocks for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, and Bytespider with Allow and Disallow rules. This gives you access control — you can block crawlers from private areas, rate-limit training data harvesting, or open your public content explicitly.

What robots.txt cannot do is tell a model what your content is about or which pages matter most. A crawler that respects your robots.txt directives will still arrive at your homepage with no structured guide to what it should read. In our tests, sites with only robots.txt directives and no llms.txt had lower AI-sourced traffic than comparable sites with both files, even when the robots.txt was perfectly configured. Use robots.txt to control access; use llms.txt to curate understanding.

The llms-full.txt format is a single large file containing the complete Markdown content of every page listed in your llms.txt index. Where llms.txt tells a model where your best pages are, llms-full.txt gives it the full text of those pages in one request — reducing the number of round trips an agent needs to make when answering a product-specific question.

This is particularly valuable for developer tools: when an engineer asks a coding assistant a question about your API, the agent can load your entire documentation into context from one URL rather than crawling dozens of individual pages. The trade-off is file size — a comprehensive llms-full.txt for a large knowledge base can exceed several megabytes, which is slow to fetch and parse. Limit it to your 30–50 most important pages and regenerate it on a scheduled basis. Mintlify handles this automatically; for other platforms, a build-time script is the cleanest solution.

MCP (Model Context Protocol) servers represent the next evolution beyond llms.txt. Where llms.txt is a static file a crawler reads once, an MCP server exposes your knowledge base as a live, queryable endpoint that AI agents can call in real time during a workflow. Mintlify now provisions these automatically for documentation sites; for custom implementations, you build an MCP server that your llms.txt points agents toward.

This is genuinely powerful for enterprise products: an AI coding assistant working inside a developer’s IDE can query your MCP server to get up-to-date API references, pricing data, or troubleshooting guides mid-workflow, without the user doing anything. The implementation burden is significant and the tooling is still maturing, but for high-value SaaS products where being part of an agent’s workflow is strategic, the investment is justified. For most sites, a well-written llms.txt file combined with strong content authority — accelerated by Ryze AI’s GEO automation — will deliver more measurable lift with far less engineering overhead.