A good llms.txt is a short list of pages you can prove answer real search queries, each with a written description of what it answers; the version most WordPress SEO plugins auto-generate is a recency-ordered list of recent posts padded with tag and category archives. The difference is not the format — both are valid markdown — it is the selection signal, and the popular generators pick the wrong one. As of mid-2026, Yoast's file lists your five most-recently-edited posts plus your five largest archives and rewrites itself weekly; Rank Math defaults to as many as 100 entries, each described by the post's own intro text.
We build an llms.txt generator into a WordPress plugin, so here is the honest lesson first: the file format is trivial — a title, a description, a bulleted list of links. Everything that makes an llms.txt good or useless happens in the selection: which pages earn a slot, in what order, described how. Auto-dumps answer that question with "whatever you edited last," and that single default is what turns a curation tool into noise.
What separates a good llms.txt from a URL dump?
Three things, and none of them is formatting: which pages are on the list, how they are ordered, and what each description says. A good file is a capped shortlist of pages that already earn impressions and clicks, ordered best-first, each line naming the question that page answers. A dump is everything-recent, ordered by edit date, described by whatever text the plugin could scrape — usually the excerpt.
Curated by performance
- 20–40 pages that already earn impressions and clicks
- Each line describes the question the page answers
- Archives, tags, author and pagination URLs excluded
- Stable — changes only when your best pages change
Auto-generated dump
- The 5 most-recently-edited posts, or up to 100 by default
- Descriptions are the post's excerpt or intro text
- Category and tag archives included out of the box
- Rewrites itself on a schedule, drifting toward new, not good
The proposal behind llms.txt — Jeremy Howard's September 2024 spec — exists so an AI system with a limited context window can find your canonical material without crawling hundreds of templated pages. A file full of tag archives and pagination defeats its own purpose: it hands the machine exactly the templated URLs it was meant to skip. The descriptions matter for the same reason the pages themselves should be written answer-first — the reader here is a model deciding what to fetch, and it acts on what the line says, not on how recently you touched the page.
How do Yoast and Rank Math actually build the file?
By recency and completeness, not by performance — which is the root of the dump problem. As of mid-2026, Yoast selects the five posts with the most recent last-modified dates (plus pages and custom post types, with cornerstone content promoted first) and adds the five categories or tags that have the most content attached, refreshing the file weekly by a scheduled action. Rank Math lets you choose which post types and taxonomies to include, describes each entry with its intro text, and defaults its item cap to 100, excluding only pages you have set to noindex.
To be fair, both give you manual controls: Yoast has a manual page-selection mode and honors your cornerstone flags, and Rank Math lets you lower the cap and drop the taxonomies. But the defaults ship a dump, and even the manual paths select by editorial flags, not by evidence — neither plugin knows which of your pages actually perform in search, so neither can rank the list by it. If you are weighing these plugins on their AI features more broadly, we broke down what Yoast and Rank Math can and can't do for AI visibility separately.
Why is the selection signal the whole game?
Because an llms.txt buys you one cheap, infrequent read, and you want it spent on the pages that already answer questions well — and the only non-guessing evidence of which pages those are is your own search performance. A post you edited yesterday with no impressions is not your best page; a two-year-old guide pulling 900 impressions a month is, even though no generator ordering by last-modified date will ever surface it. Recency is a proxy for "fresh," never for "good," and "include everything" is a proxy for nothing at all.
Your Search Console data already ranks your pages by demonstrated demand: impressions show what gets surfaced, clicks show what gets chosen, and the query list shows what each page actually answers. That is the same data you would use to find the pages leaking clicks; here you use the top of that list instead of the bottom. Order your file by proven performance and a machine that reads only the first handful of links still lands on your strongest material — which is the entire point of ordering it at all.
What does a done-right llms.txt look like in practice?
A capped, performance-ranked shortlist with real descriptions and no archive noise. After shipping this feature, the criteria we settled on are the ones a machine can actually act on — not filler that pads a count:
- Every link is a page you can show performs in search
- The list is capped — tens of pages, not hundreds
- No category, tag, author or pagination URLs
- Each description names the question the page answers
- Ordered best-first, so a short read still hits your top pages
- Stable — not silently regenerated on every publish
Concretely: pull your top pages by actual Search Console performance, keep the list in the tens rather than the hundreds, exclude every category, tag, author and pagination URL, and write each description to state the question the page answers instead of reusing its excerpt. Then leave it alone — a file that rewrites itself on every publish drifts toward your newest content, which is rarely your best. This performance-first selection is what our generator does in Contexta: it ranks your pages by their real Search Console impressions and clicks, takes the top of that list, and writes each description from what the page answers rather than its excerpt — capped, with archives skipped by default — so the file reflects proven pages, not your last edit.
Should you fix the auto-generated file or turn it off?
If your site is not documentation or a reference that people feed directly into AI tools, the honest answer is that you probably do not need the file at all — the case for that, with the consumption data behind it, is in whether a WordPress site needs llms.txt in 2026. If you do keep it on, the rule is simple: a curated file pointing at your five best pages beats an auto-dump pointing at fifty mixed ones, and an auto-dump you cannot be bothered to fix is worse than no file — because the one time something fetches it, you have spent your only impression on archives.
So do three things or turn it off: switch off the tag and category archives, cap the count, and confirm it is selecting your real best pages rather than your most-recently-edited ones. If your generator cannot do those three, hand-write the file instead — it is small enough — or disable the feature until the tool can. A wrong index is the one llms.txt outcome that actively misrepresents your site; no index at least stays neutral.
FAQ
Does an auto-generated llms.txt hurt my SEO?
No — it will not hurt your Google rankings, because Google has stated that llms.txt does not affect ranking and is not used for AI Overviews or AI Mode. The harm is subtler: a dump of recent posts and archive pages wastes the rare time an AI system does fetch the file, landing it on templated URLs instead of your best content. It misrepresents your site rather than penalizing it.
How many links should an llms.txt have?
Fewer than the plugin defaults suggest — most 2026 guidance lands around 20 to 50 curated links, and a tight set of 5 to 10 core pages often beats a padded list of 100. The number matters less than the selection: every link should be a page that earns its place, ordered best-first. A short, strong file is more useful to a model with a limited context window than a complete one.
Should I let Yoast or Rank Math generate llms.txt automatically?
Only if you change the defaults first — turn off the tag and category archives, lower the item cap, and check that it is selecting your genuinely best pages rather than your five most-recently-edited ones. As shipped, Yoast orders by last-modified date and Rank Math defaults to up to 100 entries, so both produce a dump out of the box. The automation is fine; the default selection signal is the problem.
What makes a good llms.txt description?
A line that states the specific question or task the page handles, so a model can decide whether to fetch it — not the page's marketing excerpt. 'Compares WooCommerce shipping plugins by real cost and setup time' tells an AI system what it will find; a scraped intro sentence usually does not. Write descriptions for the machine that reads them, the same way you would write an answer-first opening for a human.