Yoast SEO’s llms.txt feature does not give you a working AI discovery file. We enabled it on a live WooCommerce store running Yoast SEO v28.3 and audited exactly what it published. The file went live at /llms.txt and returned a clean 200, but it held 41 entries and 22 of them had empty links written as [Name](), every entry under Pages and Products was blank, and it leaked internal template names that were never meant to be public. The 19 entries that did carry a URL were category and tag archives, not the pages or products that matter. Yoast’s idea is right. The automatic output is not.
This is not a takedown of Yoast. We run the plugin ourselves, and Yoast is one of the few companies even trying to ship AI discovery inside a normal WordPress workflow. But we hit a concrete problem on a real store, so here is what we found.
The Short Version
- What Yoast promises: a llms.txt file generated and refreshed automatically, with no technical setup and no cost.
- What ours shipped: 41 entries, 22 with empty URLs, and every Pages and Products entry blank.
- The 19 working links: category and tag archives, not key pages or products.
- The leak: internal entries including “Single Product Template – Adaptors” and “Retired Product Template,” one with a live typo.
- The missing part: no shipping, returns, contact, currency, or region facts, which is what AI assistants actually need.
- The fix: write a short llms.txt by hand and serve it from your site root. Yoast’s own developer docs describe a manual file as the escape hatch.
What Yoast Claims, in Its Own Words
Start with the pitch, because the pitch is reasonable. On its feature page, Yoast says the llms.txt feature means “no technical or manual setup required,” that the file is “generated and refreshed automatically,” that you can “preview the file before publishing,” and that it is “built into Yoast, free for everyone.” You can let Yoast “automatically select key content” or “switch to manual mode and choose yourself.” There is even a developer specification for building your own file (Yoast, yoast.com/features/llms-txt/).
Read that list again and notice the strongest claim is also the weakest one: automatic selection of your key content. That is the exact step where our audit went wrong.
Myth 1: One Click and Your AI Discovery Is Handled
The belief: turn the toggle on, publish, and AI tools now understand your site.
The reality: the file can go live and still be close to useless. On the store we manage, Yoast v28.3 generated /llms.txt on cue and served it with a 200. Then we opened it. Of the 41 entries, 22 pointed nowhere. The markdown link pattern was there, but the address between the parentheses was empty, so every line read like [Something](). A model reading that file learns a name and no location. An empty link is not a formatting bug, it is the whole point of the file failing.
Automatic mode is the likely culprit. The generator selected entries it could not resolve into URLs. A human writing the file would have caught that in seconds, which is why curation beats automation here.
Myth 2: More Entries Means More Visibility
The belief: a long llms.txt gives an AI assistant more to work with.
The reality: length without signal is noise. The 19 entries in our file that did have working URLs were category and tag archives. Those are index pages, not answers. If a shopper asks an assistant “does this store ship to Canada,” a link to a tag archive named “adaptors” does not help. If they ask about a specific product, an archive listing says nothing about price, availability, or policy.
The product entries that did render exposed a second problem. They dumped raw, escaped HTML product descriptions straight into the file. That is the opposite of the clean, plain-text markdown llms.txt is supposed to provide. Instead of stripping markup down to readable text, the output carried tags and escape characters into a file whose entire purpose is to be easy for a language model to parse.
Myth 3: It Lists Your Key Pages and Products
The belief: the automatic selection includes the pages you care about most.
The reality: the pages you care about most are the ones that came out empty. Every entry under the Pages heading and every entry under the Products heading had a blank URL. That is the part of the file that matters most for a store, and it failed completely. The working links were all archives.
There was a second surprise in the same file. The generator exposed internal template entries that should never be public, including “Single Product Template – Adaptors” and “Retired Product Template.” One entry even carried a live typo, spelled “Templete.” These are internal build artifacts. A public file that describes your store to AI assistants should not advertise your retired templates or your spelling mistakes.
Finally, the file skipped the facts an AI needs to answer a buying question: shipping options and costs, the return policy, how to make contact, which currency prices are in, and which regions you serve. Without those, an assistant fills the gaps with a guess.
What a Good llms.txt Actually Looks Like
A good llms.txt is small: a hand-curated list of your most important pages with a one-line description of each, plus a short block of core facts. It fits on one screen. A model should read the top of it and immediately know what you do, where you ship, how returns work, and how to reach a human.
- One line about the site: what the business does and who it serves, in plain language.
- Key pages: eight to fifteen links that matter, each with a one-line description. Not every page, just the ones that answer real questions.
- Core facts: shipping regions and costs, returns, contact, currency, and support hours.
- Optional deeper files: if you have a lot of content, link a few .md files for the sections that matter most.
Notice what is absent: no internal templates, no raw HTML, no archive dumps, no empty links. Curation is not a limitation here, it is the feature. Yoast’s own copy makes that case for us. AI tools “only scan small parts” of a site in real time, so a document that points them at the right parts is worth more than a long one that points everywhere.
How to Fix It in About 20 Minutes
You do not need a plugin, and you do not need to wait for Yoast to improve the generator. Serve your own file from the site root.
Step one: write it. Keep it to the shape above. Here is the difference between what our audit produced and what it should have said.
If you would rather not write it from scratch, there is a free llms.txt generator at marcusli.com/llms-txt-generator.html. It reads your public sitemap, lets you pick what to feature, and hands you a finished file to upload yourself. It also works on any host, which matters here: some hosting quietly blocks the file Yoast tries to write into the site root, so the feature fails with nothing to show for it.
Step two: serve it. On most hosts, drop a plain llms.txt into your web root as a static file and it will be served as text/plain. If a theme or plugin already claims the path, add a small server rule (an Nginx location block, or a redirect in your host panel) that hands /llms.txt to your static file. This is the manual route, and it is the one Yoast’s own developer documentation points to when the generated file is not enough.
Step three: verify it, because a file that exists is not the same as a file that works. Load /llms.txt and confirm three things: a 200 status, a plain-text content type, and no empty links.
One honest note from our own setup: on rerealize.com, which also runs Yoast, /llms.txt returns a 404. We have not published a generated file here at all, because we want the hand-written version. So the “automatic” promise did not even fire on every install we checked. If you are relying on the toggle, confirm the file is really there before you assume anything.
Do Not Treat This as a Ranking Play
It would be easy to oversell llms.txt, so let us be precise. It is an emerging convention, not an established standard. Major assistant makers have not publicly committed to reading it, so any claim that it “gets you into ChatGPT” is unproven. It is not a Google ranking factor.
Treat it as one layer in a stack. The layers that already work come first. Give the crawlers you want clear access in your robots.txt and keep the ones you do not want out, which we covered in our guide to updating robots.txt for AI crawlers. Add structured data so machines get your facts in a format they already trust. If your hosting layer blocks AI crawlers by default, our piece on Cloudflare blocking AI crawlers explains why. Then put a clean llms.txt on top as a human-readable summary. If you are still choosing an SEO stack, our Yoast vs Rank Math vs All in One SEO comparison covers where automation helps and where it does not.
Before any AI assistant reads a summary file, the crawlers have to be allowed in. Our step-by-step guide to a current robots.txt for AI crawlers shows which bots to open and which to block.
Frequently Asked Questions
Yes. Yoast SEO includes a llms.txt feature that generates the file automatically once you enable it, and it is free. The problem is what the file contains. On our audit of a live store, 22 of 41 entries had empty URLs and every key page and product entry was blank.
No. Google does not use llms.txt for search ranking. It is a separate convention aimed at AI assistants, and it is still emerging. Do not expect it to affect your position in search results.
Write a short file by hand and serve it from your site root as a static file, or with a small server rule if something else claims the path. Keep it to a one-line summary, a short list of key pages with one-line descriptions, and a block of core facts such as shipping, returns, contact, and currency. Then verify it returns a 200 status, a plain-text content type, and no empty links.
An auto-generated file with empty links and internal template entries is worse than serving nothing, because it tells AI tools your store is poorly structured. Replace it with a hand-written file, or turn the feature off until you have one ready.
Final Thoughts
Yoast deserves credit for shipping llms.txt at all. Most SEO plugins have not, and the feature is free. We log the problem because the direction is right and the execution is not there yet. A file that leaks internal templates, dumps escaped HTML, and leaves the most important entries blank is not AI discovery. It is a checklist item that looks done.
AI discovery is a curation problem before it is a plugin problem. The value is in the editorial judgment about which pages matter and how to describe each in one line, and that is the part automation cannot do well yet. Write the file yourself, keep it short, serve it from the root, and verify it. That twenty-minute job beats any toggle.
About The Author
