Every other essay in this series is about systems I build for clients. This one is about a small, embarrassing thing I found on my own site, because the fixing of it turned out to be a better argument than anything I could have written from research.
The file is llms.txt. It is a plain-text map of a site, intended for language models — a curated index of what is here and what it is about, in a format designed to be read rather than crawled. I set mine up when I launched, listed the three essays that existed, and then wrote twelve more without ever touching it again. For weeks it told every AI system that asked that I had written three essays.
Nothing broke. No build failed, no test went red, no monitor fired. That is precisely the property that makes this category of work rot, and it is the actual subject of this piece.
Two jobs that used to be one
Search optimisation assumed a human at the end of the query. You ranked, someone clicked, they landed on your page and you got whatever you were going to get. Every incentive pointed at the click.
Increasingly there is no click. There is a model reading a handful of sources and composing an answer, or an agent doing something on a user's behalf. Which splits the old job in two, and the two halves want different things from you.
Being citable is the first: when a model composes an answer in your domain, does your material end up among the sources it draws on, and is it attributed? Being actionable is the second: when an agent arrives to do something rather than read something, can it? Most sites are built for neither, because both were previously subsumed by "rank well and get the click."
The research here is thinner than the marketing around it, but not absent. The clearest work is the Princeton-led GEO paper from KDD 2024, which tested whether content changes shift visibility in generative-engine answers and found they can — reporting gains of up to 40% in visibility from targeted optimisation.1 The authors are careful in a way the industry built on top of them often is not: they note that efficacy varies considerably by domain, which is the difference between a finding and a promise. I would treat the specific number as an upper bound observed under experimental conditions rather than something you should expect.
What I take from it is directional and, I think, safe: the things that make content legible to a model are broadly the things that made it good anyway. Claims attached to sources. Numbers with provenance. Structure that says what a thing is. It is not a hack, which is also why it is not exciting.
What I actually found
So I audited my own site with an unusually specific question: if a machine came here to understand what I have written, what would it find?
Four problems, in ascending order of how much they annoyed me.
The curated map was stale. Three essays of fifteen. Anything relying on llms.txt to know what was here had a picture of my writing from launch week.
There was no structured data at all. Not a single line of JSON-LD. Every essay was a page of prose with a title tag, and nothing anywhere said this is an article, published on this date, written by this person, in this category. All of that was obvious to a human reader and invisible to a parser.
The feed and the sitemap were hand-written XML. Every new essay meant hand-editing two files and hoping I got the date, the weekday in the RFC-822 timestamp, and the escaping right.
And the real cause: essay metadata lived in four places. The index page had one copy, the essay route had another, the RSS had a third, the sitemap a fourth. Four hands, four chances to disagree.
That last one is the diagnosis, and I did not spot it by inspection. I spotted it because I had recently published five essays in a week and edited those same four files five times, which is the kind of repetition that eventually files a complaint. llms.txt did not go stale because I am careless. It went stale because it was the one surface with no forcing function — the other three at least got touched.
What I changed
The fix is unglamorous and took an afternoon, which is roughly the point.
One source of truth. All essay metadata — slug, title, date, reading time, category, the card copy and the meta description — now lives in a single typed array. Adding an essay is one entry there plus the HTML file. Nothing else.
Everything machine-readable is generated from it. The sitemap, the RSS feed and llms.txt are now derived rather than typed, which means they cannot disagree with the site, because there is no longer anything for them to disagree with. The three files that were hand-maintained are gone from the repository.
Structured data on every page. Site-level identity — a Person and a WebSite, linked — in the root layout, and a BlogPosting on each essay carrying headline, description, publication date, author, canonical URL and category. This is the part that turns a page of prose into something a parser can assert facts about.
A richer map for the readers that ask for one. The generated llms.txt now lists all fifteen essays with category, date, reading time and description, and states plainly that the full text is free with no paywall or email wall, that sources are cited, and that essays arguing a position rather than reporting experience say so. If a model is deciding whether to trust and cite something, those are the facts I would want it to have.
One small thing I did not expect to enjoy: the category filter on the writing index is now derived from the data too. It used to be a hardcoded list of buttons, which is how I managed to publish a Note — the first in that category — that was reachable under "All" and under no other filter. The essay rendered perfectly. The category label displayed correctly. Only the filter was wrong, silently, in exactly the way llms.txt was wrong.
And then, while I was proof-reading this essay about four surfaces going stale, I scrolled to the bottom of my own page and found a fifth. The footer has a row of featured essays. It listed Software factories, Loop engineering and Services become software — the same three from launch week, hardcoded, sitting under a heading that implied they were representative. Every visitor to every page on this site had been looking at it. I had been looking at it. It is now the three most recent, derived like everything else, and I have stopped assuming I have found the last one.
The general lesson
I have written a lot lately about gates that quietly stop working — reviewers who approve more and inspect less as the queue grows. This is the same failure with a different subject. A surface that nothing checks will drift, and the drift will not announce itself.
What makes machine-readable files an especially bad case is that the audience cannot complain. A human hitting a broken page emails you. A model reading a stale index simply forms a slightly wrong picture of you and moves on, and you never learn that it did. The feedback loop is not slow; it is absent. Which means the only defence available is structural: derive the thing, or accept that it will be wrong and you will not know when.
If you want the compressed version — anything a machine reads and no human checks should be generated, not written. That is most of what I did here, and I suspect it is most of what "AI SEO" honestly amounts to once you strip out the parts being sold to you.
What to actually do
That is my process, which is not the same as a checklist you can use. So here is the checklist, in three tiers. The first is mechanical and you should simply do it. The second has real evidence behind it. The third is how you avoid taking my word — or any vendor's — for the second.
Tier one: be parseable. None of this is clever and all of it is verifiable.
Server-render your text. If your content only exists after JavaScript runs, assume a meaningful share of machine readers never see it. This is the single largest technical failure and it is invisible in a browser, because your browser runs the JavaScript. Put structured data on every page — an Article or BlogPosting with headline, description, date, author and canonical URL, plus a Person or Organization identifying who is speaking. This is what turns prose into something a parser can assert facts about rather than guess at. Set one canonical URL per page. Keep a sitemap and a feed that are actually current, which for most people means generating them. And publish an llms.txt — a plain-text curated map of what is on the site and what it is for. It is a convention rather than a standard, it costs an hour, and it is the cheapest way to hand a model an accurate picture of you instead of making it infer one.
The crawler mistake. This one gets its own paragraph because it is the most consequential thing on the list and it is routinely backwards. AI crawlers come in two families doing different jobs. Training crawlers — GPTBot, ClaudeBot, Google-Extended — gather material that may be used to train models. Search crawlers — OAI-SearchBot, Claude-SearchBot, PerplexityBot — fetch pages to answer a question a human is asking right now. Blocking the first family is a perfectly legitimate decision about whether your work becomes training data. Blocking the second removes you from citation eligibility altogether. They are frequently swept together into a single "block the AI bots" rule, which gives away the thing you wanted in exchange for the thing you had not thought about. Decide the two questions separately, because they are separate questions. For what it is worth, my own robots.txt named three training crawlers and no search crawlers — harmless, since a wildcard rule allows everything anyway, but it is a fair record of which family I had in mind, and it was the wrong one.
Tier two: be citable. Here the GEO paper is genuinely useful, because it tested nine specific content changes and reported which moved visibility and which did not.
Look at what the winners have in common. Quote the person who actually said it. Give the number. Name the source it came from. Write clearly. Every method that worked is a version of show your work — which is why I called this unexciting earlier and why I am comfortable recommending it. And look at what lost: keyword stuffing, the reflex the previous era rewarded, measured at roughly nine percent worse than doing nothing.1 That is the sentence I would put in front of anyone still running a 2019 playbook. The old tactics are not merely diminished. One of them is now a penalty.
Two qualifications from the same authors, both of which matter more than the headline. Effectiveness varies by domain — statistics performed strongest in law and government material, quotations in history and society, source citation on factual questions — so the ranking above is a starting point, not a recipe. And the finding I would have wanted first: lower-ranked pages gained substantially more than top-ranked ones. If you are already the definitive result for your topic, this is marginal. If you are fourth or fifth, it is the difference between being cited and being summarised past.
Tier three: verify it. Four checks, none of which require buying anything.
Fetch your own page the way a machine does — plain HTTP, no JavaScript — and read what comes back. If your argument is not in that text, nothing else on this list matters. Validate your structured data rather than assuming it parses; malformed JSON-LD fails silently and looks identical to no JSON-LD. Ask the assistants questions in your own domain and note who gets cited and why. It is slow and manual and it is the most direct read available. And instrument your referrals before you change anything, so you have a baseline. I did not, which is exactly why this essay ends without a result.
What I have not done, and what I cannot yet tell you
Two honest gaps.
The first is the actionable half. Everything above improves whether a machine can understand this site. None of it addresses whether an agent could do anything here — subscribe, retrieve a specific essay in a structured form, act on my behalf. There is a live standards conversation about exposing site capabilities to agents rather than making them drive a UI built for humans, and I have deliberately not built on any of it yet. It is not settled enough to be worth the commitment, and I would rather write about it when it is.
The second is measurement, and it is the one that should temper everything above. I do not know whether any of this worked. I have no before-and-after data on AI citations, because I did not instrument it before making the changes — which, given that I have just spent a thousand words on the importance of not flying blind, is a fair thing to hold against me. What I can say is that the site now accurately describes itself, which was not true a week ago, and that this is worth doing regardless of what it does for visibility.
The honest caveat: this is a report of maintenance on a personal site, not a case study with outcomes. The published research on generative-engine optimisation is early, its headline numbers come from experimental conditions its own authors flag as domain-dependent, and a great deal of what is currently sold as AI SEO is extrapolated well past what that work supports. I have tried to keep the claims here to what I did and why, rather than what it will get me. If the citation picture changes measurably in either direction, I will report it here — including if the answer turns out to be that none of it mattered.