I understand why people argue about llms.txt. It is a simple file, so it becomes a simple fight.
One side wants a new standard. The other side opens server logs and says: almost nobody reads it.
I think both sides are touching the same problem, but from the wrong distance.
The public experiments are not very kind to llms.txt as a standalone object. OtterlyAI ran a
90-day test and reported more than 62,100 AI bot visits, with only 84 visits to /llms.txt.
That is roughly 0.1 percent. Other people in SEO are finding the same kind of thing in their own logs:
the major AI crawlers do not seem to treat this file as a required stop today.
Our own older Agntbase server summary was not much more flattering. Out of 5,685 AI bot requests,
/llms.txt was requested 10 times. That is about 0.18 percent. A hub profile JSON route was requested twice.
That is not a number I would ever use to say: the bots are clearly reading everything we publish.
The honest reading is uncomfortable, but useful. A file can be technically correct and still not be part of the normal route a crawler takes. Publishing something at the root of a domain does not mean every AI system will respect it, discover it, or use it in an answer.
I still do not think the answer is: forget machine-readable files, forget the agent layer, wait until the big platforms tell us exactly what to do.
Waiting for perfect proof feels safe, but it can also become a nice way to arrive late. Joost de Valk made a good point in his piece about standards: standards do not prove themselves in an empty room. Somebody has to publish the shape before the ecosystem knows whether it wants that shape.
But that does not mean every new file deserves worship. The file is not sacred. The route is what matters. The maintained business context behind the route is what matters even more.
A business is not a blog post. It is not only a homepage. It is a living thing with a name, locations, products, people, prices, policies, reviews, old mentions, new changes, source conflicts and things it should not claim.
A human can understand a lot of this from context. A human sees the Instagram profile, the Google reviews, the founder on LinkedIn, a few photos, a price page, a complaint, a local article, and somehow builds a feeling. Machines do not have that same native business memory. They need more of it made explicit.
That is where the conversation becomes more serious than llms.txt. The question is not whether one
text file increases AI visibility. I would not expect it to. The question is whether the business has a clear,
maintained machine-readable path to what is official, current, source-backed and safe to use.
I do not mean throwing ten random files on a server and hoping the model becomes grateful.
I mean a route from discovery to understanding. A crawler, assistant, partner system or agent should be able to land on the public site or social profile, find a clear machine-readable entry point, reach the canonical business profile, see which sources support the facts, understand what changed recently, and know what actions are allowed next.
The formats can change. Maybe llms.txt survives. Maybe it becomes a minor hint. Maybe something
else wins. I care less about the filename than about the business having a controlled version of itself that
machines can read without guessing from scattered fragments.
Agntbase has a hub with thousands of company profiles and a growing set of machine-readable routes around it. The early lesson is not romantic. Top-level routes are easier to get read. Deep profile routes are harder. If a profile is buried, weakly linked, or not part of an obvious discovery path, it may sit there quietly.
That changed how I think about the product. A generator of files is not enough. The useful thing is the control room around the profile: source management, freshness, owner-confirmed facts, profile changes, bot reads, failed reads, source conflicts and the ability to update the official version when the public web drifts.
This is also why bot analytics has to be more careful than a single number called AI traffic. GPTBot, OAI-SearchBot and ChatGPT-User do not mean the same thing. A training crawler, a search/retrieval crawler and a user-triggered fetch are different signals. If we mix them all together, we get a big number and very little understanding.
SEO people are right to be suspicious of one-file shortcuts. The web has seen enough of them. But I do not think “AI bots ignore one proposed file today” is the end of the story.
The real movement is bigger. Businesses are going to need a layer where their official facts, sources, reputation signals, freshness and permissions are maintained for systems that do not browse like people. Sometimes that layer will live on the company site. Sometimes it will be connected through a hub, a registry, a commerce platform, an API, a social profile or a partner system.
The important part is that somebody owns it. Somebody has to decide what is true, what is outdated, what is disputed, what is safe to say, what agents are allowed to do, and where the next action should go.
That is the work I am trying to build toward with Agntbase. Not a promise that one file will make a company appear in every AI answer. A maintained business layer that makes the company easier to read, verify, compare and act on when AI systems need a cleaner path than the messy public web.
This note was written after comparing Agntbase hub logs with the current public debate around llms.txt.
The original proposal describes /llms.txt as a way to help LLMs use a website at inference time:
llmstxt.org.
OtterlyAI's 90-day experiment reported 84 /llms.txt visits out of more than 62,100 AI bot visits:
OtterlyAI experiment.
Joost de Valk's argument about standards and adoption is here:
Standards don't prove themselves.
Shopify's own developer forum also reflects the current uncertainty: no major LLM provider has publicly confirmed
that it honors these files during crawling:
Shopify developer discussion.