A pigeon balanced on a tension wire, illustrating an uncertain but not implausible standard

Every proposed standard has to earn its adoption. Some do, others sit in limbo for years.

llms.txt is a plain text file, proposed by data scientist Jeremy Howard, that lives at yourdomain.com/llms.txt and gives an AI system a short, markdown-formatted summary of your site: what it is and where to look for more. Instead of the AI guessing at your site’s structure by crawling every page, you hand it a roadmap.

That’s the sales copy. Last year, Google’s John Mueller said no AI system currently uses it. Flavio Longato, LLM strategist at Adobe, audited 30 days of raw server logs across 1,000 domains. GPTBot, ClaudeBot, and PerplexityBot never requested the file. Not once. The traffic that did show up was mostly Google’s crawler and SEO tools, nothing resembling an LLM fetching context.

SEOs argue about this, not always politely. Some call maintaining llms.txt “parroting myths.” Others point out Perplexity doesn’t even respect robots.txt, so expecting it to honor a newer, unofficial file is optimistic. One claim floating around: Claude is the only major LLM that’s actually adopted it, though that doesn’t jibe cleanly with Longato’s own logs.

Here’s the case for it. Crystal Carter at Wix crawled over 1,400 llms.txt files and found real, deliberate use among major players. Anthropic has maintained one since 2024. Google’s Agent Development Kit publishes one so coding tools can use it as context. Weather.com uses one to give agents correct URL structure, and Perplexity has been found citing the file directly, independent of a regular search result.

Broad LLM training crawlers mostly ignore this protocol. Specific agentic tools (some built by the same companies publicly skeptical of it) are actively using it for grounding and instruction.

This is a 600 lb pigeon on a tension wire. We’ve been here before, and the SEO community knows it: everyone said the same thing about robots.txt, right up until suddenly it wasn’t optional anymore.

Not every proposal follows that path. Google Authorship was a real initiative, backed seriously, that died a few years in. And humans.txt, a smaller proposal meant to credit a site’s creators, never became a standard either. It just never went away. You can still find it in the wild with a simple search operator, sitting on sites that adopted it a decade ago and never took it down. llms.txt could go any of those three ways. Nobody knows which yet.

It’s cheap to create and costs nothing to maintain, so there’s little reason not to have one. Treat it as a low-cost bet with genuinely mixed early evidence, running alongside the sitemap and schema work we know matters regardless of how this plays out. But llms.txt is not a strategy.

Post 5 in a series on what AI agents mean for websites. Post 4 covered sitemaps and schema. Next: the agentic version of llms.txt, and what it means to move from machine-readable to machine-actionable.