llms.txt isn’t dead: How we put dev docs in AI’s spotlight

It didn’t take long for the future of AI visibility to be written-off.

In September 2024, Jeremy Howard proposed llms.txt, a way to provide large language models (LLMs) with a raw Markdown “map” of a website’s content.

Fast-forward to 2026, however, and the search engine optimization (SEO) community has decided: “llms.txt is dead”, “LLMS.txt is freaking BS and nobody needs it”.

Sure enough, household-name LLMs (the chatbots you and your family use) do not discover or use llms.txt today in any meaningful way.

So, why did my team lean hard into llms.txt this year? Let me tell you why we decided to use a standard that some people already consider dead.

The problem we wanted to solve

The main issue was simple: we wanted developers using AI-assisted coding tools to draw on the latest docs for our products, instead of guessing from training data.

Indeed, while llms.txt may now look ineffective for SEO, developer tooling - like AI-powered IDEs, MCP and Claude Code - relishes the opportunity to ingest clean Markdown.

We wrote recently about how coding assistants tend to fall back to older, better-known patterns instead of the latest recommended ones. That is especially noticeable in technical products, where a suggestion can be close enough to look right while still relying on deprecated APIs or outdated setup steps.

To give users up-to-date super-powers, coding assistants need to ingest a lot of documentation quickly. Of course, Zyte API’s docs are available in HTML, which is great for people - but not ideal for LLMs. An LLM does not need the menus, sidebars, scripts, and page furniture; it needs the core content.

So our goal was to keep the human-facing docs as they are, while also exposing cleaner representations that are easier for LLM models to consume.

How we implemented it

The good news is that our docs setup made this fairly approachable.

docs.zyte.com runs on Sphinx, a documentation generator for Python; we use the same tool for our open source tools. Because we already had a shared Sphinx-based docs stack, adding an initial LLM-friendly rendering path was not too hard.

We used the existing sphinx-markdown-builder and sphinx-llms-txt plugins to produce the following:

1# Zyte
2
3> Zyte documentation for web scraping tools and services: Scrapy Cloud (usage and API reference), Zyte API (usage and reference), Zyte Data schemas and extraction guides, tutorials, integrations, and developer resources for building and running scrapers.
4
5## Docs
6
7- [Zyte documentation](index.md)
8- [Get started with web scraping](web-scraping/get-started.md)
9- [Web scraping tutorials](web-scraping/tutorials/index.md)
10- [Web scraping tutorial](web-scraping/tutorials/main/index.md)
11- [Start a Scrapy project](web-scraping/tutorials/main/setup.md)
12- [Deploy and run on Scrapy Cloud](web-scraping/tutorials/main/cloud.md)
13...
14...

We did not change the normal HTML docs layout. The goal was not to rewrite the web docs for LLMs. It was to publish the same content in formats that are less painful for LLM-based tools to read.

That initial implementation was good enough to ship, but not good enough for us to stop there. We were not entirely happy with the results. So we did what open source users tend to do: we sent fixes upstream.

In particular, we opened pull requests against the tools we were using:

• sphinx-markdown-builder PR #49
• sphinx-llms-txt PR #69

Hopefully those improvements will get merged upstream, because this is the kind of thing that should be easy for other docs teams to adopt too.

Why this matters for our AI tooling

This work is not just about documentation formats. It feeds directly into how we build AI tooling around our products.

One example is our Claude Code skills for Zyte API. We are trying to keep those skills as light as possible. One way is to put as much product knowledge as we can into the docs instead. That gives us two benefits:

1. The skills stay smaller and easier to maintain, while still being able to guide the model toward the right official documentation.he docs improvements are not limited to one integration.
2. Alternative skills, other coding assistants, or even raw LLM workflows can benefit from the same documentation surface. And when our products or open source tools change, updating the docs updates the source of truth for all of them.

This is also where automated checks become useful. As we build out these workflows, we use automated checks to look at things like time to complete tasks and cost, and to check whether the model is actually reaching the official docs most of the time. That helps us measure whether the approach is practical, not just whether it sounds nice in theory.

What we are seeing so far

The results so far are mostly qualitative, but still useful.

When we give LLM models and coding assistants that use them a cleaner way to reach the current docs, they are less likely to fall back to outdated patterns. That does not eliminate mistakes, but it changes the type of mistake. Instead of inventing an answer from stale memory, the model has a better chance of grounding itself in the docs we actually maintain.

That matters because many bad AI coding suggestions are not obviously broken. They are almost right. They use an old import path, an outdated argument, or an integration pattern that used to work. Those are exactly the kinds of mistakes that become less common when current docs are easier to consume.

What comes next

The next step for us in terms of documentation, based on what we learn from those automated checks, is to start making content improvements that benefit both LLMs and humans.

If those checks show that a page is unclear, too verbose, or hard for a model to navigate, that is a useful signal. We can fix the docs once and let every tool that relies on them benefit from the improvement.

So yes, in one sense llms.txt may still be a zombie standard. LLMs do not seem to go looking for llms.txt by themselves today. But in our case that is fine, because we are already building the tooling layer that can point them there.

If that means models spend less time fighting HTML and more time reading the docs, that is already a win.