Treat your AI skills like software, starting with evals

We test our AI scraping skills like software. Here's how, and why.

I asked Mikhail to share his perspective on what I should create, as a developer advocate, that would benefit the web scraping community the most.

Mikhail is a Tech Lead at Zyte. He is the person people mean when they say "the innovator."

He laughed and said he did not know. Everything right now is AI, only AI. Something interesting, then. Something non-AI. And it is hard to even think of something like that.

That melted me, because it is exactly why I had gone to him. I was exhausted by AI slop, and I went looking for the one person who might point me somewhere else, and his honest answer was that the somewhere-else has itself become the hard thing to find.

What followed was not planned. The conversation emerged out of pure curiosity about the why behind the skill we built. I asked him to help me deeply understand how it was created, from the first idea to every decision taken along the way, and the reasoning behind each one. Why this shape? Why these building blocks? What were the calls the team made, and what were they trading off?

The web scraping industry has spent the past year talking about one word: access. Give an agent access to the web, point it at a page, let it fetch what you need. Every vendor has a version of that pitch. It is a valid pitch. We launched our own Model Context Protocol (MCP) server too. But I am still stuck on the skills, the process of how we built and why the output of this skill is- production-grade level code. Yes 🙂

Zyte took a different route. Instead of a thin layer over web access, the team built a Claude skill that generates the whole thing: an end-to-end, production-grade scraping project, assembled the way an experienced scraping developer would assemble it, out of reusable building blocks. I wanted to understand why. Not the marketing why, the engineering why, the decisions sitting underneath it.

And the reason that skill can be trusted, the thing Mikhail kept circling back to, is not the part anyone demos. It is that- Zyte tests it like software.

Soak that in!

and let me explain.

The bet Zyte made that the rest of the industry didn't

Start with what the skill actually is, because its shape explains everything that follows.

When you run Zyte's official Claude Code skills, a prompt like "scrape this site" does not hand you a one-off script. It walks the same path a careful developer walks: separate the crawling from the extraction, use page objects so the two stay decoupled, deploy to Scrapy Cloud, and reach for Zyte API when a site starts throwing bans. Mikhail described the goal plainly.

What we wanted was an end-to-end solution. You write a prompt, and you get the solution in the end. And we wanted this solution to follow the patterns we follow internally. It should separate crawling code from extraction code cleanly, which is why we use page objects. We also teach the agent how to use our tools: how to deploy to Scrapy Cloud, how to use Zyte API if you face bans. It's all opt-in.

That last phrase matters more than it looks. Nothing is forced. The skill is not a black box that takes over your project. It is a set of production patterns the agent can pick up, the same patterns a scraping team would reach for on a project that has to keep running after the demo is over.

Underneath the end-to-end path sits a second idea that turned out to be the real engineering work: building blocks.

The idea was to figure out the building blocks, and then for those building blocks, make sure they work well. We try to optimize their quality and their cost.

A building block is a small, reusable job: turn a page into a schema, extract fields from HTML, schedule a spider, handle a block. Each one can be made reliable, fast, and cheap on its own, then composed into the larger workflow. That framing is what separates a skill you can maintain from a clever prompt you got working once. And the only way to know whether a building block actually works well is to measure it. Which is where the conversation turned.

The thing almost no one is testing

Here is the shift that Mikhail kept circling back to. In classical machine learning, evaluation was unavoidable. You collected a dataset, split it into training and test sets, trained a model, and checked it against the data it had never seen. You could not really skip the evaluation step, because the dataset was already sitting there in front of you.

Agents removed the dataset. You write a prompt, describe a task, and the thing works. The barrier to building dropped through the floor, and something quietly went with it.

It doesn't mean you don't need evaluation. It just means evaluation doesn't come as naturally. And it seems like many people just skip it. Almost everyone, I think, skips evaluation.

He is right, and you can check it yourself. Open the popular public skill repositories and look for the tests. Most of them do not have any. The skill is a folder of instructions, and that is the whole artifact. It works, it ships, and nobody knows what would break if they changed a line of it.

That is a fine way to build something for yourself. It is a poor way to build something a team has to maintain, which is the distinction Mikhail drew that has stayed with me since.

These skills are also software, but a very different kind of software. You need some equivalent of unit tests and integration tests for them, especially if you want to maintain them long term and develop as a team. You make some changes, and you have no idea whether it breaks something or not.

An AI skill fails differently from a normal function. A function throws an error, a test goes red, and you know. A skill can degrade in silence. It can stop triggering, ask for information it already had, leak a credential into a transcript, or return a plausible answer that is quietly incomplete. If you have no evaluation around it, every edit is a guess.

A skill has two jobs, and both can fail

The first job is one most people never think to test: the skill has to fire at the right time, and stay quiet at the wrong time.

Zyte calls these trigger evaluations. You write a set of prompts you expect to invoke the skill, and a set you expect not to.

If the user asks to deploy the Scrapy spider to Scrapy Cloud, we should invoke the skill. If the user asks what Scrapy is, we should not invoke the skill.

The second is a request to do a job**.** The first is a request for an explanation. A skill that triggers on both makes every casual question heavier and slower; a skill that triggers on neither is dead weight. When a trigger eval fails, the fix is usually the skill's description, the short piece of text the agent reads to decide whether this skill is relevant. You tighten it, rerun, and watch the positive and negative prompts sort themselves correctly.

The second job is the obvious one: once the skill fires, does it actually complete the task?

Zyte's task evaluations run the real agent harness against a fixture project, simulate a developer typing a real request, and then inspect what happened. Was the right skill invoked? Did the agent invent a question about the project ID when it could have read it from the config? Were there permission errors? Did the final result match what the project actually contains? This is closer to integration testing than to prompt tuning, and for production scraping that is the correct level of seriousness.

The failure that should scare you

If you want the single example that made the whole idea click for me, it is this one. Ask an agent to extract structured data from a page, and it will hand you something that looks completely correct.

If you just ask the agent, usually it gives you very plausible results. But if you actually check it on many websites, usually it's incomplete, and sometimes it's semi-random. You ask it once, it returns 10 fields. Second time, 12 fields. The third time, 10 fields, but different fields from the previous run.

Sit with that for a second. The agent is not crashing. It is not returning an error you can catch. It is returning a clean, believable product record, and the set of fields inside it shifts run to run. That is the failure mode nobody puts in a demo, because in a demo it looks fine.

The cause is mundane and worth knowing.

The HTML files are quite large, and the model usually avoids reading them in full. It's very hard to make a model read the whole HTML file. Instead it searches for some strings and reads only parts of the file, and because of this, it can quite likely miss information that's there.

A real product page is enormous, most of it navigation, scripts, and styling noise. Faced with all that, the model does the reasonable-looking thing and skims, searching for the strings it expects, reading around them, and stopping. Anything that lives outside the neighborhood it happened to look in gets missed. So the building block Zyte is optimizing is not "can the agent extract a price." It is "can the agent extract every field, on this page and the next 200, the same way each time." That means pushing the model to read enough of the file, preprocessing the HTML to strip the noise before it ever reaches the model, and then measuring completeness across many sites rather than eyeballing one good result.

For production extraction, completeness and consistency are the whole game. A plausible answer that is quietly missing three fields is worse than an obvious failure, because the obvious failure gets fixed and the plausible one ships.

Run the same task without the skill

This was the part I found genuinely clever, and it is the cheapest test in the whole setup. To know whether a skill is worth having, Zyte runs the exact same task with the skill removed.

It's really important to evaluate without the skill as well, because that's a pretty strong baseline.

Modern agents are capable. Hand one a project folder, a shell, and some documentation, and it will often figure out a surprising amount on its own. So the question is not "can the skill do the task." The question is whether the skill beats an agent that has no skill at all. Better can mean more reliable, cheaper, faster, safer with credentials, or simply more consistent. Sometimes the honest answer is that the skill is not helping.

Often you just need to drop a lot of stuff from the skill, because some things can make the skill work worse than if you just let the agent do it.

A skill that has to justify its own existence against a strong baseline is a skill you can trust. And the baseline exposes things a happy-path test never would. The no-skill agent frequently completes the task, then does something you very much do not want on the way.

You don't want the Scrapy Cloud API key to be visible to the agent anywhere, because if it appears, it can end up in the output, stored on local disk, and it can leak.

The agent needs the key to run the job, so left to its own devices it goes hunting, finds it, and reads it straight into the transcript. Because let's accept it- agents love API keys. Same successful outcome, a credential quietly exposed. That is exactly the kind of difference you only catch when you are comparing with and without, on purpose.

The model is only half the story

The instinct, when something is inconsistent, is to blame the model and reach for a smarter one. Mikhail's correction here reframed the problem for me.

It's not just the model. It's also the harness, the agent that runs it. The same model in Claude Code and Codex can work differently because they have different tools and different system prompts.

So Zyte evaluates across models and across harnesses, because behavior is a product of both. And the choice of which model to evaluate against is itself a deliberate decision. The team runs against Sonnet, not the strongest model available, on purpose.

The reason we run against Sonnet is that we expect it to be the lowest bar. If things work with Sonnet, they'll also work with Opus.

A very capable model can paper over vague instructions by inferring what you meant, which hides the weakness instead of revealing it. A realistic but not superhuman model fails in the exact spots where the skill is underspecified.

It probably fails because the instructions aren't clear enough. So we go back to the skill, change the instructions, and rerun to see if the change helped.

There is a balance, and Mikhail was careful about it. Test against a model too weak for anyone to actually use, and you overcorrect, stuffing the skill with instructions that bloat it and can make a genuinely smart model perform worse, since it never needed the hand-holding and now burns tokens reading it. The target is a model people really run.

The last piece is how you grade a run at all, given that agents are non-deterministic and the same task can be solved five different ways. Hard-coded assertions get brittle fast, so Zyte uses an LLM judge: normalize the transcript, hand it to a separate model, and ask one precise question. Did the final message report the scheduled job with the correct ID? Did the API key appear anywhere it should not? Each check is its own pass or fail, and they add up to a report. The judge does not need to be the smartest model in the room.

We usually run the evaluation itself on Sonnet, and for the judges we use Haiku to save cost, because it's good enough.

What this actually proves

I went into that conversation wanting to understand a design decision. I came out with something more useful, which is a way to tell a serious skill from a demo.

AI is not removing the expertise that scraping developers spent years building. It is relocating it. Some of that judgment still lives in code, and some of it now lives in skills: the patterns, constraints, and recovery strategies that teach an agent to behave like someone who has shipped scrapers before. That relocation is exactly why evaluation matters more now, not less. If expertise is the thing you are packaging, evals are how you keep it honest.

Zyte's evaluation setup is internal for the moment. There is a real reason for that, and Mikhail was straightforward about it.

There's nothing secret in it. We could open-source it if needed. It works, we use it internally, it's just not separated out yet.

So take this as the behind-the-scenes account rather than a tool you can install today. What it shows is the posture: while a lot of the industry ships the skill and moves on, Zyte team writes the trigger evals, the task evals, the security checks, and the no-skill baseline, then runs them across models and harnesses and reruns them every time they touch the skill.

Which leaves me with the question I keep coming back to. If a skill has no evaluation behind it, how would you tell the difference between one that works and one that only worked the day someone demoed it?

If you are building scraping workflows with agents and want the access layer to be the boring, solved part, you can start a free Zyte API trial and put your own building blocks to the test. The extraction is where the interesting failures live. That is where the evals belong.