Developers are changing how they build software, asking agents to plan, create, run, test, and improve real workflows. Web scraping is no exception.
Earlier this year, Zyte launched a Zyte plugin for Claude Code, our latest step toward production-ready, agentic web scraping. The idea was simple: give an AI coding agent the web expertise it needs to move from a prompt to a working Scrapy project.
Today, we’re expanding that approach with three new additions to the Zyte Agentic Web Data line-up:
- Agent Skills - portable Zyte web scraping skills for any compatible AI coding agent.
- Zyte skills for GitHub Copilot CLI - a dedicated plugin for GitHub Copilot CLI users.
- Zyte plugin for Codex - an integration specifically for Codex workflows.
Together, these additions make Zyte’s web scraping expertise available in more of the agentic coding environments developers already use. The new integrations add to the existing line-up:
- Zyte Claude Code plugin - Zyte know-how and skills in a dedicated Claude package.
- Web Scraping Copilot - Visual Studio Code extension that generates extraction logic and helps with project creation right in your code editor.
Three new agent integrations
1. Agent Skills
The biggest addition is Agent Skills.
Agent Skills package Zyte’s web scraping knowledge into portable skills that can be used by any AI coding agent that supports the Agent Skills standard.
Install them with:
1npx skills add zytedata/skillsYou can also find them on:
This is the most agent-neutral way to benefit from Zyte’s scraping code know-how. If your team has standardized on an AI coding agent that supports Agent Skills, you can bring a spider-building workflow into that environment without switching tools.
With Agent Skills, the agent can help you move from a URL and a data requirement to a complete Scrapy project with web-poet page objects, pytest fixtures, and a wired-up spider.
With our skills, you can also simply ask your tools to deploy your spider and run it to get data.
Use Agent Skills if you're looking for a much faster and easier way to build a spider and you're using a coding agent which does not yet have a native integration (Claude Code, GitHub Co-Pilot CLI etc).
2. Codex and GitHub Copilot CLI plugins
The second and third additions are Zyte Agentic Web Data for Codex and GitHub Copilot CLI.
Codex is another important environment for agentic software development. OpenAI’s coding agent that now sits inside ChatGPT. With this integration, developers using Codex can access the same web scraping workflow.
We’re also adding integration with GitHub Copilot CLI. This brings Zyte’s agentic scraping workflow to developers who work with GitHub Copilot from the command line.
The experience is designed for a familiar developer flow: stay in the terminal, point the agent at a site, describe the data you need, and generate a working Scrapy project.
Codex - find out more:
- Zyte docs: Codex plugin
- GitHub: https://github.com/zytedata/codex-skills
GitHub Copilot CLI - find out more:
- Zyte docs: GitHub Copilot CLI
- GitHub: https://github.com/zytedata/copilot-skills
Agentic scraping: The big idea
We first launched Web Scraping Copilot for Visual Studio Code in 2025. That, plus our Claude Code plugin, proved something important: general-purpose coding agents can become dramatically more useful for web data work when they are given specialist scraping knowledge.
A coding agent may know Python, it may even know Scrapy - but production web scraping requires more than generating plausible code. Reliable spiders need the right project structure, extraction patterns, page objects, test fixtures, dependency handling, rendering decisions, anti-bot configuration, and a workflow that lets you inspect and improve the result.
That is what Zyte brings to agentic development.
Instead of starting from a blank prompt, Zyte gives the agent a structured web scraping workflow:
- Explore the target site.
- Define the data you want.
- Propose and refine an extraction schema.
- Generate an inspectable Scrapy project.
- Run, test, deploy, and iterate.
You still get the speed of an AI agent. But you also get the structure, control, and reliability developers need when the output matters.
How it works
When you use Zyte’s agentic web data workflow, the agent can create two key artifacts:
- An extraction spec - This captures the schema, saved pages, expected values, and decisions made during the exploration and validation process.
- A generated Scrapy project - This includes web-poet page objects, pytest fixtures, and a spider ready to run locally or deploy to Scrapy Cloud.
This separation matters. The extraction spec gives you a durable description of the data you want, while the generated Scrapy project gives you inspectable code you can run and maintain.
That means the workflow is not just “ask an agent, get some output”. It is a path toward a real web data project.
Enhancements to skills
We have been improving the skills themselves, across all integrations.
Recent updates include:
Smarter Scrapy Cloud runs
When a Scrapy Cloud job starts, the skill can now wait for it to finish, report the results, and, if the output looks wrong, attempt to fix the problem and rerun the job. That means the workflow is moving beyond code generation toward run-check-improve loops.
More efficient extraction from list pages
If all required fields can be extracted from list pages, the skills can now generate code that does that without requesting detail pages unnecessarily. That can make spiders faster, simpler, and more efficient.
Better login handling
The Zyte login flow now uses OAuth and stores credentials in the project’s .env file for reuse by scraping skills. That reduces setup friction and makes repeated workflows easier.
More reliable Scrapy Cloud deployment
The skills now generate frozen requirements.txt files where needed, point scrapinghub.yml at them, and provide clearer guidance for refreshing dependencies. They also include smarter Scrapy stack selection and more troubleshooting guidance. In practice, this helps reduce the gap between “the agent generated a project” and “the project runs reliably in the cloud.”
The next step for agentic web data
The first wave of AI coding assistants helped developers write code faster. The next wave is about agents that can plan, execute, test, and improve real workflows.
For web data, that means moving from “generate me a scraper” to something more useful:
- Describe the data.
- Validate the output.
- Generate the project.
- Run it.
- Improve it.
- Operate it.
That is the direction we are building toward with our Zyte Agentic Web Data line-up.
And with Agent Skills, GitHub Copilot CLI, and Codex support, that workflow is now available in more places developers already work.
