Built for real-world scraping
Bring web data extraction into your IDE or coding assistant.
Access, extract, and structure data as part of your normal workflow.
Claude tries to write a Scrapy spider. With the Zyte skill, it gets it right.
Bare Claude knows Python. It doesn't know Scrapy the way Zyte engineers do. The plugin teaches it — async patterns, scrapy-poet page objects, the right libraries, real tests, anti-bot wired in. The spider that comes out of the prompt actually runs. Describe data. Get data.
Your agent on its own
Generic AI coding assistant
With Zyte skills
Domain-expert AI system
A spider that runs first time
Get a runnable Scrapy spider for a new site without correction turns.
Gets modern Scrapy patterns wrong
Writes simple & non testable code, and fails to include key addons.
Spider runs first time.
Correct async Scrapy patterns, scrapy-poet page objects, targeted selectors, testable code with pytest.
Code that holds up to review
Extraction shaped like a senior scraping engineer would write it.
Best-guess extraction
Often decides your data shape for you, and hallucinates selectors, leaving you with questionable quality.
Built to Zyte's quality bar
Code shaped like the spiders Zyte engineers ship: accurate fields, clean patterns, scalable production spiders.
Anti-bot handled out of the box
No 403s and no unblocking layer to write yourself.
No anti-bot solution
You wire one in by hand or watch the spider fail at the first ban.
Zyte API wired in by default
Bring your own if you prefer, but it works out of the box.
Testable code generation
pytest to guarantee your code.
Doesn't know how to create HTML fixtures.
No tests, or tests built.
Tests scaffolded with the spider
pytest tests scaffolded against saved HTML fixtures. Tests fail? Prompt a fix and run again.
Scrapy stack awareness
The right library at the right version, wired through correctly with our most important addons.
Produces generic Scrapy.
Doesn't know scrapy-poet, zyte-common-items, scrapy-zyte-api, or the addons system.
Reaches for the right library and version
Wires them through the addons system, not legacy middleware.
Adding 15 years of web scraping expertise to your AI workflow
Claude Code Plugin
Agent Skills
Github CLI Plugin
Web Scraping Copilot
Codex Plugin
From installation to working spider in minutes
Claude Code Plugin
See how the Zyte Claude Code plugin turns a website URL into a working, tested Scrapy scraper in under 15 minutes — ready to deploy and run.
Web Scraping Copilot
Web Scraping Copilot is built to reduce friction between idea and extraction. No context switching. No black boxes. No hidden automation.
Explore the documentation
Get started with Zyte plugins, from setup to building production-ready scraping workflows.
Insights from web data experts
Explore opinions, lessons, and emerging approaches to extracting and managing web data at scale.
Introducing Web Scraping Copilot 1.0
FAQ - Everything you need to know
What is Zyte's coding agent add-on for web scraping?
A set of skills that teach coding agents how to build production-ready web scrapers. Point an agent at a URL, describe the data you want, and it explores the site, proposes a schema for your approval, then generates a complete Scrapy project with web-poet page objects — no manual scaffolding required. It's available in three forms so you can use it in whichever coding agent you already work in:
Claude Code plugin — for Claude Code specifically.
Universal Skills — the same skills, packaged to the open Agent Skills standard, so they work with any coding agent that supports it (Cursor, Codex, Cline, and others).
GitHub Copilot CLI plugin — the same functionality, packaged for Copilot CLI.
Which coding agents are supported?
Claude Code is supported natively via the Claude Code plugin. GitHub Copilot CLI is supported natively via its own plugin. For any other coding agent that supports the Agent Skills standard, install the Universal Skills package — the underlying skills are identical, only the packaging differs.
What's the difference between the Claude Code plugin, Universal Skills, and the Copilot CLI plugin?
They all run the same underlying skills and produce the same output — a working Scrapy project with page objects and tests. The difference is purely packaging and agent-specific optimizations (e.g. usage of sub-agents):
Use the Claude Code plugin if you work in Claude Code.
Use the Copilot CLI plugin if you work in GitHub Copilot CLI.
Use Universal Skills if you work in any other coding agent that supports the Agent Skills standard (or if you want one install that isn't tied to a specific agent).
You won't get different results by picking one over another — pick based on which agent you already use.
Who is it for?
Built for developers and data teams who need reliable, production-ready web data fast, regardless of which coding agent they've standardized on. It's especially useful if you want Scrapy + web-poet scaffolding without writing boilerplate from scratch, a reusable extraction spec before committing to any specific implementation, or a "data-first" workflow where you validate what you're extracting before generating code.
What do I need to get started?
That depends on which package you use:
Claude Code plugin: Claude Code (CLI or desktop app) and the uv Python package manager (the plugin will prompt you to install it if it's missing).
Copilot CLI plugin: GitHub Copilot CLI and uv.
Universal Skills: a coding agent that supports the Agent Skills standard, and uv.
A Zyte API key is recommended for all three — it handles fetching and rendering pages — but isn't strictly required to run any of them.
Is a Zyte API key required?
No, all three can run without one. We recommend using the Zyte API for fetching and extraction, though — it handles JavaScript rendering, anti-bot measures, and reliability, giving you significantly better results on complex sites.
Are the plugins free to use?
Yes, all three are free to use. As above, we recommend pairing them with the Zyte API for fetching and extraction, which is billed separately based on usage.
How does the workflow actually work?
The process has two clearly separated stages, regardless of which agent you're using. First, the extraction spec: the agent fetches pages, discovers fields, and iterates with you until you approve the schema and expected values. Second, code generation: the approved spec is used to produce a complete Scrapy project (page objects, test fixtures, spider).
This stage does not crawl the web — it works entirely from the outputs of stage 1. Crawling only happens when you run the generated spider locally, or deploy and run it on Scrapy Cloud.
What does the output look like?
You get two artifacts: a .scrape/{site-name}/ folder containing the extraction spec (schema, saved pages, and approved expected values), and a fully generated Scrapy project with web-poet page objects, pytest fixtures, and a wired-up spider ready to run — including an option to deploy to Scrapy Cloud. This is identical whether you generated it from Claude Code, Copilot CLI, or another agent via Universal Skills.
Can I use the extraction spec without the generated Scrapy code?
Yes. The spec is a technology-neutral artifact that isn't tied to any framework or agent. It can feed a plain-Python implementation, direct Zyte API extraction, or a fully manual process. The generated Scrapy project is one possible consumer — not the only one.
How does it handle websites that change over time?
The spec is the durable part of your project. When a site changes, you re-run the spec stage (or update field selectors in the page objects) without discarding the rest of your project structure. The deliberate separation of spec from generated code is what makes the system resilient to change.
Does it work with JavaScript-heavy sites?
Yes. Page fetching and rendering is delegated to the Zyte API, so JavaScript-rendered content is handled transparently — no extra configuration needed, on any of the three packages.
_HFpro5d6k3.png&w=256&q=75)
_E4PyVpfAxa.png&w=256&q=75)
-(1).png&w=1920&q=75)
-(1)_VZGHqxCgXV.png&w=1920&q=75)
