Claude Code can help you write Python - but it doesn’t naturally boast the hard-won, opinionated web-scraping know-how that makes spiders reliable: the Scrapy patterns, the project structure, page objects, fixtures, and smoke tests that make the spider actually run.
Today, we’re excited to release Zyte Web Data for Claude Code: Zyte’s official Claude Code plugin that takes you from a plain-English prompt to a working Scrapy spider.
What is it?
Zyte Web Data for Claude Code is a Claude Code plugin that generates production-ready Scrapy spiders with web-poet page objects from natural-language instruction to run and extract data. Describe data, get data.
Give it a URL and describe what you want to extract. It handles:
- Site exploration.
- Schema discovery and approval (you confirm the fields before code generation).
- Code generation (project scaffold, page objects and spider wiring).
- Smoke testing so you get a runnable project, not a snippet.
Optionally, you can deploy directly to Scrapy Cloud for scheduled runs, job history, and monitoring.
The problem we’re solving
The friction in web scraping is rarely the first 30 minutes. It’s the next 30 days:
- You want repeatability, not a one-off answer.
- You want inspectable code, not a black box.
- You want a pipeline you can deploy, monitor, and maintain.
Most agentic coding assistants can generate something Scrapy-shaped. But they often miss the details that matter in real projects, like correct base classes, coherent structure across files, tests, monitoring, and the practicalities of web access (rendering, anti-bot, and configuration).
Our view is simple: Claude alone will try to write Scrapy spiders - but it gets important details wrong. With Zyte Web Data skills, it gets them right.
With the combination, we can create agentic web data workflows that can be industrialized.
Most valuable use cases
This release is especially useful for teams who:
- Build and maintain multiple spiders (and want consistency across them).
- Need fast iteration on requirements (fields, edge cases, pagination, variants).
- Care about production readiness: reliability, auditability, and maintainability.
- Want to shorten the distance between “idea” and “job running in the cloud”.
Common examples:
- Product catalogs and e-commerce monitoring.
- Competitive intelligence feeds.
- Marketplace inventory and pricing.
- News/article monitoring (with structured extraction).
How does it work?
The plugin is packaged as a set of 14 reusable skills. The main one is /scrape, which orchestrates a five-stage pipeline automatically:
- Decide which fields to extract (
/scrape-define) - Analyze the website (
/scrape-spec) - Create the Scrapy project (
/scrape-ensure-project) - Generate the extraction code (
/scrape-codegen) - Generate the spider (
/scrape-create-spider)
When the pipeline completes, you have a runnable spider and a passing-test suite:
uv run scrapy crawl <spider_name> uv run pytest fixtures/
The beauty of the environment, however, is that you don’t have to call any of these skills explicitly - Claude Code will figure out for itself when to invoke them based on the context of your task.
Zyte’s 14 skills span the entire scraping workflow…
Orchestration
| Skill | Description |
|---|---|
scrape |
End-to-end web scraping workflow — from URL to working spider with web-poet page objects |
Pipeline stages (called automatically by /scrape)
| Skill | Description |
|---|---|
scrape-define |
Quick schema definition: explore one detail page, discover fields, fast approval loop |
scrape-spec |
Explore diverse pages and validate the extraction spec: downloads pages, compares variants, optional browser review |
scrape-explore-site |
Explore a website to find and save diverse pages (start, list, detail) with classified links |
scrape-analyze-page |
Extract all available fields with values from a detail page |
scrape-ensure-project |
Ensure a Scrapy project exists with scrapy-poet and Zyte API support |
scrape-codegen |
Generate web-poet page object code from an extraction spec |
scrape-codegen-analyze |
Analyze an HTML page to produce field extraction instructions for code generation |
scrape-codegen-generate |
Generate web-poet page object code from per-page extraction analyses |
scrape-create-spider |
Generate a Scrapy spider that wires page objects together |
Utilities
| Skill | Description |
|---|---|
| scrape-add-page-object | Add an empty web-poet page object to a Scrapy project |
| scrape-review-schema | Generate an HTML review page for schema and extracted data verification |
Deployment
| Skill | Description |
|---|---|
| scrape-scrapy-cloud | Deploy projects, schedule spiders, list/stop jobs, and view items or logs on Scrapy Cloud |
| scrape-zyte-login | Set up your Zyte account and credentials |
Who is this for?
Zyte’s Claude Code plugin is for:
- Web scraping engineers who want to move faster without lowering standards
- Data engineers who need a repeatable pipeline, not a brittle script
- Developers who occasionally need web data but don’t want to become scraping specialists
- Teams building agentic systems that still require reliable web data as an input layer
If you’ve ever thought “I don’t want a demo - I want a spider that runs,” this is for you.
Try the plugin now
To install the plugin, run:
claude plugin marketplace add zyte-ai/claude-skillsclaude plugin install zyte-web-data@zyte-ai
If Claude Code is already running, reload plugins in the active session:
/reload-plugins
After installation, quick-start:
/scrape https://books.toscrape.com/ products
Finding further info
Docs and install details are online now:
- GitHub repo: https://github.com/zytedata/claude-skills
- Claude Code docs: Discovering and installing plugins
If you hit an issue (unexpected prompts, excessive wall time/cost, broken flows), please open a GitHub issue with enough detail to reproduce - feel free to anonymize target sites/data.
What’s next
This is the first step in a larger end-to-end agentic workflow for web data.
Next, we’ll continue expanding distribution and integrations so the same capabilities can be used in more agent environments, beyond Claude Code, while keeping the core principle intact:
Across all these surfaces, the aim will be the same: Describe data. Get data. But also: ship something you can run again tomorrow.
.png&w=3840&q=75)
