AI-assisted data extraction
Articles from the Zyte blog about AI-assisted data extraction.
Self-healing spiders: letting an agent fix your Scrapy crawl, with a lid on it
Two demos from our Extract Summit Austin workshop: an agent repairs a Scrapy spider after a site redesign, then heals a live crawl mid-run through Spidermon and the Scrapy MCP, with the measured costs and the mistakes that shaped the design.
The anti-ban arms race: Agentic traffic and the fight for the open web
As AI agents transform how we interact with the web, anti-bot systems threaten to fence off the digital open range.
How to convert a website to Markdown for AI: 7 methods compared
Seven realistic ways to turn web pages into clean Markdown for AI - what each method covers, where each breaks, and how to choose for training, RAG and agent pipelines.
Introducing Zyte MCP: Powerful web data gathering for your agent
Zyte MCP brings live web access, structured extraction and Scrapy Cloud operations into coding agents such as Claude Code, Cursor and Codex.
AI web scraping and scalable data collection
AI changes extraction, adaptation and the cost of adding a source. It does not change access. What is genuinely different, with the arithmetic.
New Zyte add-ons: Agent Skills, Codex, GitHub and more updates
A host of additions to Zyte’s agentic scraping toolbelt helps developers go from prompt to working spider code and beyond.
Claude Fable 5 is the new best model for writing scrapers
We ran nine models, including the new GPT-5.6, through our Zyte Scraping Code Benchmark. Claude Fable 5 puts Sol and others in the shade - but it’s pricey, and the best extraction code still depends on the best infrastructure.
The harness matters more than the model - Podcast EP07
"The model is the engine — but the harness is everything else." In Episode 7, we dig into why the infrastructure layer around your AI model matters more than the model itself, rank the best models available right now, and ask whether the open-weighted revolution is about to make frontier subscriptions obsolete.
What's becoming of web scraping developers in the age of AI agents?
AI agents can generate code, suggest selectors, and draft crawl logic. What they can't do is design the system that decides when to stop, what to trust, and how to recover when the web pushes back. That job still belongs to a human.
What multi-agent orchestration looks like in a large-scale web scraping project
Multi-agent orchestration is having its moment. The diagrams are everywhere now. Boxes for planners, boxes for hands, boxes for daemons, arrows to a shared brain, a human floating at the top. They keep getting prettier. The part where the web pushes back is still the part nobody draws.
I built scraping agents for 30 days - here’s what I learned
For the last 30 days, I did one thing almost exclusively: I built scraping systems with AI agents, from the ground up, across real targets, with real deadlines. Not prototypes designed to impress in a demo, not isolated experiments running against a toy website, but production-grade pipelines that needed to ship and keep running.
The Scrapy whisperer: Adrian Chaves on Web Scraping Copilot
An interview with Scrapy maintainer Adrian Chaves on Zyte’s Web Scraping Copilot, AI-generated parsing code, and building reliable scraping workflows.
More articles on AI-assisted data extraction
- Code is cheap, show me the talk: How copilots are re-engineering developers
- Claude skills, MCP or Web Scraping Copilot: Which should you choose?
- Supercharging web scraping with Claude skills
- Claude Sonnet 4.6 is the new best model for writing scrapers
- AI and the web: What 2025 changed and what comes next
- Gemini 3.0 Pro is the new best model for writing scrapers
- Why AI agents struggle with web scraping (and how to help them)
- Five key takeaways from Extract Summit 2025
- Introducing Web Scraping Copilot - A rocket boost for data extractors
- Extract clean content automatically with Zyte API’s new pageContent data type
- Why your agent deserves a wallet
- Agentic web scraping: Hype, reality and what happens next
- Four sweet spots for AI in web scraping
- Data on command: The natural-language web scraping revolution
- From products to SERPs: AI scraping now does it all
- The Right AI For the Right Problem: How Zyte Solved Web Data's Trilemma of Cost, Quality, and Flexibility
- Beyond Hello World: The Operational Gaps in LLM-Powered Scraping Tools
- Faster and cheaper: How AI is breaking barriers in web data sourcing
- No wait, no upfront costs: How Zyte transformed its data delivery business with AI
- How to meet your data extraction deadlines with AI power
- Zyte: Transforming AI web scraping for Digital Shelf companies
- AI Scraping now available in Zyte API
- Zyte A.I. Automatic Extraction API | e-commerce & articles