AI
Articles from the Zyte blog about AI.
12 things I use Zyte MCP for, ranked, with prompts and rules you can copy
What can you use Zyte MCP for? 12 uses, 11 tested, ranked, each with a copy-paste prompt, a CLAUDE.md rule, and a skill idea.
How I use Zyte MCP to plan web scrapers
Zyte MCP lets my AI agent investigate a site before I write the scraper, so I can decide where the data should come from, compare an API with the rendered page for missing fields, and end up with a crawl plan I can act on.
Self-healing spiders: letting an agent fix your Scrapy crawl, with a lid on it
Two demos from our Extract Summit Austin workshop: an agent repairs a Scrapy spider after a site redesign, then heals a live crawl mid-run through Spidermon and the Scrapy MCP, with the measured costs and the mistakes that shaped the design.
The anti-ban arms race: Agentic traffic and the fight for the open web
As AI agents transform how we interact with the web, anti-bot systems threaten to fence off the digital open range.
How to convert a website to Markdown for AI: 7 methods compared
Seven realistic ways to turn web pages into clean Markdown for AI - what each method covers, where each breaks, and how to choose for training, RAG and agent pipelines.
Train AI on industry-leading Markdown with Zyte’s new extraction mode
Zyte API can now extract a web page’s main content directly as clean, structured Markdown for AI systems, RAG pipelines and developer workflows.
Introducing Zyte MCP: Powerful web data gathering for your agent
Zyte MCP brings live web access, structured extraction and Scrapy Cloud operations into coding agents such as Claude Code, Cursor and Codex.
AI web scraping and scalable data collection
AI changes extraction, adaptation and the cost of adding a source. It does not change access. What is genuinely different, with the arithmetic.
Stopping a crawl when the data stops looking real with Jev
A Scrapy pipeline that asks a fast, calibrated AI model whether each scraped field still looks real, and stops the crawl when too many don't. What it caught, what it misses, and whether building it was worth it.
uv Python cheatsheet: what changed in 0.12 and what still trips you up
uv 0.12 changed several defaults that Python developers rely on—from how `uv init` structures projects to how hash checking and lockfiles behave. This practical cheatsheet tests the differences across versions and highlights the command-line traps most likely to surprise you.
.png&w=3840&q=75)
Give your AI agent a browser: search, compare, and check product availability with Zyte CDP
What if your AI agent could use a real browser to search, click, compare products, and collect data, without you writing a selector for every step? By connecting it to a remote Zyte CDP browser, you can turn plain-language instructions into practical browser automation.
Four in 10 sites block AI bots with robots.txt
Nearly four in 10 of the world's top sites are now closed to well-behaved AI crawlers. Here's what operators actually do with robots.txt, and which AI agents they block or welcome.
More articles on AI
- Harness Engineering #3- Headless mode: the minimal agent harness
- GPT-5.6, Fable 5, and GLM-5.2 entered a bar “crawl” and got hit by The Rate Limit
- Treat your AI skills like software, starting with evals
- The best agent skill is the one that says the least
- My personal agent setup: the architecture that runs my household and my DevRel work
- What's becoming of web scraping developers in the age of AI agents?
- Why 10 million tokens won’t save your AI agent (and what will)
- Are programming practices relevant anymore?
- Is your AI coding assistant stuck in the past?
- Data in 2026: Bets and forecasts from web experts
- Scraping a synthetic web: Dead Internet Theory meets web data extraction
- Why your agent deserves a wallet
- Agentic web scraping: Hype, reality and what happens next