Scraping strategy
Articles from the Zyte blog about Scraping strategy.
Inside Scrapy's RemoteControl extension
Scrapy 2.19's RemoteControl extension runs a small authenticated HTTP server inside every crawl. How it starts, how it's secured, what the code you send can reach, and what you can do with it on a live crawl, plus the traps and what changes under scrapy-zyte-api.
The web is being priced, not blocked
Zyte's State of Web Access research finds new barriers making web scraping more difficult. We interview the researcher who says that doesn't mean the opportunity is over.
Rendering Javascript pages without giving up Scrapy
How to render dynamic content and work with a browser through playwright and scrapy.
A guide to Scrapy item types
Scrapy supports multiple item types, but which should you use, and why.
Podcast Ep08 - Scrapy, Python and mushroom soup
Scrapy's core handles crawling well and deliberately leaves almost everything else out: no bundled browser, no opinion about how you shape your data, no built-in answer for every anti-bot wrinkle. What it gives you instead is a clean way to add those things at the edges, exactly when you need them and never before.
Treat your AI skills like software, starting with evals
Most AI skills are never tested — and it shows. Here's how Zyte evaluates scraping skills like real software, catching failures demos miss.
The great wall of data: The complexities of web scraping in the Asian market
While the technological arms race of web data access is universal, the battleground in Asia has its own unique rules of engagement.
Brand visibility in the digital era: How web data help brands see the full picture
Discover how web data helps brands improve visibility, track competitors, monitor availability, and analyze reviews to win on the digital shelf.
How online retailers use web data to compete on price, promotion, and availability
Discover how retailers leverage web data to optimize pricing, track competitor stock, detect trends, and improve sales performance.
How web data turns e-commerce listings into retail intelligence
Discover how web data enables digital shelf analytics vendors to track prices, availability, and product trends at scale—fueling real-time retail intelligence and competitive advantage.
The seven habits of highly effective data teams
Discover the seven habits that set high-performing data teams apart—from treating data as a product to ensuring data trust, quality, and decision impact. Learn how leading teams scale reliable data systems.
Five key takeaways from Extract Summit 2025
From AI-accelerated scraping to “dead internet” risks and rising access wars, these five takeaways from Extract Summit 2025 show where web data is heading next.