Web data
Articles from the Zyte blog about Web data.
What web access actually costs - the tier picture
The barrier headlines sound extreme, but the cost data is more differentiated: 87.9% of top sites are Simple or Easy to access - and that means cost-effective.
The trouble with variants: Solving the web's infinite ecommerce explosion
Product variants can turn a straightforward ecommerce scraping project into millions of combinations. Learn how to classify, scope and control them.
Need for speed: How data inertia kills business ambition
Slow web data pipelines can hold back new markets, conversion gains and product launches.
The web's layered defences - how access barriers combine
Individual barriers may get the headlines, but it's stacked access control mechanisms that pose the biggest block.
The anti-ban arms race: Agentic traffic and the fight for the open web
As AI agents transform how we interact with the web, anti-bot systems threaten to fence off the digital open range.
TLS fingerprinting, the handshake that reveals your true intent
TLS fingerprinting fires before you have even asked for a page, and tells you nothing when it does. The least common barrier, and the hardest to diagnose.
JavaScript renders almost half of sites invisible
JavaScript renders 40.6% of top sites unreadable to a plain HTTP request, not by design but as a side effect of how the modern web is built.
Rate limiting is a roadblock, not a speed limit
Rate limiting sounds like a speed limit, but 93.7% of rate-limited sites block on the very first request. It is an identity check, not a throttle.
Introducing Zyte MCP: Powerful web data gathering for your agent
Zyte MCP brings live web access, structured extraction and Scrapy Cloud operations into coding agents such as Claude Code, Cursor and Codex.
CAPTCHA is more than just a puzzle piece
The famously awkward security mechanism has quietly become an invisible behavioral scoring system. Two vendors enable 89% of CAPTCHAs, and the smallest sites use it most.
One Scrapy Spider, Three Browser Setups: Playwright, Patchright, and Zyte
One Scrapy spider, three browser setups: stock Playwright, Patchright via the new PLAYWRIGHT_BROWSER_PROVIDER hook, and a remote browser on Zyte over CDP. Same spider and selectors throughout, only the browser changes.
Inside antibot, the arms race you can't see
Only 18.5% of top sites run dedicated antibot, but every one of them chose to. Why it's the most intentional barrier in the stack, and the strongest signal of a hardened site.