PINGDOM_CHECK

Web data collection

Articles from the Zyte blog about Web data collection.

A line-up of robot babushka or Matryoshka dolls.
Access handling

What web access actually costs - the tier picture

The barrier headlines sound extreme, but the cost data is more differentiated: 87.9% of top sites are Simple or Easy to access - and that means cost-effective.

Robert Andrews
A shopper faces an overwhelming cascade of ecommerce T-shirt product cards in many colours and configurations, illustrating product-variant explosion.
Web data collection

The trouble with variants: Solving the web's infinite ecommerce explosion

Product variants can turn a straightforward ecommerce scraping project into millions of combinations. Learn how to classify, scope and control them.

Robert Andrews12 min read
Two runners carrying backpacks full of web page cards race along a purple track towards a glowing digital finish line.
Web data collection

Need for speed: How data inertia kills business ambition

Slow web data pipelines can hold back new markets, conversion gains and product launches.

James Kehoe8 min read
A stack of six in fortifies a wall blocking a computer from accessing pages.
Access handling

The web's layered defences - how access barriers combine

Individual barriers may get the headlines, but it's stacked access control mechanisms that pose the biggest block.

Robert Andrews
Akshay Philar speaking at Extract Summit about the anti-ban arms race and the open web.
Anti-ban

The anti-ban arms race: Agentic traffic and the fight for the open web

As AI agents transform how we interact with the web, anti-bot systems threaten to fence off the digital open range.

Akshay Philar9 min read
A key made of a fingerprint is examined by a cryptographic light as it gains access to the network.
Access handling

TLS fingerprinting, the handshake that reveals your true intent

TLS fingerprinting fires before you have even asked for a page, and tells you nothing when it does. The least common barrier, and the hardest to diagnose.

Robert Andrews
Missing JavaScript has made almost half a web page invisible.
Access handling

JavaScript renders almost half of sites invisible

JavaScript renders 40.6% of top sites unreadable to a plain HTTP request, not by design but as a side effect of how the modern web is built.

Robert Andrews
A speed limit sign's speed guidance is replaced with a 403 code.
Access handling

Rate limiting is a roadblock, not a speed limit

Rate limiting sounds like a speed limit, but 93.7% of rate-limited sites block on the very first request. It is an identity check, not a throttle.

Robert Andrews
Zyte MCP chat box connected to fetch, search and extract tools.
Data gathering for AI

Introducing Zyte MCP: Powerful web data gathering for your agent

Zyte MCP brings live web access, structured extraction and Scrapy Cloud operations into coding agents such as Claude Code, Cursor and Codex.

Valter Sciarrillo7 min read
A classic toy robot waves a protest banner showing "I'm not a robot"
Access handling

CAPTCHA is more than just a puzzle piece

The famously awkward security mechanism has quietly become an invisible behavioral scoring system. Two vendors enable 89% of CAPTCHAs, and the smallest sites use it most.

Robert Andrews
A spyglass reveals a robot secretly using a web browser with its hand on the mouse.
Access handling

Inside antibot, the arms race you can't see

Only 18.5% of top sites run dedicated antibot, but every one of them chose to. Why it's the most intentional barrier in the stack, and the strongest signal of a hardened site.

Robert Andrews
A firewall surrounds a server computer
Access handling

Firewalls run the web, just not on purpose

Web Application Firewalls run on 92.4% of the top websites, but most arrived bundled with a CDN and were never tuned to resist automated access.

Robert Andrews
1234

More articles on Web data collection