The State of Web Access: How different industries behave to bots

The data from The State of Web Access, Zyte's new audit of 11,100 websites' access controls, looks, at first glance, like a thicket of technical complexity.

We examined Web Application Firewalls, JavaScript rendering requirements, CAPTCHAs, rate limiters, TLS fingerprinting, dedicated antibot engines - six distinct categories of control, deployed in varying combinations across 110 industries and 230 countries.

But underneath that complexity, a single principle organizes almost everything we found: access controls concentrate where data has commercial value, and relax where it doesn't.

Businesses, far more than security teams responding to abstract threats, build the web's access layer - through rational decisions about what information they can afford to share with the world, and what they cannot.

The default layer and the strategic layer

Not every barrier in our dataset is a deliberate choice.

Web Application Firewalls are the most widely deployed access control we measured, found in some form on roughly nine in ten popular landing pages. But for most of those sites, the WAF is not a strategic deployment - it arrives pre-bundled with a content delivery network and runs at default settings. Many operators would struggle to describe what their WAF is actually doing.

This matters because it means the baseline access friction on the web is largely structural, not intentional. The first layer of defense most sites carry was never installed to keep anyone out specifically. It exists because CDN vendors package it in by default, and because leaving it on costs nothing.

The interesting question is what happens above that baseline.

Where the intentional choices begin

Once you look beyond WAFs, a much cleaner pattern emerges.

Antibot solutions can be a byproduct

Dedicated antibot services - the most sophisticated tier of access control, using behavioral analysis to identify automated traffic - appear on roughly one in five landing pages in our dataset. But their distribution is not random.

High-traffic sites are 50% more likely to deploy them. Consumer electronics, computer hardware, adult content, and gambling lead antibot adoption. These are sectors facing acute, specific threats: account compromise, payment fraud, inventory hoarding by bots targeting high-demand products.

Antibot solutions are a considered investment, made by organizations that have calculated the cost of not having it.

Rate limiting is adopted unevenly

Rate limiting tells a similar story from a different angle. Industries that benefit from frequent indexing - media, education, recruitment - adopt rate limiting relatively lightly. Industries with large, frequently updated inventories that attract systematic extraction - real estate, travel, automotive - adopt it heavily.

The threshold at which sites activate rate limiting reflects exactly how much request volume they're prepared to absorb before the economics of open access stop making sense.

CAPTCHA appeals to different industries

Even CAPTCHA, the most visible and user-disruptive of the controls we measured, distributes in a way that makes commercial sense.

• Larger, higher-traffic sites tend not to rely on it - they can afford more sophisticated infrastructure. Smaller sites in regulated industries (tobacco leads, with 52% CAPTCHA adoption) use it as a cost-effective compliance and gating mechanism.
• Lower-traffic sites are 39% more likely to use CAPTCHAs than high-traffic ones. The friction is calibrated to what the site can spend.

The industry argument

The clearest evidence that access controls follow business logic rather than technical convention is what happens when you look across industries.

• Apparel and fashion carries the highest mean recommended access tier of any industry in our dataset. Fashion data - trend intelligence, competitor pricing, inventory levels - is the raw material of a competitive market where margins are thin and information asymmetry is money.
• Brands and retailers have learned, often painfully, that leaving product and pricing data openly accessible means handing intelligence to competitors in real time. The defenses are a direct response to that calculation.

Real estate, consumer electronics, and travel form the next cluster of defended industries. In each case, the pattern is the same: the data being protected has direct, quantifiable commercial value to anyone who can systematically extract it.

• Real estate agents defend listing and pricing data because controlling information asymmetry is their business model.
• Travel sites defend inventory and pricing because competitor surveillance and screen-scraping are endemic in the sector.

Then look at the other end of the spectrum. Public sector sites, agriculture, and media and publishing sit among the least defended industries in the dataset. Those sites have certainly heard of WAFs; they stay open because their business models depend on being found.

The infrastructure equation

Zyte API classifies access difficulty in five tiers - from Simple (plain HTTP) through Easy (JavaScript rendering), Moderate (browser with fingerprint handling), Complex (residential proxies or advanced evasion), to Advanced (full specialised infrastructure).

The mean tier for each industry reflects the minimum infrastructure required for the average site in that sector. A mean above 2.0 means a headless browser is the baseline, not the exception.

Most defended (mean recommended Zyte API tier):

Least defended (mean recommended Zyte API tier):

Being closed has its downsides, depending on your business. A news publisher that makes its content hard to access is one that could disappear from search results. A government information site that blocks automated access is failing at its core function of disseminating public information.

The economic logic of openness is as strong in these sectors as the economic logic of closure is in fashion and real estate - it just points in the opposite direction.

The robots.txt signal

Perhaps the clearest evidence of intentionality comes not from technical barriers at all, but from robots.txt - the plaintext file sites use to declare their access preferences.

Three-quarters of sites in our dataset maintain a robots.txt file. And the patterns in those files reveal exactly the kind of deliberate, commercially-motivated reasoning we see in the barrier data.

Publishers are blocking AI crawlers specifically, while continuing to permit search engine access. Among newspaper sites, 69% now name specific AI agents in their robots.txt. The figure for search crawlers is lower. Publishers have decided - consciously, explicitly - that search engines send traffic and AI agents extract content without returning value. That is a business decision made in plain text.

What this means for anyone building on web data

If access difficulty follows commercial logic, it becomes predictable.

A site defending high-value inventory data will be defended systematically, with multiple layered controls, actively maintained. A site in a sector that depends on discoverability will be comparatively open, even if it carries a default WAF.

Understanding the business model of the site you're trying to access tells you, with reasonable accuracy, what you're going to encounter before you write a line of code.

That knowledge has consequences for how data teams approach infrastructure investment. The question is not so much "how do I handle web access barriers?" It's "what does this industry's economics tell me about what I'm likely to face, and what's the right tool for that specific context?"

The State of Web Access 2026 data - covering all six barriers across 11,100 sites, 110 industries, and 230 countries - is built to answer exactly that question. The full research is available at zyte.com/sowa/2026.