Author
Robert Andrews
Senior editor
Robert is a journalist and editor turned content strategist who eats and sleeps the web. Previously senior editor at Google, News Corp, ContentNext and others.
As Zyte's senior editor, Robert covers the state of the data-access industry — legal developments affecting AI and scraping, industry benchmark reports, and forward-looking analysis like "Data in 2026: Bets and forecasts from web experts" and "Zyte's 2025 review: Year of locks and unlocks." His 16 articles also include practical engineering content (Scrapy's 2026 async crawling release, Zapier integration) and event recaps from Extract Summit.
Inside antibot, the arms race you can't see
Only 18.5% of top sites run dedicated antibot, but every one of them chose to. Why it's the most intentional barrier in the stack, and the strongest signal of a hardened site.
Firewalls run the web, just not on purpose
Web Application Firewalls run on 92.4% of the top websites, but most arrived bundled with a CDN and were never tuned to resist automated access.
Four in 10 sites block AI bots with robots.txt
Nearly four in 10 of the world's top sites are now closed to well-behaved AI crawlers. Here's what operators actually do with robots.txt, and which AI agents they block or welcome.
75% of the web uses robots.txt - here's how
Three in four of the world's top sites publish a robots.txt, yet few name individual crawlers. A look inside the web's advisory layer: coverage, sophistication and limits.
Revealed: How retailers use tech to block the bots
The web’s big general marketplaces exist to be browsed by everyone. But they are also the second most defended sector online, shutting the door on AI crawlers.
The web is being priced, not blocked
Zyte's State of Web Access research finds new barriers making web scraping more difficult. We interview the researcher who says that doesn't mean the opportunity is over.
Fashion websites are the hardest to size up
While publishers fight over AI crawlers, fashion has built some of the most heavily defended shop windows on the web. So, which controls are in fashion, in fashion?
The State of Web Access: How different industries behave to bots
Zyte's large-scale audit of web access controls shows how different kinds of businesses exhibit different policies.
The State of Web Access: How today's web raises its barriers
The largest ever audit of web access control mechanisms lifts the lid on how sites are regulating programmatic visits.
Five reasons to attend this year’s Extract Summit
In its eighth year, Extract Summit is again shaping up to be the web data industry's premier gathering.
Data is the new water
For 20 years, we were told “data is the new oil”. That’s no longer true. In the era of the fluid web, information is more fundamental, cleaner and abundant than that.
Legal clarity comes with compliance demands
Explore how new regulations like the EU AI Act and California AB 2013 are reshaping AI data compliance in 2026. Learn why provenance, transparency, and lawful sourcing are now critical.
Web traffic is splintering into access lanes
Explore how AI agents are reshaping web traffic into hostile, negotiated, and invited access lanes. Learn what this means for bots, scraping, and the future of data access.
Automation drives power in the data arms race
Anti-bot systems now evolve in minutes, not weeks. Discover why automated, self-healing scraping systems are essential to survive the 2026 data arms race and how to adapt.
The seven habits of highly effective data teams
Discover the seven habits that set high-performing data teams apart—from treating data as a product to ensuring data trust, quality, and decision impact. Learn how leading teams scale reliable data systems.
AI is the new engine for web scraping
From LLM-powered extraction to agentic pipelines, here's how AI is reshaping every stage of the web scraping workflow in 2026 -- and what it means for your stack.
Data outcomes are top of the scraping stack
By 2026, scraping shifts from proxy management to API-first data outcomes. Learn why unified scraping APIs are replacing traditional stacks.
No-code web scraping workflows are here: Introducing the Zyte integration for Zapier
Discover Zyte’s Zapier integration to automate web data extraction and connect it to 10,000+ apps. Build powerful workflows without coding and turn web data into actionable insights instantly.
Is your AI breaking the law? Legal experts’ advice for web scrapers
Legal experts discuss how AI, web scraping, copyright law, and the EU AI Act intersect—covering fair use, data provenance, and compliance risks for businesses.
Sun, sea and code: What we built at Zyte’s API hackathon
Discover the 7 creative projects built at Zyte’s API Hackathon in Turkey, from security scanning tools to price comparison engines and smart caching systems.
Scrapy in 2026: New release brings modern async crawling standards
Scrapy 2.14.0 modernizes the framework with native async/await, smarter scheduling, and cleaner spider configuration. Here’s what the new release means for production crawlers in 2026.
Data in 2026: Bets and forecasts from web experts
What will 2026 have in store for the world of web data? We asked eight experts for their predictions.
Zyte’s 2025 review: Year of locks and unlocks
2025 reshaped web scraping. From AI-assisted extraction and escalating bot defenses to clearer legal frameworks and cheaper APIs, Zyte reviews the forces redefining access to web data—and what comes next.
Zyte leads the pack in Proxyway’s 2025 Web Scraping API Report
Proxyway’s 2025 Web Scraping API Report ranks Zyte #1 for unblocking success, speed, cost efficiency, and AI-powered data extraction. See the full breakdown.
Beyond the block: The front line of data access
A deep dive into the evolving battle for web data access—featuring insights from Castle, Scrapoxy, and Zyte at Extract Summit 2025. Learn how AI, anti-bots, economics, and authentication standards like Web Bot Auth are transforming scraping, security, and the future of the open internet.
How to build a daily industry news digest
Learn how data analyst Anshika Khandelwal automated a daily AI funding news digest using n8n and Zyte API. Discover how to pull articles, classify funding stories, and deliver a curated newsletter that saves 10+ hours per week.
Five key takeaways from Extract Summit 2025
From AI-accelerated scraping to “dead internet” risks and rising access wars, these five takeaways from Extract Summit 2025 show where web data is heading next.
Death of the Proxy? There’s An API for That
The proxy era is ending as web scraping shifts from managing IP pools to smarter, API-driven solutions.
Why the Best Engineers Are Actually Lazy
There’s a popular archetype in web scraping circles: the heroic engineer who fights CAPTCHAs at 3 a.m., hand-tunes a proxy farm before breakfast, then rewrites four spiders after lunch because the target sites pushed new JavaScript.
The future of Scrapy: Smarter, faster and ready for AI-powered scraping
What does the future hold for the tool some describe as “the gift that revolutionised web scraping”?
Rise of the Data Vendor: How Outsourcing is Transforming Supply and Fuelling Businesses
With the emergence of managed data extraction vendors, businesses no longer need to gather web data themselves.