Future of the web
Articles from the Zyte blog about Future of the web.
The anti-ban arms race: Agentic traffic and the fight for the open web
As AI agents transform how we interact with the web, anti-bot systems threaten to fence off the digital open range.
Four in 10 sites block AI bots with robots.txt
Nearly four in 10 of the world's top sites are now closed to well-behaved AI crawlers. Here's what operators actually do with robots.txt, and which AI agents they block or welcome.
75% of the web uses robots.txt - here's how
Three in four of the world's top sites publish a robots.txt, yet few name individual crawlers. A look inside the web's advisory layer: coverage, sophistication and limits.
Efficient Web Scraping with LLMs
In this hands-on workshop hosted by Iván Sánchez, you'll learn how to harness the incredible power of Large Language Models (LLMs) to tackle your toughest scraping challenges.
Fireside chat with Shane Evans on web scraping APIs
Watch Shane Evans, CEO and co-founder of Zyte, in an exclusive Fireside Chat as he delves into the evolution of web scraping APIs, the groundbreaking technology revolutionizing how businesses extract and utilize web data.
