Subscribe to our Blog
The latest from Robert Andrews

Leadership
Death of the Proxy? There’s An API for That
The proxy era is ending as web scraping shifts from managing IP pools to smarter, API-driven solutions.
Robert AndrewsAugust 26, 2025

Leadership
Why the Best Engineers Are Actually Lazy
There’s a popular archetype in web scraping circles: the heroic engineer who fights CAPTCHAs at 3 a.m., hand-tunes a proxy farm before breakfast, then rewrites four spiders after lunch because the target sites pushed new JavaScript.
Robert AndrewsAugust 18, 2025

Open Source
The future of Scrapy: Smarter, faster and ready for AI-powered scraping
What does the future hold for the tool some describe as “the gift that revolutionised web scraping”?
Robert AndrewsJune 23, 2025

Open Source
Rise of the Data Vendor: How Outsourcing is Transforming Supply and Fuelling Businesses
With the emergence of managed data extraction vendors, businesses no longer need to gather web data themselves.
Robert AndrewsJune 20, 2025