Field notes from the world of data extraction.
Articles, interviews and analysis on how data is gathered, used and fought over — written by the people closest to it.
How to build your first Scrapy extension
My Scrapy project now plays a triumphant fanfare when a crawl finishes clean and a sad trombone when it doesn't, which is also how I finally learned what Scrapy's extension points are actually for.
My personal agent setup: the architecture that runs my household and my DevRel work
"Four people, four diets, two work schedules, and a baby who answers to nobody. That's what finally made me build a personal agent." A walkthrough of the actual architecture I run to hold my household and my DevRel work together — profiles, skills, memory, and the web-data layer that makes it all reach the live web.
Data is the new water
For 20 years, we were told “data is the new oil”. That’s no longer true. In the era of the fluid web, information is more fundamental, cleaner and abundant than that.
Building robust agentic AI workflows with rapid web data
AI agents need access to public web data, right now. Tools connected to web scraping APIs empower agents to return live data quickly.
Now what exactly is loop engineering? And where do Anthropic's Fable 5 model and web scraping fit in?
A viral clip from Claude Code's creator put a name to something a lot of us have been circling: loop engineering. Here's why web scraping may be its best-fit domain — and what that means in practice.
Enhancing AI model performance with fresh web data
No-one likes an out-of-touch AI assistant. Fortunately, rapid refreshing can keep AI models aware of the very latest public information.
What's becoming of web scraping developers in the age of AI agents?
AI agents can generate code, suggest selectors, and draft crawl logic. What they can't do is design the system that decides when to stop, what to trust, and how to recover when the web pushes back. That job still belongs to a human.
Building superior AI models with quality web data
Training data quality can make or break AI model effectiveness. So how are engineers sourcing the web’s best input?
Web scraping on an iPhone? Yes, really!
When you can scrape the web by API, a world of possibility opens up. Yes, you can extract live web data using iOS Shortcuts.
Introducing Zyte Web Data for Claude Code: Production-ready scraping from a prompt
Developers are embracing agentic coding tools - but data engineers need tools with specialist scraping skills.
Copilot like a pro: Eight tips that supercharged my workflow
AI-assisted coding is a revelation. But are you getting the most out of your IDE’s sidebar sidekick?
Automate deployment of your web scraper on VPS with Ubuntu 24.04 cloud-init
Your VPS is ready, but now you need to work through the same sequence you have run a dozen times before: apt update, apt install python3-pip, pip install scrapy, playwright install chromium, the Chromium dependency list that never installs cleanly on the first try, Redis, possibly Postgres, whatever else this particular project needs.
_HFpro5d6k3.png&w=256&q=75)
_E4PyVpfAxa.png&w=256&q=75)
-(1).png&w=1920&q=75)
-(1)_VZGHqxCgXV.png&w=1920&q=75)
