Developer interest

Articles from the Zyte blog about Developer interest.

⌕

The harness matters more than the model - Podcast EP07

"The model is the engine — but the harness is everything else." In Episode 7, we dig into why the infrastructure layer around your AI model matters more than the model itself, rank the best models available right now, and ask whether the open-weighted revolution is about to make frontier subscriptions obsolete.

John Rooney11 min readJune 27, 2026

Developer interest

How to build your first Scrapy extension

My Scrapy project now plays a triumphant fanfare when a crawl finishes clean and a sad trombone when it doesn't, which is also how I finally learned what Scrapy's extension points are actually for.

Ayan Pahwa11 min readJune 18, 2026

Developer interest

Now what exactly is loop engineering? And where do Anthropic's Fable 5 model and web scraping fit in?

A viral clip from Claude Code's creator put a name to something a lot of us have been circling: loop engineering. Here's why web scraping may be its best-fit domain — and what that means in practice.

Ayan Pahwa12 min readJune 10, 2026

Developer interest

Automate deployment of your web scraper on VPS with Ubuntu 24.04 cloud-init

Your VPS is ready, but now you need to work through the same sequence you have run a dozen times before: apt update, apt install python3-pip, pip install scrapy, playwright install chromium, the Chromium dependency list that never installs cleanly on the first try, Redis, possibly Postgres, whatever else this particular project needs.

Ayan Pahwa9 min readMay 31, 2026

Developer interest

NotAnInterview: “I Have Superpowers Now"

The problem was a project with 12,000 websites to crawl, and there’s no world where you write custom spiders for 12,000 websites, not with a human team and certainly not sustainably. So Javier built a workflow: a set of AI prompts that could analyze a website, figure out its structure, and generate a crawl configuration that a generic spider could then use.

Neha Setia Nagpal9 min readMay 27, 2026

Developer interest

Flatcar Linux for web scrapers: deploy immutable containers with just one config file

The next time you spin up a VPS to give it a persistent home, you spend the better part of an afternoon rebuilding from memory. Here's a tool to help using Flatcar Linux

Ayan Pahwa11 min readMay 25, 2026

Developer interest

My agentic coding setup: Claude Code, multi-agent orchestration, and how I actually work

Ayan's 4 agent team, using Claude's /goal, and the models and coding agents he uses to code effectively.

Ayan Pahwa25 min readMay 22, 2026

Web scraping APIs

Are programming practices relevant anymore?

From LLM-powered extraction to agentic pipelines, here's how AI is reshaping every stage of the web scraping workflow in 2026 -- and what it means for your stack.

Mikhail Korobov5 min readApril 7, 2026

Developer interest

Is your AI coding assistant stuck in the past?

AI coding tools like Copilot can suggest outdated code and practices. Learn why this happens and how developers can ensure they use the latest standards.

Theresia Tanzil5 min readMarch 27, 2026

Zyte Blog — field notes from the world of data extraction

Developer interest

Stop using Python requests for web scraping: Use these modern modules instead

While the 'Requests' library remains the default choice for many Python developers due to its reliability and extensive documentation, the Python HTTP landscape has evolved considerably. Modern alternatives now offer significant advantages, including built-in asynchronous support, HTTP/2 compatibility, enhanced performance, and up-to-date TLS handling.

Ayan Pahwa6 min readMarch 11, 2026

AI-assisted data extraction

AI and the web: What 2025 changed and what comes next

2025 was the year AI learned to reason. From reasoning-first LLMs to autonomous agents and a reshaped web economy, this retrospective explores what changed—and what’s coming next.

Iván Sánchez10 min readDecember 22, 2025