Neha Setia Nagpal

The best agent skill is the one that says the least

More instruction, worse output. Zyte's head of R&D on why telling your agent exactly what to do can blind it to the obvious answer.

My personal agent setup: the architecture that runs my household and my DevRel work

"Four people, four diets, two work schedules, and a baby who answers to nobody. That's what finally made me build a personal agent." A walkthrough of the actual architecture I run to hold my household and my DevRel work together — profiles, skills, memory, and the web-data layer that makes it all reach the live web.

What's becoming of web scraping developers in the age of AI agents?

AI agents can generate code, suggest selectors, and draft crawl logic. What they can't do is design the system that decides when to stop, what to trust, and how to recover when the web pushes back. That job still belongs to a human.

What multi-agent orchestration looks like in a large-scale web scraping project

Multi-agent orchestration is having its moment. The diagrams are everywhere now. Boxes for planners, boxes for hands, boxes for daemons, arrows to a shared brain, a human floating at the top. They keep getting prettier. The part where the web pushes back is still the part nobody draws.

NotAnInterview: “I Have Superpowers Now"

The problem was a project with 12,000 websites to crawl, and there’s no world where you write custom spiders for 12,000 websites, not with a human team and certainly not sustainably. So Javier built a workflow: a set of AI prompts that could analyze a website, figure out its structure, and generate a crawl configuration that a generic spider could then use.

I'm not the same developer I was before LLMs

I've been running a series of conversations with developers at Zyte to understand what's actually changed in the way they work since LLMs showed up. Not the headlines. The day-to-day. What they delegate, what they don't, what they notice, what surprises them. This one was different on two counts.

AI won’t fix your data quality (until you answer these three questions)

In our interview, a QA expert warns - before you delegate web scraping quality assurance to AI, make sure you can describe what ‘good’ looks like for yourself.

The Scrapy whisperer: Adrian Chaves on Web Scraping Copilot

An interview with one of the popular framework’s co-maintainers on Zyte’s new, AI-powered Scrapy sidekick.

More data, more trouble: How a perfect corpus corrupted my AI dream

What a failed experiment taught me about curated data, prompting, and when scraping actually matters.

Claude skills, MCP or Web Scraping Copilot: Which should you choose?

Three ways to bring Zyte-powered web data into your AI workflow — from production spiders to conversational extraction.

A Deep Dive into Zyte's Open-Source Libraries

Discover how Zyte’s open-source libraries like ClearHTML, Extruct, Chomp.js, and more simplify web data extraction and processing.

Analyze web data quickly with Jupyter Notebooks and Zyte API

With AI Scraping in Zyte API, you can pull data from any e-commerce website straight into your Jupyter notebooks.

Overcoming web scraping challenges of Puppeteer and Playwright

Discover the challenges of scaling web scraping with Playwright & Puppeteer, from browser farm management to IP rotation and anti-scraping tactics.

How Session Management Minimizes Bans and Enhances Data Quality in Web Scraping

User sessions can help overcome website bans, handle IP rate limits, streamline cookie management, and avoid detection.

Best web scraping methods for JavaScript-heavy websites

Discover key techniques to efficiently extract data from JavaScript-heavy websites.

Selenium, Puppeteer, Playwright: Which tool is right for web scraping at scale?

Discover the strengths and limitations of Selenium, Puppeteer, and Playwright for web scraping at scale.

Why are Sessions Crucial in Web Data Extraction?

Whether you're scraping a simple webpage or navigating a complex multi-step process, leveraging sessions is key to ensuring success.

4 essential Scrapy plugins for building efficient and effective spiders

Here are four essential Scrapy plugins we use to build efficient web crawlers for our customers.

The Scraper’s System Part 2: Explorer’s Compass to analyze websites

In the first part, we discussed a template to define the clear purpose of your web scraping system that can help you design your crawlers better and prepare you for the uncertainty involved in a large scale web scraping project.

How Zyte API takes care of the fundamental needs of your web scraping project!

How Web Scraping and Graph Databases Can Power Recommendation Engines

I recently had the pleasure of participating in the third episode of Graphversation, a monthly live stream series that brings together graph experts and Neo4j enthusiasts for engaging and enlightening discussions about the captivating world of graphs.

Scraper's System: Secrets to architect scalable web scraping

Staring at the Statement of Work opened on your laptop screen, your thoughts must be wandering in all directions - where do you start?

How To Use Selenium With Zyte Smart Proxy Manager

Using Selenium and/or Zyte Smart Proxy Manager? You just stumbled upon the right blog.

How to use Playwright with Smart Proxy Manager

We have launched a new Zyte SmartProxy Playwright and we’re sure you’re going to love it!

The best way to Use Puppeteer With Zyte Smart Proxy Manager

We are super excited to share some good news for all the Puppeteer users who are looking for an easy-to-integrate anti-ban solution for extracting data from javascript-heavy websites.

Developer’s guide to rotating proxies in Python

What is a rotating proxy? Why is IP rotation important for scraping?

New to scraping and rotating proxies? Start here to learn what is IP rotation why it is important for web scraping.

Find the best pint using web scraping on St. Patrick’s day

At Zyte we are known for our ability to help companies make mission-critical business decisions through the use of web scraping.

Why you need the best proxies for web scraping

Before we begin, take a look at this short video - it's the scene from Harry Potter where he gets The Invisibility Cloak. It’ll help us better understand the concepts behind proxies.