Neha Setia Nagpal

Subscribe to our Blog

The latest from Neha Setia Nagpal

What multi-agent orchestration looks like in a large-scale web scraping project

NotAnInterview: “I Have Superpowers Now"

I'm not the same developer I was before LLMs

AI won’t fix your data quality (until you answer these three questions)

The Scrapy whisperer: Adrian Chaves on Web Scraping Copilot

More data, more trouble: How a perfect corpus corrupted my AI dream

Claude skills, MCP or Web Scraping Copilot: Which should you choose?

A Deep Dive into Zyte's Open-Source Libraries

Analyze web data quickly with Jupyter Notebooks and Zyte API

Overcoming web scraping challenges of Puppeteer and Playwright

How Session Management Minimizes Bans and Enhances Data Quality in Web Scraping

Best web scraping methods for JavaScript-heavy websites

Selenium, Puppeteer, Playwright: Which tool is right for web scraping at scale?

Why are Sessions Crucial in Web Data Extraction?

4 essential Scrapy plugins for building efficient and effective spiders

The Scraper’s System Part 2: Explorer’s Compass to analyze websites

How Zyte API takes care of the fundamental needs of your web scraping project!

How Web Scraping and Graph Databases Can Power Recommendation Engines

Scraper's System: Secrets to architect scalable web scraping

How To Use Selenium With Zyte Smart Proxy Manager

How to use Playwright with Smart Proxy Manager

The best way to Use Puppeteer With Zyte Smart Proxy Manager

Developer’s guide to rotating proxies in Python

What is a rotating proxy? Why is IP rotation important for scraping?

Find the best pint using web scraping on St. Patrick’s day

Why you need the best proxies for web scraping

Multi-agent orchestration is having its moment. The diagrams are everywhere now. Boxes for planners, boxes for hands, boxes for daemons, arrows to a shared brain, a human floating at the top. They keep getting prettier. The part where the web pushes back is still the part nobody draws.

The problem was a project with 12,000 websites to crawl, and there’s no world where you write custom spiders for 12,000 websites, not with a human team and certainly not sustainably.
So Javier built a workflow: a set of AI prompts that could analyze a website, figure out its structure, and generate a crawl configuration that a generic spider could then use.

I've been running a series of conversations with developers at Zyte to understand what's actually changed in the way they work since LLMs showed up. Not the headlines. The day-to-day. What they delegate, what they don't, what they notice, what surprises them.
This one was different on two counts.

In our interview, a QA expert warns - before you delegate web scraping quality assurance to AI, make sure you can describe what ‘good’ looks like for yourself.

An interview with one of the popular framework’s co-maintainers on Zyte’s new, AI-powered Scrapy sidekick.

What a failed experiment taught me about curated data, prompting, and when scraping actually matters.

Three ways to bring Zyte-powered web data into your AI workflow — from production spiders to conversational extraction.

Discover how Zyte’s open-source libraries like ClearHTML, Extruct, Chomp.js, and more simplify web data extraction and processing.

With AI Scraping in Zyte API, you can pull data from any e-commerce website straight into your Jupyter notebooks.

Discover the challenges of scaling web scraping with Playwright & Puppeteer, from browser farm management to IP rotation and anti-scraping tactics.

User sessions can help overcome website bans, handle IP rate limits, streamline cookie management, and avoid detection.

Discover key techniques to efficiently extract data from JavaScript-heavy websites.

Discover the strengths and limitations of Selenium, Puppeteer, and Playwright for web scraping at scale.

Whether you're scraping a simple webpage or navigating a complex multi-step process, leveraging sessions is key to ensuring success.

Here are four essential Scrapy plugins we use to build efficient web crawlers for our customers.

In the first part, we discussed a template to define the clear purpose of your web scraping system that can help you design your crawlers better and prepare you for the uncertainty involved in a large scale web scraping project.

I recently had the pleasure of participating in the third episode of Graphversation, a monthly live stream series that brings together graph experts and Neo4j enthusiasts for engaging and enlightening discussions about the captivating world of graphs.

Staring at the Statement of Work opened on your laptop screen, your thoughts must be wandering in all directions - where do you start?

Using Selenium and/or Zyte Smart Proxy Manager? You just stumbled upon the right blog.

We have launched a new Zyte SmartProxy Playwright and we’re sure you’re going to love it!

We are super excited to share some good news for all the Puppeteer users who are looking for an easy-to-integrate anti-ban solution for extracting data from javascript-heavy websites.

New to scraping and rotating proxies? Start here to learn what is IP rotation why it is important for web scraping.

At Zyte we are known for our ability to help companies make mission-critical business decisions through the use of web scraping.

Before we begin, take a look at this short video - it's the scene from Harry Potter where he gets The Invisibility Cloak. It’ll help us better understand the concepts behind proxies.

Neha is a Senior Developer Advocate at Zyte. She works at the intersection of web scraping, developer experience, and AI, helping developers understand what is possible with web data and how to achieve it. She creates technical experiences: blogs, events, workshops, and demos that draw inspiration from unexpected places. fiction, films, conversations that stuck, and experiments that broke. She connects the dots and transforms them into content that makes complex things click.