Explore resources by topic or category

Smarter tools and AI-driven automation are rewriting the rules of web scraping. As costs fall and setup barriers vanish, smaller teams can now compete at scale, reshaping how the web’s data economy works.

Web scraping tools save hours of work by automating data extraction, testing web applications, and performing repetitive tasks.

It was 6 years ago when Zyte released Dateparser, an open source library that parses human-readable dates, and in October 2020 we released version 1.0.0, a very important milestone.

When crawling the web, there’s always a speed limit. A spider can't fetch faster than the host willing to send the pages.

As a python developer at Zyte (formerly Scrapinghub), I spend a lot of time in the Scrapy shell.

If you’ve been using Scrapy for any period of time, you know the capabilities a well-designed Scrapy spider can give you.

If you know anything about Zyte , you know that we are obsessed with data quality and data reliability.

Absolutely not! Website changes (sometimes very subtly), anti-bot countermeasures, and temporary problems often reduce the quality and reliability of our data.

This is a guest post from the folks over at Intoli, one of the awesome companies providing Scrapy commercial support and longtime Scrapy fans.

The first rule of web crawling is you do not harm the website. The second rule of web crawling is you do NOT harm the website. We’re supporters of the democratization of web data, but not at the expense of the website’s owners.

Welcome to This Month in Open Source at Zyte! In this regular column, we share all the latest updates on our open source projects including Scrapy, Splash, Portia, and Frontera.

We eat our own spider food since Scrapy is our go-to workhorse on a daily basis. However, there are certain situations where Scrapy can be overkill and that’s when we use Parsel.