Explore resources by topic or category

Learn how Claude skills can automate HTML fetching, AI parsing, selector generation, and structured data extraction to build faster, smarter web scraping workflows.

Demo project scrape2postgresql shows how to scrape structured data with Scrapy, store it in PostgreSQL, and run both the spider and database in separate containers using Docker Compose.

What does the future hold for the tool some describe as “the gift that revolutionised web scraping”?

With the emergence of managed data extraction vendors, businesses no longer need to gather web data themselves.

The Strategic Case for Buying Web Data: Quality, Focus, and Scale

Learn how successful open-source projects balance community value with sustainable growth. Industry leaders share insights on monetization, maintenance, and building thriving communities.

Discover how Zyte’s open-source libraries like ClearHTML, Extruct, Chomp.js, and more simplify web data extraction and processing.

Discover the strengths and limitations of Selenium, Puppeteer, and Playwright for web scraping at scale.

Here are four essential Scrapy plugins we use to build efficient web crawlers for our customers.

Web scraping tools save hours of work by automating data extraction, testing web applications, and performing repetitive tasks.

When crawling the web, there’s always a speed limit. A spider can't fetch faster than the host willing to send the pages.

If you’ve been using Scrapy for any period of time, you know the capabilities a well-designed Scrapy spider can give you.