Modern Scrapy for experienced developers: A new series

This is the first of a 4 part mini series to help bring you up to speed with Scrapy, how we use it to scrape the modern web, and how you can extend it to meet your needs. By the end of this series you will be able to write production grade Scrapy code ready for large scale web scraping.

f you have used Scrapy before, you probably remember the satisfying part: write a callback, pull a few fields out of a response, yield a dictionary, and watch a JSON file appear. That workflow still works. The trouble starts when the spider has to run again next week, on a larger catalog, with a missing price, a timeout, or a request that needs different behavior from the rest of the project.

The useful mental shift is to stop treating Scrapy as a convenient loop around requests and selectors. It is a crawler with places for configuration, logging, data processing, and request handling. Using those places is usually less work than rebuilding them in a spider callback.

A working spider is not yet a working Scrapy project

Consider a small catalog spider. It follows product links and extracts a name and price. There is nothing wrong with starting here, and the Scrapy tutorial remains a good way to refresh the basic flow.

Run it with scrapy crawl products -O products.jsonl, and you have data. For a short-lived investigation, that may be all you need.

For a project you intend to keep, ask different questions. Which settings apply to every spider? How will you notice that a selector stopped matching? Where should price normalization live? What is the polite request rate for this site? How can you change the output destination without editing extraction code?

Scrapy already has answers. The rest of this article puts them in the places where they remain useful after the initial crawl.

Put crawl behavior in settings, where it can be reviewed and overridden

settings.py is not a graveyard of copied options. It is the project contract: the behavior that should be consistent across spiders unless one has a specific reason to differ. Scrapy defines a clear precedence order, with command-line settings above spider settings, which sit above project settings and defaults. That makes a temporary diagnostic run possible without committing a one-off change to the repository. See the settings documentation for the full order and built-in reference.

Here is a sensible starting point for the catalog project:

The values are examples, not universal defaults. A site may publish crawl guidance that calls for a lower rate, and an internal API might support more concurrency. The important part is that the decision is explicit, visible in review, and separate from parsing code. AutoThrottle adjusts delay using the observed latency while respecting the limits you set. It is a better starting point than guessing a fixed delay, but it is not permission to crawl a site aggressively.

Use the command line when you need a temporary change. This run writes logs to a file without changing the project settings:

Do not use per-spider settings just because a setting is nearby. Use them when the difference belongs to that spider. For example, a small, high-value spider might need a separate feed while the project defaults remain unchanged:

The feed exports documentation covers storage backends, batching, field order, and other options that become relevant before you write your own exporter.

Make logs explain the crawl, not narrate it

The default crawl log already contains useful request, response, retry, and item statistics. Your spider should add the missing business context. self.logger is a normal Python logger configured by Scrapy, so it fits into the same output and log-level controls as the rest of the crawl. Scrapy's logging guide explains the project-wide configuration.

Logging every successful product is usually noise. Logging an extraction condition that may change the quality of the dataset is useful:

Keep the values as logger arguments rather than formatting the string yourself. Apart from being standard Python logging practice, it keeps the call easy to scan and lets the logging system decide whether the message needs formatting.

When a page is puzzling, use the Scrapy shell before changing the spider. It shortens the loop from "run a crawl, inspect output, edit code" to "inspect the exact response and selector."

If the shell does not contain the data you see in a browser, that is evidence about the page delivery path, not a reason to immediately add a browser. Part four of this series will cover that choice.

Use pipelines for data rules that apply to every item

It is tempting to clean prices in parse_product(). It is also how normalization gets copied into the next three spiders. A pipeline is the right boundary for item-by-item work that should be consistent across the project: validating required fields, normalizing formats, removing duplicates, or sending items to storage. Scrapy calls each enabled item pipeline in order after a spider yields an item.

This example normalizes a simple dollar price and drops items that cannot be used. In a real project, agree on the currency and numeric representation with the data consumer rather than silently assuming dollars.

Enable it in the project settings:

The number controls order. That matters when one pipeline depends on work done by another, such as normalizing a URL before a deduplication pipeline sees it. Avoid creating a pipeline for every tiny transformation, though. Page-specific extraction stays in the spider; a rule shared by the project belongs here.

Know what middleware is for before you need it

Pipelines work on items. Middleware works on the crawl itself. Downloader middleware can modify requests before download or responses after download, which makes it a useful home for cross-cutting concerns such as project-wide headers, authentication, or response diagnostics. Spider middleware sits around a spider's input and output.

You do not need to write custom middleware to benefit from it. Scrapy's own retry, redirect, cookies, HTTP cache, and robot exclusion behavior use this component model. The practical lesson for a returning developer is to resist adding request behavior to every callback. First check whether a built-in setting covers it. If it does not, middleware is usually a cleaner home than a growing stack of if statements in the spider.

The same restraint applies to packages and generated code. Tools can get you to a runnable spider quickly, but a runnable spider still needs project-level decisions about output, rate, retries, and data rules. That distinction appears in this recent look at AI-generated Scrapy projects and is why a project structure matters even when an IDE helps write the first callback.

Keep the first project boring on purpose

There is a version of this article that adds a browser, a queue, a custom downloader, a database, and a monitoring system before the first crawl finishes. That is not the version to copy.

Start with a project setting for behavior you want to keep, a log message for a condition you will need to investigate, a pipeline for a data rule shared across spiders, and feed exports for ordinary output. Once those boundaries are in place, adding a specialized component is a design choice rather than a rescue operation.

If you want help building and maintaining the project from inside your editor, Web Scraping Copilot is designed for Scrapy workflows. Zyte's guide to bringing web scraping into the IDE and the Web Scraping Copilot 1.0 introduction both explore that workflow. The code still deserves the same review: inspect the settings, run the shell, and decide where the data rules live.

What comes next: make the spider easier to change

In the next article, we will take the product extraction out of the callback and use scrapy-poet Page Objects to make it reusable and testable. That is where Scrapy's extension ecosystem starts paying for itself, because the framework gives you a clear place to put project-wide behavior before you need a larger architecture.

For now, take one spider you already have and make four changes: move its stable behavior into settings.py, replace its important print() calls with logs, move one shared data rule into a pipeline, and configure a feed export. Those are small changes, but they are the difference between a script that scraped a site once and a Scrapy project you can trust to revisit.