The future of Scrapy: Smarter, faster and ready for AI-powered scraping

It’s the little framework that was created for a single task but grew into the web scraping tool of choice for millions of developers.

Ten years after it reached version 1.0, Scrapy has amassed 82 million downloads and almost 11,000 commits, thanks to more than 575 open-source contributors.

But what does the future hold for the tool some describe as “the gift that revolutionised web scraping”?

Recent updates

In the 2020s, Scrapy has already been evolving as a modern framework, with more recent additions including:

1. Asynchronous powerhouse: Scrapy has been adding support for asyncio, allowing use of myriad asyncio-based libraries.

2. Headless browsers and beyond: Modern web scraping often involves rendering JavaScript-heavy pages. Scrapy already plays nicely with tools like Playwright, making it competitive with newer, browser-based scraping frameworks. The Scrapy API supports custom downloaders, making it easy to develop support for Playwright and future browser automation tools.

3. Ecosystem of extensions: Scrapy’s plugin architecture has evolved, with powerful middlewares and add-ons that extend its functionality, from handling CAPTCHAs to integrating with cloud-based storage and data pipelines.

That is thanks partly to Scrapy’s steward, Zyte, but also to its active community of developers building on what was always an inherent extensibility.

Meanwhile, to celebrate the tenth anniversary of Scrapy’s 1.0 version, June 2025 also saw a major facelift for Scrapy’s own website.

Principles for the future

The future will build on Scrapy as a foundational scraping framework, in a largely evolutionary approach.

“The focus ahead is on modernizing Scrapy while staying true to its original principles,” according to Scrapy’s co-originator Shane Evans, CEO, Zyte.

Mikhail Korobov, head of development, Zyte, agrees. “We will keep doing what it’s doing, but better,” he says. That means:

• Continuing Scrapy’s heritage of extensibility.

• Out-of-the-box usability - effective default options to speed up time to results.

• Compatibility with the general Python ecosystem.

• A simple programmatic API for developers.

“In general, we prefer extensibility over hard-coded, out-of-box features,” Korobov adds.

Future features

The team has some specific developments in mind. As Evans puts it: “That means enhancing support for modern web technologies (especially JavaScript-heavy sites), improving integrations with headless browsers like Playwright, and continuing to streamline the developer experience – particularly around configuration and observability.”

Execution control

“We are planning enhancements to retry logic, rate limiting, delay handling, etc,” Korobov says.

The changes would improve the way Scrapy deals with failure, speed, and politeness so it can scrape the web more smoothly, more reliably, and more respectfully.

Modernised engine

“We’re rewriting Scrapy core from Twisted to asyncio primitives; the plan is to eventually make the Twisted reactor optional,” says Korobov.

Such a move would help make Scrapy simpler, faster, and more compatible with asynchronous Python code that has become the popular modern go-to.

Better organization

Mikhail Korobov reveals: “We’re exploring different ways to organize the web scraping code, such as page objects (web-poet library), spider templates, etc.

“Scrapy spiders are easy to get started, but having all the code in a single spider class also can get in the way.

“These “new” paradigms will allow developers to make web scraping projects more maintainable in the long run. They also turn out to be a better fit for various AI tools.”

AI-ready, closer to the user

Beyond those updates, two key motions are on the horizon, each aimed at making scraping easier.

Artificial intelligence is now a key area of focus. As Shane Evans says: “We’re thinking about how Scrapy can stay relevant in a world increasingly shaped by AI.”

The world is witnessing an explosion of tools like Cursor, Windsurf, OpenAI’s Codex and Claude Code for accelerating and improving code writing.

At Zyte, AI-assisted coding capabilities are already making Zyte’s own developers up to twice as productive.

Now the company wants to bring the same powers to writing scraping code.

So Zyte is working to:

1. Enhance its AI tools for easy spider code creation, available to both non-technical users and in developer tooling.

2. Make Scrapy features more immediately accessible in code editors.

“We’d like to help people to generate web scraping code, and Scrapy is a foundation framework for this, alongside libraries like web-poet,” Korobov explains.

Ready for the future

Scrapy, then, is ready for a future in which its historic extensibility and its central utility at the heart of scraping operations get augmented by a new wave of services, making its capabilities more immediate than ever.

“I think it plays really well with a lot of the AI tooling that we're seeing nowadays,” Evans says.

While the future may see scraping developers conjure spider code effortlessly, Evans believes, underneath the hood, it will be Scrapy “at the core of it”.