Python 3 Support with Scrapy 1: Embracing the Future

Python 3 is coming to Scrapy

Scrapy with beta Python 3 support is finally here! Released through Scrapy 1.1.0rc1, this is the result of several months of hard work on the part of the Scrapy community and Zyte engineers.

This is a huge milestone for all you Scrapy users (and those who haven’t used Scrapy due to the lack of Python 3). Scrapy veterans and new adopters will soon be able to move their entire stack to Python 3 once the release becomes stable. Keep in mind that since this a release candidate, it is not ready to be used in production.

Updates

Benefits of Scrapy 1.1.0rc1

In addition to Python 3 support, Scrapy 1.1.0rc1 includes a lot of enhancements and bug fixes:

• Scrapy now respects robots.txt by default
• Item loaders now support nested loaders
• Anonymous S3 connections are now allowed
• XML node names containing dots are now accepted
• A new response.text attribute to get the response body as unicode
• Selectors were extracted from Scrapy into a separate library called parsel, allowing you to benefit from using the power of Scrapy selectors in your own projects

Python 3 Limitations

This beta release includes some limitations for Python 3 (which will not affect Python 2 users):

• Scrapy doesn’t work yet in Windows with Python 3
• S3 downloads are not supported yet
• Sending emails is not supported yet
• Telnet is not supported yet
• FTP download handler is not supported yet
• Some non-ASCII URLs may fail in Scrapy with Python 3

If you want the full details of all the benefits of this new release, check out the release notes.

Test it Out

And now it’s your turn, Scrapy users! The sooner this is battle-tested, the sooner a stable release comes your way, so please give it a try by installing or upgrading Scrapy via pip:

1$ pip install scrapy==1.1.0rc1

Wrap Up

We appreciate any feedback or bug reports as they will help a lot in making Python 3 support with Scrapy stable. If you run into any problems, please let us know via the issue tracker. We’re looking forward to releasing the official Scrapy 1.1, so get to experimenting!