New Scrapy blog and Scrapy 0.10 release
Hi everyone!
After two months of work, we're happy to be announcing the release of Scrapy 0.10, the fourth major release after our first 0.7 release almost a year go. We're also using this opportunity to announce the launch the official Scrapy blog, which you are reading right now. And you may have heard of the new Scrapy snippets site to share code examples.
The 0.10 release includes several bug fixes, and a bunch of new features. Here we summarize the most important ones.
New command-line tool
There is a new command line tool for creating and controlling your Scrapy projects. Instead of using the old scrapy-ctl.py script you'll now use the scrapy command which "auto discovers" the project you're working on. So there's no need for a separate script per project anymore. Check the new command-line tool documentation for a list of available commands.
The new command also works out of the box on Windows, if you're using the Windows installer.
Feed exports
Feed exports provide a flexible way to generate data feeds with the scraped data. It supports multiple formats (JSON, XML, CSV) and storage backends (filesystem, FTP, S3). Both formats and storages are pluggable so you can plug your own one.
Persistent spider queues
Scrapy now comes with a persistent spider queue (sometimes referred as Execution queues in 0.9) out of the box (implemented using SQLite), which allows you to schedule spiders to a Scrapy process that is already running.
To illustrate, running this:
scrapy crawl myspider
Is the same as running this:
scrapy queue add myspider
scrapy crawl
You can also start Scrapy in server mode with:
scrapy runserver
And schedule your spiders later with:
scrapy queue add myspider
Scrapy service
Scrapy 0.10 introduces Scrapyd, a new application for running Scrapy as a service and deploying Scrapy projects. Insophia provides Ubuntu packages for Scrapyd (the package source is in the /debian dir) so you can install it with apt-get install scrapyd and start using it. Hopefully, with some community help, we'll get packages for other platforms.
Ubuntu packages
New official Ubuntu packages (of Scrapy and Scrapyd) are provided through APT repos which are continuously updated with the latest bug fixes. If you install Scrapy through the APT repos you'll get bash autocompletion out of the box.
Other Scrapy 0.10 improvements
- Simplified Images pipeline to make it easier to use - see documentation
- Some Scrapy shell usability improvements (the shell should look and feel nicer now)
- Support for returning deferreds on most signals (which enables support for implementing asynchronous extensions)
- Scrapy log refactoring to support hooking up your own log observers (though this is not yet documented)
And much more. For more info see:
Finally, a big thanks to Mydeco (which is already using Scrapy 0.10 in production) for the beta-testing.
So what are you waiting for?. Go download it, and please report any issues you find in the scrapy-users group or the bug tracker. You can also fork Scrapy on Github and fix them.
Previous stable release
As usual, you can find the documentation for the previous stable release at http://doc.scrapy.org/0.9/ and download it from the releases archive at http://scrapy.org/releases/. The Mercurial repo for the previous stable release will also remain available at http://hg.scrapy.org/scrapy-0.9/
UPDATE: The release files have been rebuilt to fix the issue reported here by adding this change.
