CrawlerΒΆ

These documents give you an overview of the core Scrapy Cluster Crawler and how to work within both Scrapy and Scrapy Cluster to fit your needs.

Design
Learn about the design considerations for the Scrapy Cluster Crawler
Quick Start
How to use and run the distributed crawlers
Controlling
Learning how to control your Scrapy Cluster will enable you to get the most out of it
Extension
How to use both Scrapy and Scrapy Cluster to enhance your crawling capabilites
Settings
Explains all of the settings used by the Crawler