Scrapy Cluster 1.0 Documentation

This documentation provides everything you need to know about the Scrapy based distributed crawling project, Scrapy Cluster.

Getting Started

Learn about the Scrapy Cluster Architecture.
Quick Start
A Quick Start guide to those who want to jump right in.

Architectural Components

Kafka Monitor
The gateway into your Scrapy Cluster.
The crawling plant behind the scenes.
Redis Monitor
Provides the insight into how your crawls are doing.


Advanced Topics
Advanced items to consider when running Scrapy Cluster.
Scrapy Cluster is licensed under the MIT License.