Scrapy Cluster 1.0 Documentation¶
This documentation provides everything you need to know about the Scrapy based distributed crawling project, Scrapy Cluster.
Getting Started¶
- Overview
- Learn about the Scrapy Cluster Architecture.
- Quick Start
- A Quick Start guide to those who want to jump right in.
Architectural Components¶
- Kafka Monitor
- The gateway into your Scrapy Cluster.
- Crawler
- The crawling plant behind the scenes.
- Redis Monitor
- Provides the insight into how your crawls are doing.
Miscellaneous¶
- Advanced Topics
- Advanced items to consider when running Scrapy Cluster.
- License
- Scrapy Cluster is licensed under the MIT License.