Kafka MonitorΒΆ

The Kafka Monitor serves as the entry point into the crawler architecture. It validates API requests so that any point afterwards can ensure the data is in the correct format.

Design
Learn about the design considerations for the Kafka Monitor
Quick Start
How to use and run the Kafka Monitor
API
The default Kafka API the comes with Scrapy Cluster
Plugins
Gives an overview of the different plugin components within the Kafka Monitor, and how to make your own.
Settings
Explains all of the settings used by the Kafka Monitor