These documents provide information about the agnostic utility classes used within Scrapy Cluster that may be useful in other applications.
- Argparse Helper
- Simple module to assist in argument parsing with subparsers.
- Log Factory
- Module for logging multithreaded or concurrent processes to files, stdout, and/or json.
- Method Timer
- A method decorator to timeout function calls.
- Redis Queue
- A module for creating easy redis based FIFO, Stack, and Priority Queues.
- Redis Throttled Queue
- A wrapper around the Redis Queue module to enable distributed throttled pops from the queue.
- Settings Wrapper
- Easy to use module to load both default and local settings for your python application and provides a dictionary object in return.
- Stats Collector
- Module for statistics based collection in Redis, including counters, rolling time windows, and hyperloglog counters.
- Zookeeper Watcher
- Module for watching a zookeeper file and handles zookeeper session connection troubles and re-establishment of watches.