Settings

This page covers all of the various settings contained within the Kafka Monitor. The sections are broken down by functional component.

Redis

REDIS_HOST

Default: 'localhost'

The Redis host.

REDIS_PORT

Default: 6379

The port to use when connecting to the REDIS_HOST.

Kafka

KAFKA_HOSTS

Default: 'localhost:9092'

The Kafka host. May have multiple hosts separated by commas within the single string like 'h1:9092,h2:9092'.

KAFKA_INCOMING_TOPIC

Default: 'demo.incoming'

The Kafka Topic to read API requests from.

KAFKA_GROUP

Default: 'demo-group'

The Kafka Group to identify the consumer.

KAFKA_FEED_TIMEOUT

Default: 5

How long to wait (in seconds) before timing out when trying to feed a JSON string into the KAFKA_INCOMING_TOPIC

KAFKA_CONN_TIMEOUT

Default: 5

How long to wait (in seconds) before timing out when trying to connect to the Kafka cluster.

Plugins

PLUGIN_DIR

Default: 'plugins/'

The folder containing all of the Kafka Monitor plugins.

PLUGINS

Default:

{
    'plugins.scraper_handler.ScraperHandler': 100,
    'plugins.action_handler.ActionHandler': 200,
    'plugins.stats_handler.StatsHandler': 300,
}

The default plugins loaded for the Kafka Monitor. The syntax for this dictionary of settings is '<folder>.<file>.<class_name>': <rank>. Where lower ranked plugin API’s are validated first.

Logging

LOGGER_NAME

Default: 'kafka-monitor'

The logger name.

LOG_DIR

Default: 'logs'

The directory to write logs into. Only applicable when LOG_STDOUT is set to False.

LOG_FILE

Default: 'kafka_monitor.log'

The file to write the logs into. When this file rolls it will have .1 or .2 appended to the file name. Only applicable when LOG_STDOUT is set to False.

LOG_MAX_BYTES

Default: '10MB'

The maximum number of bytes to keep in the file based log before it is rolled.

LOG_BACKUPS

Default: 5

The number of rolled file logs to keep before data is discarded. A setting of 5 here means that there will be one main log and five rolled logs on the system, generating six log files total.

LOG_STDOUT

Default: True

Log to standard out. If set to False, will write logs to the file given by the LOG_DIR/LOG_FILE

LOG_JSON

Default: False

Log messages will be written in JSON instead of standard text messages.

LOG_LEVEL

Default: 'INFO'

The log level designated to the logger. Will write all logs of a certain level and higher.

Note

More information about logging can be found in the utilities Log Factory documentation.

Stats

STATS_TOTAL

Default: True

Calculate total receive and fail stats for the Kafka Monitor.

STATS_PLUGINS

Default: True

Calculate total receive and fail stats for each individual plugin within the Kafka Monitor.

STATS_CYCLE

Default: 5

How often to check for expired keys and to roll the time window when doing stats collection.

STATS_DUMP

Default: 60

Dump stats to the logger every X seconds. If set to 0 will not dump statistics.

STATS_TIMES

Default:

[
    'SECONDS_15_MINUTE',
    'SECONDS_1_HOUR',
    'SECONDS_6_HOUR',
    'SECONDS_12_HOUR',
    'SECONDS_1_DAY',
    'SECONDS_1_WEEK',
]

Rolling time window settings for statistics collection, the above settings indicate stats will be collected for the past 15 minutes, the past hour, the past 6 hours, etc.

Note

For more information about stats collection, please see the Stats Collector documentation.