Quick Start

First, create a localsettings.py to track your overridden custom settings. You can override any setting you find within the Rest service’s settings.py, and a typical file may look like the following:

REDIS_HOST = 'scdev'
KAFKA_HOSTS = 'scdev:9092'

rest_service.py

The main mode of operation for the Rest service is to generate a Restful endpoint for external applications to interact with your cluster.

$ python rest_service.py

Typical Usage

Typical usage of the Rest service is done in conjunction with both the Kafka Monitor and Redis Monitor. The Rest service is a gateway for interacting with both of those components, and provides little functionality when used by itself.

Warning

This guide assumes you have both a running Kafka Monitor and Redis Monitor already. Please see the Kafka Monitor Quick Start or the Redis Monitor Quick Start for more information.

Open two terminals.

Terminal 1:

Run the rest service

$ python rest_service.py

Terminal 2:

Curl the basic endpoint of the component.

$ curl scdev:5343
{
  "kafka_connected": true,
  "my_id": "34806877e13f",
  "node_health": "GREEN",
  "redis_connected": true,
  "uptime_sec": 23
}

You should see a log message come through Terminal 1 stating the message was received and the data transmitted back.

2016-11-11 09:51:26,358 [rest-service] INFO: 'index' endpoint called

Feed a request

$ curl scdev:5343/feed -H "Content-Type: application/json" -d '{"uuid":"abc123", "appid":"stuff"}'

The request provided is outlined at the Kafka Monitor’s API, when using the feed endpoint you need to ensure your request conforms to specification outlined on that page.

You should see a log message come through Terminal 1 stating the message was received and the data transmitted back.

2016-11-06 14:26:05,621 [rest-service] INFO: 'feed' endpoint called

After a short time, you should see the result come through your curl command in terminal 2.

{
  "data": {
    "appid": "stuff",
    "crawler": {
      "machines": {
        "count": 0
      },
      "queue": {
        "total_backlog": 0
      },
      "spiders": {
        "link": {
          "count": 1
        },
        "total_spider_count": 1,
        "unique_spider_count": 1
      }
    },
    "kafka-monitor": {
      "fail": {
        "21600": 1,
        "3600": 1,
        "43200": 1,
        "604800": 1,
        "86400": 1,
        "900": 1,
        "lifetime": 1
      },
      "plugins": {
        "StatsHandler": {
          "21600": 2,
          "3600": 2,
          "43200": 2,
          "604800": 2,
          "86400": 2,
          "900": 2,
          "lifetime": 2
        }
      },
      "total": {
        "21600": 3,
        "3600": 3,
        "43200": 3,
        "604800": 3,
        "86400": 3,
        "900": 3,
        "lifetime": 3
      }
    },
    "redis-monitor": {
      "nodes": {
        "afa660f7e348": [
          "3333a4d63704"
        ]
      },
      "plugins": {
        "StatsMonitor": {
          "21600": 2,
          "3600": 2,
          "43200": 2,
          "604800": 2,
          "86400": 2,
          "900": 2,
          "lifetime": 2
        }
      },
      "total": {
        "21600": 2,
        "3600": 2,
        "43200": 2,
        "604800": 2,
        "86400": 2,
        "900": 2,
        "lifetime": 2
      }
    },
    "server_time": 1478714930,
    "stats": "all",
    "uuid": "abc123"
  },
  "error": null,
  "status": "SUCCESS"
}

At this point, your Rest service is operational.