Scrapy Cluster 1.0 Documentation¶

kafka_monitor.py¶

The Kafka Monitor consists of the following two different functionalities

python kafka-monitor.py run
Usage:
    monitor run --settings=<settings>
    monitor feed --settings=<settings> <json_req>

When in run mode, the class monitors the designated kafka topic and validates it against a formal Json Schema Specification. After the formal json request is validated, the designated SCHEMA_METHOD handler function is then called to process your data. We designed it in this fashion so you could deploy then same simple python file, but dynamically configure it to validate and process your request based on a tunable configuration file. The formal specifications for submitted crawl requests and action requests are outlined in below section.

When in feed mode, the class acts as a simple interface to validate and submit your command line json request. An example is shown in the above section when you submit a new crawl request. Anything in the formal json specification can be submitted via the command line or via a Kafka message generated by another application. The feeder is only used when you need to submit manual crawls to the API for debugging, and any major application should connect via Kafka.

Incoming Crawl Request Kafka Topic:

• demo.incoming_urls - The topic to feed properly formatted crawl requests to

Outbound Crawl Result Kafka Topics:

• demo.crawled_firehouse - A firehose topic of all resulting crawls within the system. Any single page crawled by the Scrapy Cluster is guaranteed to come out this pipe.
• demo.crawled_<appid> - A special topic created for unique applications that submit crawl requests. Any application can listen to their own specific crawl results by listening to the the topic created under the appid they used to submit the request. These topics are a subset of the crawl firehose data and only contain the results that are applicable to the application who submitted it.

Example Crawl Requests:

python kafka-monitor.py feed '{"url": "http://www.apple.com/", "appid":"testapp", "crawlid":"myapple"}' -s settings_crawling.py

• Submits a single crawl of the homepage of apple.com

python kafka-monitor.py feed '{"url": "http://www.dmoz.org/", "appid":"testapp", "crawlid":"abc123", "maxdepth":2, "priority":90}' -s settings_crawling.py

• Submits a dmoz.org crawl spidering 2 levels deep with a high priority

python wat-scrapy-kafka-monitor.py feed '{"url": "http://aol.com/", "appid":"testapp", "crawlid":"a23bbqwewqe", "maxdepth":3, "allowed_domains":["aol.com"], "expires":1423591888}' -s settings_crawling.py

• Submits an aol.com crawl that runs for (at the time) 3 minutes with a large depth of 3, but limits the crawlers to only the aol.com domain so as to not get lost in the weeds of the internet.

Example Crawl Request Output from the kafkadump utility:

{
    u'body': u'<real raw html source here>',
    u'crawlid': u'abc1234',
    u'links': [],
    u'response_url': u'http://www.dmoz.org/Recreation/Food/',
    u'url': u'http://www.dmoz.org/Recreation/Food/',
    u'status_code': 200,
    u'status_msg': u'OK',
    u'appid': u'testapp',
    u'headers': {
        u'Cteonnt-Length': [u'40707'],
        u'Content-Language': [u'en'],
        u'Set-Cookie': [u'JSESSIONID=FB02F2BBDBDBDDE8FBE5E1B81B4219E6; Path=/'],
        u'Server': [u'Apache'],
        u'Date': [u'Mon, 27 Apr 2015 21:26:24 GMT'],
        u'Content-Type': [u'text/html;charset=UTF-8']
    },
    u'attrs': {},
    u'timestamp': u'2015-04-27T21:26:24.095468'
}

For a full specification as to how you can control the Scrapy Cluster crawl parameters, please refer to the scraper_schema.json documentation.

kafkadump.py¶

The Kafka dump utility stemmed from the need to quickly view the resulting kafka crawl results. This is a simple utility designed to do two things:

python kafkadump.py --help

Usage:
    kafkadump list --host=<host>
    kafkadump dump <topic> --host=<host> [--consumer=<consumer>]

When ran with the command list the utility will dump out all of the topics created on your cluster.

When ran with the dump command, the utility will connect to Kafka and dump every message in that topic starting from the beginning. Once it hits the end it will sit there and wait for new data to stream through. This is especially useful when doing command line debugging of the cluster to ensure that crawl results are flowing back out from the system, or for monitoring the results of information requests.

scraper_schema.json¶

The Scraper Schema defines the level of interaction an application gets with the Scrapy Cluster. The following properties are available to control the crawling cluster:

Required

• appid: The application ID that submitted the crawl request. This should be able to uniquely identify who submitted the crawl request
• crawlid: A unique crawl ID to track the executed crawl through the system. Crawl ID’s are passed along when a maxdepth > 0 is submitted, so anyone can track all of the results from a given seed url. Crawl ID’s also serve as a temporary duplication filter, so the same crawl ID will not continue to recrawl pages it has already seen.
• url: The initial seed url to begin the crawl from. This should be a properly formatted full path url from which the crawl will begin from

Optional:

• spiderid: The spider to use for the crawl. This feature allows you to chose the spider you wish to execute the crawl from
• maxdepth: The depth at which to continue to crawl new links found on pages
• priority: The priority of which to given to the url to be crawled. The Spiders will crawl the highest priorities first.
• allowed_domains: A list of domains that the crawl should stay within. For example, putting [ "cnn.com" ] will only continue to crawl links of that domain.
• allow_regex: A list of regular expressions to apply to the links to crawl. Any hits within from any regex will allow that link to be crawled next.
• deny_regex: A list of regular expressions that will deny links to be crawled. Any hits from these regular expressions will deny that particular url to be crawled next, as it has precedence over allow_regex.
• deny_extensions: A list of extensions to deny crawling, defaults to the extensions provided by Scrapy (which are pretty substantial).
• expires: A unix timestamp in seconds since epoch for when the crawl should expire from the system and halt. For example, 1423669147 means the crawl will expire when the crawl system machines reach 3:39pm on 02/11/2015. This setting does not account for timezones, so if the machine time is set to EST(-5) and you give a UTC time for three minutes in the future, the crawl will run for 5 hours and 3 mins!
• useragent: The header request user agent to fake when scraping the page. If none it defaults to the Scrapy default.
• attrs: A generic object, allowing an application to pass any type of structured information through the crawl in order to be received on the other side. Useful for applications that would like to pass other data through the crawl.

action_schema.json¶

The Action Schema allows for extra information to be gathered from the Scrapy Cluster, as well as stopping crawls while they are executing. These commands are executed by the Redis Monitor, and the following properties are available to control.

Required

• appid: The application ID that is requesting the action.
• spiderid: The spider used for the crawl (in this case, link)
• action: The action to take place on the crawl. Options are either info or stop
• uuid: A unique identifier to associate with the action request. This is used for tracking purposes by the applications who submit action requests.

Optional:

• crawlid: The unique crawlid to act upon. Only needed when stopping a crawl or gathering information about a specific crawl.

distributed_scheduler.py¶

A Scrapy Scheduler that tries to find new requests from the Redis queue associated with the spider’s name. Since we are using a link spider, the queue it will look under will be called link:queue.

The scheduler has all of the same overridden functions you would expect, except for has_pending_requests(). In our tests we found inconsistencies in Scrapy when returning True when grabbing a new item from within that function or letting the item be returned from next_request. For now we return False so that the spider goes back to idle, but it is worth investigating more in the future.

items.py¶

Holds a single basic item yielded by the crawlers to the item pipeline. The RawResponseItem returns other metadata associated with the crawl including:

• response url
• status code
• status message
• headers
• body text
• links found
• passed through attributes

pipelines.py¶

The pipelines file is a basic Scrapy Item Pipeline with a single class to take the yielded item and send it back to Kafka. It also checks to make sure that the Kafka topic exists before sending the message to it.

redis_dupefilter.py¶

An extremely basic class that serves as a crawl link duplication filter utilizing a Redis Set. This allows two important things:

• Any unique crawlid will not recrawl a url it has already seen
• New crawl requests with a different crawlid can crawl those same links, without being effected by other crawl duplication filters

This allows for a crawl job over a variety of links to not waste resources by crawling the same things. If you would like to recrawl those same urls, simply submit the same url with a different crawl identifier to the API. If you would like to continue to expand your crawl frontier, submit a crawl with the same identifier.

redis_queue.py¶

A utility class that utilizes Pickle encoding to store and retrieve arbitrary sets of data in Redis. The queues come in three basic forms:

• RedisQueue - A FIFO queue utilizing a Redis List
• RedisStack - A Stack implementation utilizing a Redis List
• RedisPriorityQueue - A prioritized queue utilizing a Redis Sorted Set. This is the queue utilized by the scheduler for prioritized crawls

All three of these classes can handle arbitrary sets of data, and handle the pickle encoding and decoding for you.

redis_retry_middleware.py¶

This class is a Scrapy Downloader Middleware that catches 504 timeout exceptions thrown by the spider. These exceptions are handled differently from other status codes because the spider never even got to the url, so the downloader throws an error.

The url is thrown back into the cluster queue at a lower priority so the cluster can try all other higher priority urls before the one that failed. After a certain amount of retries, the url is given up on and discarded from the queue.

redis_spider.py¶

A base class that extends the default Scrapy Spider so we can crawl continuously in cluster mode. All you need to do is implement the parse method and everything else is taken care of behind the scenes.

link_spider.py¶

An introduction into generic link crawling, the LinkSpider inherits from the base class RedisSpider to take advantage of a simple html content parse. The spider’s main purpose is to generate two things:

1. Generate more urls to crawl, found by grabbing all the links on the page
2. Generate a RawResponseItem to be processed by the item pipeline.

These two things enable generic depth based crawling, and the majority of the code used within the class is to generate those two objects. For a single page this spider might yield 100 urls to crawl and 1 html item to be processed by the Kafka pipeline.

lxmlhtml.py¶

This is actually a duplicate of the Scrapy LxmlParserLinkExtractor but with one slight alteration. We do not want Scrapy to throw link extraction parsing errors when encountering a site with malformed html or bad encoding, so we changed it to ignore errors instead of complaining. This allows for the continued processing of the scraped page all the way through the pipeline even if there are utf encoding problems.

In the future this may just be an extended class but for now it is the full copy/paste.

redis_monitor.py¶

The Redis Monitor acts upon two different action requests, and also sends out notifications when a crawl job expires from the queues.

All requests adhere to the following three Kafka topics for input and output:

Incoming Action Request Kafka Topic:

• demo.inbound_actions - The topic to feed properly formatted action requests to

Outbound Action Result Kafka Topics:

• demo.outbound_firehose - A firehose topic of all resulting actions within the system. Any single action conducted by the Redis Monitor is guaranteed to come out this pipe.
• demo.outbound_<appid> - A special topic created for unique applications that submit action requests. Any application can listen to their own specific action results by listening to the the topic created under the appid they used to submit the request. These topics are a subset of the action firehose data and only contain the results that are applicable to the application who submitted it.

Information Action

The info action can be conducted in two different ways.

Application Info Request

python kafka-monitor.py feed -s settings_actions.py '{"action":"info", "appid":"testapp", "uuid":"someuuid", "spiderid":"link"}

This returns back all information available about the appid in question. It is a summation of the various crawlid statistics.

Application Info Response

{
    u'server_time': 1429216294,
    u'uuid': u'someuuid',
    u'total_pending': 12,
    u'total_domains': 0,
    u'total_crawlids': 2,
    u'appid': u'testapp',
    u'crawlids': {
        u'2aaabbb': {
            u'low_priority': 29,
            u'high_priority': 29,
            u'expires': u'1429216389'
            u'total': 1
        },
        u'1aaabbb': {
            u'low_priority': 29,
            u'high_priority': 39,
            u'total': 11
        }
    }
}

Here, there were two different crawlid‘s in the queue for the link spider that had the specified appid. The json return value is the basic structure seen above that breaks down the different crawlid‘s into their total, their high/low priority in the queue, and if they have an expiration.

Crawl ID Info Request

python kafka-monitor.py feed -s settings_actions.py '{"action":"info", "appid":"myapp", "uuid":"someuuid", "crawlid":"abc123", "spiderid":"link"}'

This is a very specific request that is asking to poll a specific crawlid in the link spider queue. Note that this is very similar to the above request but with one extra parameter. The following example response is generated:

Crawl ID Info Response from Kafka

{
    u'server_time': 1429216864,
    u'crawlid': u'abc123',
    u'total_pending': 28,
    u'low_priority': 39,
    u'high_priority': 39,
    u'appid': u'testapp',
    u'uuid': u'someuuid'
}

The response to the info request is a simple json object that gives statistics about the crawl in the system, and is very similar to the results for an appid request. Here we can see that there were 28 requests in the queue yet to be crawled of all the same priority.

Stop Action

The stop action is used to abruptly halt the current crawl job. A request takes the following form:

Stop Request

python kafka-monitor.py feed -s settings_actions.py '{"action":"stop", "appid":"testapp", "uuid":"someuuid2", "crawlid":"ABC123", "spiderid":"link"}'

After the request is processed, only current spiders within the cluster currently in progress of downloading a page will continue. All other spiders will not crawl that same crawlid past a depth of 0 ever again, and all pending requests will be purged from the queue.

Stop Response from Kafka

{
    u'total_purged': 524,
    u'uuid': u'someuuid',
    u'spiderid': u'link',
    u'appid': u'testapp',
    u'action': u'stop',
    u'crawlid': u'ABC123'
}

The json response tells the application that the stop request was successfully completed, and states how many requests were purged from the particular queue.

Expire Notification

An expire notification is generated by the Redis Monitor any time an on going crawl is halted because it has exceeded the time it was supposed to stop. A crawl request that includes an expires attribute will generate an expire notification when it is stopped by the Redis Monitor.

Expire Notification from Kafka

{
    u'total_expired': 75,
    u'crawlid': u'abcdef-1',
    u'spiderid': u'link',
    u'appid': u'testapp',
    u'action': u'expired'
}

This notification states that the crawlid of “abcdef-1” expired within the system, and that 75 pending requests were removed.