April 15, 2012 11:23 / 0 comments / irc python redis

I recently rewrote my personal site using flask and peewee, breaking a good amount of stuff in the process. I was trying to track down the errors by tailing log files, but that didn't help alert me to new errors that someone visiting the site might stir up. I thought about setting up error emails a-la django, which is a tried and true method...but then I happened on a different approach. I won't say it's the most elegant solution, but it was a quick hack and the results have been awesome. I wrote a custom logging handler that pushes JSON-encoded log record data to a redis pub/sub channel. I then have an IRC bot that subscribes to this channel and when it receives a message generates a paste of the traceback and pings me with a link to the traceback.

The following screenshot shows the entire workflow:

Redis Errors

And here is the output that is dumped into the paste:

Redis Error (Pastebin)

Coding this up

There are really just 2 components we'll need to write - the handler and the bot. The most interesting is the logging handler, since it can be wired up to anything. The most fun part is, of course, the IRC bot. I chose IRC since, well, I enjoy writing irc bots (1, 2), but it could be anything. I've also included a tiny flask application so you can see how I've wired up the handler.

To build this sample code you'll need the following libraries, all are installable with pip:

IRC bot:

Sample app:

Redis pub/sub logging handler

The logging handler subclasses the standard library logging.Handler, and when asked to emit a record simply encodes it as json and publishes it to the channel. Note that it is also passing an additional key, 'formatted', which contains a formatted copy of the record (this is used so we can easily get a formatted traceback):

import json
import logging
import redis


class RedisHandler(logging.Handler):
    def __init__(self, channel, conn, *args, **kwargs):
        logging.Handler.__init__(self, *args, **kwargs)
        self.channel = channel
        self.redis_conn = conn

    def emit(self, record):
        attributes = [
            'name', 'msg', 'levelname', 'levelno', 'pathname', 'filename',
            'module', 'lineno', 'funcName', 'created', 'msecs', 'relativeCreated',
            'thread', 'threadName', 'process', 'processName',
        ]
        record_dict = dict((attr, getattr(record, attr)) for attr in attributes)
        record_dict['formatted'] = self.format(record)
        try:
            self.redis_conn.publish(self.channel, json.dumps(record_dict))
        except redis.RedisError:
            pass

A trivial application

Next I'll show a trivial flask application so you can see how I've wired up the handler. All it does is throw an error at the root url:

from flask import Flask

app = Flask(__name__)

@app.route('/')
def homepage():
    raise ValueError('uh oh')

if __name__ == '__main__':
    app.run()

So how do you wire up the handler? I do this right after instantiating the app:

# imports required by the logging handler, add these at top of module
import logging
from redis import Redis
from redis_handler import RedisHandler

# instantiate a redis connection and a logging handler
redis_conn = Redis()
handler = RedisHandler('error-channel', redis_conn)

# only capture errors
handler.setLevel(logging.ERROR)

# lastly, add our handler to the logger for our app (flask sets this up)
app.logger.addHandler(handler)

The fun part, the IRC bot

The irckit library I wrote is handy for writing small bots. Below is the functional shell of our bot:

#!/usr/bin/env python

from gevent import monkey
monkey.patch_all()

import gevent
import httplib2
import json
import redis

from irc import IRCBot, run_bot
from urllib import urlencode


class ErrorBot(IRCBot):
    def __init__(self, *args, **kwargs):
        super(ErrorBot, self).__init__(*args, **kwargs)

        # api key used for pastebin
        self.api_key = 'your-pastebin-key'
        self.api_url = 'http://pastebin.com/api/api_post.php'

        # who to send links to
        self.master = 'your-nick-here'

        # redis pubsub channel
        self.channel = 'error-channel'

        # redis connection & pubsub
        self.redis_conn = redis.Redis()
        pubsub = self.redis_conn.pubsub()

        gevent.spawn(self.sub, pubsub)

    def sub(self, pubsub):
        pubsub.subscribe(self.channel)
        for msg in pubsub.listen():
            print msg['data']

    def command_patterns(self):
        return ()

host = 'irc.freenode.net'
port = 6667
nick = 'super-error-bot'
run_bot(ErrorBot, host, port, nick, [])

Nothing super interesting here yet, we're just starting up the bot and creating a separate greenlet (a lightweight thread) to subscribe to our error channel and print messages to stdout. You may have noticed that you'll need a pastebin api key, head over to http://pastebin.com/api to get one.

Now let's add the code that will deserialize the json, post the traceback, and lastly send the "master" a private message with a link to the traceback. Add this method to the bot:

def paste(self, msg_data):
    json_data = json.loads(msg_data)
    msg = json_data['formatted']

    sock = httplib2.Http()
    request_headers = {'Content-type': 'application/x-www-form-urlencoded'}
    post_data = urlencode({
        'api_dev_key': self.api_key,
        'api_option': 'paste',
        'api_paste_code': msg,
        'api_paste_private': '1', # 1 = unlisted, 2 = private
        'api_paste_expire_date': '1D',
    })

    headers, resp = sock.request(self.api_url, 'POST', post_data, headers=request_headers)
    status = int(headers['status'])
    if status == 200:
        self.respond(resp.strip(), nick=self.master)
    else:
        self.respond('error posting %s' % (status), nick=self.master)

And finally replace the "print" statement with a call to the new paste method:

def sub(self, pubsub):
    pubsub.subscribe(self.channel)
    for msg in pubsub.listen():
        gevent.spawn(self.paste, msg['data'])

That's it!

Reading more

Interested in adding something like this to your django app? As of 1.3 django introduced pretty good support for configuring loggers. Check the docs for examples.

Also David Cramer has written a popular project sentry that does realtime error logging, exposed via a web interface.

Finally, wraithan has written a redis powered bot (using my library, incidentally!) that supports modular extensions called ZENIrcBot.

More like this

Comments (0)


Commenting has been closed, but please feel free to contact me