Building a markov-chain IRC bot with python and Redis

January 24, 2012 22:59 / irc python / 0 comments

As an IRC bot enthusiast and tinkerer, I would like to describe the most enduring and popular bot I've written, a markov-chain bot. Markov chains can be used to generate realistic text, and so are great fodder for IRC bots. The bot I am writing of has been hanging out in my town's channel for the past year or so and has amassed a pretty awesome corpus from which it generates messages. Here are few of his greatest hits:

whatyousay muses on his recent downtime:

matt: whatyousay goes down like a short DHCP lease.

whatyousay: like a hair eating ninja

whatyousay responds to adam's inquiry:

adam: whatyousay: do you think I was being suggestive?

whatyousay: think i just got trolled by adam.

Daniel and I are talking about gaming and whatyousay takes it too far

cleifer: all i want for christmas is a headshot

daniel: cleifer I will grant you that gift

whatyousay: i will bathe in your blood

Ben tries to complement whatyousay, who is in a reflective mood

ben: whatyousay is the coolest office pet ever.

whatyousay: is the reason i started programming, sad as that is.

How it works

The idea behind the bot's operation is that, based on things that have been said previously, new sentences can be generated. The "chain" part of "markov chain" illustrates how distinct elements (words, in our case) often follow one another in patterns, forming chains of words. A chain that is 2 words long looks something like this (forgive the crap diagram):

As sentences are added to the corpus, the number of branches at each step grows. The program only needs a single "key" to start with, and it can then follow branches randomly until it reaches a dead end, building up a new sentence along the way.

So how is this data structured in our code? We'll skip the Redis implementation details for now, and instead think of the bot's corpus as a big python dictionary. A new phrase pops up in the IRC channel -- what happens?

cleifer: this is a test

The first step is to split the phrase into rolling groups of words that are the length of our chain + 1, including one group that contains just the last two words plus a "stop word":

Group 1: ['this', 'is', 'a']
Group 2: ['is', 'a', 'test']
Group 3: ['a', 'test', <stop>]

Here is the code:

def split_message(self, message):
    # split the incoming message into words, i.e. ['what', 'up', 'bro']
    words = message.split()

    # if the message is any shorter, it won't lead anywhere
    if len(words) > self.chain_length:

        # add the stop word onto the message
        # ['what', 'up', 'bro', '\x02']
        words.append(self.stop_word)

        # len(words) == 4, so range(4-2) == range(2) == 0, 1, meaning
        # we return the following slices: [0:3], [1:4]
        # or ['what', 'up', 'bro'], ['up', 'bro', '\x02']
        for i in range(len(words) - self.chain_length):
            yield words[i:i + self.chain_length + 1]

Then, we'll store the first two words (where two is the length of our chain) as the key in the dictionary and add the third word to a list at that key:

{
    ('this', 'is'): ['a'],
    ('is', 'a'):    ['test'],
    ('a', 'test'):  [<stop>],
}

In this way, by doing a simple search, we can reconstruct the sentence by starting with the key ('this', 'is'). Suppose another phrase comes in that reads:

cleifer: but this is something else

The dictionary will now look like this:

{
    ('this', 'is'): ['a', 'something'], # <-- added "something"
    ('is', 'a'):    ['test'],
    ('a', 'test'):  [<stop>],
    ('but', 'this'): ['is'],         # <-- this line and below are new
    ('is', 'something'): ['else'],
    ('something', 'else'): [<stop>],
}

We could now generate a sentence like "but this is a test" by randomly following from one word to another assuming we started with "but this".

Storing data in Redis

Redis acts, in many ways, like a big python dictionary that can store several types of useful data structures. For our purposes, we will use the set data type. The top-level keyspace will contain our "keys", which will be encoded links in our markov chain. At each key there will be a set of words that have followed the words encoded in the key. To generate "random" messages, we'll use the "SRANDMEMBER" command, which returns a random member from a set.

The code to generate a message based on a "seed" key looks like this:

def generate_message(self, seed):
    key = seed

    # keep a list of words we've seen
    gen_words = []

    # only follow the chain so far, up to <max words>
    for i in xrange(self.max_words):

        # split the key on the separator to extract the words -- the key
        # might look like "this\x01is" and split out into ['this', 'is']
        words = key.split(self.separator)

        # add the word to the list of words in our generated message
        gen_words.append(words[0])

        # get a new word that lives at this key -- if none are present we've
        # reached the end of the chain and can bail
        next_word = self.redis_conn.srandmember(self.make_key(key))
        if not next_word:
            break

        # create a new key combining the end of the old one and the next_word
        key = self.separator.join(words[1:] + [next_word])

    return ' '.join(gen_words)

Putting it together

The final part is to wire this up as an IRC bot and allow it to interact with other users in the channel. For this I used a library I've built, which you can find on github or by installing using pip:

pip install irckit

The only pieces missing are the parts that:

log messages from the user, adding them to the corpus
generate new messages based on some context

These bits are covered in one big method, simply called "log". The idea is that, as we're splitting up and storing the incoming message, we can use those little pieces to generate a bunch of new messages:

def log(self, sender, message, channel):
    # speak only when spoken to, or when the spirit moves me
    say_something = self.is_ping(message) or (
        sender != self.conn.nick and random.random() < self.chattiness
    )

    messages = []

    # use a convenience method to strip out the "ping" portion of a message
    if self.is_ping(message):
        message = self.fix_ping(message)

    if message.startswith('/'):
        return

    # split up the incoming message into chunks that are 1 word longer than
    # the size of the chain, e.g. ['what', 'up', 'bro'], ['up', 'bro', '\x02']
    for words in self.split_message(self.sanitize_message(message)):
        # grab everything but the last word
        key = self.separator.join(words[:-1])

        # add the last word to the set
        self.redis_conn.sadd(self.make_key(key), words[-1])

        # if we should say something, generate some messages based on what
        # was just said and select the longest, then add it to the list
        if say_something:
            best_message = ''
            for i in range(self.messages_to_generate):
                generated = self.generate_message(seed=key)
                if len(generated) > len(best_message):
                    best_message = generated

            if best_message:
                messages.append(best_message)

    if len(messages):
        return random.choice(messages)

All together, here is the code for the redis-backed markov bot.

Thanks for reading, I hope you've enjoyed this post. This project has been a fun one for me and my coworkers. Feel free to submit any suggestions/corrections/etc in the comment section!

Comments (0)

Commenting has been closed.

Building a markov-chain IRC bot with python and Redis

How it works

Storing data in Redis

Putting it together

Further reading

Comments (0)