Examples of using Walrus, a lightweight Redis Toolkit


walrus is my go-to toolkit for working with Redis in Python, and hopefully this post will convince you that it can be your go-to as well. I've tried to include lots of high-level Python APIs built on Redis primitives and the result is quite a lot of functionality. In this post I'll take you on a tour of the library and show examples of how it might be useful in your next project.

To follow along, you can install walrus locally by running:

$ pip install walrus

Also make sure that redis is running locally.

Introduction to Walrus

At the lowest level, Walrus is just a wrapper on top of Andy McCurdy's redis-py. Walrus adds new functionality, but does not detract from what's already there, so if you're familiar with redis-py, you will be familiar with Walrus.

To get started, let's import walrus and check it out:

>>> from walrus import Walrus
>>> db = Walrus()
>>> db

As you can see, the Walrus() instance is just a fancy wrapper.


At the next level, Walrus provides Pythonic container types for the various Redis data-types. Walrus provides Pythonic wrappers for:

These containers use Python magic methods to make them behave like the Python analogues. For instance, the Hash acts a lot like a dict:

>>> h = db.Hash('huey-info')
>>> h['color'] = 'white'
>>> h.update(temperament='feisty', eyes='blue')
<Hash "my-hash": {'eyes': 'blue', 'foo': 'bar', 'temperament': 'feisty'}>
>>> len(h)
>>> 'eyes' in h
>>> sorted(h)
[('eyes', 'blue'), ('foo', 'bar'), ('temperament', 'feisty')]

For details on the various container types and their APIs, check out the containers documentation.

High-level APIs

Walrus provides high-level APIs for the following functionality:


Let's begin with caching and work our way through to Models, which are the most complex and also the most interesting I think.

The caching APIs are pretty simple. There are the standard get, set, and delete we're all familiar with from our memcached days, but in addition there is a handy decorator that can be used to effectively "memoize" the results of a function call. There is also a cached_property decorator that works just like the function decorator but exposes the cached method call as a property. Also possibly of interest is the cache_async() decorator, which will execute the wrapped function in a separate thread.

Here's a very simple example demonstrating the use of the cache:

>>> import datetime
>>> cache = db.cache()
>>> @cache.cached()
... def what_time_is_it(seed=0):
...     return datetime.datetime.now()

>>> what_time_is_it()
datetime.datetime(2016, 1, 13, 23, 27, 44, 160312)
>>> what_time_is_it()  # We get the cached value.
datetime.datetime(2016, 1, 13, 23, 27, 44, 160312)

>>> what_time_is_it(123)  # We get a new value since the function args changed.
datetime.datetime(2016, 1, 13, 23, 28, 02, 211871)
>>> what_time_is_it(123)  # We get the previous value since the seed is the same.
datetime.datetime(2016, 1, 13, 23, 28, 02, 211871)

>>> what_time_is_it()  # Back to the original value.
datetime.datetime(2016, 1, 13, 23, 27, 44, 160312)

For more information, check out the caching docs.


The walrus autocomplete is much more powerful than the autocomplete examples you typically see that involve the use of ZRANGEBYLEX or something similar. walrus uses a complex scoring algorithm to ensure that multi-word titles are sorted correctly when being returned to the user, ensuring that the closest matches come first. The walrus autocomplete engine can also store rich metadata along with the titles. Let's say you're building a movie search box using the autocomplete. You could store the movie's year, a URL to a thumbnail of the box-cover, and the URL to the movie detail page in the autocomplete index, that way when you are showing a user their results you can avoid a second round-trip to the database for that metadata.

Let's look at a simple example that does not use metadata. We'll just store simple titles and the search engine will return simple string results.

>>> ac = db.autocomplete()
>>> ac.store('charlie and huey are friends')
>>> ac.store('huey is not friends with mickey')
>>> ac.store('zaizee loves huey')
>>> ac.store('zaizee and huey are cats')

>>> [result for result in ac.search('hue')]
[u'huey is not friends with mickey',
 u'charlie and huey are friends',
 u'zaizee and huey are cats',
 u'zaizee loves huey']

Note how the first result starts with the phrase we were looking for, "hue*". In the subsequent results, "huey" is the third word, the results are then sorted alphabetically by word. This type of sorting, along with metadata storage, are the primary reasons to use the walrus autocomplete engine over a simpler ZRANGEBYLEX solution.

For more information, check out the autocomplete docs.


Rate-limiting is a kind of thorny problem and I'll say up front that walrus implements a fairly simplisitic rate-limiting implementation. The gist is that the rate limiter will allow up to N number of events per t seconds for a given key, which may be an IP address, etc. The time period is rolling, so we're looking at the difference between the Nth-last event and the first in order to determine if the limit has been exceeded.

Let's create another function that tells the time and put a global rate limit on it of 2 calls every 10 seconds:

>>> rl = db.rate_limit('rl-1', limit=2, per=10)
>>> @rl.rate_limited()
... def the_time():
...     return datetime.datetime.now()

>>> the_time()
datetime.datetime(2016, 1, 13, 23, 45, 58, 530863)

>>> the_time()
datetime.datetime(2016, 1, 13, 23, 45, 59, 148192)

>>> the_time()
RateLimitException                        Traceback (most recent call last)
<ipython-input-30-4a5a055f25ab> in <module>()
----> 1 the_time()

/home/charles/pypath/walrus/rate_limit.pyc in inner(*args, **kwargs)
     93                     raise RateLimitException(
     94                         'Call to %s exceeded %s events in %s seconds.' % (
---> 95                             fn.__name__, self._limit, self._per))
     96                 return fn(*args, **kwargs)
     97             return inner

RateLimitException: Call to the_time exceeded 2 events in 10 seconds.

>>> the_time()  # Waited a bit...
datetime.datetime(2016, 1, 13, 23, 46, 9, 369301)

If your function takes parameters, then by default walrus will hash them together to create a unique key for the rate limiter. This key functions the same as a cache key might when using the cached() decorator. Rate limits are often a necessary evil in web development, so having one handy, albeit a simple implementation, is definitely convenient.

For more information, check out the rate limiting documentation.


Really there's not much to say here... You give it a key and call incr() or decr(). If you'd like to read up on them, check out the docs. Here's a quick example:

>>> c1 = db.counter('my-counter')
>>> c1.incr()
>>> c1.incr()
>>> c2 = db.counter('my-counter')  # Same counter key!
>>> c2.value()
>>> c2.incr()
>>> c1.value()


In a networked or distributed environment, sometimes it's necessary to serialize access to particular resources to avoid race conditions. Python provides nice locking APIs, but they only work for the given interpreter process. If you've got multiple web servers, a task queue, etc, and they all need to be able to coordinate access to a shared resource, then using Redis as a lock server can be a good solution. The hitch is that we need to be sure our Redis implementation isn't vulnerable to race conditions. To this end, walrus implements the lock acquisition and release as Lua scripts which are guaranteed to be atomic.

The walrus Lock object has an API that mimicks threading.Lock with the exception that locks are given a name so other processes can reference them. Beyond that, they can be used as context managers, decorators, or just use the acquire / release methods.

>>> l = db.lock('secret-file')
>>> with l:
...     read_from_file()
...     write_changes_to_file()

>>> l.acquire()
>>> l.acquire(block=False)  # If we didn't specify False, we'd be stuck.
>>> l.release()

To read more, check out the locks documentation.


As is typical with Python ORMs these days, walrus does not deviate far from the standard declarative metaclass hackery combined with lists of Field instances describing the various model attributes.

On the backend, models are stored in hashes where the attributes correspond to the keys of the hash. Models can also have fields that are themselves containers, and these are stored separately, so a "Blog Post" model might have a "Set" of tags which is not stored in the hash, but in a separate Set object. In addition, Models support secondary indexes, which provide the ability to execute arbitrarily complex queries.

Let's look at a simple example that flexes all the muscles. The model code will be as follows:

import datetime
from walrus import *

db = Walrus()

class User(Model):
    database = db
    namespace = 'my-app'  # Optional.

    # Here we specify that the user's username will be the primary key we use
    # to look them up. If a primary key is not supplied, then walrus will create
    # an auto-incrementing integer primary key named "_id".
    username = TextField(primary_key=True)

    # We specify that `dob` is indexed. This means we can perform equality and
    # range-type queries on the (reported) ages of the users.
    dob = DateField(index=True)

class Note(Model):
    database = db
    namespace = 'my-app'  # Optional.

    # There is no foreign key monkey-business in walrus. But since the username is
    # the primary key of the User model, in order to link a note to a user, we can
    # just store their username. To ensure that notes are searchable by username,
    # we add a secondary index to it using `index=True`.
    username = TextField(index=True)

    # In order to support full-text search over the notes, we'll use `fts=True` to
    # create a special type of secondary index that allows full-text search.
    content = TextField(fts=True, stemmer=True)

    # The timestamp will also be stored in an indexed field to allow querying for
    # notes within a given range.
    timestamp = DateTimeField(index=True, default=datetime.datetime.now)

    # We will store whether or not a note is published using a boolean field.
    # Since we intend to query on it, we'll add a secondary index.
    published = BooleanField(index=True, default=True)

    # Lastly we have an instance of a container field. Tags will be stored in a
    # Redis set.
    tags = SetField()

Before we run queries against our data, let's populate a couple users and notes:

>>> charlie = User.create(username='charlie', dob=datetime.date(1983, 1, 1))
>>> huey = User.create(username='huey', dob=datetime.date(2010, 7, 1))
>>> Note.create(username='charlie', content='my awesome first note')
>>> Note.create(username='charlie', content='my fantastic second note')
>>> Note.create(username='charlie', content='my terrible third note')
>>> Note.create(username='huey', content='meow this is awesome')
>>> Note.create(username='huey', content='purr walrus is fantastic')
>>> Note.create(username='huey', content='purr hiss', published=False)

For our first example, let's just find the published notes by huey:

>>> Note.query((Note.published == True) & (Note.username == 'huey'))
<generator object query at 0x7f95c0f026e0>

As you can see, the output is a generator. To get the actual results we'll need to iterate over the generator:

>>> results = _  # Get the previous line's return value, our generator
>>> [note.content for note in results]
[u'meow this is awesome', u'purr walrus is fantastic']

Cool! Did you notice that we combined two operations using the & operator? With walrus you can combine as many different query expressions as you want, and using the secondary indexes, walrus will resolve each expression into a Set. These small sets are combined with one another such that & results in an intersection and | (pipe) results in a union. In this way arbitrarily complex queries are not only possible, but quite easy to implement.

For the next query, let's use the full-text search index to find published notes containing the word "awesome". For an added bonus, we'll order them sorting from newest-to-oldest:

>>> results = Note.query((Note.published == True) & (Note.content.search('awesome')), order_by=Note.timestamp.desc())
>>> [x.content for x in results]
[u'meow this is awesome', u'my awesome first note']

Pretty neat! If we wanted to query for users whose DOB is less than 2000, we can write:

>>> og_users = User.query(User.dob < datetime.date(2000, 1, 1))
>>> [user.username for user in og_users]

If you'd like to learn more about walrus models, check out the documentation or have a look at the example twitter app or diary app.

That's it!

Thanks for taking the time to read this post. If you weren't already familiar with walrus, then hopefully it may be useful to you. Redis is an amazing database and it lends itself well to little wrappers like this, and I look forward to continuing to enhance walrus as Redis adds new features or as I get new ideas for cool things to build on top of Redis.

Are there any missing pieces you'd like to see added to walrus? Please don't hesitate to leave a comment or contact me with your ideas.

I did not go into detail, but walrus also supports a handful of alternative "redis-like" databases (currently ledisdb, vedis, and rlite). If there are any other Redis-like databases out there that I missed, please let me know and I'll see about adding support.

Thanks again for reading, happy hacking!

Comments (2)

Stephen McDonald | jan 15 2016, at 05:15pm

Also check out my own hot-redis, extremely similar project :-)


Don Holloway | jan 14 2016, at 06:55am

Nice job! I will check it out this weekend.

Commenting has been closed, but please feel free to contact me