Rewriting Huey for a better API

For a while I've been itching to rewrite Huey, and just last week released 0.4 which is an almost total rewrite. I initially started Huey for performing tasks like checking comments for spam, sending emails, generating thumbnails, and basically anything that would slow down the pagespeed on my sites. This is still what I see as the primary use-case for huey -- performing small tasks outside the request/response cycle and running jobs on a schedule (I have a site that scrapes the county sheriff's site and keeps a log of arrests in my town). The goal for the rewrite was not to change the purpose of Huey, rather it was to change the API.

How much does API matter?

Look at a project like Kenneth Reitz' requests. This project bills itself as "HTTP for Humans", and the landing page for the documentation is filled with praise for its API. Everything about this project shows that the author's intent was to design a clean, spare API around making HTTP requests from python, and as a result the project has been enormously successful. However, the ability to make HTTP requests is present in the python standard library, and has been for a long time. Kenneth Reitz simply noticed that the existing APIs were terrible and decided to do something about it.

The API matters for projects like Huey

For a project like Huey, the API is extremely important. Huey provides a subset of the functionality provided by more sophisticated libraries like Celery, so what differentiates it is the simplicity of integrating it into your project. By exposing a very simple API, it is easy for implementers to imagine how they will use Huey with their project. A nice API also softens the initial learning curve when getting started with a new library. I would like to point out, though, that Celery has gotten much more simple to use with the release of 3.0 -- the API is very clean!

Building a better API for job queueing

For some time I've been aware that Huey's APIs could be cleaned up. There was some duplication of configuration data, and the configuration in general seemed poorly designed. The "task" decorators had a required parameter and overall there was too much implementation leaking into the API. It was so convoluted, in fact, that I decided not to put the "standard 5-line example" on the project's README!

Here is how the API would have looked before the rewrite:

from huey.backends.redis_backend import RedisQueue, RedisDataStore
from huey.bin.config import BaseConfiguration
from huey.decorators import queue_command
from huey.queue import Invoker

queue = RedisQueue('my-app')
data_store = RedisDataStore('my-app')

class QueueConfiguration(BaseConfiguration):
    QUEUE = queue
    RESULT_STORE = data_store
    THREADS = 4

invoker = Invoker(queue, data_store)

def some_task(foo, bar):

That is terrible. Here is how the new API looks:

from huey import RedisHuey

huey = RedisHuey('my-app')

def some_task(foo, bar):

Things I did to improve the API:

Reading more

If you're interested in checking out Huey, take a look at the getting started guide. If you're already using an older version of Huey, check out the upgrading documentation.

Thanks for reading!

Comments (0)

Commenting has been closed, but please feel free to contact me