Rewriting Huey for a better API
For a while I've been itching to rewrite Huey, and just last week released 0.4 which is an almost total rewrite. I initially started Huey for performing tasks like checking comments for spam, sending emails, generating thumbnails, and basically anything that would slow down the pagespeed on my sites. This is still what I see as the primary use-case for huey -- performing small tasks outside the request/response cycle and running jobs on a schedule (I have a site that scrapes the county sheriff's site and keeps a log of arrests in my town). The goal for the rewrite was not to change the purpose of Huey, rather it was to change the API.
How much does API matter?
Look at a project like Kenneth Reitz' requests. This project bills itself as "HTTP for Humans", and the landing page for the documentation is filled with praise for its API. Everything about this project shows that the author's intent was to design a clean, spare API around making HTTP requests from python, and as a result the project has been enormously successful. However, the ability to make HTTP requests is present in the python standard library, and has been for a long time. Kenneth Reitz simply noticed that the existing APIs were terrible and decided to do something about it.
The API matters for projects like Huey
For a project like Huey, the API is extremely important. Huey provides a subset of the functionality provided by more sophisticated libraries like Celery, so what differentiates it is the simplicity of integrating it into your project. By exposing a very simple API, it is easy for implementers to imagine how they will use Huey with their project. A nice API also softens the initial learning curve when getting started with a new library. I would like to point out, though, that Celery has gotten much more simple to use with the release of 3.0 -- the API is very clean!
Building a better API for job queueing
For some time I've been aware that Huey's APIs could be cleaned up. There was some duplication of configuration data, and the configuration in general seemed poorly designed. The "task" decorators had a required parameter and overall there was too much implementation leaking into the API. It was so convoluted, in fact, that I decided not to put the "standard 5-line example" on the project's README!
Here is how the API would have looked before the rewrite:
from huey.backends.redis_backend import RedisQueue, RedisDataStore from huey.bin.config import BaseConfiguration from huey.decorators import queue_command from huey.queue import Invoker queue = RedisQueue('my-app') data_store = RedisDataStore('my-app') class QueueConfiguration(BaseConfiguration): QUEUE = queue RESULT_STORE = data_store THREADS = 4 invoker = Invoker(queue, data_store) @queue_command(invoker) def some_task(foo, bar): pass
That is terrible. Here is how the new API looks:
from huey import RedisHuey huey = RedisHuey('my-app') @huey.task() def some_task(foo, bar): pass
Things I did to improve the API:
- Remove the configuration altogether and encapsulate queue configuration inside
- Consumer-specific options, like number of worker threads, are now specified as command-line options when running the consumer.
- The decorator is exposed as a method on the
Hueyobject, eliminating the need to specify the "invoker" parameter.
- The simple case is simple -- to use Redis simply instantiate the
Thanks for reading!
Commenting has been closed, but please feel free to contact me