After several months of running the task queue bundled with django-utils, I decided to re-evaluate certain aspects of the design. This post describes those changes.
I'd been scrounging around for a smallish project, when I happened on the idea of writing a spider with a simple web interface. I had recently released a task queue, so I wanted to incorporate that to do the actual crawling, while a django view served up the results as they arrived in the database. The end result is a new project I'm calling django-spider, you can check it out on GitHub. This post will discuss some of the aspects of the design.
It's quite common when building out a website to trigger actions during the normal request/response cycle that may be time-consuming. Examples of these actions might be:
I remember last year about this time my coworkers and I got pretty excited about Celery, a distributed task queue, that provided a really nice API for executing tasks out-of-process. Basically just decorate functions with the @task decorator and so long as everything is configured properly, they will execute out of process. Celery is an actively-developed project with great documentation and an incredibly rich feature-set, but all those features come with the added cost of lots of configuration and the need for integration with a number of projects (celery, django-celery, kombu, django-kombu, pyparsing, mailer).
I needed a lightweight task queue for some side-projects and rather than trying to integrate all the various celery dependencies (and pinning all the correct versions) I did what anyone would do: rolled my own.