I'd been scrounging around for a smallish project, when I happened on the idea of writing a spider with a simple web interface. I had recently released a task queue, so I wanted to incorporate that to do the actual crawling, while a django view served up the results as they arrived in the database. The end result is a new project I'm calling django-spider, you can check it out on GitHub. This post will discuss some of the aspects of the design.