At my job we've been doing a quarterly hackday for almost a year now. My coworkers have made some amazing stuff, and its nice to have an entire day dedicated to hacking on ... well, whatever you want. Tomorrow marks the 4th hackday and I need to scrounge up a good project, but in the meantime I thought I'd write a post about what I did last time around -- a lightweight python task queue that has an API similar to celery.

I've called it huey (which also turns out to be the name of my kitten).

Design goals

The goal of the project was to keep it simple while not skimping on features. At the moment the project does the following:

• multi-threaded task execution
• scheduled execution at a given time
• periodic execution, like a crontab
• retrying tasks that fail
• task result storage

Backend storages implement a simple API, currently the only implementation uses Redis but adding one that uses the database would be a snap.

The other main goal of the project was to have it work easily for any python application (I've been into using flask lately), but come with baked-in support for django. Because of django's centralized configuration and conventions for loading modules, the django API is simpler than the python one, but hopefully both are reasonably straightforward.

As an IRC bot enthusiast and tinkerer, I would like to describe the most enduring and popular bot I've written, a markov-chain bot. Markov chains can be used to generate realistic text, and so are great fodder for IRC bots. The bot I am writing of has been hanging out in my town's channel for the past year or so and has amassed a pretty awesome corpus from which it generates messages. Here are few of his greatest hits:

Over the last two months I've spent a lot of time working on improvements to peewee, a lightweight ORM written in python.

Some of these features are present in Django and were added for better parity, some I found a need for while working on other projects, and others were requested by opening an issue on GitHub or bringing it up on IRC (#peewee on freenode). If you're interested in trying peewee out, it ships with an example app which is described here.

Here's a rundown on what has been added recently:

• atomic updates
• select related models
• basic transaction support

For a change, I've been doing all of my new app development using flask, a python web framework built atop the werkzeug WSGI toolkit. Having used django for the last two years it's been fun to do something different, but at the same time stick with python.

In this post I'd like to show a couple of the small projects I've written using flask over the past few weeks.

Recently I stumbled across the twitter bootstrap project, which is a set of cross-browser compliant stylesheets and scripts. I liked them so much that I've ported the admin templates to use bootstrap. Here's a little screenshot of the design refresh taken from the example app:

http://media.charlesleifer.com/images/photos/flask-peewee-admin.jpg

I hope this will make the admin easier to work with in the long-run!

I'd like to write a post about a project I've been working on for the past month or so. I've had a great time working on it and am excited to start putting it to use. The project is called flask-peewee -- it is a set of utilities that bridges the python microframework flask and the lightweight ORM peewee. It is packaged as a flask extension and comes with the following batteries included:

• Admin interface a-la django
• RESTful API toolkit a-la tastypie
• Authentication system

Over the past month I've been working on adding support for both MySQL and PostgreSQL to peewee. I'm happy to say that after a couple weekend hack sessions all tests are now passing.

With the 0.3.0 release of django-relationships, I've made a couple backwards-incompatible changes which I thought I'd mention.

One of the problems mentioned by a couple people when I asked for suggestions on improving djangosnippets.org was the proliferation of tags. This is a well-known problem on sites that allow users to enter their own tags, where misspellings are frequent and its sometimes unclear whether a tag should be plural or singular.

To try and reduce the amount of different tags on djangosnippets I ended up using the jQuery UI autocomplete tools to provide users with hints when they enter tags for their snippets.

Describing some of the improvements made to the django snippets site over the past couple weeks and asking for user feedback on additional improvements they'd like to see.

It's been a while since I first wrote about setting up Solr on Ubuntu. Since then I've opted for a different approach that is both simpler and lighter-weight. This post describes briefly the steps to setting up Solr on Ubuntu.

I'm pleased to announce that I've added support for MySQL to peewee. All tests are now passing. In the process I uncovered a few small bugs which have also been fixed.

I also added some new reference documentation which describes succinctly how to do basic configuration and querying with peewee.

After several months of running the task queue bundled with django-utils, I decided to re-evaluate certain aspects of the design. This post describes those changes.

Just a quick heads-up to anyone out there using django-completion, I've released a couple important updates this weekend and you may be interested in updating your checkouts. These changes are purely additive, so don't worry about having to update your own code.

There are three important updates:

1. Bug: the (experimental) Redis backend could fail to return results under certain conditions and, if data were removed frequently, leak memory all over the place
2. Bug: partial titles generated for the Redis and Database backends were too restrictive, i.e. "testing python code" would not complete on "python" unless the AUTOCOMPLETE_MAX_WORDS setting was 2 (or 1)
3. Feature: it is now possible to restrict suggestions to certain models by passing in a list of acceptable models to the suggest() method

As of this week we instituted a regular "hackday" at my office -- anything goes, you can work on whatever you like, so at 11:30 the night before the hackday started I decided on writing a simple IRC-powered botnet.

I'd been scrounging around for a smallish project, when I happened on the idea of writing a spider with a simple web interface. I had recently released a task queue, so I wanted to incorporate that to do the actual crawling, while a django view served up the results as they arrived in the database. The end result is a new project I'm calling django-spider, you can check it out on GitHub. This post will discuss some of the aspects of the design.

It's quite common when building out a website to trigger actions during the normal request/response cycle that may be time-consuming. Examples of these actions might be:

• sending email, for example a contact form
• checking a comment for spam by sending to Akismet
• thumbnailing an image

I remember last year about this time my coworkers and I got pretty excited about Celery, a distributed task queue, that provided a really nice API for executing tasks out-of-process. Basically just decorate functions with the @task decorator and so long as everything is configured properly, they will execute out of process. Celery is an actively-developed project with great documentation and an incredibly rich feature-set, but all those features come with the added cost of lots of configuration and the need for integration with a number of projects (celery, django-celery, kombu, django-kombu, pyparsing, mailer).

I needed a lightweight task queue for some side-projects and rather than trying to integrate all the various celery dependencies (and pinning all the correct versions) I did what anyone would do: rolled my own.

I'm writing this post to introduce a new project I've released, django-generic-m2m, which as its name would indicate is a generic ManyToMany implementation for django models. The goal of this project was to provide a uniform API for both creating and querying generically-related content in a flexible manner. One use-case for this project would be creating semantic "tags" between diverse objects in the database.

One of the nicest UI's around when dealing with a large dataset is a good autocomplete. Facebook's search is a great example, same for Netflix, and recently Google launched "Google Instant", which returns search results as you type. Autocomplete can really complement hierarchical drill-down search (which is useful for discovery), as the goal of autocomplete is more for helping users find something they already know about with a minimum of effort.

For the past month or so I've been working on writing my own ORM in Python. The project grew out of a need for a lightweight persistence layer for use in Flask web apps. As I've grown so familiar with the Django ORM over the past year, many of the ideas in Peewee are analagous to the concepts in Django. My goal from the beginning has been to keep the implementation simple without sacrificing functionality, and to ultimately create something hackable that others might be able to read and contribute to.

You may not know it, but djangoembed can be used to OEmbed your own site's static media. We use it at work to allow users to embed photos they upload through the site.

Tetris in JavaScript using the Canvas element, 'nuff said!

The canvas element is awesome. JavaScript is fast enough that you can run some pretty computationally intensive stuff (I've seen 3D games, a NES emulator, and much more all done with JS!). This script shouldn't push your CPU to the limit, but it does show how easy it is to create cool effects with just a small amount of code.

Keeping with the theme of yesterday's post - "a stroll down memory lane" - I thought I'd re-create the Nokia Snake game (a distant relative of Nibbles) using JavaScript and the canvas element.

When I started working at my current job I was surprised to see that everyone used IRC as their primary means of communication - much more so than email or IM. I recently wrote a small irc bot library in python - it was a ton of fun and reminded me of some of the first programs I wrote that were bots and scripts for America Online.

Users of djangosnippets.org may have noticed the addition of a few search-related features over the past several months. I'd like to highlight some of the additions that have been made and show how you can implement similar functionality on your sites. All of djangosnippet's search leans on Apache Solr, a powerful search engine built on top of Apache Lucene (full-text search). Haystack is the search solution for Django apps - it provides a querying interface similar to Django's ORM, handles indexing your models for you, and supports advanced features like "more-like-this" and faceting.

This post discusses the two flavors of model inheritance supported by Django, some of their use-cases as well as some potential gotchas.

In this post I'll show how I used Hookbox, a comet server/message queue, and Flask, a lightweight python web framework, to create a simple real-time chat app. This post will walk through creating a bare-bones example, then discuss ways to add additional functionality.

Sites often have many views that operate with a similar set of assumptions. Maybe there are entire areas that the user must be logged-in to visit, or there is some repetitive boilerplate functionality that a group of views shares like being rate-limited. This post looks at ways to make this kind of functionality less repetitive by using a common Django pattern, view decorators.

As the first installment in a series on common patterns in Django development, I'd like to discuss the Pluggable Backend pattern. The pattern addresses the common problem of providing extensible support for multiple implementations of a lower level function, be it caching, database querying, etc.

All about the lightweight jQuery-powered markup editing toolkit markItUp, which I used recently to implement basic reStructuredText markup support.

I'm pleased to announce the release of djangoembed, a django app for consuming and providing rich media.

What is OEmbed?

OEmbed is a format for allowing a rich representation of a url. If you've used Facebook you've probably seen this feature before -- linking a YouTube video will embed an actual video player in the news feed, automatically. The player is represented by some HTML, plus there may be additional metadata like the author, a link to their channel, the title of the video, or even a thumbnail.

Last week, after several false starts, I moved all the sites I maintain into virtualenvs, with their own pip requirements files. My reasons for doing so are pretty simple:

• Experimenting with new/different versions of software is a pain in the ass without isolation
• A pip requirements file for each site is a very nice thing to behold
• I symlink 3rd party apps into a custom directory on my PYTHONPATH and it was getting huge
• Profit

There are quite a few great tutorials out there for getting started with these tools. I will only discuss how I got over some of the hurdles involved in using these tools, as well as a tool for automating the creation of "skeleton" django sites.

Apache Solr is a fast, open-source search solution. People are doing some very cool things with Solr. I personally have only begun to scratch the surface of what is possible with Solr, but have seen amazing returns with a relatively small investment (thanks entirely to Daniel Lindsley's excellent search framework, django-haystack). There are instructions for getting up and running with Solr + Jetty -- the purpose of this blog entry is to walk through setting up multi-core Solr with Apache Tomcat.

To get the benefit of Django 1.2's new CSRF protection, all POST forms will need a special token. Here is a quick command that runs through templates adding the token:

find . -type f -name "*.html" -exec sed -i \
's|\(<form[^>]*method="post"[^>]*>\)\({% csrf_token %}\)\?|\1{% csrf_token %}|g' \
{} \;

This post will be very brief, but I want to show a little trick I'm using on my different servers so I can tell them apart at a glance. I use a custom bash prompt which gives the hostname of each server a different color:

[charles@alpha ~] $
[charles@beta ~] $

Aggregation support was added to django's ORM in version 1.1, allowing you to generate Sums, Counts, and more without having to write any SQL. According to the docs aggregation is not supported for generic relations. This entry describes how to work around this using the .extra() method.

Most django sites I create have quite a lot in common. Beyond the handful of files generated by django-admin startproject, my projects all have a database, wsgi file, apache and nginx confs, static media and templates. All these building blocks of a site vary very little from project-to-project. django-site-gen allows you to automate the creation of the stuff that doesn't vary.

cue is a simple queue abstraction along the lines of the command pattern, and provides django apps with a way to decouple the creation of a request from its execution.

A collection of a few snippets I've made use of recently.

Topeka, the state capitol of Kansas, is about 30 minutes down the road from Lawrence, KS where I work. In a recent publicity stunt, they've been talking about renaming the capitol to 'Google' in order to bring Google fiber-optics to the city. Here's Google's response.

I recently posted on writing an app that allows you to create flexible and descriptive relationships between Django's built-in auth.users. django-relationships is the result.

Doing a quick search on the Django Developers group for 'user model' yields a bunch of requests for extensible user models. There's also an epic ticket, #3011, opened 3 years ago on this topic. I don't really feel one way or the other about it -- this is just a roundabout way of excusing the hack you're about to see.

Reusable code is one of the most talked-about silver bullets in our toolkit. Django has a huge ecosystem of reusable apps, but not nearly so many reusable fields. Apps and fields generally solve different problems, though, and while an app can be retemplated or modified to fit your use case, a field is often times so specialized or exact in its purpose that either you need it or you don't. In this entry I'll discuss a great use-case for a field we had at work building out live March Madness coverage, and a popular field that is also an app.

Writing simple templatetags is only marginally less painful that writing complex tags. django-simpletags aims at making writing simple tags simple.

Object-oriented programming stresses the idea of code reuse, through concepts like inheritance and polymorphism. View programming in django can sometimes get a boost from class-based design.

How to implement self-referencing many-to-many relationships in Django. Example use cases are modeling asymmetrical following (a-la twitter) or symmetrical friendship (a-la facebook).

Django uses several types of registration patterns for some of its most notable features. This entry looks at the way django implements its different types of registries.

django-github is a Django app for integration with GitHub.

Amazing CG!

OEmbed is a pretty neat idea. A site that serves up content decides that it wants to allow users elsewhere to integrate with their site. What's missing is the discovery step.

My site is now powered by Django! I added some features, trimmed some bloat, and hopefully have a leaner, cleaner site.

Github, the premier social coding site, has a new REST API. I've written a small API client in Python to handle retrieving data on repos, commits, and gists.

When I interviewed for my first web development job one of the first questions was "Are you comfortable using linux?" I thought "Why would I need to know how to use that?" -- newb mistake.

Hooking into django's authentication system using two approaches: a custom authentication_backend and signals.

DynDNS provides an awesome free service - a hostname that points at your home server. Most of us have a dynamic IP provided by our internet service provider, and running a home server can be a pain whenever your IP address should happen to change. DynDNS allows you to create a hostname that will point to your dynamic IP.

If you have comments on your site, or allow use generated content to appear in prominent places, it's a good idea to remove profanity. This regular expression matches case-insensitive for any bad words, and strips the entire word to the leading and trailing whitespace.

One of the biggest barriers to entry on the web might be the "register -> check email -> login -> remember account info" paradigm. At work I recently developed a proof-of-concept app that allows a single login to work with multiple sites. How? iframe

Outlining how I merged two databases for django-powered sites using a combination of SQL, django-multidb, and pg_dump.

How to implement many-to-many relationships with extra fields, or, Many-to-Many Through. Covers django models and admin.

I am just beginning the process of merging two fairly large databases. This merge is complicated by the fact that there are a number of shared tables between the two databases. I've written a short python script to generate lists of tables for each dat

Poker hand evaluator written in Python.

Lightweight BBCode parser and syntax highlighting for Django. Code blocks set off with the tag will be automatically highlighted.

My first github repo -- a collection of PHP Snippets.

I've been lucky enough to have spent this week attending Jacob Kaplan-Moss's Django Training. One of the topics we covered a bit later in the week was middleware, which in the Django usage is somewhat of a misnomer. Middleware in Django allows you to pre-process requests, post-process responses, and handle exceptions.

Project Euler is a web site devoted to mathematical problems solved through the use of computer software.

Use Django to calculate most popular tags, or any query that makes use of an aggregation function.

I use Apache's mod_rewrite extensively on my site to generate clean URLs. It is basically matches a regex to a url (similar to Django's URLConf). This snippet will automatically try adding a backslash to any URL that initially generates a 404.

Query WHOIS using PHP.

Yesterday, while trying to set up a public-facing form for a recipes app, I ran into the issue of combining Django's ModelForm (for the Recipe model) with an inline for the foreign key Ingredients. One of my astute coworkers (there are many of them) poi

This post describes how to get Live Android 0.2 up and running on Ubuntu using VirtualBox. I've got an Asus X83V notebook running Ubuntu 8.10 64-bit. Step-by-step howto with some screenshots.

PHP function to decode keywords used by search engines. Determine what search keywords drive traffic to your site.

Add BBCode support and Syntax highlighting using PHP. To make the blog even easier to use, I thought I'd get away from using HTML in my entries altogether. I also wanted a way to easily paste code and have it highlighted. Works on nested tags!

Count how many lines of code are in a project using this PHP script. Searches directories recursively.

Recursively search and replace in files using this PHP script. Follows subdirectories and uses regular expressions for search and replace. Allows file extensions to be specified.

Learn how to build a blog system using PHP and MySQL. I thought I'd share my experience after rebuilding the site blog system. This entry will cover building a simple blog system like the one found on this site, using code that is also available on thi

Added News section to the site, which acts as a simple PHP RSS aggregator (with really nice URLs). Rewrote the PHP Blog System and published the code online. Added a contact form.

1337 fluxbox desktop, showing conky, root-tail and of course DooM.

The site has been in stasis for about three months, so it's probably time to start developing a new version. After looking at the site for a while, I think it's fair to say that many aspects of this site have been (paradoxically) both over-engineered a