In this post I'll present how I built a (reasonably) powerful autocomplete engine with Redis and python. For those who are not familiar with Redis, it is a fast, in-memory, single-threaded database that is capable of storing structured data (lists, hashes, sets, sorted sets). I chose Redis for this particular project because its sorted set data type, which is a good fit for autocomplete. The engine I'll describe relies heavily on Redis' sorted sets and its set operations, but can easily be translated to a pure-python solution (links at bottom of post).
I recently rewrote my personal site using flask and peewee, breaking a good amount of stuff in the process. I was trying to track down the errors by tailing log files, but that didn't help alert me to new errors that someone visiting the site might stir up. I thought about setting up error emails a-la django, which is a tried and true method...but then I happened on a different approach. I won't say it's the most elegant solution, but it was a quick hack and the results have been awesome. I wrote a custom logging handler that pushes JSON-encoded log record data to a redis pub/sub channel. I then have an IRC bot that subscribes to this channel and when it receives a message generates a paste of the traceback and pings me with a link to the traceback.
One of the nicest UI's around when dealing with a large dataset is a good autocomplete. Facebook's search is a great example, same for Netflix, and recently Google launched "Google Instant", which returns search results as you type. Autocomplete can really complement hierarchical drill-down search (which is useful for discovery), as the goal of autocomplete is more for helping users find something they already know about with a minimum of effort.