Entries tagged with redis
Shortly after launching my Nginx-based cache + thumbnailing web-service, I realized I had no visibility into the performance of the service. I was curious what my hit-ratios were like, how much time was spent during a cache-miss, basic stuff like that. Nginx has monitoring tools, but it looks like they're only available to people who pay for Nginx Plus, so I decided to see if I could roll my own. In this post, I'll describe how I used Lua, cosockets, and Redis to extract real-time metrics from my thumbnail service.
walrus is my go-to toolkit for working with Redis in Python, and hopefully this post will convince you that it can be your go-to as well. I've tried to include lots of high-level Python APIs built on Redis primitives and the result is quite a lot of functionality. In this post I'll take you on a tour of the library and show examples of how it might be useful in your next project.
Redis is one of the more unique NoSQL offerings to have become popular over the past five years. It seems that there is no limit to the use-cases one can find for Redis. It's fantastic as a cache, doubles as a task-queue, can provide fast type-ahead search, and much more. The idea that you can store data-structures instead of rows and columns, keys and values, or JSON documents strikes me as particularly innovative. A while back I released walrus, a collection of Python utilities I'd built to simplify some of these use-cases and provide Pythonic APIs for the data-structures Redis natively supports. If you're a Python developer you might check it out.
Recently I've learned about a few new Redis-like databases: Rlite, Vedis and LedisDB. Each of these projects offers a slightly different take on the data-structure server you find in Redis, so I thought that I'd take some time and see how they worked. In this post I'll share what I've learned, and also show you how to use these databases with Walrus, as I've added support for them in the latest 0.3.0 release.
A couple weekends ago I got it into my head that I would build a thin Python wrapper for working with Redis. Andy McCurdy's redis-py is a fantastic low-level client library with built-in support for connection-pooling and pipelining, but it does little more than provide an interface to Redis' built-in commands (and rightly so). I decided to build a project on top of redis-py that exposed pythonic containers for the Redis data-types. I went on to add a few extras, including a cache and a declarative model layer. The result is walrus.
In this post I'll present how I built a (reasonably) powerful autocomplete engine with Redis and python. For those who are not familiar with Redis, it is a fast, in-memory, single-threaded database that is capable of storing structured data (lists, hashes, sets, sorted sets). I chose Redis for this particular project because its sorted set data type, which is a good fit for autocomplete. The engine I'll describe relies heavily on Redis' sorted sets and its set operations, but can easily be translated to a pure-python solution (links at bottom of post).
Redis-completion is now deprecated. The autocomplete functionality, along with a number of other features, have been integrated into a new project walrus. Check out the walrus blog announcement for more details.
I recently rewrote my personal site using flask and peewee, breaking a good amount of stuff in the process. I was trying to track down the errors by tailing log files, but that didn't help alert me to new errors that someone visiting the site might stir up. I thought about setting up error emails a-la django, which is a tried and true method...but then I happened on a different approach. I won't say it's the most elegant solution, but it was a quick hack and the results have been awesome. I wrote a custom logging handler that pushes JSON-encoded log record data to a redis pub/sub channel. I then have an IRC bot that subscribes to this channel and when it receives a message generates a paste of the traceback and pings me with a link to the traceback.
One of the nicest UI's around when dealing with a large dataset is a good autocomplete. Facebook's search is a great example, same for Netflix, and recently Google launched "Google Instant", which returns search results as you type. Autocomplete can really complement hierarchical drill-down search (which is useful for discovery), as the goal of autocomplete is more for helping users find something they already know about with a minimum of effort.