Blog Entries all / by tag / by year / popular

Nginx: a caching, thumbnailing, reverse proxying image server?

february 12, 2016 11:49am / nginx / 0 comments

A month or two ago, I decided to remove Varnish from my site and replace it with Nginx's built-in caching system. I was already using Nginx to proxy to my Python sites, so getting rid of Varnish meant one less thing to fiddle with. I spent a few days reading up on how to configure Nginx's cache and overhauling the various config files for my Python sites (yes, irony). In the course of my reading I bookmarked a number of interesting Nginx modules to return to, among them the Image Filter module.

I thought it would be neat to combine Nginx's reverse proxying, caching, and image filtering to create a thumbnailing server for my images hosted on S3. If you look closely at the <img> tag below (and throughout this site), you can see Nginx in action.



Examples of using Walrus, a lightweight Redis Toolkit

january 14, 2016 12:45am / nosql python redis walrus / 2 comments


walrus is my go-to toolkit for working with Redis in Python, and hopefully this post will convince you that it can be your go-to as well. I've tried to include lots of high-level Python APIs built on Redis primitives and the result is quite a lot of functionality. In this post I'll take you on a tour of the library and show examples of how it might be useful in your next project.


Five reasons you should use SQLite in 2016

january 06, 2016 08:49pm / berkeleydb python sqlite / 17 comments

Sqlite Logo

If you haven't heard, SQLite is an amazing database capable of doing real work in real production environments. In this post, I'll outline 5 reasons why I think you should use SQLite in 2016.


Announcing sophy: fast Python bindings for Sophia Database

december 19, 2015 03:58pm / cython kv nosql python / 0 comments


Sophia is a powerful key/value database with loads of features packed into a simple C API. In order to use this database in some upcoming projects I've got planned, I decided to write some Python bindings and the result is sophy. In this post, I'll describe the features of Sophia database, and then show example code using sophy, the Python wrapper.

Here is an overview of the features of the Sophia database:


Updated instructions for compiling BerkeleyDB with SQLite for use with Python

december 06, 2015 09:19pm / berkeleydb python sqlite / 0 comments


About three years ago I posted some instructions for building the Python SQLite driver for use with BerkeleyDB. While those instructions still work, they have the unfortunate consequence of stomping on any other SQLite builds you've installed in /usr/local. I haven't been able to build pysqlite with BerkeleyDB compiled in, because the source amalgamation generated by BerkeleyDB is invalid. So that leaves us with dynamically linking, and that requires that we use the BerkeleyDB libsqlite, which is exactly what the previous post described.

In this post I'll describe a better approach. Instead of building a modified version of libsqlite3, we'll modify pysqlite to use the BerkeleyDB libdb_sql library.


SQLite Spellfix Extension and Python

december 04, 2015 08:31pm / peewee python sqlite / 1 comments

The SQLite source tree is full of wonders. There is the the lemon parser generator, a btree implementation (well, kind of not surprising), multiple search engines, a json library, and more. It looks like Dr. Hipp is also experimenting with integrating the LSM key/value store from SQLite4 as a standalone virtual table.

I've written about the json and full-text search extensions, the lsm key/value store, and the transitive closure extension (useful when querying hierarchical data). In this post I'll be covering another interesting extension, the spellfix1 extension (documentation).


SQLite Table-Valued Functions with Python

december 04, 2015 02:31pm / cython python sqlite / 0 comments

One of the benefits of running an embedded database like SQLite is that you can configure SQLite to call into your application's code. SQLite provides APIs that allow you to create your own scalar functions, aggregate functions, collations, and even your own virtual tables. In this post I'll describe how I used the virtual table APIs to expose a nice API for creating table-valued (or, multi-value) functions in Python. The project is called sqlite-vtfunc and is hosted on GitHub.


Advanced filtering and SQLite FTS5 with Scout search engine

november 24, 2015 12:09am / peewee python scout search sqlite / 0 comments


I've been working on some new features for Scout and thought they might be worth a short blog post. The super-short version is that Scout now supports complex filtering on metadata, adding another layer of filtering besides the full-text search. Additionally, I've added support for SQLite FTS5, using it by default if it's available otherwise falling back to FTS4.


Who butchered my preface?

november 21, 2015 10:39am / thoughts / 0 comments

I admit, I'm a little on edge right now. A book I co-authored on Flask is going to be published soon and I was sent a copy of the preface to approve. When I opened the preface, I was horrified. All my original work was gone and had been replaced by a bland, nonsensical paragraph written by someone I suspect was not a native English speaker.


Using the SQLite JSON1 and FTS5 Extensions with Python

november 11, 2015 08:01am / peewee python search sqlite / 3 comments

Back in September, word started getting around trendy programming circles about a new file that had appeared in the SQLite fossil repo named json1.c. I originally wrote up a post that contained some gross hacks in order to get pysqlite to compile and work with the new json1 extension. With the release of SQLite 3.9.0, those hacks are no longer necessary.

SQLite 3.9.0 is a fantastic release. In addition to the much anticipated json1 extension, there is a new version of the full-text search extension called fts5. fts5 improves performance of complex search queries and provides an out-of-the-box BM25 ranking implementation. You can also boost the significance of particular fields in the ranking. I suggest you check out the release notes for the full list of enhancements

This post will describe how to compile SQLite with support for json1 and fts5. We'll use the new SQLite library to compile a python driver so we can use the new features from python. Because I really like pysqlite and apsw, I've included instructions for building both of them. Finally, we'll use peewee ORM to run queries using the json1 and fts5 extensions.