My List of Python and SQLite Resources

august 25, 2015 11:38pm / python sqlite / 0 comments

Sqlite Logo

This post is going to be a greatest hits of my open-source libraries and blog posts concerning the use of SQLite with Python. I'll also share a list of some other neat SQLite projects that you may not have heard of before.

Read more...


Using SQLite4's LSM Storage Engine as a Stand-alone NoSQL Database with Python

august 14, 2015 12:21am / cython nosql python sqlite / 1 comments

photos/lsm.png

SQLite and Key/Value databases are two of my favorite topics to blog about. Today I get to write about both, because in this post I will be demonstrating a Python wrapper for SQLite4's log-structured merge-tree (LSM) key/value store.

I don't actively follow SQLite's releases, but the recent release of SQLite 3.8.11 drew quite a bit of attention as the release notes described massive performance improvements over 3.8.0. While reading the release notes I happened to see a blurb about a new, experimental full-text search extension (which I wrote about in a different post), and all this got me to wondering what was going on with SQLite4.

As I was reading about SQLite4, I saw that one of the design goals was to provide an interface for pluggable storage engines. At the time I'm writing this, SQLite4 has two built-in storage backends, one of which is an LSM key/value store. Over the past month or two I've been having fun with Cython, writing Python wrappers for the embedded key/value stores UnQLite and Vedis. I figured it would be cool to use Cython to write a Python interface for SQLite4's LSM storage engine.

After pulling down the SQLite4 source code and reading through the LSM header file (it's very small!), I started coding and the result is python-lsm-db (docs).

Read the rest of the post for examples of how to use the library.

Read more...


Building the SQLite FTS5 Search Extension

july 30, 2015 03:07pm / peewee python search sqlite / 0 comments

photos/p1438286638.84.png

SQLite 3.8.11.1 contains a new, experimental version of the full-text search extension named FTS5. Reviewing the documentation for FTS5, I saw that it includes a couple cool enhancements, namely a more sophisticated query language, and built-in BM25 result ranking.

I decided to give it a try and thought I'd share my notes on compiling the extension in case anyone else is curious.

Read more...


Introduction to the fast new UnQLite Python Bindings

july 21, 2015 01:07am / cython nosql python unqlite / 2 comments

photos/unqlite-python-logo.png

About a year ago, I blogged about some Python bindings I wrote for the embedded NoSQL document store UnQLite. One year later I'm happy to announce that I've rewritten the library using Cython and operations are, in most cases, an order of magnitude faster.

This was my first real attempt at using Cython and the experience was just the right mix of challenging and rewarding. I bought the O'Reilly Cython Book which came in super handy, so if you're interested in getting started with Cython I recommend picking up a copy.

In this post I'll quickly touch on the features of UnQLite, then show you how to use the Python bindings. When you're done reading you should hopefully be ready to use UnQLite in your next Python project.

Read more...


Alternative Redis-Like Databases with Python

may 26, 2015 10:18am / nosql python redis walrus / 1 comments

photos/p1432653421.74.png

Redis is one of the more unique NoSQL offerings to have become popular over the past five years. It seems that there is no limit to the use-cases one can find for Redis. It's fantastic as a cache, doubles as a task-queue, can provide fast type-ahead search, and much more. The idea that you can store data-structures instead of rows and columns, keys and values, or JSON documents strikes me as particularly innovative. A while back I released walrus, a collection of Python utilities I'd built to simplify some of these use-cases and provide Pythonic APIs for the data-structures Redis natively supports. If you're a Python developer you might check it out.

Recently I've learned about a few new Redis-like databases: Rlite, Vedis and LedisDB. Each of these projects offers a slightly different take on the data-structure server you find in Redis, so I thought that I'd take some time and see how they worked. In this post I'll share what I've learned, and also show you how to use these databases with Walrus, as I've added support for them in the latest 0.3.0 release.

Read more...


A Tour of Tagging Schemas: Many-to-many, Bitmaps and More

april 03, 2015 09:59pm / peewee python sql / 0 comments

photos/p1428119319.19.jpg

In this post I'll describe how to implement tagging with a relational database. What I mean by tagging are those little labels you see at the top of this blog post, which indicate how I've chosen to categorize the content. There are many ways to solve this problem, and I'll try to describe some of the more popular methods, as well as one unconventional approach using bitmaps. In each section I'll describe the database schema, try to list the benefits and drawbacks, and present example queries. I will use Peewee ORM for the example code, but hopefully these examples will easily translate to your tool-of-choice.

Read more...


Meet Scout, a Search Server Powered by SQLite

march 28, 2015 11:03am / peewee python scout search sqlite / 5 comments

photos/scout.png

In my continuing adventures with SQLite, I had the idea of writing a RESTful search server utilizing SQLite's full-text search extension. You might think of it as a poor man's ElasticSearch – a very, very poor man.

So what is this project? Well, the idea I had was that instead of building out separate search implementations for my various projects, I would build a single lightweight search service I could use everywhere. I really like SQLite (and have previously blogged about using SQLite's full-text search with Python), and the full-text search extension is quite good, so it didn't require much imagination to take the next leap and expose it as a web-service.

Read on for more details.

Read more...


How to make a Flask blog in one hour or less

march 09, 2015 08:43pm / flask peewee python / 1 comments

For fun, I thought I'd write a post describing how to build a blog using Flask, a Python web-framework. Building a blog seems like, along with writing a Twitter-clone, a quintessential experience when learning a new web framework. I remember when I was attending a five-day Django tutorial presented by Jacob Kaplan-Moss, one of my favorite projects we did was creating a blog. After setting up the core of the site, I spent a ton of time adding features and little tweaks here-and-there. My hope is that this post will give you the tools to build a blog, and that you have fun customizing the site and adding cool new features.

In this post we'll cover the basics to get a functional site, but leave lots of room for personalization and improvements so you can make it your own. The actual Python source code for the blog will be a very manageable 200 lines.

Who is this post for?

This post is intended for beginner to intermediate-level Python developers, or experienced developers looking to learn a bit more about Python and Flask. For the mother of all Flask tutorials, check out Miguel Grinberg's 18 part Flask mega-tutorial.

The spec

Here are the features:

Read more...


Querying the top N objects per group with Peewee ORM

march 03, 2015 12:39am / peewee python sql / 1 comments

photos/p1425417194.66.png

This post is a follow-up to my post about querying the top related item by group. In this post we'll go over ways to retrieve the top N related objects by group using the Peewee ORM. I've also presented the SQL and the underlying ideas behind the queries, so you can translate them to whatever ORM / query layer you are using.

Retrieving the top N per group is a pretty common task, for example:

In this post we'll discuss the following types of solutions:

Read more...


Querying the top item by group with peewee ORM

february 27, 2015 09:10pm / peewee python sql / 0 comments

photos/kitties-and-toys.jpg

In this post I'd like to share some techniques for querying the top item by group using the Peewee ORM. For example,

This is a common task, but one that can be a little tricky to implement in a single SQL query. To add a twist, we won't use window functions or other special SQL constructs, since they aren't supported by SQLite. If you're interested in finding the top N items per group, check out this follow-up post.

Read more...