Connor Thomas Leifer

july 01, 2015 01:17pm / thoughts / 2 comments

photos/p1435774453.7.jpg

Read more...


When someone rips off your code and ideas

june 14, 2015 05:12pm / peewee python rants / 5 comments

I have always liked to think that imitation is the sincerest form of flattery, but today I am finding that feeling tinged with a bit of anger as well. I was playing around with GitHub's API and came across a project that borrows quite a bit of code and ideas from Peewee ORM.

At first I was excited to see another Python coder writing a lightweight ORM. I figured it would be neat to see how someone else would approach the problem. To my surprise, though, I found that most of the ideas, class hierarchies, and APIs were lifted directly from Peewee!

Read more...


Alternative Redis-Like Databases with Python

may 26, 2015 10:18am / nosql python redis walrus / 1 comments

photos/p1432653421.74.png

Redis is one of the more unique NoSQL offerings to have become popular over the past five years. It seems that there is no limit to the use-cases one can find for Redis. It's fantastic as a cache, doubles as a task-queue, can provide fast type-ahead search, and much more. The idea that you can store data-structures instead of rows and columns, keys and values, or JSON documents strikes me as particularly innovative. A while back I released walrus, a collection of Python utilities I'd built to simplify some of these use-cases and provide Pythonic APIs for the data-structures Redis natively supports. If you're a Python developer you might check it out.

Recently I've learned about a few new Redis-like databases: Rlite, Vedis and LedisDB. Each of these projects offers a slightly different take on the data-structure server you find in Redis, so I thought that I'd take some time and see how they worked. In this post I'll share what I've learned, and also show you how to use these databases with Walrus, as I've added support for them in the latest 0.3.0 release.

Read more...


Why I won't be switching to Disque

april 27, 2015 07:49pm / thoughts / 0 comments

Disque's alpha release announcement generated some buzz on HackerNews. If you missed it, Disque is a distributed message broker from Salvatore Sanfilippo, the author of Redis.

In the Limitations section of the README, Salvatore has written:

Disque was designed a bit in astronaut mode, not triggered by an actual use case of mine, but more in response to what I was seeing people doing with Redis as a message queue and with other message queues.

This admission makes me wary of using Disque, even if it reaches a stable release, because of my own experience with similar projects I've created but never actually used. These projects are usually fun opportunities for learning, but when it comes to maintenance, my experience has shown me that they quickly become a burden. Usually the problem is masked by the fact that if I'm not using it usually nobody else is either, but in the rare case I do end up with users, then eventually those users are going to submit bug reports and feature requests.

For a problem as complex as a distribute message broker, I imagine that there are going to be a lot of bug reports, strange edge-cases, and feature requests to support exotic use-cases. I hope that, in addition to his work on Redis, Salvatore can find the time to support Disque!

The other reason I don't foresee using Disque is alluded to in the author's own comments. He observes that many people are using Redis as a message broker, and decides that maybe there is a need for a "Redis of messaging". I would say the opposite is true, and that instead of another message server, people want to use Redis!

Redis integrates very nicely into the stack for web-based projects. It can be used as a cache, for locking, as a primary data store, for write-heavy portions of the application, and yes, as a message broker.

Perhaps the reason people are using Redis as a message broker is because they don't want to use something else?

Read more...


A Tour of Tagging Schemas: Many-to-many, Bitmaps and More

april 03, 2015 09:59pm / peewee python sql / 0 comments

photos/p1428119319.19.jpg

In this post I'll describe how to implement tagging with a relational database. What I mean by tagging are those little labels you see at the top of this blog post, which indicate how I've chosen to categorize the content. There are many ways to solve this problem, and I'll try to describe some of the more popular methods, as well as one unconventional approach using bitmaps. In each section I'll describe the database schema, try to list the benefits and drawbacks, and present example queries. I will use Peewee ORM for the example code, but hopefully these examples will easily translate to your tool-of-choice.

Read more...


Meet Scout, a Search Server Powered by SQLite

march 28, 2015 11:03am / peewee python scout search sqlite / 5 comments

photos/scout.png

In my continuing adventures with SQLite, I had the idea of writing a RESTful search server utilizing SQLite's full-text search extension. You might think of it as a poor man's ElasticSearch – a very, very poor man.

So what is this project? Well, the idea I had was that instead of building out separate search implementations for my various projects, I would build a single lightweight search service I could use everywhere. I really like SQLite (and have previously blogged about using SQLite's full-text search with Python), and the full-text search extension is quite good, so it didn't require much imagination to take the next leap and expose it as a web-service.

Read on for more details.

Read more...


How to make a Flask blog in one hour or less

march 09, 2015 08:43pm / flask peewee python / 1 comments

For fun, I thought I'd write a post describing how to build a blog using Flask, a Python web-framework. Building a blog seems like, along with writing a Twitter-clone, a quintessential experience when learning a new web framework. I remember when I was attending a five-day Django tutorial presented by Jacob Kaplan-Moss, one of my favorite projects we did was creating a blog. After setting up the core of the site, I spent a ton of time adding features and little tweaks here-and-there. My hope is that this post will give you the tools to build a blog, and that you have fun customizing the site and adding cool new features.

In this post we'll cover the basics to get a functional site, but leave lots of room for personalization and improvements so you can make it your own. The actual Python source code for the blog will be a very manageable 200 lines.

Who is this post for?

This post is intended for beginner to intermediate-level Python developers, or experienced developers looking to learn a bit more about Python and Flask. For the mother of all Flask tutorials, check out Miguel Grinberg's 18 part Flask mega-tutorial.

The spec

Here are the features:

Read more...


Querying the top N objects per group with Peewee ORM

march 03, 2015 12:39am / peewee python sql / 1 comments

photos/p1425417194.66.png

This post is a follow-up to my post about querying the top related item by group. In this post we'll go over ways to retrieve the top N related objects by group using the Peewee ORM. I've also presented the SQL and the underlying ideas behind the queries, so you can translate them to whatever ORM / query layer you are using.

Retrieving the top N per group is a pretty common task, for example:

In this post we'll discuss the following types of solutions:

Read more...


Querying the top item by group with peewee ORM

february 27, 2015 09:10pm / peewee python sql / 0 comments

photos/kitties-and-toys.jpg

In this post I'd like to share some techniques for querying the top item by group using the Peewee ORM. For example,

This is a common task, but one that can be a little tricky to implement in a single SQL query. To add a twist, we won't use window functions or other special SQL constructs, since they aren't supported by SQLite. If you're interested in finding the top N items per group, check out this follow-up post.

Read more...


Managing Database Connections with Peewee

february 12, 2015 01:48am / peewee python / 0 comments

photos/p1423749536.32.png

Recently I've been receiving a lot of questions about how to manage database connections with the peewee ORM. I thought I'd write a short post explaining the various ways peewee can help you manage your database connections, and outline some general best practices.

Read more...


Naive Bayes Classifier using Python and Kyoto Cabinet

february 03, 2015 12:04am / kyotocabinet nosql python / 1 comments

photos/p1422977174.11.png

In this post I will describe how to build a simple naive bayes classifier with Python and the Kyoto Cabinet key/value database. I'll begin with a short description of how a probabilistic classifier works, then we will implement a simple classifier and put it to use by writing a spam detector. The training and test data will come from the Enron spam/ham corpora, which contains several thousand emails that have been pre-categorized as spam or ham.

Read more...


Walrus: Lightweight Python utilities for working with Redis

january 11, 2015 07:49pm / nosql python redis walrus / 5 comments

photos/walrus-logo.png

A couple weekends ago I got it into my head that I would build a thin Python wrapper for working with Redis. Andy McCurdy's redis-py is a fantastic low-level client library with built-in support for connection-pooling and pipelining, but it does little more than provide an interface to Redis' built-in commands (and rightly so). I decided to build a project on top of redis-py that exposed pythonic containers for the Redis data-types. I went on to add a few extras, including a cache and a declarative model layer. The result is walrus.

Read more...


Setting up ElasticSearch with Basic Auth and SSL for use with Python

january 06, 2015 11:04pm / elasticsearch python / 7 comments

photos/p1420602336.52.jpg

I'm interested in learning to use ElasticSearch, so I thought I'd document how I set it up on my EC2 instance. Because I wanted to write code on my laptop, I needed to expose ElasticSearch over the public internet, which added a bit of extra complexity. Here is a rough outline of the process:

Read more...


Pollywog: Fun with Regular Expressions

december 30, 2014 05:03pm / pollywog python regex / 0 comments

photos/p1419822415.19.png

After writing about building a nicer regex API using operator overloading, I thought it would be fun to flesh out the code and release it as a library. I'm calling the result pollywog and you can find the code on GitHub.

To install pollywog, you can use pip or clone the GitHub repository:

$ pip install pollywog

In this post I'll show some examples of using pollywog.

Read more...


Extending SQLite with Python

december 02, 2014 09:33pm / peewee python sqlite / 1 comments

photos/sqlite-and-python.png

SQLite is an embedded database, which means that instead of running as a separate server process, the actual database engine resides within the application. This makes it possible for the database to call directly into the application when it would be beneficial to add some low-level, application-specific functionality. SQLite provides numerous hooks for inserting user code and callbacks, and, through virtual tables, it is even possible to construct a completely user-defined table. By extending the SQL language with Python, it is often possible to express things more elegantly than if we were to perform calculations after the fact.

In this post I'll describe how to extend SQLite with Python, adding functions and aggregates that will be callable directly from any SQL queries you execute. We'll wrap up by looking at SQLite's virtual table mechanism and seeing how to expose a SQL interface over external data sources.

Read more...


Querying Tree Structures in SQLite using Python and the Transitive Closure Extension

november 22, 2014 08:52pm / peewee python sqlite / 0 comments

photos/fall-foliage.jpg

I recently read a good write-up on tree structures in PostgreSQL. Hierarchical data is notoriously tricky to model in a relational database, and a variety of techniques have grown out of developers' attempts to optimize for certain types of queries.

In his post, Graeme describes several approaches to modeling trees, including:

In the comments, some users pointed out that the ltree extension could also be used to efficiently store and query materialized paths. LTrees support two powerful query languages (lquery and ltxtquery) for pattern-matching LTree labels and performing full-text searches on labels.

One technique that was not discussed in Graeme's post was the use of closure tables. A closure table is a many-to-many junction table storing all relationships between nodes in a tree. It is related to the adjacency model, in that each database row still stores a reference to its parent row. The closure table gets its name from the additional table, which stores each combination of ancestor/child nodes.

Read more...


Web-based SQLite Database Browser, powered by Flask and Peewee

november 13, 2014 08:28pm / python sqlite / 1 comments

photos/sqlite-browser.png

For the past week or two I've been spending some of my spare time working on a web-based SQLite database browser. I thought this would be a useful project, because I've switched all my personal projects over to SQLite and foresee using it for pretty much everything. It also dovetailed with some work I'd been doing lately on peewee regarding reflection and code generation. So it seemed like some pretty good bang/buck, especially given my perception that there weren't many SQLite browsers out there (it turns out there are quite a few, however). I'm sharing it in the hopes that other devs (and non-devs?) find it useful.

Read more...


Dear Diary, an Encrypted Command-Line Diary with Python

november 03, 2014 03:20pm / peewee python sqlcipher sqlite / 2 comments

photos/p1415049480.87.png

In my last post, I wrote about how to work with encrypted SQLite databases with Python. As an example application of these libraries, I showed some code fragments for a fictional diary program. Because I was thinking the examples directory of the peewee repo was looking a little thin, I decided to flesh out the diary program and include it as an example.

In this post, I'll go over the diary code in the hopes that you may find it interesting or useful. The code shows how to use the peewee SQLCipher extension. I've also implemented a simple command-line menu loop. All told, the code is less than 100 lines!

Read more...


Encrypted SQLite Databases with Python and SQLCipher

october 27, 2014 11:20pm / peewee python sqlcipher sqlite / 5 comments

photos/p1414470640.98.png

SQLCipher, created by Zetetic, is an open-source library that provides transparent 256-bit AES encryption for your SQLite databases. SQLCipher is used by a large number of organizations, including Nasa, SalesForce, Xerox and more. The project is open-source and BSD licensed. Best of all, there are open-source python bindings.

A GitHub user known as The Dod was kind enough to contribute a sqlcipher playhouse module, making it a snap to use Peewee with SQLCipher.

In this post, I'll show how to compile SQLCipher and the pysqlcipher bindings, then use peewee ORM to work with an encrypted SQLite database.

Read more...


Saturday morning hacks: Command-line client for the Flask note-taking app

october 22, 2014 08:44pm / flask peewee python saturday-morning-hacks / 0 comments

saturday hacks huey and mickey

A while back I wrote about using Flask to create a note-taking app. The goal of that app was to make it really easy for me to save little notes from my phone or computer. In the first follow-up post, I showed how to transition the JSON views to a full-featured RESTful API. I also showed how to add email reminders and to-do lists with checkable items to the notes app. Then in the most recent post, I showed how to leverage SQLite's full-text search extension to make the notes searchable. In this, the final installment, I'll show how I built a command-line client for the note-taking app.

Here is how the notes app looked when we left off at the end of part 3:

photos/p1412692599.23.png

Note-taking app code

If you'd like to follow along, you can find the source code for the most recent version of the note-taking app in this gist:

https://gist.github.com/coleifer/d93d6c43e59698d149c0

If you'd rather skip the post and get straight to the code, here is the code for the updated version:

https://gist.github.com/coleifer/948dd13f5aa98e2f1364

Read more...


Saturday morning hacks: DataSet for Peewee

october 17, 2014 05:39pm / peewee python saturday-morning-hacks / 0 comments

Saturday morning hacks

I recently became acquainted with the dataset project while browsing a curated list of awesome SQLAlchemy resources. I was intrigued by the project's simplicity, and apparently I'm not the only one as the project has quite a few followers on GitHub. Since peewee has the ingredients required to provide a similar API (reflection, schema migrations), I decided it might be a fun project to add a DataSet-like module to the playhouse extension collection.

Taking a look at DataSet

I feel like the best way to explain DataSet is to show it in action, so let's take a look at what it can do.

>>> from playhouse.dataset import DataSet

>>> db = DataSet('sqlite:///:memory:')

>>> people = db['people']  # This will create the people table.

>>> people.insert(name='charlie', gender='M')
1
>>> people.insert(name='huey', gender='M', favorite_color='blue')
2

>>> list(people)
[{'favorite_color': None, 'gender': u'M', 'id': 1, 'name': u'charlie'},
 {'favorite_color': u'blue', 'gender': u'M', 'id': 2, 'name': u'huey'}]

>>> people.update(favorite_color='green', name='charlie', columns=['name'])
1

>>> people.find_one(name='charlie')['favorite_color']
u'green'

Read more...


Saturday morning hacks: Adding full-text search to the flask note-taking app

october 08, 2014 11:57am / flask peewee python saturday-morning-hacks / 0 comments

Saturday morning hacks

In preparation for the fourth and final installment in the "Flask Note-taking app" series, I found it necessary to improve the search feature of the note-taking app. In this post we will use SQLite's full-text search extension to improve the search feature.

To recap, the note-taking app provides a lightweight interface for storing markdown-formatted notes. Because I frequently find myself wanting to take notes on the spur of the moment, the note-taking app needed to be very mobile-friendly. By using twitter bootstrap and a hefty dose of JavaScript, we made an app that matches our spec and manages to look good doing it!

In part 2, we added email reminders and check-able task lists to the note-taking app. We also converted the backend to use flask-peewee's REST API extension, which made it easy to add pagination and search. And that is how I've left it for the last three months or so.

Below is a screenshot of the latest version of the notes app. The UI is much cleaner thanks to a stylesheet from bootswatch. The bootswatch stylesheet works as a drop-in replacement for the default bootstrap CSS file.

photos/p1412692599.23.png

All together, the note-taking app has the following features:

You can browse or download the finished code from part 2 in this gist. If you're in a hurry, you can find all the code from this post in this gist.

In case you were curious, I've been using the notes app for things like:

Read more...


Saturday morning hacks: Building an Analytics App with Flask

september 30, 2014 10:40am / flask peewee python saturday-morning-hacks / 5 comments

Saturday morning hacks

A couple years back I wrote about building an Analytics service with Cassandra. As fun as that project was to build, the reality was that Cassandra was completely unsuitable for my actual needs, so I decided to switch to something simpler. I'm happy to say the replacement app has been running without a hitch for the past 5 months taking up only about 20 MB of RAM! In this post I'll show how to build a lightweight Analytics service using Flask.

Read more...


The Case for Peewee: Small, Hackable and Fun

september 26, 2014 01:40pm / peewee python thoughts / 0 comments

photos/rock-crawler.jpg

SQLAlchemy is one of those tools that is so powerful and so well-established that it's hard to imagine a time when it didn't exist. Mike Bayer, the project's primary author, after what I'm sure was a lot of real-world experience, hit on the right amount of abstraction and API that pleases both die-hard DBA types as well as more casual users.

So why would anyone use peewee?

Read more...


Postgresql HStore, JSON data-type and Arrays with Peewee ORM

september 12, 2014 11:13pm / peewee postgresql python / 2 comments

I've developed an interest in some of the more advanced features of SQLite after reading the O'Reilly title Using SQLite (Small. Fast. Reliable. Choose Any Three). For personal projects I like using SQLite, but when I need something more powerful I turn to Postgresql. Because peewee supports both of these databases (as well as MySQL), it is limited to a lowest-common-denominator feature set. While this encompasses a broad range of features, each database engine has its own extensions and I've been interested in adding some pythonic support for the cooler extensions.

Here are some of the fun things you can find in peewee's playhouse (collection of extensions):

This post will showcase the peewee support for HStore, JSON document store, and arrays. I've written elsewhere about SQLite's full-text search, so if you're a SQLite user you may want to check out that post.

To follow along at home, feel free to install peewee:

pip install peewee

Read more...


The Octopus Attack and the Sick Cat

july 30, 2014 12:04pm / kansas thoughts / 0 comments

It's been a great summer so far! Here are some of the things I've been up to.

p1406739096.16.jpg

I was trying to get some work done the other day and there was an octopus attack. I survived by eating them.

Read on for more!

Read more...


Playing with Python Magic Methods to make a nicer Regex API

july 19, 2014 11:24am / python regex / 5 comments

p1405787444.31.png

A co-worker of mine mentioned that he missed Ruby's syntactic sugar for regular expressions. I haven't used Ruby's regular expressions, but I'm familiar enough with Python's to know that the API is a bit wanting in syntactic sweetness.

In this post I'll show how you might use python's magic methods to make a nicer API for working with regular expressions.

Read more...


SQLite: Small. Fast. Reliable. Choose any three.

july 14, 2014 12:37pm / berkeleydb peewee python sqlite / 10 comments

Sqlite Logo

SQLite is a fantastic database and in this post I'd like to explain why I think that, for many scenarios, SQLite is actually a great choice. I hope to also clear up some common misconceptions about SQLite.

Read more...


JavaScript Canvas Fun: Pong

july 09, 2014 08:54am / canvas javascript pong / 0 comments

Earlier this week I posted about some old games I'd written, and I realized that I had not yet done a JavaScript version of Pong. I did versions of Tetris and Snake, perennial favorites of mine to implement, but somehow I'd forgotten about Pong. I think Pong was probably the first game I ever tried to copy, and it has a special place in my early-programmer's memory.

So I set out last night to put together a JavaScript canvas version of Pong. You can find a playable version in the post.

Read more...


Mutant killer, Mario and more: rediscovering old projects

july 06, 2014 10:44pm / programming thoughts / 0 comments

I was reorganizing some folders on my laptop and ran across some really old code I'd written. I knew the code was there, but I hadn't looked at it in years and thought it would be fun to take a peek, so I created a WindowsXP virtual machine and fiddled around trying to get the various programs to run.

Read more...


Completely un-scientific benchmarks of some embedded databases with Python

june 29, 2014 01:31am / nosql python / 5 comments

I've spent some time over the past couple weeks playing with the embedded NoSQL databases Vedis and UnQLite. Vedis, as its name might indicate, is an embedded data-structure database modeled after Redis. UnQLite is a JSON document store (like MongoDB, I guess??). Beneath the higher-level APIs, both Vedis and UnQLite are key/value stores, which puts them in the same category as BerkeleyDB, KyotoCabinet and LevelDB. The Python standard library also includes some dbm-style databases, including gdbm.

For fun, I thought I would put together a completely un-scientific benchmark showing the relative speeds of these various databases for storing and retrieving simple keys and values.

Here are the databases and drivers that I used for the test:

I'm running these tests with:

For the test, I simply recorded the time it took to store 100K simple key/value pairs (no collisions). Then I recorded the time it took to read back all these values. The results are in seconds elapsed:

p1404105100.86.png

Read more...


Python bindings for UnQLite, an embedded NoSQL database/JSON document store

june 26, 2014 11:46am / nosql python unqlite / 1 comments

unqlite python logo

I'm happy to write that I've just released some python bindings for UnQLite, an embedded NoSQL database and JSON document store. UnQLite might be characterized as the SQLite of NoSQL databases, though it's JSON document-store and Jx9 scripting language make it a pretty unique offering. UnQLite is created by Symisc Systems, who are also responsible for Vedis, an embedded Redis-like database (I also wrote some python bindings for vedis). Here is a quick overview of some of UnQLite's features, as described on the project homepage:

In the rest of this post I will show some basic usage of the unqlite-python library. If you'd like to follow along, you can use pip to install unqlite:

pip install unqlite

You can find the project source code hosted on GitHub and the documentation is available on readthedocs.

Read on for the details!

Read more...


Python bindings for Vedis, the Embedded NoSQL Database

june 19, 2014 10:22am / nosql python vedis / 0 comments

vedis-python logo

Over the past week I've been writing some python bindings to the embedded NoSQL database Vedis, a transactional data-store modeled after Redis. Like Redis, Vedis could be characterized as an advanced key-value store that supports hash, set and list data-structures. Vedis has over 70 available commands for working with the various data types. Unlike Redis, which is run as a separate server process, Vedis is embedded in the host process like SQLite. Vedis works with either in-memory databases or on-disk databases. Vedis is transactional (ACID) and also thread-safe. If you'd like more information, check out the Vedis FAQ.

Vedis-python

Vedis-python allows you to use Vedis in your Python apps. Vedis-python supports all the Vedis data-types, and also allows you extend Vedis by writing your own commands in Python. As I mentioned, this project is very new so while I have written pretty extensive unit tests, the library has certainly not been battle-tested yet.

If you'd like to give it a try, you can use pip to install vedis-python. At the time of writing the current version is 0.1.5.

$ pip install vedis-python

Just a word of caution, I've tested the installation on various flavors of Linux (including on my raspberry pi), and Mac OSX, but have not tested on Windows.

Read the rest of the post for the details.

Read more...


Gangster as hell

june 12, 2014 06:48pm / huey mickey / 0 comments

photos/2013-04-11_18.47.40.jpg

Read more...


jist: a command-line utility for managing multi-file, multi-directory private gists

june 04, 2014 02:48pm / gist python / 2 comments

Gist logo

I'd like to share a little command-line utility I wrote for managing multi-file and multi-directory private gists on GitHub. If you're not familiar with GitHub Gist, it's basically a git-backed pastebin. One of the benefits of Gist is that it supports private gists for free, allowing you to create private repos for your code snippets. To prevent abuse, GitHub does not allow you to create gists containing subdirectories.

I like to keep my list of public GitHub repositories very tidy, so I frequently use Gists for smaller projects. Last week I wanted to share the code for the note-taking app I blogged about. I didn't want to put the code into a GitHub repo, so I decided to create a gist. Unfortunately, the project contained templates and javascript that needed to go in subdirectories. To work around Gist's subdirectory restriction I used a naming convention to indicate that these files belonged in subdirectories, e.g.:

Then I had a lightbulb moment -- why not write a script to do all this automatically?

Read the rest of the post for the details.

Read more...


Saturday morning hacks: Revisiting the notes app

may 30, 2014 04:17pm / flask javascript peewee python saturday-morning-hacks / 1 comments

Saturday morning hacks

My post from last month, Saturday Morning Hack, a Little Note-Taking App with Flask, was pretty well-received. Since I've made a number of improvements to the app, I thought I would write one more post to share some of the updates I've made to this project, in the hopes that they may be of interest to you.

A live demo is up and running on Python Anywhere, so feel free to check that out before continuing on with the post: http://beetlejuicer.pythonanywhere.com/

To briefly recap the previous post, I discussed how I built a lightweight note-taking app which I could use from my phone or desktop. It has a nice ajax-ey interface and some simple markdown helpers written with javascript. In addition to supporting markdown, it also supports oembed for automatically embedding YouTube videos and the like. Here is what it looked like when we left off a few weeks ago:

Notes on Desktop

And this is how it looks now!

New and improved notes app

So what's new? Well, I've made a couple changes under-the-hood, and added some entirely new features to the UI.

This was super fun to hack on so I thought I'd share the new code and describe how I added these features. Honestly, I didn't really end up adding much in terms of implementation. Huey handles scheduling and sending the email reminders, even automatically retrying messages that fail to send. Similarly, Flask-Peewee's REST API provides search and pagination out-of-the-box, so all I had to do was write the JavaScript to communicate with it. Thanks to these libraries, I was able to focus on the things that made this project unique, and hopefully you enjoy reading about the code.

Read the rest of the post for the details.

Read more...


That one time I used a coroutine

may 26, 2014 12:55pm / coroutine python / 0 comments

Coroutines are an interesting feature of the Python language that I have, in practice, found very little occasion to use. Then one day I was stuck working on a problem and I realized suddenly that coroutines were exactly the thing I needed. In this post I'll describe how coroutines helped me to solve a tricky problem.

Note

David Beazley has written an excellent guide to coroutines in Python titled A Curious Course on Coroutines and Concurrency. I strongly recommend checking it out if this is your first brush with coroutines, or if it's just been a while and you'd like a refresher.

Read more...


Saturday morning hack: Automatically import your app's models when starting IPython

may 18, 2014 02:45pm / python saturday-morning-hacks / 0 comments

Saturday morning hacks

I use Flask and peewee for all my personal projects, and wanted an easy way to automatically open an IPython shell with all my project's models in the namespace. If you've used the excellent django-extensions project, you may be familiar with the shell_plus command, which does the same thing. If your project has multiple models spread across several modules, this kind of hack can save you a lot of keystrokes.

Read more...


Using SQLite Full-Text Search with Python

may 12, 2014 07:12pm / peewee python search sqlite / 0 comments

Full-text search with SQLite

In this post I will show how to use SQLite full-text search with Python (and a lot of help from peewee ORM). We will see how to index content for searching, and how to order search results using two ranking algorithms.

Last week I migrated my site from Postgresql to SQLite. I had been using Redis to power my site's search, but since SQLite has an awesome full-text search extension, I decided to give it a try. I am really pleased with the results, and being able to specify boolean search queries is an added plus. Here is a brief overview of the types of search queries SQLite supports:

Check out the full post for details on adding full-text search to your project.

Read more...


Saturday morning hack: personalized news digest with boolean query parser

may 10, 2014 10:28am / flask peewee python saturday-morning-hacks / 3 comments

Saturday morning hacks

Because I had so much fun writing my last Saturday morning hack, I thought I would share another little hack. I was thinking that I really enjoy my subscription to Python weekly and wouldn't it be great if I had a personal email digest containing just the types of things that interest me? I regularly cruise reddit and hacker hater news but in my opinion there's a pretty low signal-to-noise ratio. Occasionally I stumble on fascinating content and that's what keeps me coming back.

I wanted to write an app that would search the sites I read and automatically create an email digest based on search terms that I specified. I recently swapped my blog over to SQLite and I love that the SQLite full-text search extension lets you specify boolean queries. With that in mind, I decided that I would have a curated list of boolean search queries which would be used to filter content from the various sites I read. Any articles that match my search would then be emailed to me.

Here are some of my search terms, which I am viewing in the flask-peewee admin interface:

Search term admin

If you're interested in learning how to build your own version of this project, check out the rest of the post.

Read more...


Migrating to SQLite

may 08, 2014 06:42pm / peewee python sqlite / 0 comments

Sqlite Logo

Small. Fast. Reliable. Choose any three.

I made the decision this week to migrate my personal sites and several other sites I host onto SQLite. Previously almost everything I hosted had been using Postgresql. The move was motivated by a couple factors:

At times it has seemed to me that there is a tacit agreement within the Flask / Django communities that if you're using SQL you should be using Postgresql. Postgresql is an amazing piece of engineering. I have spent the last five years of my career working exclusively with it, and I am continually impressed by its performance and the constant stream of great new features.

So why change things?

Well, as my list indicates, there are a handful of reasons. But the primary reason was that I wanted something lightweight. I'm running a fairly low-traffic, read-heavy site, so Postgresql was definitely overkill. My blog is deployed on a VPS with very limited resources, so every MB of RAM counts. Additionally, I wasn't using any special Postgresql features so there was nothing holding me back.

(continued...)

Read more...


Ricing the Phone

may 08, 2014 01:35pm / android phone / 0 comments

If you follow my blog you've probably seen a few posts at regular intervals containing screenshots of my desktop. I enjoy customizing my desktop for usability and visual appeal, and since I spend 8+ hours a day on my laptop it's important that things be just right.

Recently I've gotten into ricing my phone homescreen as well. I've got a Moto G handset, a great cheap phone which I like so much better than my previous galaxy nexus. Unlike the majority of phones these days it's not so gigantic that I can hardly hold it in one hand.

Anyways, after rooting my phone and installing SlimKat, a custom KitKat ROM, I spent a bit of time tweaking my homescreen. Here are the results!

Homescreen SlimKat

Read more...


A Flask Front-End and Chrome Extension for YouTube-DL

may 04, 2014 09:27pm / flask python youtube / 4 comments

YouTube screenshot

I was cruising through some old bookmarked YouTube videos this evening and was sad to see that a few of my favorites were no longer available. I was able to search and find them again, but I decided it might not be a bad idea to save my favorites to my media server. I installed youtube-dl and was impressed by how well it just worked. Since I wanted to store these videos on my media server and watch them on Plex I then scp-ed the video files to the media server. Then it hit me -- why not just write a tiny web frontend for youtube-dl? Once I had a web app, then I could write a chrome extension to communicate with it...

With Flask the whole process took about 20 minutes, so I thought I'd share in case anyone else would find this useful.

Read on for the details.

Read more...


Saturday morning hack: a little note-taking app with Flask

april 27, 2014 08:55am / flask javascript peewee python saturday-morning-hacks / 5 comments

Saturday morning hacks

A couple Saturdays ago I spent the morning hacking together a note-taking app. I'm really pleased with the result, so I thought I'd share the code in case anyone else might find it useful.

The note-taking project idea came about out of necessity -- I wanted something that worked well from my phone. While I have a personal wiki site I've used for things like software installation notes or salsa recipes, I've also noticed that because it's so cumbersome to use from my phone, I often end up emailing things to myself. Plus a wiki implies a kind of permanence to the content, making it not a great fit for these impromptu notes. I also like to use markdown to format notes, but markdown isn't too easy on a phone because of the special characters or the need to indent blocks of text. With these considerations in mind, I set out to build a note-taking app that would be easy to use from my phone.

Here is how the app appears on a narrow screen like my phone:

Notes on Phone

And here it is on my laptop:

Notes on Desktop

Because markdown is a bit difficult to use when you're not in a nice text editor like vim, I've added some simple toolbar buttons to the editor:

Notes Toolbar

Read the full post for all the details!

Read more...


New Schema Migrations Module for Peewee ORM

april 20, 2014 07:34pm / peewee python / 0 comments

I'm pleased to announce that peewee now has more robust support for schema migrations. Basic migration support for Postgres has existed for some time, but peewee now supports SQLite and MySQL as well. In addition, there were a couple of issues with the original migrations API. Schema migrations have been one of the most-requested features, so I'm hopeful that with the addition of this feature, peewee users will have one more reason to enjoy using the library!

Check out the rest of the post for more details.

Read more...


Python's yield from

april 13, 2014 12:13pm / python / 0 comments

The yield from syntax, introduced in PEP 380, is getting a lot of attention lately due to its important role in the new asyncio package. I did not immediately understand what this syntax provides, but I have a handy way of thinking about it which I thought I'd share on my blog.

Imagine you have an arbitrarily nested list structure like so:

lists = [
    1, 2, 3,
    [4, 5, [6, 7], 8],
    [[[9, 10], 11]],
    [[]],
    12,
]

You can flatten this data-structure by writing a recursive generator thanks to the new yield from syntax:

def flatten(items):
    for item in items:
        if isinstance(item, (list, tuple)):
            yield from flatten(item)
        else:
            yield item

The output would then be:

>>> [item for item in flatten(lists)]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]

To achieve this using Python 2.x, which does not have yield from, you would instead write the recursive call like this:

if isinstance(item, (list, tuple)):
    for subitem in flatten(item):
        yield subitem

Read more...


In Honor of Spring...

march 17, 2014 11:18am / desktop leet linux / 0 comments

Two new themes.

Green Desktop

Early Spring Desktop

Read more...


Lawrence, KS

february 26, 2014 11:18am / kansas thoughts / 5 comments

I am proud to live in Lawrence, KS, a college town of about 100,000 which has been my home for the majority of my life. Perhaps the most striking feature about my home is the amazing sky here -- nowhere else I've lived comes close:

Cloudy winter day

Sunset over school

Being in the tech industry, I'm often asked if I have plans to move away to a place with more jobs. I always answer simply and somewhat apologetically that I intend to stay in Kansas. Answering that way is so much less embarassing than explaining why I love Kansas. My home is very much a part of me, though, and I'd like to write just once about why I am so happy to live here.

Read more...


How do you use peewee?

february 22, 2014 01:11pm / peewee / 5 comments

When I first wrote peewee I set out to accomplish a simple task: make it easy to execute queries in my Flask apps. I was a bit familiar with SQLAlchemy, but wanted something lightweight and thought it would be a quick project. While the first version only took a couple days to write, over the past two or three years peewee has been my favorite project to work on. I've been very surprised to see that it's user base has grown, and would like to ask anyone who is using peewee:

How do you use peewee?

I'd like to add a "testimonials" section to the documentation that describes the interesting projects people have written using peewee. If you don't mind sharing, I'd love to hear about your project.

Read more...


Window functions, case statements, and savepoints in peewee

february 21, 2014 10:44am / peewee python sql / 0 comments

In case you've missed the last few releases, I've been busy adding some fun new features to peewee. While the changelog and the docs explain the new features and describe their usage, I thought I'd write a blog post to provide a bit more context.

Most of these features were requested by peewee users. I depend heavily on users like you to help me improve peewee, so thank you very much! Not only have your feature requests helped make peewee a better library, they've helped me become a better programmer.

So what's new in peewee? Here is something of an overview:

Hopefully some of those things sound interesting. In this post I will not be discussing everything, but will hit some of the highlights.

Read more...