Python's yield from

April 13, 2014 12:13 / 0 comments

The yield from syntax, introduced in PEP 380, is getting a lot of attention lately due to its important role in the new asyncio package. I did not immediately understand what this syntax provides, but I have a handy way of thinking about it which I thought I'd share on my blog.

Imagine you have an arbitrarily nested list structure like so:

lists = [
    1, 2, 3,
    [4, 5, [6, 7], 8],
    [[[9, 10], 11]],
    [[]],
    12,
]

You can flatten this data-structure by writing a recursive generator thanks to the new yield from syntax:

def flatten(items):
    for item in items:
        if isinstance(item, (list, tuple)):
            yield from flatten(item)
        else:
            yield item

The output would then be:

>>> [item for item in flatten(lists)]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]

To achieve this using Python 2.x, which does not have yield from, you would instead write the recursive call like this:

if isinstance(item, (list, tuple)):
    for subitem in flatten(item):
        yield subitem

Window functions, case statements, and savepoints in peewee

February 21, 2014 10:44 / 0 comments

In case you've missed the last few releases, I've been busy adding some fun new features to peewee. While the changelog and the docs explain the new features and describe their usage, I thought I'd write a blog post to provide a bit more context.

Most of these features were requested by peewee users. I depend heavily on users like you to help me improve peewee, so thank you very much! Not only have your feature requests helped make peewee a better library, they've helped me become a better programmer.

So what's new in peewee? Here is something of an overview:

Hopefully some of those things sound interesting. In this post I will not be discussing everything, but will hit some of the highlights.


"djpeewee": use the peewee ORM with your Django models

November 19, 2013 22:40 / 5 comments

I sat down and started working on a new library shortly after posting about Django's missing API for generating SQL. djpeewee is the result, and provides a simple translate() function that will recursively translate a Django model graph into a set of "peewee equivalents". The peewee versions can then be used to construct queries which can be passed back into Django as a "raw query".

Here are a couple scenarios when this might be useful:

  • Joining on fields that are not related by foreign key (for example UUID fields).
  • Performing filters on calculated values.
  • Performing aggregate queries on calculated values.
  • Using SQL statements that Django does not support such as CASE.
  • Utilizing SQL functions that Django does not support, such as SUBSTR.
  • Replacing nearly-identical SQL queries with reusable, composable data-structures.

I've included this module in peewee's playhouse, which is bundled with peewee.


The search for the missing link: what lies between SQL and Django's ORM?

November 12, 2013 11:54 / 7 comments

I had the opportunity this week to write some fairly interesting SQL queries. I don't write "raw" SQL too often, so it was fun to use that part of my brain (by the way, does it bother anyone else when people call SQL "raw"?). At Counsyl we use Django for pretty much everything so naturally we also use the ORM. Every place I've worked there's a strong bias against using SQL when you've got an ORM on board, which makes sense -- if you choose a tool you should standardize on it if for no other reason than it makes maintenance easier.

So as I was saying, I had some pretty interesting queries to write and I struggled to think how to shoehorn them into Django's ORM. I've already written about some of the shortcomings of Django's ORM so I won't rehash those points. I'll just say that Django fell short and I found myself writing SQL. The queries I was working on joined models from very disparate parts of our codebase. The joins were on values that weren't necessarily foreign keys (think UUIDs) and this is something that Django just doesn't cope with. Additionally I was interested in aggregates on calculated values, and it seems like Django can only do aggregates on a single column.

As I was prototyping, I found several mistakes in my queries and decided to run them in the postgres shell before translating them into my code. I started to think that some of these errors could have been avoided if I could find an abstraction that sat between the ORM and a string of SQL. By leveraging the python interpreter, the obvious syntax errors could have been caught at module import time. By using composable data structures, methods I wrote that used similar table structures could have been more DRY. When I write less code, I think I generally write less bugs as well.

That got me started on my search for the "missing link" between SQL (represented as a string) and Django's ORM.


Using peewee to explore CSV files

November 07, 2013 06:19 / 3 comments

I recently heard a talk from a coworker wherein one of the things he discussed was automatically converting CSV data for use with a SQLite database. I thought this would be a great thing to add to peewee, especially as lately I've found myself on several occasions working with CSV and battling with it in a spreadsheet. It would be much easier to load it into a database and then query it using a tool I'm familiar with.

Which brings me to playhouse.csv_loader, a new module I've added to the playhouse package of extras. It's hopefully really easy to use. Here is an example of how you might use it:

>>> from playhouse.csv_loader import *
>>> db = SqliteDatabase(':memory:')  # Create an in-memory sqlite database

# Load the CSV file into the in-memory database and return a Model suitable
# for querying the data.
>>> ZipToTZ = load_csv(db, 'zipcode_to_timezone.csv')

# Get the timezone for a zipcode.
>>> ZipToTZ.get(ZipToTZ.zip == 66047).timezone
'US/Central'

# Get all the zipcodes for my town.
>>> [row.zip for row in ZipToTZ.select().where(
...     (ZipToTZ.city == 'Lawrence') && (ZipToTZ.state == 'KS'))]
[66044, 66045, 66046, 66047, 66049]

Using python to generate awesome linux desktop themes

August 22, 2013 10:56 / 5 comments

I remember spending hours when I was younger cycling through the various awesome color themes on my 386, in the glory days of windows 3.1. Remember hotdog stand?

Hotdog Stand

Well, I haven't changed much. I still enjoy making tweaks to the colors and appearance of my desktop. In this post I'll talk about a script I wrote that makes it easy for me to modify all the various colors and configuration files which control the appearance of my desktop.


Becoming a Django Apologist

July 24, 2013 08:33 / 1 comments

It's been roughly four years since my introduction to the Django framework and I thought I'd write a little post to commemorate this. In my mind nothing had as big an impact on my career as my decision to work at the Journal World. When I started there I knew basically nothing about software engineering or open-source, and it is entirely thanks to my excellent (and patient) coworkers there that I was able to learn about these things.


Rewriting Huey for a better API

May 15, 2013 11:50 / 0 comments

For a while I've been itching to rewrite Huey, and just last week released 0.4 which is an almost total rewrite. I initially started Huey for performing tasks like checking comments for spam, sending emails, generating thumbnails, and basically anything that would slow down the pagespeed on my sites. This is still what I see as the primary use-case for huey -- performing small tasks outside the request/response cycle and running jobs on a schedule (I have a site that scrapes the county sheriff's site and keeps a log of arrests in my town). The goal for the rewrite was not to change the purpose of Huey, rather it was to change the API.


Structuring flask apps, a how-to for those coming from Django

April 27, 2013 13:21 / 0 comments

The other day a friend of mine was trying out flask-peewee and he had some questions about the best way to structure his app to avoid triggering circular imports. For someone new to flask, this can be a bit of a puzzler, especially if you're coming from django which automatically imports your modules. In this post I'll walk through how I like to structure my flask apps to avoid circular imports. In my examples I'll be showing how to use "flask-peewee", but the same technique should be applicable for other flask plugins.

I'll walk through the modules I commonly use in my apps, then show how to tie them all together and provide a single entrypoint into your app.


"wallfix", using python to set my wallpaper

April 22, 2013 09:54 / 0 comments

I had fun writing about my "cd" helper, so I thought I'd share another productivity helper I wrote for setting my wallpaper. It's a little silly, but I insist on my wallpaper being used for my lockscreen and my login window as well -- that way the entire time I'm on my computer the background is "seamless". Before I wrote this script it used to take me probably 3 or 4 minutes to change wallpapers!