For a while I've been itching to rewrite Huey, and just last week released 0.4 which is an almost total rewrite. I initially started Huey for performing tasks like checking comments for spam, sending emails, generating thumbnails, and basically anything that would slow down the pagespeed on my sites. This is still what I see as the primary use-case for huey -- performing small tasks outside the request/response cycle and running jobs on a schedule (I have a site that scrapes the county sheriff's site and keeps a log of arrests in my town). The goal for the rewrite was not to change the purpose of Huey, rather it was to change the API.
The other day a friend of mine was trying out flask-peewee and he had some questions about the best way to structure his app to avoid triggering circular imports. For someone new to flask, this can be a bit of a puzzler, especially if you're coming from django which automatically imports your modules. In this post I'll walk through how I like to structure my flask apps to avoid circular imports. In my examples I'll be showing how to use "flask-peewee", but the same technique should be applicable for other flask plugins.
I'll walk through the modules I commonly use in my apps, then show how to tie them all together and provide a single entrypoint into your app.
I had fun writing about my "cd" helper, so I thought I'd share another productivity helper I wrote for setting my wallpaper. It's a little silly, but I insist on my wallpaper being used for my lockscreen and my login window as well -- that way the entire time I'm on my computer the background is "seamless". Before I wrote this script it used to take me probably 3 or 4 minutes to change wallpapers!
My password "system" used to be that I had three different passwords, all of which were variations on the same theme. I maintained a list of sites I had accounts on and for each site gave a hint which of the three passwords I used. What a terrible scheme.
A couple weeks ago I decided to do something about it. I wanted, above all, to only have to remember a single password. Being lately security-conscious, I also recognized the need for a unique password on every site.
In this post I'll show how I used python to create a password management system that allows me to use a single "master" password to generate unique passwords for all the sites and services I use.
Everyone uses cd a lot, I'm no exception. Because I use virtualenvs for my
python projects, I'm often "cutting" through several layers of crap to get to
what I actually want to edit. This was a good opportunity for a helper script!
The two biggest annoyances I was trying to alleviate were:
cd ../../some-other-dir/foo/.
It would be nice to just type the part that matters and not the whole
thing.The solution I came up with stores directories I use (the entire path), and then I can perform a search of that history using a partial path.
In this post I'd like to talk about some of the shortcomings of the Django ORM, the ways peewee approaches things differently, and how this resulted in peewee having an API that is both more consistent and more expressive.
I saw a post on hackernews this morning where a guy had built a little screenshot uploader for dropbox. Unfortunately, his script is for Mac OS and I use linux.
So, for linux folks out there, here is a little wrapper around scrot, a linux screenshot utility. It will allow you to capture the full screen, the current window, or free-select a region, then take the resulting image and put it in your dropbox folder or upload it to Imgur:
#!/usr/bin/env python
import base64
import json
import optparse
import os
import subprocess
import sys
import time
import urllib
import urllib2
BINARY = 'scrot'
HOME = os.environ['HOME']
# Imgur API -- register your app and paste the client id and secret:
CLIENT_ID = ''
CLIENT_SECRET = ''
# Location of your dropbox folder and your dropbox user id:
DROPBOX_DIR = os.path.join(HOME, 'Dropbox/Public/screens/')
DROPBOX_URL_TEMPLATE = 'http://dl.dropbox.com/u/%s/screens/%s'
DROPBOX_UID = ''
def upload_file(filename):
with open(filename, 'rb') as fh:
contents = fh.read()
payload = urllib.urlencode((
('image', base64.b64encode(contents)),
('key', CLIENT_SECRET),
))
request = urllib2.Request('https://api.imgur.com/3/image', payload)
request.add_header('Authorization', 'Client-ID ' + CLIENT_ID)
try:
resp = urllib2.urlopen(request)
except urllib2.HTTPError, exc:
return False, 'Returned status: %s' % exc.code
except urllib2.URLError, exc:
return False, exc.reason
resp_data = resp.read()
try:
resp_json = json.loads(resp_data)
except ValueError:
return False, 'Error decoding response: %s' % resp_data
if resp_json['success']:
return True, resp_json['data']['link']
return False, 'Imgur failure: %s' % resp_data
def get_parser():
parser = optparse.OptionParser('Screenshot helper')
parser.add_option('-s', '--select', action='store_true', default=True,
dest='select', help='Select region to capture')
parser.add_option('-f', '--full', action='store_true', dest='full',
help='Capture entire screen')
parser.add_option('-c', '--current', action='store_true', dest='current',
help='Capture currently selected window')
parser.add_option('-d', '--delay', default=0, dest='delay', type='int',
help='Seconds to wait before capture')
parser.add_option('-p', '--public', action='store_true', dest='dropbox',
help='Store in dropbox public folder')
parser.add_option('-x', '--no-upload', action='store_false', default=True,
dest='upload', help='Do not upload to imgur')
parser.add_option('-k', '--keep-local', action='store_true', default=False,
dest='keep', help='Keep local copy after upload')
return parser
def get_scrot_command(filename, options):
args = [BINARY]
if options.current:
args.append('-u')
elif not options.full:
args.append('-s')
if options.delay:
args.append('-d %s' % options.delay)
args.append(dest)
return args
if __name__ == '__main__':
parser = get_parser()
options, args = parser.parse_args()
filename = 's%s.png' % time.time()
if options.dropbox:
dest = os.path.join(DROPBOX_DIR, filename)
else:
dest = os.path.join(HOME, 'tmp', filename)
if not options.current and not options.full:
print 'Select a region to capture...'
scrot_args = get_scrot_command(dest, options)
p = subprocess.Popen(scrot_args)
p.wait()
if options.dropbox:
print DROPBOX_URL_TEMPLATE % (DROPBOX_UID, filename)
if options.upload:
success, res = upload_file(dest)
if not success:
print 'Error uploading image: %s' % res
print 'Image stored in: %s' % dest
sys.exit(1)
else:
if not options.keep:
os.unlink(dest)
print res
else:
print dest
I'm working on a little photography website for my Dad and thought it would be neat to extract color information from photographs. I tried a couple of different approaches before finding one that works pretty well. This approach uses k-means clustering to cluster the pixels in groups based on their color. The center of those resulting clusters are then the "dominant" colors. k-means is a great fit for this problem because it is (usually) fast.
Today I merged in the "unstable/2.0" branch of peewee. I'm very excited about the changes and I hope you will be, too.
I have written a documentation page on upgrading which gives the rationale behind the rewrite and some examples of the new querying API. Please feel free to take a look but much of the information presented in this post is lifted directly from the docs.
The biggest changes between 1.0 and 2.0 are in the syntax used for constructing queries. The first iteration of peewee I threw up on github was about 600 lines. I was passing around strings and dictionaries and as time went on and I added features, those strings turned into tuples and objects. This meant, though, that I needed code to handle all the possible ways of expressing something. Look at the code for parse_select.
I learned a valuable lesson: keep data in datastructures until the absolute last second.
With the benefit of hindsight and experience, I decided to rewrite and unify the API a bit. The result is a tradeoff. The newer syntax may be a bit more verbose at times, but at least it will be consistent.
The other day I noticed I had a couple thumbdrives kicking around with various versions of my "absolutely do not lose" files...stuff like my private keys, tax documents, zips of papers I wrote in college, etc. These USB drives were all over the house, and many contained duplicate versions of the same files. I thought it would be neat to write a little app to give me a web-based interface to store and manage these files securely. In this post I'll talk about how I built a web-based file storage app using flask, pycrypto, and amazon S3.