Last week, after several false starts, I moved all the sites I maintain into virtualenvs, with their own pip requirements files. My reasons for doing so are pretty simple:

• Experimenting with new/different versions of software is a pain in the ass without isolation
• A pip requirements file for each site is a very nice thing to behold
• I symlink 3rd party apps into a custom directory on my PYTHONPATH and it was getting huge
• Profit

There are quite a few great tutorials out there for getting started with these tools. I will only discuss how I got over some of the hurdles involved in using these tools, as well as a tool for automating the creation of "skeleton" django sites.

The hard parts

Dude where's my code

The first thing I ran into was making sure my newly-minted virtualenv got all the packages it needed. I didn't realize that you needed to pass in the "-E" switch to pip to signify that you want the packages installed in a certain virtualenv:

pip install -E path/to/virtualenv -r path/to/requirements.txt

Additionally, there are several environment variables you can set to alter the way pip behaves. To get pip to always use the currently active virtualenv, use:

export PIP_RESPECT_VIRTUALENV=true

Pip installs packages

I also sort of glossed the fact that "pip installs packages", and therefore won't know what to do with a folder or two of code. It needs a setup.py -- for some help on this, you can do what I did and copy someone else's, or read the docs.

GitHub or Bitbucket

If you use github or bitbucket, it's easy to point your requirements file at third party repos, just be sure to specify an egg-name:

-e git+git@github.com:coleifer/django-relationships.git#egg=relationships
-e git+http://github.com/alex/django-taggit.git#egg=taggit
-e hg+http://bitbucket.org/andrewgodwin/south/#egg=south

These three examples illustrate cloning from a read/write repo over ssh, a read-only repo over http, and a mercurial repo over http. With git, behind-the-scenes pip is just git clone-ing, so if it works with git clone, it should work with pip.

Wsgi

In order to get your site running against the code in your virtualenv, you need to do just a little bit of hacking on your sys.path to instruct it to look in your virtualenv. The site module has a function addsitedir which will take a directory and add it to the pythonpath, additionally processing any .pth files. This function will append to the sys.path, so if you're concerned about the ordering here or anticipate collisions you may need to do a bit more hacking, however the following works for me:

import os, sys
import site

# since, by convention, my wsgi file lives in ~/envs/projectname/apache/, this is how
# I get to ~/envs/projectname/
root_dir = os.path.dirname(os.path.dirname(__file__))

# the site module has a handy function addsitedir, which not only adds the directory to the
# pythonpath, but also processes any .pth files it finds
site.addsitedir(os.path.join(root_dir, 'lib/python2.6/site-packages/'))

# then, for good measure, add the root dir of the env and the project dir
sys.path.insert(0, root_dir)
sys.path.insert(0, os.path.join(root_dir, 'projectname'))

# point django at the projects' settings file
os.environ['DJANGO_SETTINGS_MODULE'] = 'projectname.settings'
import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()

Automation

A while back I wrote about a tool called django-site-gen, which is basically just a souped-up django-admin startproject. It is highly configurable, but probably wouldn't help you much if you intended to use pip + virtualenv. Until now.

django-site-gen will create:

python generate_site.py twitter 9001

• envs/twitter/ - the root directory of django site - a virtualenv
• envs/twitter/apache/ - where the wsgi file lives
• envs/twitter/conf/ - apache and nginx confs for local and production
• envs/twitter/logs/ - apache and nginx logs
• envs/twitter/twitter/ - where the site, site apps, settings, etc live
• envs/twitter/twitter/media/ - site media served by nginx
• envs/twitter/twitter/templates/ - site template dirs

It is highly configurable, all constants except for site_name and port (which are specified at execution-time) are stored in an easy-to-edit conf.yaml file.

settings:
    user: charles
    email: coleifer@gmail.com
    base_site: /home/charles/envs/
    requirements_file: requirements.txt
...
directories:
    - site_root: '%(base_site)s%(site_name)s/'
    - project_root: '%(site_root)s%(site_name)s/'
    - wsgi: '%(project_root)sapache/'
...

Then there is a mapping a file name to template:

files:
    '%(project_root)ssettings.py': settings.py.conf
    '%(wsgi)s%(site_name)s.wsgi': wsgi.conf

This instructs the generate_site script to use the settings.py.conf template to generate a file named "settings.py" that will live in the project_root, which is specified above in the directories section. The generate_site script goes through all the files listed in the files section, rendering the template and saving the rendered output in the directory you specify. You'll notice that the directories is, in YAML terms, a list of dictionaries. This is to preserve ordering, allowing directories to extend previously defined directories.

Taking a look at the wsgi.conf template, you can see that the constants defined in the conf.yaml can be used to generate the stuff that changes from project-to-project:

import os, sys
import site

# redirect print statements to apache log
sys.stdout = sys.stderr

root_dir = '{{ site_root }}'
site.addsitedir(os.path.join(root_dir, 'lib/python2.6/site-packages/'))
sys.path.insert(0, root_dir)
sys.path.insert(0, os.path.join(root_dir, '{{ site_name }}'))

os.environ['DJANGO_SETTINGS_MODULE'] = '{{ site_name }}.settings'
import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()

One additional feature of django-site-gen is that, if you have some skeleton templates/media you like to use, the generate_site script will copy them wholesale into the new site's virtualenv.

This project is pretty hackable, so if you're interested, please consider forking on GitHub.

Edit 7/3/10

It has been pointed out that virtualenvwrapper is handy for quick switching between virtualenvs as well as for providing a way to hook into the activation/deactivation process with custom scripts. Check the docs for a full reference of features.

Edit 7/4/10

I've merged the virtualenv branch into master, also adding support for generation of a gunicorn conf file. The Nginx configuration file generated will reverse proxy to either apache or gunicorn. There is also now support for autogeneration of Flask skeleton apps.