June 04, 2014 14:48 / 2 comments / gist python

Gist logo

I'd like to share a little command-line utility I wrote for managing multi-file and multi-directory private gists on GitHub. If you're not familiar with GitHub Gist, it's basically a git-backed pastebin. One of the benefits of Gist is that it supports private gists for free, allowing you to create private repos for your code snippets. To prevent abuse, GitHub does not allow you to create gists containing subdirectories.

I like to keep my list of public GitHub repositories very tidy, so I frequently use Gists for smaller projects. Last week I wanted to share the code for the note-taking app I blogged about. I didn't want to put the code into a GitHub repo, so I decided to create a gist. Unfortunately, the project contained templates and javascript that needed to go in subdirectories. To work around Gist's subdirectory restriction I used a naming convention to indicate that these files belonged in subdirectories, e.g.:

Then I had a lightbulb moment -- why not write a script to do all this automatically?

Jist

jist is a small ~200 line python script that makes it easy to create and manage private gists. It works by flattening sub-directories before pushing to GitHub, then re-expanding them afterwards. jist supports three primary operations: init, push and clone.

How you might use it

Let's say we have a project in a directory named app. Inside the app directory are folders containing templates, static assets, source code, and more. To create a private gist containing all these files we would simply run:

$ jist init app/
Pushed changes to git@gist.github.com:08d9e56d4a2195355db5.git
https://gist.github.com/coleifer/08d9e56d4a2195355db5

jist will flatten the files, push them to a new gist, then restore the files to their original structure. Later we make some improvements and want to update the gist. To do this, we use jist push:

$ cd app/
# make some changes...
$ jist push

Again, jist will flatten the files, commit and push to our private Gist, then restore the files to their original structure.

Suppose a friend wants to checkout our code. We would share the gist's ID with them and they would clone our gist using jist clone:

$ jist clone <gist id>

This would clone the repo and expand the flattened files into their appropriate strucutre.

jist help

While init, push and clone are the porcelain, there are also some plumbing commands available for doing things like flattening, expanding, etc. For all the details, just run jist help.

SSH Keys

In order to create new Gists you will need to add your public SSH key to GitHub (if you haven't already). To do this navigate to the account details page (the wrench/screwdriver icon), select SSH Keys and upload your public key:

GitHub SSH setup

Creating a personal access token

In order to be able to create new gists from the command-line, we need to use an API token. GitHub makes this very easy to do, so I will walk you through the steps.

First log in to your github account and click the wrench/screwdriver icon to navigate to your account settings. In the account settings page, click the Applications tab on the left-hand navigation. Then select the Generate new token button:

GitHub Applications Page

You will come to a page where you can configure permissions for your new API token. For our purposes we only need permission to create gists, so select the Gist checkbox:

GitHub generate application token

Give your new token a name and save. You will be taken to a page where you can copy your new token. Do not lose this token since this is the only time you will be able to view it:

Gist Github access token

Configuring the API token

You can pass your token into jist using command line arguments:

jist -u myusername -k thetoken <command>

Because entering these values can get very repetitive, jist will also read these values from your global git config. To add these values to your gitconfig file, run the following commands:

$ git config --global jist.user my_github_username
$ git config --global jist.token my_api_token

And that's all you need!

The code

Below is the entire source code for the jist utility:

#!/usr/bin/env python

"""
Jist
====

Create private gists with directories.

MIT license.
"""

import json
import logging
import optparse
import os
import subprocess
import sys
import urllib2


logger = logging.getLogger('jist')
run_logger = logging.getLogger('jist.run')

def run(*args):
    run_logger.debug(' '.join(args))
    p = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    stdout, stderr = p.communicate()
    exit_code = p.wait()
    if exit_code != 0:
        logger.error(stderr.strip() or stdout.strip())
        sys.exit(1)
    return stdout.strip()

def command(highlight=False):
    def decorator(fn):
        fn._is_command = True
        fn._is_highlighted = highlight
        return fn
    return decorator

class Gist(object):
    user_agent = 'jist/0.1'

    def __init__(self, username=None, key=None, quiet=False):
        self.username = username or self.read_config_value('user')
        self.key = key or self.read_config_value('token')
        self.quiet = quiet

    def print_message(self, msg):
        if not self.quiet:
            print(msg)

    def read_config_value(self, key):
        return run('git', 'config', '--global', 'jist.%s' % key)

    def get_api_headers(self):
        return {
            'X-Github-Username': self.username,
            'Content-Type': 'application/json',
            'Authorization': 'token %s' % self.key}

    def guess_dest(self, gist_id):
        try:
            api_detail = self.get_gist_details(gist_id)
        except:
            logger.exception('Unable to fetch gist details from GitHub API.')
            return

        if api_detail['description']:
            first_word = api_detail['description'].split()[0].lower()
            logger.debug('Gist description "%s"', first_word)
            return first_word

        logger.debug('Gist does not contain description, using filename.')
        if api_detail['files']:
            return max([
                filename.lower().split('.')[0]
                for filename in api_detail['files']])

    def get_gist_details(self, gist_id):
        url = 'https://api.github.com/gists/%s' % gist_id
        request = urllib2.Request(url, headers={'User-Agent': self.user_agent})
        try:
            fh = urllib2.urlopen(request)
        except urllib2.HTTPError as exc:
            logger.debug('Received %s requesting %s', exc.code, url)
            raise
        else:
            return json.loads(fh.read())

    def _clone(self, gist_id, dest):
        remote = 'git@gist.github.com:%s.git' % gist_id
        run('git', 'clone', remote, dest)

    @command()
    def pull(self):
        """
        Update local checkout with the latest changes from GitHub. Execute
        this command from within the repository's root directory.

        jist pull
        """
        run('git', 'pull', 'origin', 'master')

    @command(highlight=True)
    def clone(self, gist_id, dest=None):
        """
        Clone a private gist with the given ID. Any flattened directories
        will be expanded.

        jist clone [gist id] [optional: path/for/code]
        """
        if dest is None:
            dest = self.guess_dest(gist_id) or 'gist-%s' % gist_id[:6]

        if os.path.exists(dest):
            logger.info('%s already exists, not cloning.' % dest)
        else:
            self._clone(gist_id, dest)

        os.chdir(dest)
        self.pull()
        self.expand()

    @command()
    def expand(self, path=None):
        """
        Expand flattened files into directories. For example `foo___bar.js`
        would be expanded to `foo/bar.js`. If no path is specified, command
        will run in the current working directory.

        jist expand [optional: path/to/expand]
        """
        if path is not None:
            os.chdir(path)

        cwd = os.getcwd()
        logger.debug('Expanding: %s', cwd)
        for src in os.listdir(cwd):
            if os.path.isdir(src):
                logger.debug('Skipping %s, directory', src)
                continue
            parts = src.split('___')
            path, filename = '/'.join(parts[:-1]), parts[-1]
            if path:
                if not os.path.exists(path):
                    logger.debug('Making new directory %s', path)
                    os.makedirs(path)
                elif os.path.isfile(path):
                    raise Exception(
                        'Directory %s and file of same name found.', path)

                dest = os.path.join(path, filename)
                logger.info('Renaming %s -> %s' % (src, dest))
                os.rename(src, dest)

    @command(highlight=True)
    def push(self, path=None, force_push=True):
        """
        Commit and push changes to GitHub. Any directories will be flattened
        before commiting and pushing, then re-expanded afterwards.

        jist push [optional: path/to/code]
        """
        if path is not None:
            os.chdir(path)

        self.flatten()
        try:
            self.commit()
            run('git', 'push', '-f', 'origin', 'master')
        except:
            self.expand()
            raise
        self.expand()
        remote = run('git', 'config', '--get', 'remote.origin.url')
        self.print_message('Pushed changes to %s' % remote)

    @command()
    def flatten(self, path=None):
        """
        Flatten all directories, renaming files so the directories can be
        reconstructed using `expand`.

        jist flatten [optional: path/to/files]
        """
        if path is not None:
            os.chdir(path)

        cwd = os.getcwd()
        logger.debug('Flattening files in %s' % cwd)

        for dirpath, dirnames, filenames in os.walk('.'):
            if dirpath.startswith('./.git'):
                continue

            for base_filename in filenames:
                file_path = os.path.join(dirpath, base_filename)
                filename = os.path.relpath(file_path, '.')
                flattened = filename.replace('/', '___')
                if filename != flattened:
                    logger.info('Renaming %s -> %s' % (filename, flattened))
                    os.rename(filename, flattened)

    @command()
    def commit(self):
        """
        Commit changes to current working directory.

        jist commit
        """
        run('git', 'add', '.')
        run('git', 'commit', '-a', '-m', 'updates')

    @command(highlight=True)
    def init(self, path=None, description=None):
        """
        Initialize a Git repo and create a new Gist.

        jist init [optional: path/to/files] [optional: gist description]
        """
        if path is not None:
            os.chdir(path)

        # Initialize a git repo in the specified directory.
        cwd = os.getcwd()
        logger.info('Initializing git repository in %s' % cwd)
        run('git', 'init')

        # Create a new Gist using GitHub's API.
        gist_id = self.create(description=description)

        # Add the gist URL as a remote.
        remote_url = 'git@gist.github.com:%s.git' % gist_id
        run('git', 'remote', 'add', 'origin', remote_url)

        self.push()
        self.print_message('https://gist.github.com/%s/%s' % (
            self.username, gist_id))

    def create(self, description='just a jist'):
        data = {
            'description': description,
            'public': False,
            'files': {'jist': {'content': 'jist 0.1 placeholder file.'}}}
        request = urllib2.Request(
            'https://api.github.com/gists',
            data=json.dumps(data),
            headers=self.get_api_headers())
        fh = urllib2.urlopen(request)
        response = json.loads(fh.read())
        return response['id']

    @command()
    def help(self):
        """
        Print available commands.
        """
        commands = []
        for key, value in Gist.__dict__.items():
            if getattr(value, '_is_command', False):
                commands.append(
                    (0 if value._is_highlighted else 1, key, value.__doc__))

        for color_code, command, docstring in sorted(commands):
            print('\x1b[%s;34m%s\x1b[0m' % (color_code ^ 1, command))
            print(docstring)


def get_option_parser():
    parser = optparse.OptionParser(
        usage='Usage: %prog [options] command param1 [param2 [param3]]')
    parser.add_option('-v', '--verbose', action='store_true', dest='verbose')
    parser.add_option('-d', '--debug', action='store_true', dest='debug')
    parser.add_option('-q', '--quiet', action='store_true', dest='quiet')
    parser.add_option('-k', '--key', dest='key')
    parser.add_option('-u', '--username', dest='username')
    return parser

if __name__ == '__main__':
    parser = get_option_parser()
    options, args = parser.parse_args()

    if not options.quiet:
        handler = logging.StreamHandler()
        handler.setFormatter(logging.Formatter(logging.BASIC_FORMAT))
        logger.addHandler(handler)
        if options.verbose:
            logger.setLevel(logging.INFO)
        if options.debug:
            logger.setLevel(logging.DEBUG)

    if len(args) == 0:
        sys.stderr.write('Error, missing command argument.\n')
        parser.print_usage(sys.stderr)
        sys.exit(1)
    else:
        command = args[0]

    gist = Gist(
        username=options.username,
        key=options.key,
        quiet=options.quiet)

    method = getattr(gist, command, None)
    if not callable(method):
        sys.stderr.write('Error, unrecognized command "%s"' % command)
        sys.stderr.flush()
        sys.exit(1)

    method(*args[1:])

Or, if you prefer, you can checkout the jist repo.

Thank you!

Thanks for taking the time to read this post, I hope you found it interesting. If you have any comments or suggestions on ways I could improve this script, please leave a comment!

Links

More like this

Comments (2)

Sam | jun 04 2014, at 04:32pm

That is quite cute! I've just started using gists for a lot more than small one off notes.

An alternative of course is the unlimited free private repos at bitbucket, which I've just found out about.

Anyway, if jist plays nice with mr (https://joeyh.name/blog/entry/introducing_mr/ ) it removes the need to remember to use jist instead of git...

Charles | jun 04 2014, at 04:57pm

Thanks for your comment Sam. I have a bitbucket account but I just haven't gotten around to actually using it, even though, as you mentioned, they give you unlimited free private repos. Maybe I'll give it a try!


Commenting has been closed, but please feel free to contact me