Fully Automated Python Package Releases

I've recently been working on a set of internal tools for automating common processes. It's leveraging some great open source projects (Fabric, Ansible, Docker, Vagrant, and more) but the core functionality is in a python package which we're deploying to a private PyPi repo. (Not devpi yet, but hopefully soon.)

There are quite a few internal users of our package, so we were finding ourselves needing to do point releases pretty often. Since we were already using fabric, we put a basic version of our release process into a fabfile.py, and it's grown to the point that a lot of useful things happen when you run fab release.

  • We use bumpversion to figure out the current and next version numbers, configurably updating the right major/minor/point values.
  • It opens your default $EDITOR with an already-added new section in the HISTORY.rst, then commits that edit.
  • The version is bumped and the release is tagged.
  • The software is built
  • The build is shipped to the private PyPI Repo
  • The documentation is built and then shipped
  • Finally, the right channel in our #slack chat room is notified.

All that's left to do is push all the changes to origin. This is done manually in case anything went horribly wrong; it's easy to revert the changes to your local repository, much uglier when a mess has gone upstream.

You can view the whole release fabfile.py as a gist. This feels like the kind of thing that could be (and probably has been) abstracted and generalized, but this works for now.

There's nothing too complicated in the file, but there are some nice tricks. One of the most fun capabilities is the Slack integration, which we recently did. All credit to slack, it was incredibly easy.

@task
def notify_slack(version=None):
    """ Notify the slack channel of a new release """

    # hardcoded URI from the slack integration panel
    url = ("https://ourteam.slack.com/services/hooks/incoming-webhook"
           "?token=tokenid")

    payload = {
        'username': 'ourpackage-fabfile',
        'icon_emoji': ':shipit:',
        'text': 'Deployed version {} of ourpackage-python'.format(version)}

    requests.post(url, data=json.dumps(payload))

Automating the release process has made it take seconds and, more importantly, work identically every time. It's well worth doing. (I just can't shake the feeling that there's a more generic way to do it.)

Update: I was pointed to zest.releaser which has the same basic features, minus the slack integration. You can also use it with gocept.zestreleaser.customupload to enable the scp uploading we're doing here via fabric.

Telling Technology Stories with IPython Notebook

In my role at Media Temple, I end up doing a lot of ad-hoc analysis, looking at things a diverse as how our infrastructures are growing, to changes in the market, to trying to figure out how we can help our customers better, and so on. As I discussed in the talk, it was becoming a more and more frustrating process. I'd increasingly seen people mentioning something called the IPython Notebook, and I finally decided to give it a try.

It was love at first install. It helped dramatically with so many of the kinds of tasks I was working on, and yet almost nobody I talked to had heard of it or actually tried it out. So, I submitted talk proposals to a few conferences, and was pleasantly suprised when it was accepted by both OSCON and LinuxCon. (Extra special thanks to my local Python Meetup, #pdxpython, for letting me practice before OSCON, and giving me some extremely constructive feedback which really improved the talk.)

I got a lot of nice feedback about the talk, including this extremely kind comment from Fernando Perez, the project's founder:

/images/fperez_tweet.png

The OSCON session was recorded, and thanks to their speaker-friendly video agreement, I'm allowed to share their video with you here.

The best way to see the slides is live, with a running IPython Notebook. You can get all the code (and the instructions to get started) in the talk's github repo.

If you want a preview, you can check out the fullscreen slides or see it inline right here, but please note that much of the talk is 'live' IPython Notebook demos, and the links to them will not work.

Contents © 2014 Joshua Barratt - Powered by Nikola