virtualenv HOWTO

An introduction to distutils/pip and virtualenv to manage dependencies in Python.

Unix basics

The $PATH environment variable contains a list of directories where executable programs are searched for. 

        $ ls
        slides.html
        $ echo $PATH
        /usr/local/bin:/usr/bin:/bin:…
        $ which ls
        /bin/ls
        $ /bin/ls
        slides.html

Unix basics: shebang

When an executable program start with #! (called Shebang) the OS will actually append the program name to the rest of the line and execute that. 

        $ which meld
        /usr/bin/meld
        $ head -n 1 /usr/bin/meld
        #!/usr/bin/python
        $ meld &
        $ pgrep -l meld
        24666 /usr/bin/python /usr/bin/meld

Unix basics: shebang, continued

The shebang must be an absolute path to an executable, optionally with parameters.

/usr/bin/env can search $PATH like shells do. 

        $ head -n 1 /usr/bin/ipython
        #!/usr/bin/env python

Python basics

sys.path is a list of directories where modules are searched for when importing. 

        >>> import sys, pprint
        >>> pprint.pprint(sys.path)
        ['',
         '/usr/lib/python2.6',
         '/usr/lib/python2.6/lib-dynload',
         '/home/simon/.local/lib/python2.6/site-packages',
         '/usr/lib/python2.6/dist-packages', …]
        >>> pprint.__file__
        '/usr/lib/python2.6/pprint.pyc'

virtualenv

Give the virtualenv command a directory name to create a new environment. 

        $ virtualenv ./test-env
        New python executable in ./test-env/bin/python…
        $ ls test-env
        bin  include  lib
        $ ls test-env/bin
        activate  easy_install  pip  python  …
        $ head -n 1 test-env/bin/pip
        #!/home/simon/test-env/bin/python

virtualenv: activate

Run the activate script with the source command (also .) to add the environment to $PATH. 

        $ . test-env/bin/activate
        (test-env)$ echo $PATH
        /home/simon/test-env/bin:/usr/local/bin:/usr/bin:…
        (test-env)$ which python
        /home/simon/test-env/bin/python
        (test-env)$ deactivate
        $

activate is for convenience only. Using test-env/bin/python in a shell or a shebang (like pip does) would Just Work™.

pip: installing packages

pip can download and install Python packages and their dependencies in a virtualenv, without polluting the system or being root. 

        (test-env)$ pip install Frozen-Flask
        Downloading/unpacking Frozen-Flask…
        Downloading/unpacking Flask (from Frozen-Flask)…
        Downloading/unpacking Werkzeug>=0.6.1 (from Flask→Frozen-Flask)…
        Downloading/unpacking Jinja2>=2.4 (from Flask→Frozen-Flask)…
        Successfully installed Frozen-Flask Flask Werkzeug Jinja2

virtualenv: sys.path

The environment’s python automatically has its sys.path changed: 

        (test-env)$ python
        >>> import sys, pprint
        >>> pprint.pprint(sys.path)
        ['',
         '/home/simon/test-env/lib/python2.6',
         '/home/simon/test-env/lib/python2.6/site-packages',
         '/usr/lib/python2.6',
         '/usr/lib/python2.6/dist-packages', …]
        >>> import flask
        >>> flask.__file__
        '/home/simon/test-env/lib/python2.6/site-packages/flask/__init__.pyc'

setup.py: describe your dependencies

Add a setup.py file to your own projects to describe their dependencies: 

        (test-env)$ mkdir -p test-project/myapp
        (test-env)$ touch test-project/myapp/__init__.py
        (test-env)$ cat > test-project/setup.py
        from setuptools import setup, find_packages
        setup(
            name='MyApp',
            packages=find_packages(),
            install_requires=["Flask", "docutils"])

pip editables

pip install -e or --editable with a directory name installs a package by adding to sys.path instead of copying files, so you don’t need to reinstall for every change. 

        (test-env)$ pip install -e ./test-project/myapp
        …Successfully installed docutils MyApp
        (test-env)$ python
        >>> import sys, myapp
        >>> myapp.__file__
        '/home/simon/test-project/myapp/__init__.pyc'
        >>> sys.path
        [… '/home/simon/test-project', …]

Wrap up

Trick #1: activate and bash history

You will re-activate the same virtualenv often (at least for every new terminal you open.) To make that easier, source activate with a path that does not depend on the current directory: 

        $ . ~/test-env/bin/activate

Next time, find it with your shell’s command history (often CTRL+R.)

Trick #2: /usr/bin/env and virtualenvs

When a script uses /usr/bin/env python instead of eg. /usr/bin/python as a shebang, it will use the activated virtualenv’s python.

I changed /usr/bin/ipython that way on my system so that I can use it without installing it in every virtualenv.

Trick #3: pip download cache

When using more than one virtualenv, you will install the same packages over and over. pip’s download cache can speed things up a bit: 

        $ mkdir -p ~/.pip
        $ cat > ~/.pip/pip.conf
        [install]
        download-cache=~/.pip/download-cache

Trick #4: virtualenvwrapper

virtualenvwrapper manages virtualenvs for you.