virtualenv HOWTO

An introduction to distutils/pip and virtualenv to manage dependencies in Python.

Unix basics

The $PATH environment variable contains a list of directories where executable programs are searched for.

        $ ls
        $ echo $PATH
        $ which ls
        $ /bin/ls

Unix basics: shebang

When an executable program start with #! (called Shebang) the OS will actually append the program name to the rest of the line and execute that.

        $ which meld
        $ head -n 1 /usr/bin/meld
        $ meld &
        $ pgrep -l meld
        24666 /usr/bin/python /usr/bin/meld

Unix basics: shebang, continued

The shebang must be an absolute path to an executable, optionally with parameters.

/usr/bin/env can search $PATH like shells do.

        $ head -n 1 /usr/bin/ipython
        #!/usr/bin/env python

Python basics

sys.path is a list of directories where modules are searched for when importing.

        >>> import sys, pprint
        >>> pprint.pprint(sys.path)
        >>> pprint.__file__


Give the virtualenv command a directory name to create a new environment.

        $ virtualenv ./test-env
        New python executable in ./test-env/bin/python
        $ ls test-env
        bin  include  lib
        $ ls test-env/bin
        activate  easy_install  pip  python
        $ head -n 1 test-env/bin/pip

virtualenv: activate

Run the activate script with the source command (also .) to add the environment to $PATH.

        $ . test-env/bin/activate
        (test-env)$ echo $PATH
        (test-env)$ which python
        (test-env)$ deactivate

activate is for convenience only. Using test-env/bin/python in a shell or a shebang (like pip does) would Just Work™.

pip: installing packages

pip can download and install Python packages and their dependencies in a virtualenv, without polluting the system or being root.

        (test-env)$ pip install Frozen-Flask
        Downloading/unpacking Frozen-Flask
        Downloading/unpacking Flask (from Frozen-Flask)
        Downloading/unpacking Werkzeug>=0.6.1 (from Flask→Frozen-Flask)
        Downloading/unpacking Jinja2>=2.4 (from Flask→Frozen-Flask)
        Successfully installed Frozen-Flask Flask Werkzeug Jinja2

virtualenv: sys.path

The environment’s python automatically has its sys.path changed:

        (test-env)$ python
        >>> import sys, pprint
        >>> pprint.pprint(sys.path)
         '/usr/lib/python2.6/dist-packages', ]
        >>> import flask
        >>> flask.__file__
        '/home/simon/test-env/lib/python2.6/site-packages/flask/__init__.pyc' describe your dependencies

Add a file to your own projects to describe their dependencies:

        (test-env)$ mkdir -p test-project/myapp
        (test-env)$ touch test-project/myapp/
        (test-env)$ cat > test-project/
        from setuptools import setup, find_packages
            install_requires=["Flask", "docutils"])

pip editables

pip install -e or --editable with a directory name installs a package by adding to sys.path instead of copying files, so you don’t need to reinstall for every change.

        (test-env)$ pip install -e ./test-project/myapp
        Successfully installed docutils MyApp
        (test-env)$ python
        >>> import sys, myapp
        >>> myapp.__file__
        >>> sys.path
        [ '/home/simon/test-project', ]

Wrap up

Trick #1: activate and bash history

You will re-activate the same virtualenv often (at least for every new terminal you open.) To make that easier, source activate with a path that does not depend on the current directory:

        $ . ~/test-env/bin/activate

Next time, find it with your shell’s command history (often CTRL+R.)

Trick #2: /usr/bin/env and virtualenvs

When a script uses /usr/bin/env python instead of eg. /usr/bin/python as a shebang, it will use the activated virtualenv’s python.

I changed /usr/bin/ipython that way on my system so that I can use it without installing it in every virtualenv.

Trick #3: pip download cache

When using more than one virtualenv, you will install the same packages over and over. pip’s download cache can speed things up a bit:

        $ mkdir -p ~/.pip
        $ cat > ~/.pip/pip.conf

Trick #4: virtualenvwrapper

virtualenvwrapper manages virtualenvs for you.