Installing Libraries for Python Outside of System Directories

Python is being used more frequently in HPC applications. Whether a job is being run by the scheduler or pre/post-processing on login nodes, there’s a chance you may run into it. With Python comes the need for libraries. Installing the libraries in system directories normally isn’t possible, but there is a good solution for that.

The solution common to the Python world is to use Virtual Environments[1], or virtualenv. When installed, virtualenv will let you create a custom Python environment in your own directory structure, allowing you to use specific versions of libraries and install libraries without needing system administrator intervention. It also allows for easy switching between Python 2 and Python 3.

Installing Virtualenv
The initial installation of virtualenv is the only part that will need to be done by a system administrator. There are a few ways to do it. First, install the OS distribution provided version with “yum install python-virtualenv”. The plus is that it will be updated along with any OS updates, and compatibility tracks the distribution installed Python version(s).  The downside is it won’t be the latest. The latest can be installed from the command line with “pip install virtualenv”, or if pip isn’t installed, “easy_install virtualenv”.

Creating a New Environment
Once installed, it’s very easy to create a new environment.  Note that you can have multiple environments, and each environment will need to reside in its own directory structure, which you can place anywhere. To create a new environment (in this example called “orange”), simply run “virtualenv orange”. This will create a new directory in your current directory named orange.  If you want multiple virtual environments for different purposes or library versions, it may be easier to manage if you create a directory like “venv”, change to that directory, and create all your environments in there.

Loading and Activation
You can then load that environment with “source orange/bin/activate”.  Your prompt will change, giving you a hint of which virtualenv is active.  You may now change your working directory to anywhere and still be using that virtualenv.  Now you can easy_install or pip install any libraries you want, and they’ll be installed only for this virtualenv.  (You can even “easy_install pip” if you don’t have pip on your system!)  For jobs that are running on the cluster, source the same file in your job submission script, and you’ll be ready to run remotely.

Removing Virtualenv
To clear the virtualenv from your current shell, simply run “deactivate” from the command line.  The libraries and rest of the environment can be permanently removed from your system by removing the virtualenv directory created earlier.

[1] http://docs.python-guide.org/en/latest/dev/virtualenvs/