Custom Python Modules and Virtualenv

Virtualenv is a tool for creating isolated Python environments.  It is bundled with the python modules on the ARSC systems.  ARSC encourages its users to use virtualenv to install any necessary third-party Python modules.

Creating your virtual environment

The following steps describe how to create your own Python installation in your $HOME directory on ARSC systems using virtualenv.

1. Log in to any login node on the system which is not busy.  You can get a rough estimate after logging in of how many users are on a node by running 'who', and of overall CPU utilization by running 'top'.  Note: Your Linux workstation $HOME directory is your Pacman $HOME directory, so you may log in to a workstation and treat it as your Pacman login node.

2. Source the modules "init" file on the system.  The "news modules" news item contains the correct syntax for the shell and system you are currently logged into.

3. Clear all loaded modules using "module purge".

4. Load the site Python module using "module load python".  

5. Make a new directory in your home directory, with a suitably unambiguous name ("python" may be ambiguous).  In this article, we will call that directory "mypython".

6. Run the virtualenv tool as shown below to copy all relevant files your 'mypython' directory.   NOTE: The --system-site-packages flag causes your virtual environment to inherit and depend on packages present in the global installation on your system!  If you use any of these packages and want a more complete snapshot of Python that you can keep isolated and stable, remove the --system-site-packages flag from the command below.  This will remove the linkage, but will not locally install the global packages for you.  You will need to manually copy or install all packages you want to use from the global site-packages directory (numpy, scipy, etc) to your local site-packages directory.

virtualenv ~/mypython --system-site-packages

Activating and deactivating your virtual environment

To activate your new virtual environment, source the provided "activate" script as shown below.

source ~/mypython/bin/activate

To deactivate the virtual environment from the shell at any time, simply type "deactivate".

Running Python from your virtual environment

To use the Python installation you have created, activate your virtual environment as described above and run "python" as you would outside of your virtual environment.  The activation script has temporarily prepended its "bin" folder to your $PATH, so this virtualenv version of Python will be called instead of the system installed version of Python.  

Installing custom Python modules in your virtual environment

The following shell commands give an example for how to install a custom module "mymodule" (tarballed as "mymodule-x.x.x.tar.gz") to your virtual environment.

source ~/mypython/bin/activate cd ~/mypython/lib/python2.7/site-packages/ cp ~/path/to/mymodule-x.x.x.tar.gz . tar -xzvf mymodule-x.x.x.tar.gz cd mymodule-x.x.x/ python setup.py install

If you would like more information on installing Python modules, please contact User Support.  

Back to Top