Python

Pip/Virtualenv

Pip is the preferred package manager for Python and each cluster provides several Python versions through the associated module which comes with pip. In order to install new packages, you will first have to create a personal space for them to be stored. The preferred solution (as it is the preferred solution on Compute Canada clusters) is to use virtual environments.

First, load the python module you want to use:

module load python/3.6

Then, create a virtual environment in your home directory:

virtualenv $HOME/<env>

where <env> is the name of your environment. Finally, activate the environment:

source $HOME/<env>/bin/activate

You can now install any python package you wish using the pip command, e.g. pytorch:

pip install torch torchvision

or Tensorflow:

pip install tensorflow-gpu

Conda

Another solution for Python is to use miniconda or anaconda which are also available through the module command: (the use of conda is not recommended for Compute Canada Clusters due to the availability of custom-built packages for pip)

 module load miniconda/3
[=== Module miniconda/3 loaded ===]
To enable conda environment functions, first use:

 conda-activate

Then like advised, if you want to enable conda activate/deactivate functions, start the following alias once

conda-activate

To create an environment (see here for details) do:

conda create -n <env> python=3.6

where <env> is the name of your environment. You can now activate it by doing:

conda activate <env>

You are now ready to install any python package you want in this environment. For instance, to install pytorch, you can find the conda command of any version you want on pytorch’s website, e.g:

conda install pytorch torchvision cudatoolkit=10.0 -c pytorch

Don’t forget to clean the environment after each install:

conda clean --all