Python

Python versions

Typically several versions of Python are supported - and provided - on DICE. The default (as invoked by the python command) will be whatever is provided as standard for the distribution. Here's a summary of installed versions:

Python modules on DICE

Python has a straightforward and powerful mechanism, "pip" to allow users to install their own modules, described below. This is usually the simplest way to get things done.

We can usually install any other libraries you require where they are available pre-packaged as part of the distribution, or easily built using the standard python setup tools. Asking us to handle the installation of Python modules can sometimes make your life easier: not only will we do the initial install onto all DICE machines, but we can also handle any security updates or changes required for operating system upgrades. We'll try to do this whenever it provides a wider benefit to all DICE users.

We aim to keep up-to-date with the latest versions of important modules but this is not always possible. Sometimes new versions will not be compatible with the version of Python or the Linux distribution used for the DICE environment. Also, if there is a teaching requirement for a specific version of a module then that will be frozen for the whole of the teaching year. Also, because providing modules for non-default versions of Python can be disproportionately expensive, we usually only install core libraries for alternative Python versions. However we understand not all modules can be built without assistance, and we'll provide any build dependencies needed for such modules (where possible).

Installing python modules yourself

If your project needs newer versions of Python modules, there is a very easy way to install them in your home directory or group space: use the virtualenv / pyvenv and virtualenvwrapper tools. A web search reveals excellent in-depth tutorials so we will just give a brief taster here.

See also the page on Installing Software which covers the use of pip without virtualenv.

Getting Started with virtualenv and pyvenv

For the first example let's assume you want to install the latest version of the awesome all-singing interpreter bpython (though be aware we do already provide bpython, and IPython, and JupyterHub, and plenty more...) for your project. Firstly we create a new virtual environment (named myproject) and make it active.

cd
# one of...
  pyvenv myproject  # for python 3
  virtualenv --distribute myproject  # for python 2
# then...
cd myproject
source ./bin/activate

You should notice your shell prompt change after sourcing the activate script: it will gain a (myproject) prefix which helps you to identify which virtual environment you are actively using.

You can now install pretty much any module you like using the pip or easy_install tools, which will have been added to your standard path when the virtual environment was activated. For example:

pip install bpython

Choosing a python version

One good reason to use virtual environments is to use a specific version of Python. If you're using virtualenv, just use the --python command line option when creating your project:

virtualenv --distribute --python=/usr/bin/python3.6 foobar

To use the modules you have installed in your virtual environment, as before, remember to first source the activate script.

virtualenv and Python 3 ("pyvenv")

As of our Python 3.x installations, virtualenv functionality is provided as part of the core python installation as the tool pyvenv. It lacks most of the additional options of virtualenv as it is tied to the calling version of python but works, and is activated, in exactly the same way, for example:

pyvenv foobar
source ./foobar/bin/activate

Tools to use with virtualenv

virtualenvwrapper

To further simplify the process there is a group of tools which are usually referred to as virtualenvwrapper. This is basically a set of shell scripts which make it easy to create multiple virtual environments and then switch them between them as necessary.

1. To get started, make a top-level directory to contain your virtual environments:
   

   mkdir $HOME/virtualenvs
   

2. Set an environment variable for your shell to specify the location of this top-level directory and then call the initialisation script:
   

   if [ -x /usr/bin/virtualenvwrapper.sh ] ; then
     export WORKON_HOME=$HOME/virtualenvs
     source /usr/bin/virtualenvwrapper.sh
   fi
   

That needs to be done in the configuration file for your chosen shell (usually .brc) so that it is run everytime you login. The first time the virtualenvwrapper.sh script is run it will create various directories within your top-level virtual environment directory.

Here are a few example commands:

To create a new virtual environment:

mkvirtualenv myproject

To activate a virtual environment:

workon myproject

It's also useful to note that running workon with no virtual environment name will give you a list of all those available (if any).

To deactivate a virtual environment:

deactivate 

Notes for SL7 users

Bear in mind that the final version of python we will provide on SL7 is 3.6, matching the SCL version, it is now unlikely you will need to use these workarounds, which we'll retained here for SL7 compatibility purposes.

Software collections & integrated development environments

Please see our help page on Software Collections for advice on using the scl utility. Note however that sometimes the scl command may not be required within software such as PyCharm; for this IDE, you only need to add the python binary path (/opt/rh/rh-python36/root/bin/python) to the list of Python Interpreters within Settings.

Combining SCL and virtual environments

If you are using one of the SCL-provided versions of Python (e.g. 3.5 on SL7) you will need to prefix the virtualenv commands above with scl enable rh-python35 <command>, or else run them all inside an SCL shell with scl enable rh-python36 bash (then using ^D or 'exit' to leave the inner shell when you are done).