Skip to content

Python on MeluXina

The MeluXina system environment provides the Python programming language.

EasyBuild module description

Python is a programming language that lets you work more quickly and integrate your systems more effectively.

You can use Python to run your applications.

Python usage

Interactive

Reserve an interactive session:

salloc -A COMPUTE_ACCOUNT -t 01:00:00 -q dev --res cpudev -p cpu -N 1

The example above will allocate one CPU node in interactive mode (dev QoS with cpudev reservation) . You can use Python in two different ways:

  • Interactively by login into Python programming language environment through terminal.
  • By running a Python script containing all your command necessary for your program to run.

See examples below:

Using Python

Check Python versions available on the system with module command:

module avail Python
Terminal output example
Python:
---------------------------------------------------------------------------------------------
Description:
Python is a programming language that lets you work more quickly and integrate your systems more effectively.

Versions:
   Python/2.7.18-GCCcore-10.2.0
   Python/3.8.6-GCCcore-10.2.0
Other possible modules matches:
   Biopython  flatbuffers-python  protobuf-python

---------------------------------------------------------------------------------------------
To find other possible module matches execute:

  $ module -r spider '.*Python.*'

---------------------------------------------------------------------------------------------
For detailed information about a specific "Python" package (including how to load the modules) use the module's full name.
Note that names that have a trailing (E) are extensions provided by other modules.
For example:

$ module spider Python/3.8.6-GCCcore-10.2.0
---------------------------------------------------------------------------------------------

Load the Python module.

module load Python

Log into the Python environment

python

Terminal output example
Python 3.6.8 (default, Aug 18 2020, 08:33:21) 
[GCC 8.3.1 20191121 (Red Hat 8.3.1-5)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print(sys.version)
3.6.8 (default, Aug 18 2020, 08:33:21) 
[GCC 8.3.1 20191121 (Red Hat 8.3.1-5)]

Python can also be used interactively to run your python script programs.

#Load Python module
module load Python

#Run your Python script
python my_python_script.py

Batch

Python can also be used in a batch job using Slurm. The script below runs a Python program through a script on one CPU node allocated for 1 hour.

#!/bin/bash -l
#SBATCH -N 1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH -p cpu
#SBATCH -q test
#SBATCH --time 01:00:00

#Load Python module
module load Python

#Check Python version
python -c  'import sys; print(sys.version)'

#Execute the program
python my_python_script.py

Python environment

Our Python installations only come with some standard packages (including the standard library). Your applications may use and need packages or versions (for different reasons such as bug fixes or obsolete version) that are not part of our standard packages. Our Python installation can therefore not meet all requirements of every application. One solution is to create a self-contained tree that contains a Python installation for a particular version of Python, plus a number of additional packages: Python virtual environment.

Python Environment

A virtual environment is a Python tool for dependency management and project isolation. They allow Python site packages (third party libraries) to be installed locally in an isolated directory for a particular project, as opposed to being installed globally (i.e. as part of a system-wide Python).

Creating virtual environment

Run the following command to create a python virtual environment:

python3 -m venv my_python-env

The command above will create the my_python-env directory if it doesn't exist, and some other directories containing the standard library and the Python (3) interpreter. Activate the virtual environment to modify your current environment. Therefore, running python will get you the version installed in the virtual environment (python 3 in our example).

source my_python-env/bin/activate

python
terminal output example
Python 3.6.8 (default, Mar 19 2021, 05:13:41) 
[GCC 8.4.1 20200928 (Red Hat 8.4.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/usr/lib64/python36.zip', '/usr/lib64/python3.6', '/usr/lib64/python3.6/lib-dynload', '$HOME/my_python-env/lib64/python3.6/site-packages', '$HOME/my_python-env/lib/python3.6/site-packages']
>>> 

It can be noticed that the standard library is loaded from $HOME/my_python-env/ virtual environment directory.

Installing your own packages

Interactive

It is possible to install any packages required by your Python program in your virtual environment directory using the pip program/command.

python -m pip install pandas scipy seaborn

or request a specific version of the package:

python -m pip install pandas==1.1.5 scipy==1.5.4 seaborn==v0.11.2
Terminal output example
Successfully installed pandas-1.1.5 scipy-1.5.4 seaborn-0.11.2

Batch mode

The python virtual environment can also be set up from a batch(slurm) job. The example below shows a python program requiring specific packages from a standard library:

#!/bin/bash -l
#SBATCH -N 1
#SBATCH --ntasks=128
#SBATCH --cpus-per-task=1
#SBATCH -p cpu
#SBATCH -q test
#SBATCH --time 01:00:00

#Load Python module
module load Python

#Check Python version
python -c  'import sys; print(sys.version)'

#Create the virtual environment
python -m venv my_python-env

#Source to activate the virtual envronment
source my_python-env/bin/activate

#Install the dependencies(listed in requirement.txt) with pip in the virtual environment
python -m pip install -r requirements.txt

#Execute the program
python my_python_script.py

requirement.txt contains all the necessary packages for the python program to be run.

The script only works on one node simulation. For multi-nodes solution, please contact us to get support.