Python

In development

JupyterLab is accessible for testing, and the service may be subject to disruption. To request access, or to provide feedback, please use our service desk. We are glad to hear from you and learn from your experience!

---

Overview

This documentation uses the following variables:

Variable	Description	Example value
${kernel_python}	The base Python module to use	Python/3.9.5-GCCcore-10.3.0
${kernel_root}	The root path to the kernels / virtual environments	${HOME}/.venvs
${kernel_name}	The name of the kernel / virtual environment	cuda

Variables definitions

Your can set the variables in your current Shell:

[u000000@mel0123 ~] kernel_python="Python/3.9.5-GCCcore-10.3.0"
[u000000@mel0123 ~] kernel_root="${HOME}/.venvs"
[u000000@mel0123 ~] kernel_name="cuda"

Operations

Creating a custom kernel

Create a virtual environment

Load the base Python version:

module load ${kernel_python}

Initialize the virtual environment:

python3 -m venv ${kernel_root}/${kernel_name}

Create a kernel entry point

Create and adapt the following script as ${kernel_root}/${kernel_name}/bin/kernel:

#!/bin/bash -li

# Select software stack
# Important: Always choose the same stack as the one used to create the
# virtual environment
module unuse /apps/USE/easybuild/release/latest/modules/all
module use /apps/USE/easybuild/staging/2021.5/modules/all/

# Load modules
# Customize statements as needed
module load CUDA/11.3.1

# Run Python
# Replace KERNEL_ROOT and KERNEL_NAME with the appropriate values
KERNEL_ROOT/KERNEL_NAME/bin/python "$@"

Set the proper permissions:

chmod +x ${kernel_root}/${kernel_name}/bin/kernel

Set up the virtual environment

Activate the virtual environment

source ${kernel_root}/${kernel_name}/bin/activate

Important

Your Shell prompt should now include the virtual environment name, e.g.:

[u000000@mel0123 ~]
[u000000@mel0123 ~] source ${kernel_root}/${kernel_name}/bin/activate
(cuda) [u000000@mel0123 ~]

Install ipykernel

python3 -m pip install -U ipykernel 2>/dev/null

Errors while installing ipykernel

Depending on the base Python you are using, you may receive errors from pip when installing ipykernel. You may safely ignore those errors.

Install the kernel

python3 -m ipykernel install --name "${kernel_name}" --user

Get the kernel configuration file

kernel_json="$(jupyter-kernelspec list | grep -i ${kernel_name}[^/] | awk '{print $2}')/kernel.json"

Patch the kernel configuration file

sed -i "s|/.*/bin/python.*\"|${kernel_root}/${kernel_name}/bin/kernel\"|" "${kernel_json}"

Install packages into a kernel virtual environment

First, activate the kernel's virtual environment:

source ${kernel_root}/${kernel_name}/bin/activate

Then install Python packages with pip:

python3 -m pip install PACKAGE_NAME

Package installation from notebook

We strongly advise against installing Python packages from a notebook cell, e.g.:

!pip install PACKAGE_NAME

This method does not guarantee that the packages are installed in the running kernel / virtual environment.

---