OpenMPI on MeluXina

The MeluXina system environment provides the OpenMPI implementation.

EasyBuild module description

The Open MPI Project is an open source Message Passing Interface implementation that is developed and maintained by a consortium of academic, research, and industry partners. Open MPI is therefore able to combine the expertise, technologies, and resources from all across the High Performance Computing community in order to build the best MPI library available. Open MPI offers advantages for system and software vendors, application developers and computer science researchers.

You can use OpenMPI to build your MPI programs written in C/C++ and fortran.

OpenMPI usage

Interactive

Reserve an interactive session:

salloc -A COMPUTE_ACCOUNT -t 01:00:00 -q dev --res cpudev -p cpu -N 1

The example above will allocate one CPU node in interactive mode (dev QoS with cpudev reservation). Load the OpenMPI (Will load default version if not specified, we recommend to load a specific version) module as in below script.

module load OpenMPI
mpicc hello.c -o ./HelloTest_C

Batch

You can run your MPI program in a batch job using Slurm. The script below compile a simple C HelloWorld test on one CPU node and 5 tasks allocated for 5 minutes.

#!/bin/bash -l
#SBATCH -N 1
#SBATCH --ntasks=5
#SBATCH --cpus-per-task=1
#SBATCH -p cpu
#SBATCH -q test
#SBATCH --time 00:05:00

#Load OpenMPI module
module load OpenMPI

#Check OpenMPI version
mpicc --version

#Compile C program with OpenMPI
mpicc hello.c -o ./hello_c

#Execute the program
srun hello_c

It is recommended to use a Makefile to compile your program. Your Makefile should look like the example below:

#Defines compiler
CC=mpicc

# These are the options we pass to the compiler.
# -std=c++14 means we want to use the C++14 standard.
# -stdlib=libc++ specifies that we want to use the standard library implementation called libc++
# -g specifies that we want to include "debugging symbols" which allows us to use a debugging program.
# -O0 specifies to do no optimizations on our code.
CFLAGS = -std=c++14 -stdlib=libc++ -g -O0

all: hello

hello: hello.o
        $CC $CFLAGS -o hello_c hello.o

hello.o: hello.cpp
          $CC $CFLAGS -c hello.c

It is also possible to compile via a Makefile inside a batch job:

#!/bin/bash -l
#SBATCH -N 1
#SBATCH --ntasks=5
#SBATCH --cpus-per-task=1
#SBATCH -p cpu
#SBATCH -q test
#SBATCH --time 00:05:00

#Load OpenMPI module
module load OpenMPI

#Check OpenMPI version
mpicc --version

#Compile C program with OpenMPI (in parallel)
make -j

#Execute the program
srun hello_c

Known issues

When using our OpenMPI build, you may encounter the following issue related to ucx:

[1637941300.411897] [mel2037:9795 :0]       mm_posix.c:194  UCX  ERROR open(file_name=/proc/9791/fd/41 flags=0x0) failed: No such file or directory
[1637941300.411932] [mel2037:9795 :0]          mm_ep.c:154  UCX  ERROR mm ep failed to connect to remote FIFO id 0xc000000a4000263f: Shared memory error
[1637941300.412053] [mel2037:9790 :0]       mm_posix.c:194  UCX  ERROR open(file_name=/proc/9791/fd/41 flags=0x0) failed: No such file or directory

The problem will be solved by exporting the environment variable: export OMPI_MCA_pml=ucx.