MPI

The following MPIs can be used with the ABCI system.

To use one of these libraries, it is necessary to configure the user environment in advance using the module command. If you run the module command in an interactive node, environment variables for compilation are set automatically. If you run the module command in a compute node, environment variables both for compilation and execution are set automatically.

[username@es1 ~]$ module load hpcx/2.12

[username@es1 ~]$ module load intel-mpi/2021.11

The following is a list MPI versions installed in the ABCI system.

NVIDIA HPC-X

Module Version	Open MPI Version	Compute Node (V)	Compute Node (A)
2.12	4.1.5a1	Yes	Yes

Using HPC-X

This section describes how to use the NVIDIA HPC-X module.

ABCI provides the following types of HPC-X modules.Please load the module according to your application.

Module Name	Description
hpcx	Standard
hpcx-mt	Multi-Threading support
hpcx-debug	for debug
hpcx-prof	for profiling

When executing the mpirun and mpiexec commands in a job, a host file is also specified in the -hostfile option. The host file is set in the $SGE_JOB_HOSTLIST environment variable.

[username@es1 ~]$ qrsh -g groupname -l rt_F=2 -l h_rt=01:00:0
[username@g0001 ~]$ module load hpcx/2.12
[username@g0001 ~]$ mpirun -np 2 -map-by ppr:1:node -hostfile $SGE_JOB_HOSTLIST ./hello_c
Hello, world, I am 0 of 2, (Open MPI v4.1.5a1, package: Open MPI root@hpc-kernel-03 Distribution, ident: 4.1.5a1, repo rev: v4.1.4-2-g1c67bf1c6a, Unreleased developer copy, 144)
Hello, world, I am 1 of 2, (Open MPI v4.1.5a1, package: Open MPI root@hpc-kernel-03 Distribution, ident: 4.1.5a1, repo rev: v4.1.4-2-g1c67bf1c6a, Unreleased developer copy, 144)

NVIDIA HPC-X provides the NCCL-SHARP plug-in. The plug-in supports different versions of NCCL for different versions of HPC-X. See the table below for compatibility between HPC-X and NCCL.

HPC-X Version	NCCL Version
2.12	2.12

For information on how to use SHARP and the NCCL-SHARP plug-in, see Using SHARP.

For more information about NVIDIA HPC-X, please refer to the official documentation.

Intel MPI

intel-mpi/	Compute Node (V)	Compute Node (A)
2021.11	Yes	Yes