Skip to content

7. GPU

The following libraries provided by NVIDIA are available on the ABCI System:

To use these libraries, it is necessary to set up the users environment using the module command in advance. The module command allows users to automatically set environment variables for execution, such asPATH, and environment variables for compilation, such as search paths for header files and libraries.

[username@g0001 ~]$ module load cuda/10.0/10.0.130.1
[username@g0001 ~]$ module load cudnn/7.4/7.4.2
[username@g0001 ~]$ module load nccl/2.4/2.4.7-1

The following is a list of CUDA Toolkit, cuDNN, and NCCL that can be used with the ABCI system.

CUDA Toolkit

Major version Minor version Available from NVIDIA Available on ABCI
cuda/8.0 8.0.44 Yes -
cuda/8.0 8.0.61 Yes -
cuda/8.0 8.0.61.2 Yes Yes
cuda/9.0 9.0.176 Yes Yes
cuda/9.0 9.0.176.1 Yes -
cuda/9.0 9.0.176.2 Yes -
cuda/9.0 9.0.176.3 Yes -
cuda/9.0 9.0.176.4 Yes Yes
cuda/9.1 9.1.85 Yes -
cuda/9.1 9.1.85.1 Yes -
cuda/9.1 9.1.85.2 Yes -
cuda/9.1 9.1.85.3 Yes Yes
cuda/9.2 9.2.88.1 - Yes
cuda/9.2 9.2.148 Yes -
cuda/9.2 9.2.148.1 Yes Yes
cuda/10.0 10.0.130 Yes Yes
cuda/10.0 10.0.130.1 Yes Yes
cuda/10.1 10.1.105 Yes -
cuda/10.1 10.1.168 Yes -
cuda/10.1 10.1.243 Yes Yes
cuda/10.2 10.2.89 Yes Yes

cuDNN

Version cuda/8.0 cuda/9.0 cuda/9.1 cuda/9.2 cuda/10.0 cuda/10.1 cuda/10.2
5.1.10 Yes - - - - - -
6.0.21 Yes - - - - - -
7.0.5 Yes Yes Yes - - - -
7.1.3 Yes Yes Yes - - - -
7.1.4 - Yes - Yes - - -
7.2.1 - Yes - Yes - - -
7.3.0 - - - - - - -
7.3.1 - Yes - Yes Yes - -
7.4.1 - - - - - - -
7.4.2 - Yes - Yes Yes - -
7.5.0 - Yes - Yes Yes Yes -
7.5.1 - Yes - Yes Yes Yes -
7.6.0 - Yes - Yes Yes Yes -
7.6.1 - Yes - Yes Yes Yes -
7.6.2 - Yes - Yes Yes Yes -
7.6.3 - Yes - Yes Yes Yes -
7.6.4 - Yes - Yes Yes Yes -
7.6.5 - Yes - Yes Yes Yes Yes
8.0.2 - - - - - Yes Yes

NCCL

Version cuda/8.0 cuda/9.0 cuda/9.1 cuda/9.2 cuda/10.0 cuda/10.1 cuda/10.2
1.3.5-1 Yes Yes Yes Yes Yes - -
2.0.5-3 - - - - - - -
2.1.15-1 Yes Yes Yes - - - -
2.2.12-1 - - - - - - -
2.2.13-1 Yes Yes - Yes - - -
2.3.4-1 - Yes - Yes Yes - -
2.3.5-2 - Yes - Yes Yes - -
2.3.7-1 - Yes - Yes Yes - -
2.4.2-1 - - - Yes Yes Yes -
2.4.7-1 - - - Yes Yes Yes -
2.4.8-1 - - - Yes Yes Yes -
2.5.6-1 - Yes - - Yes Yes Yes
2.6.4-1 - - - - Yes Yes Yes
2.7.8-1 - - - - - Yes Yes

GDRCopy

Version cuda/8.0 cuda/9.0 cuda/9.1 cuda/9.2 cuda/10.0 cuda/10.1 cuda/10.2
2.0 Yes Yes Yes Yes Yes Yes Yes

Changing GPU Compute Mode

Only when exclusively allocating compute nodes with -l rt_F=N option, you can change GPU Compute Mode by using -v GPU_COMPUTE_MODE=num option. The following three Compute Modes can be specified.

Option Description
-v GPU_COMPUTE_MODE=0 DEFAULT mode.
Multiple contexts are allowed per device.
-v GPU_COMPUTE_MODE=1 EXCLUSIVE_PROCESS mode.
Only one context is allowed per device, usable from multiple threads at a time.
-v GPU_COMPUTE_MODE=2 PROHIBITED mode.
No contexts are allowed per device (no compute apps).

Execution example in an interactive job:

[username@es1 ~]$ qrsh -g grpname -l rt_F=1 -l h_rt=1:00:00 -v GPU_COMPUTE_MODE=1 /bin/bash

Execution example in a batch job:

#!/bin/bash

#$ -l rt_F=1
#$ -l h_rt=1:00:00
#$ -j y
#$ -cwd
#$ -v GPU_COMPUTE_MODE=1
/usr/bin/nvidia-smi