Skip to content

9. Linux Containers

Singularity

Singularity is available on the ABCI System. Available versions are Singularity version 2.6 and Singularity PRO 3.5. To use Singularity, set up user environment by the module command.

Singularity 2.6

[username@g0001~]$ module load singularity/2.6.1

Singularity PRO 3.5

[username@g0001~]$ module load singularitypro/3.5

More comprehensive user guide for Singularity will be found:
Singularity 2.6 User Guide
Singularity PRO 3.5 User Guide

To run NGC-provided Docker images on ABCI by using Singularity: NVIDIA GPU Cloud (NGC)

Running a container with Singularity

When you use Singularity, you need to start Singularity container using singularity run command in job script. The container image is downloaded at first startup and cached in home area. The second and subsequent times startup is faster by using cached data.

Example) Execution of Singularity

The following sample is execution of Singularity using caffe2 container image published in Docker Hub. python sample.py is executed on Singularity container started by singularity run command.

Singularity 2.6

[username@es1 ~]$ qrsh -l rt_F=1 -l h_rt=1:00:00
[username@g0001~]$ module load singularity/2.6.1
[username@g0001~]$ singularity run --nv docker://caffe2ai/caffe2:latest
Docker image path: index.docker.io/caffe2ai/caffe2:latest
Cache folder set to /fs3/home/username/.singularity/docker
Creating container runtime...
...
[username@g0001~]$ python sample.py
True

Singularity PRO 3.5

[username@es1 ~]$ qrsh -l rt_F=1 -l h_rt=1:00:00
[username@g0001~]$ module load singularitypro/3.5
[username@g0001~]$ singularity run --nv docker://caffe2ai/caffe2:latest
...
Singularity> python sample.py
True

Create a Singularity image

Singularity container image can be stored as a file. On ABCI system, you cannot build a container image from the recipe. To use your custom container image, you adapt your own server environment to the ABCI environment (the version of singularity, framework, and mpi), build a container image on it, and then move the container image to ABCI system.

Example) Create a Singularity image file

Singularity 2.6

[username@es1 ~]$ module load singularity/2.6.1
[username@es1 ~]$ singularity pull --name caffe2.img docker://caffe2ai/caffe2:latest
Docker image path: index.docker.io/caffe2ai/caffe2:latest
Cache folder set to /fs3/home/username/.singularity/docker
...
[username@es1 ~]$ ls caffe2.img
caffe2.img

Singularity PRO 3.5

[username@es1 ~]$ module load singularitypro/3.5
[username@es1 ~]$ singularity pull caffe2.img docker://caffe2ai/caffe2:latest
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
...
[username@es1 ~]$ ls caffe2.img
caffe2.img

Example) Start a container using Singularity image file

Singularity 2.6

[username@es1 ~]$ module load singularity/2.6.1
[username@es1 ~]$ singularity run ./caffe2.img

Singularity PRO 3.5

[username@es1 ~]$ module load singularitypro/3.5
[username@es1 ~]$ singularity run ./caffe2.img

Example) Using a Singularity image file in a job script

Singularity 2.6

[username@es1 ~]$ cat job.sh
(snip)
source /etc/profile.d/modules.sh
module load singularity/2.6.1 openmpi/3.1.6

mpiexec -n 4 singularity exec --nv ./caffe2.img \
    python sample.py

Singularity PRO 3.5

[username@es1 ~]$ cat job.sh
(snip)
source /etc/profile.d/modules.sh
module load singularitypro/3.5 openmpi/3.1.6

mpiexec -n 4 singularity exec --nv ./caffe2.img \
    python sample.py

Docker

In the ABCI System, job can be executed on Docker container. When you use Docker, you need to set up user environment by the module command and specify -l docker option and -l docker_image option at job submission.

Warning

Docker container can not be used on memory intensive node in the ABCI system.

option description
-l docker job is executed on Docker container
-l docker_images specify using Docker image

The available Docker image can be referred by show_docker_images command.

[username@es1 ~]$ show_docker_images
REPOSITORY                TAG             IMAGE ID     CREATED       SIZE
jcm:5000/dhub/ubuntu      latest          113a43faa138 3 weeks ago   81.2MB

Warning

In the ABCI System, Users can use only Docker images provided in the system.

Example) job script using Docker

The following job script executes python3 ./test.py on Docker container.

[username@es1 ~]$ cat run.sh
#!/bin/sh
#$-cwd
#$-j y
#$-l rt_F=1
#$-l docker=1
#$-l docker_images="*jcm:5000/dhub/ubuntu*"

python3 ./sample.py

Example) Submission of job script using Docker

[username@es1 ~]$ qsub run.sh
Your job 12345 ("run.sh") has been submitted

Warning

Docker container is only available on a node-exclusive job.