

9. Linux Containers

Singularity

Singularity is available on the ABCI System. Available versions are Singularity version 2.6 and SingularityPRO 3.5. To use Singularity, set up user environment by the module command.

Singularity 2.6

[username@g0001~]$ module load singularity/2.6.1

SingularityPRO 3.5

[username@g0001~]$ module load singularitypro/3.5

More comprehensive user guide for Singularity will be found:

• Singularity 2.6 User Guide
• SingularityPRO 3.5 User Guide

To run NGC-provided Docker images on ABCI by using Singularity: NVIDIA NGC

Running a container with Singularity

When you use Singularity, you need to start Singularity container using singularity run command in job script. The container image is downloaded at first startup and cached in home area. The second and subsequent times startup is faster by using cached data.

Example) Execution of Singularity

The following sample is execution of Singularity using caffe2 container image published in Docker Hub. python sample.py is executed on Singularity container started by singularity run command.

Singularity 2.6

[username@es1 ~]$ qrsh -l rt_F=1 -l h_rt=1:00:00
[username@g0001~]$ module load singularity/2.6.1
[username@g0001~]$ singularity run --nv docker://caffe2ai/caffe2:latest
Docker image path: index.docker.io/caffe2ai/caffe2:latest
Cache folder set to /fs3/home/username/.singularity/docker
Creating container runtime...
...
[username@g0001~]$ python sample.py
True

SingularityPRO 3.5

[username@es1 ~]$ qrsh -l rt_F=1 -l h_rt=1:00:00
[username@g0001~]$ module load singularitypro/3.5
[username@g0001~]$ singularity run --nv docker://caffe2ai/caffe2:latest
...
Singularity> python sample.py
True

Create a Singularity image (pull)

Singularity container image can be stored as a file. This procedure shows how to create a Singularity image file using pull.

Example) Create a Singularity image file using pull

Singularity 2.6

[username@es1 ~]$ module load singularity/2.6.1
[username@es1 ~]$ singularity pull --name caffe2.img docker://caffe2ai/caffe2:latest
Docker image path: index.docker.io/caffe2ai/caffe2:latest
Cache folder set to /fs3/home/username/.singularity/docker
...
[username@es1 ~]$ ls caffe2.img
caffe2.img

SingularityPRO 3.5

[username@es1 ~]$ module load singularitypro/3.5
[username@es1 ~]$ singularity pull caffe2.img docker://caffe2ai/caffe2:latest
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
...
[username@es1 ~]$ ls caffe2.img
caffe2.img

Example) Start a container using Singularity image file

Singularity 2.6

[username@es1 ~]$ module load singularity/2.6.1
[username@es1 ~]$ singularity run ./caffe2.img

SingularityPRO 3.5

[username@es1 ~]$ module load singularitypro/3.5
[username@es1 ~]$ singularity run ./caffe2.img

Example) Using a Singularity image file in a job script

Singularity 2.6

[username@es1 ~]$ cat job.sh
(snip)
source /etc/profile.d/modules.sh
module load singularity/2.6.1 openmpi/3.1.6

mpiexec -n 4 singularity exec --nv ./caffe2.img \
    python sample.py

SingularityPRO 3.5

[username@es1 ~]$ cat job.sh
(snip)
source /etc/profile.d/modules.sh
module load singularitypro/3.5 openmpi/3.1.6

mpiexec -n 4 singularity exec --nv ./caffe2.img \
    python sample.py

Create a Singularity image (build)

In the SingularityPRO 3.5 environment of the ABCI system, You can build container image files using fakeroot option.

Warning

You cannot build a container image from the recipe file in Singularity 2.6 environment. To use your custom container image, you adapt your own server environment to the ABCI environment (the version of singularity, framework, and mpi), build a container image on it, and then move the container image to ABCI system.

Example) Create a Singularity image file using build

SingularityPRO 3.5

[username@es1 ~]$ module load singularitypro/3.5
[username@es1 ~]$ singularity build --fakeroot ubuntu.sif ubuntu.def
INFO:    Starting build...
(snip)
INFO:    Creating SIF file...
INFO:    Build complete: ubuntu.sif
[username@es1 singularity]$

If the above command is executed under the group area (/ groups1,/groups2), an error occurs. This can be avoided by executing the newgrp command after checking the group to which the id -a command belongs, as shown below, before executing the singularity command.
In the example below, gaa00000 is the group to which it belongs.

[username@es1 groupname]$ id -a
uid=0000(aaa00000aa) gid=0000(aaa00000aa) groups=0000(aaa00000aa),00000(gaa00000)
[username@es1 groupname]$ newgrp gaa00000

Docker

In the ABCI System, job can be executed on Docker container. When you use Docker, you need to set up user environment by the module command and specify -l docker option and -l docker_image option at job submission.

Warning

Docker container can not be used on memory-intensive node in the ABCI system.

option	description
-l docker	job is executed on Docker container
-l docker_images	specify using Docker image

The available Docker image can be referred by show_docker_images command.

[username@es1 ~]$ show_docker_images
REPOSITORY                TAG             IMAGE ID     CREATED       SIZE
jcm:5000/dhub/ubuntu      latest          113a43faa138 3 weeks ago   81.2MB

Warning

In the ABCI System, Users can use only Docker images provided in the system.

Example) job script using Docker

The following job script executes python3 ./test.py on Docker container.

[username@es1 ~]$ cat run.sh
#!/bin/sh
#$-cwd
#$-j y
#$-l rt_F=1
#$-l docker=1
#$-l docker_images="*jcm:5000/dhub/ubuntu*"

python3 ./sample.py

Example) Submission of job script using Docker

[username@es1 ~]$ qsub run.sh
Your job 12345 ("run.sh") has been submitted

Warning

Docker container is only available on a node-exclusive job.