4. Storage
4.1. Home Area
Home area is the disk area shared by each interactive and compute node, which is available to all ABCI users by default. The disk quota is limited to 200GiB.
Warning
In Home area, there is an upper limit on the number of files that can be created under one directory. The upper limit depends on the length of the file names located in the directory and is approximately 4M to 12M files. If an attempt is made to create a file that exceeds the limit, an error message "No spcae left on device" is output and the file creation fails.
[Advanced Option] File Striping
Home area is provided by the Lustre file system. The Lustre file system distributes and stores file data onto multiple disks. On home area, you can choose two distribution methods which are Round-Robin (default) and Striping.
How to Set Up File Striping
$ lfs setstripe [options] <dirname | filename>
| Option | Description |
|---|---|
| -S | Sets a stripe size. -S #k, -S #m or -S #g option sets the size to KiB, MiB or GiB respectively. |
| -i | Specifies the start OST index to which a file is written. If -1 is set, the start OST is randomly selected. |
| -c | Sets a stripe count. If -1 is set, all available OSTs are written. |
Tips
To display OST index, use the lfs df or lfs osts command
Example) Set a stripe pattern #1. (Creating a new file with a specific stripe pattern.)
[username@es1 work]$ lfs setstripe -S 1m -i 10 -c 4 stripe-file [username@es1 work]$ ls stripe-file
Example) Set a stripe pattern #2. (Setting up a stripe pattern to a directory.)
[username@es1 work]$ mkdir stripe-dir [username@es1 work]$ lfs setstripe -S 1m -i 10 -c 4 stripe-dir
How to Display File Striping Settings
To display the stripe pattern of a specified file or directory, use the lfs getstripe command.
$ lfs getstripe <dirname | filename>
Example) Display stripe settings #1. (Displaying the stripe pattern of a file.)
[username@es1 work]$ lfs getstripe stripe-file stripe-file lmm_stripe_count: 4 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 10 obdidx objid objid group 10 3024641 0x2e2701 0 11 3026034 0x2e2c72 0 12 3021952 0x2e1c80 0 13 3019616 0x2e1360 0
Example) Display stripe settings #2. (Displaying the stripe pattern of a directory.)
[username@es1 work]$ lfs getstripe stripe-dir stripe-dir stripe_count: 4 stripe_size: 1048576 stripe_offset: 10
4.2. Group Area
Group area is the disk area shared by each interactive and compute node. To use Group area, "Usage Manager" of the group needs to apply "Add group disk" via ABCI User Portal. Regarding how to add group disk, please refer to Disk Addition Request in the ABCI Portal Guide.
4.3. Local Storage
In ABCI System, 1.6 TB NVMe storage is installed into each compute node and 1.8 TB SATA 3.0 storage is installed into each memory intensive node. There are two ways to utilize this storage as follows:
- Using as a local storage of each compute node (Using as a local storage).
- Using as a distributed shared file system, which consists of multiple NVMe storages in multiple compute nodes (Using as a BeeOND storage).
The follwing table shows how to utilize local storage by two types of node.
| compute node | memory intensive node | |
|---|---|---|
| using as a local storage | ok | ok |
| using as a BeeOND storage | ok | - |
4.3.1. Using as a local storage
The NVMe storage can be used as a local storage for each compute node.
In this case, user can access to local storage by using environment variables SGE_LOCALDIR.
When using this method, you need not to specify any additional options.
Note that the amount of the local storage you can use is determined by "Resource type". For more detail on "Resource type", please refer to Job Executing Resource.
Warning
The file stored under $SGE_LOCALDIR directory is removed when job finished. The required files need to be moved to Home area or Group area in job script using cp command.
4.3.2. Using as a BeeOND storage
The set of NVMe storages of job assigned compute nodes can be used as a
distributed shared file system (BeeGFS) on demand.
When using on demand BeeGFS (BeeOND), you need to submit job with -l USE_BEEOND=1 option.
And you need to specify -l rt_F option in this case, because node must be exclusively allocated to job.
The created distributed shared file system area can be accessed from /beeond.
Example) sample of job script (use_beeond.sh)
#!/bin/bash #$-l rt_F=2 #$-l USE_BEEOND=1 #$-cwd echo test1 > /beeond/foo.txt echo test2 > /beeond/bar.txt cp -rp /beeond/foo.txt $HOME/test/foo.txt
Example) Submitting a job
[username@es1 ~]$ qsub -g grpname use_beeond.sh
Example) Status after execution of use_beeond.sh
[username@es1 ~]$ ls $HOME/test/ foo.txt <- The file remain only when it is copied explicitly in script.
Warning
The file stored under /beeond directory is removed when job finished. The required files need to be moved to Home area or Group area in job script using cp command.
[Advanced Option] Configure BeeOND Servers
A BeeOND filesystem partition consists of two kinds of services running on compute nodes: one is storage service which stores files, and the other is metadata service which stores file matadata. Each service runs on compute nodes. We refer to a compute node which runs storage service as a storage server and a compute node which runs metadata service as a metadata server. Users can specify number of storage servers and metadata servers.
The default values for counts of metadata server and storage server are as follows.
| Parameter | Default |
|---|---|
| Count of metadata servers | 1 |
| Count of storage servers | Number of nodes requested by a job |
To change the counts, define following environment variables. These environment variables have to be defined at job submission, and changing in job script takes no effect. When counts of servers are less than the number of requested nodes, servers are lexicographically selected by their names from assigned compute nodes.
| Environment Variable | Description |
|---|---|
| BEEOND_METADATA_SERVER | Count of metadata servers in integer |
| BEEOND_STORAGE_SERVER | Count of storage servers in integer |
The following example create a BeeOND partition with two metadata servers and four storage servers. beegfs-df is used to see the configuration.
Example) sample of job script (use_beeond.sh)
#!/bin/bash #$-l rt_F=8 #$-l USE_BEEOND=1 #$-v BEEOND_METADATA_SERVER=2 #$-v BEEOND_STORAGE_SERVER=4 #$-cwd beegfs-df -p /beeond
Example output
METADATA SERVERS: TargetID Cap. Pool Total Free % ITotal IFree % ======== ========= ===== ==== = ====== ===== = 1 normal 1489.7GiB 1322.7GiB 89% 149.0M 149.0M 100% 2 normal 1489.7GiB 1322.7GiB 89% 149.0M 149.0M 100% STORAGE TARGETS: TargetID Cap. Pool Total Free % ITotal IFree % ======== ========= ===== ==== = ====== ===== = 1 normal 1489.7GiB 1322.7GiB 89% 149.0M 149.0M 100% 2 normal 1489.7GiB 1322.7GiB 89% 149.0M 149.0M 100% 3 normal 1489.7GiB 1322.7GiB 89% 149.0M 149.0M 100% 4 normal 1489.7GiB 1322.7GiB 89% 149.0M 149.0M 100%
[Advanced Option] File Striping
Files are split into small chunks and stored in multiple storage servers in a BeeOND partition. Users can change file striping configuration of BeeOND.
The default configuration of the file striping is as follows.
| Parameter | Default | Description |
|---|---|---|
| Stripe size | 512 KB | File chunk size |
| Stripe count | 4 | Number of storage servers that store chunks of a file |
Users can configure file striping per-file or per-directory using beegfs-ctl command.
The following example sets file striping configuration of /beeond/data directory as 8 stripe count and 4MB stripe size.
Example) sample of job script (use_beeond.sh)
#!/bin/bash #$-l rt_F=8 #$-l USE_BEEOND=1 #$-cwd BEEOND_DIR=/beeond/data mkdir ${BEEOND_DIR} beegfs-ctl --setpattern --numtargets=8 --chunksize=4M --mount=/beeond ${BEEOND_DIR} beegfs-ctl --mount=/beeond --getentryinfo ${BEEOND_DIR}
Output example
New chunksize: 4194304 New number of storage targets: 8 EntryID: 0-5D36F5EC-1 Metadata node: gXXXX.abci.local [ID: 1] Stripe pattern details: + Type: RAID0 + Chunksize: 4M + Number of storage targets: desired: 8 + Storage Pool: 1 (Default)