Skip to content

Using s3fs-fuse

ABCI provides an s3fs-fuse module that allows you to mount your ABCI Cloud Storage bucket as a local file system. This section describes how to use the s3fs-fuse module.

Access Key

An access key is required to use s3fs-fuse.

Please refer to the ABCI Portal Guide for how to issue an access key. After issuing the access key, use the AWS CLI to set the access key. Please refer to How to Use ABCI Cloud Storage for how to set the access key.

Here, it is assumed that the access key is set in the default profile.

Loading module

After logging in to the interactive node, load the s3fs-fuse module. Also load the aws-cli module to create a bucket and so on.

[username@es1 ~]$ module load aws-cli s3fs-fuse

Creating bucket

Create a bucket for mounting.

[username@es1 ~]$ aws --endpoint-url s3 mb s3://s3fs-bucket
make_bucket: s3fs-bucket

Mounting bucket

Create a mount point on the HOME directory and mount the s3fs-bucket bucket with the s3fs command.

[username@es1 ~]$ mkdir s3fs_dir
[username@es1 ~]$ s3fs s3fs-bucket ~/s3fs_dir -o url= -o use_path_request_style

If you want to use an access key other than the default profile, specify the-o profile = profile name option.

[username@es1 ~]$ s3fs s3fs-bucket ~/s3fs_dir -o url= -o profile=aaa00000.2 -o use_path_request_style

File operations

After mounting the bucket, you can add and remove objects from the bucket in the same way as you would with a file.

[username@es1 ~]$ cp ~/my-file ~/s3fs_dir/
[username@es1 ~]$ ls ~/s3fs_dir/my-file
[username@es1 ~]$ rm ~/s3fs_dir/my-file

Unmounting bucket

Use the fusermount command to unmount the bucket.

[username@es1 ~]$ fusermount -u ~/s3fs_dir

If you mount a bucket using s3fs-fuse in a job obtained by the On-demand or Spot service, it will be automatically unmounted at the end of the job. However, if you mount the bucket using s3fs-fuse on the interactive node, it will not be unmounted automatically, so unmount it when you no longer need it.

s3fs command options

The options for the s3fs command are shown below. See the man s3fs or s3fs-fuse website for more information.

Option Description Example
url Endpoint URL used to connect. -o url=
profile Profile name used for authentication. -o profile=aaa00000.2
dbglevel Debug message level. -o dbglevel=info
curldb Enabling libcurl debug messages. -o curldb