Remote File Service

The Mantik Remote File Service is designed to help you manage data. With this service, you can easily perform tasks such as transferring files or directories from your local machine to a remote storage device. Likewise, it enables you to perform actions such as creating or deleting files and directories across remote file systems.

Mantik supports a Remote File Service for data management with S3 and UNICORE. The following set up is needed when using the Mantik Remote File Service:

First, you need to install the required s3 extra dependencies via

pip install "mantik[s3]"

Then

  1. Set all environment variables required by Mantik. You can find more information about this here

  2. Set your AWS username and password (of the IAM user which has access rights to a given S3 bucket) using the following environment variables:

  • AWS_ACCESS_KEY_ID

  • AWS_SECRET_ACCESS_KEY

The Mantik Remote File Service supports standard File Service operations such as creating, copying, removing files and directories. For both Remote File Services the same operations are supported. In the following we walk through some of the most common operations.

To run any of the Mantik Remote File Service operations, the command’s format is mantik s3-file-service <operation command> --connection-id. Note that to specify that a path is remote, start the path with s3://.

To copy a file from a local machine to the remote, the source path to the local file must be indicated first, followed by the target path and the file name in the remote.

mantik s3-file-service copy-file /path/local.file s3://bucket-name/local.file

Analogously a directory can be copied from a local machine to a remote by specifying the source and target paths.

mantik s3-file-service copy-directory /path/localdir s3://bucket-name/localdir

To remove a file or a directory on the bucket, specify the file’s or directories target path and execute:

mantik s3-file-service remove-file s3://bucket-name/local.file
mantik s3-file-service remove-directory s3://bucket-name/remotedir

Note that copy operation from a remote source to a remote target are also supported:

mantik s3-file-service copy-file s3://bucket-name/local1.file s3://bucket-name/local2.file
mantik s3-file-service copy-directory s3://bucket-name/remotedir1 s3://bucket-name/remotedir2

Instead of setting the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, you can provide a connection-id every time you run a file service command. This will retrieve your bucket credentials from the Mantik API. See Connections for more information on how to create a connection.

mantik s3-file-service copy-file --connection-id <a connection id> /path/local.file s3://bucket-name/local.file

To run any of the Mantik Remote File Service operations, first initialise the data client with the remote backend, and to be able to interact with the Client instance.

import mantik
s3_data_client = mantik.remote_file_service.data_client.DataClient.from_env(
    connection_id=<a connection id>,
    remote_fs_type=mantik.remote_file_service.s3_file_service.S3FileService
)

Note that to specify that a path is remote, start the path with s3://.

To copy a file from a local machine to the remote, the source path to the local file must be indicated first, followed by the target path and the file name in the remote.

s3_data_client.copy_file(source="/bucket-name/local.file", target="s3://bucket-name/local.file")

Analogously a directory can be copied from a local machine to a remote by specifying the source and target paths.

s3_data_client.copy_directory(source="/bucket-name/dir", target="s3://bucket-name/dir")

To remove a file or a directory in the remote execute:

s3_data_client.remove_file(target="s3://bucket-name/local.file")
s3_data_client.remove_directory(target="s3://bucket-name/dir")

Note that copy operations on from a remote source to a remote target are also supported:

s3_data_client.copy_file(source="s3://bucket-name/remote1.file", target="s3://bucket-name/remote2.file")
s3_data_client.copy_directory(source="s3://bucket-name/remotedir1", target="s3://bucket-name/remotedir2")

Instead of setting the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, you can provide a connection-id every time you run a File Service function. This will retrieve your credentials from the Mantik API. See Connections for more information on how to create a connection.

s3_data_client.copy_file(source="/bucket-name/local.file", target="s3://bucket-name/local.file", connection_id=<a connection id>)

Note that the default local base path is your current working directory. To change this modify accordingly the environment variable MANTIK_LOCAL_FS_BASE_PATH. The S3 Remote File Service only accepts absolute remote paths.

First, you need to install the required unicore extra dependencies via

pip install "mantik[unicore]"

Then

  1. Set all environment variables required by Mantik. You can find more information about this here

  2. Set your UNICORE username and password using the following environment variables:

  • MANTIK_UNICORE_USERNAME

  • MANTIK_UNICORE_PASSWORD

  1. Set the UNICORE authentication URL to the environment variable MANTIK_UNICORE_AUTH_SERVER_URL. The UNICORE File Service it is based on UFTP, therefore this refers to the UFTP authentication URL of your HPC. In case you want to transfer files to JSC the URL is https://uftp.fz-juelich.de:9112/UFTP_Auth/rest/auth/JUDAC

The Mantik Remote File Service supports standard File Service operations such as creating, copying, removing files and directories. For both Remote File Services the same operations are supported. In the following we walk through some of the most common operations.

To run any of the Mantik Remote File Service operations, the command’s format is mantik unicore-file-service <operation command> --connection-id. Note that to specify that a path is remote, start the path with remote:

To copy a file from a local machine to the remote, the source path to the local file must be indicated first, followed by the target path and the file name in the remote.

mantik unicore-file-service copy-file /path/local.file remote:path/remote.file

Analogously a directory can be copied from a local machine to a remote by specifying the source and target paths.

mantik unicore-file-service copy-directory /path/localdir remote:path/remotedir

To remove a file or a directory on the bucket, specify the file’s or directories target path and execute:

mantik unicore-file-service remove-file remote:path/remote.file
mantik unicore-file-service remove-directory remote:path/remotedir

Note that copy operations on from a remote source to a remote target are also supported:

mantik unicore-file-service copy-file remote:path/local1.file remote:path/local2.file
mantik unicore-file-service copy-directory remote:path/remotedir1 remote:path/remotedir2

Instead of setting the environment variables MANTIK_UNICORE_USERNAME and MANTIK_UNICORE_PASSWORD, you can provide a connection-id every time you run a File Service command. See Connections for more information on how to create a connection.

mantik unicore-file-service copy-file --connection-id <a connection id> /path/local.file remote:path/remote.file

To run any of the Mantik Remote File Service operations, first initialise the data client with the remote backend, and to be able to interact with the client instance. For both Remote File Service the same operations are supported.

import mantik
unicore_data_client = mantik.remote_file_service.data_client.DataClient.from_env(
    connection_id=...,  # optionally fill in your connection id
    remote_fs_type=mantik.remote_file_service.unicore_file_service.UnicoreFileService,
)

Note that to specify that a path is remote, start the path with remote: respectively.

To copy a file from a local machine to the remote, the source path to the local file must be indicated first, followed by the target path and the file name in the remote.

unicore_data_client.copy_file(source="/path/local.file", target="remote:path/remote.file")

Analogously a directory can be copied from a local machine to a remote by specifying the source and target paths.

unicore_data_client.copy_directory(source="/path/localdir", target="remote:path/remotedir")

Note that copy operations on from a remote source to a remote target are also supported:

unicore_data_client.copy_file(source="remote:path/remote1.file", target="remote:path/remote2.file")
unicore_data_client.copy_directory(source="remote:path/remotedir1", target="remote:path/remotedir2")

To remove a file or a directory in the remote execute:

unicore_data_client.remove_file(target="remote:path/local.file")
unicore_data_client.remove_directory(target="remote:path/remotedir")

Instead of setting the environment variables MANTIK_UNICORE_USERNAME and MANTIK_UNICORE_PASSWORD, you can provide a connection-id every time you run a File Service function. See Connections for more information on how to create a connection.

unicore_data_client.copy_file(source="/path/local.file", target="remote:path/remote.file", connection_id=<a connection id>)

Note that the default local base path is your current working directory. To change this modify accordingly the environment variable MANTIK_LOCAL_FS_BASE_PATH. The default remote base path is the remote root directory, which can be changed by setting MANTIK_REMOTE_FS_BASE_PATH.

Overview of mantik API calls when using the Remote File Service (Advanced users)

  1. Login into the API by token retrieval (GET /mantik/tokens/create)

  2. If a connection id is provided, the connection credentials for a specific connection id are requested via the API, in order to be able to authenticate the user on the provided connection.(GET /mantik/tokens/create) If no connection id is provided, no API call is made, and instead, the credentials are retrieved from the environment variables.