2. Transfer Large FilesΒΆ

This section covers how to set up a code repository and how to transfer large files to a remote compute resource. Mantik only supports the transfer of large files for UNICORE. If you are using firecREST, please consult the CSCS tutorial on Data Transfer available here.

Note

For this section you need to have git and apptainer installed locally.

Installation links:

2.1. Clone the Wine Quality Executor Tutorial RepositoryΒΆ

Now that you created a reference to our experiment repository, you need to clone the GitLab repository locally, since some build artifacts (the apptainer image) have to be directly uploaded to the remote infrastructure (JUWELS) with the Mantik Remote File Service, before submitting the run.

On your laptop, clone this repository:

git clone https://gitlab.com/mantik-ai/tutorials.git
cd tutorials/wine-quality-estimator/

Now you have our tutorials repository on your laptop, and you are inside the wine-quality-estimator project.

2.2. Build Apptainer ImageΒΆ

For the details and instructions check out Apptainer User Guide.

This repository contains a recipe.def file, which is a recipe to create an apptainer image, which contains the environment in which you are going to execute the training script on JUWELS.

In order to build the apptainer image run the following command:

apptainer build mlproject/wine-quality-executor.sif mlproject/recipe.def

2.3. Transfer Image to JSCΒΆ

In order to transfer the created image directly to JUWELS you will use the Mantik CLI.

Note

Make sure that python is installed on your laptop (otherwise click here).

Note

We suggest to run the following command in a python environment using something such as venv or virtualenv or conda, since mantik has many dependencies.

Install the Mantik python client and CLI:

pip install mantik

Now you need transfer the apptainer image from local file to the remote HPC cluster, since apptainer images are usually large files. Large files are transferred with the Mantik Remote File Service directly to the remote instance to save traffic and time in the upload.

Note

There is a limit to the amount of data that can be transferred when submitting a job, so it is good practice to move all large files (apptainer images, training data) first with the Mantik Remote File Service CLI.

As a first step you need to initialize the Mantik client by providing your credentials:

export MANTIK_USERNAME=<your mantik username>
export MANTIK_PASSWORD=<your mantik password>
mantik init

2.3.1. Get Connection ID and Project IDΒΆ

In order to transfer the files you need the UUID of your Connection to JULICH. To retrieve it navigate again to your Connections in the Settings, where you can click on eye-icon.svg to view the details of the Connection.

How to get the Project ID:

  • go to My Project,

  • click on View details

  • go to Settings

  • click on General. There you will see the Project ID that you can copy for use below.

Now you can transfer the file:

export MANTIK_PROJECT_ID=<UUID of the project you created by following this tutorial>
export MANTIK_UNICORE_AUTH_SERVER_URL="https://uftp.fz-juelich.de:9112/UFTP_Auth/rest/auth/JUDAC"
mantik unicore-file-service copy-file --connection-id="<Connection ID>" mlproject/wine-quality-executor.sif remote:<IMAGE PATH ON JUWELS>/wine-quality-executor.sif

The output will be

INFO:mantik.utils.mantik_api.client:A new data repository with ID: <DATA REPOSITORY ID> and name: <DEFAULT: REMOTE PATH> at <REMOTE LOCATION> has been created
File(path=<REMOTE PATH>, metadata=FileMeta(owner=<UNICORE USERNAME>, mode='Unknown', last_changed=<CURRENT TIME>, is_remote=True)

Note

In the above example we assume your current directory to be tutorials/wine-quality-estimator.

2.4. Check created Data RepositoryΒΆ

The previous command not only transferred a file to the remote location, it also created a reference to it on the Mantik platform.

If you navigate to Data on the side menu, you should see a new Data Repository entry as a reference to the transferred file.