2. Transfer Large FilesΒΆ
This section covers how to set up a code repository and how to transfer large files to a remote compute resource. Mantik only supports the transfer of large files for UNICORE. If you are using firecREST, please consult the CSCS tutorial on Data Transfer available here.
Note
For this section you need to have git and apptainer installed locally.
Installation links:
2.1. Clone the Wine Quality Executor Tutorial RepositoryΒΆ
Now that you created a reference to our experiment repository, you need to clone the GitLab repository locally, since some build artifacts (the apptainer image) have to be directly uploaded to the remote infrastructure (JUWELS) with the Mantik Remote File Service, before submitting the run.
On your laptop, clone this repository:
git clone https://gitlab.com/mantik-ai/tutorials.git
cd tutorials/wine-quality-estimator/
Now you have our tutorials repository on your laptop, and you are inside the wine-quality-estimator
project.
2.2. Build Apptainer ImageΒΆ
For the details and instructions check out Apptainer User Guide.
This repository contains a recipe.def
file, which is a recipe to create an apptainer image,
which contains the environment in which you are going to execute the training script on JUWELS.
In order to build the apptainer image run the following command:
apptainer build mlproject/wine-quality-executor.sif mlproject/recipe.def
2.3. Transfer Image to JSCΒΆ
In order to transfer the created image directly to JUWELS you will use the Mantik CLI.
Note
Make sure that python is installed on your laptop (otherwise click here).
Note
We suggest to run the following command in a python environment
using something such as venv
or virtualenv
or conda
, since mantik has many dependencies.
Install the Mantik python client and CLI:
pip install mantik
Now you need transfer the apptainer image from local file to the remote HPC cluster, since apptainer images are usually large files. Large files are transferred with the Mantik Remote File Service directly to the remote instance to save traffic and time in the upload.
Note
There is a limit to the amount of data that can be transferred when submitting a job, so it is good practice to move all large files (apptainer images, training data) first with the Mantik Remote File Service CLI.
As a first step you need to initialize the Mantik client by providing your credentials:
export MANTIK_USERNAME=<your mantik username>
export MANTIK_PASSWORD=<your mantik password>
mantik init
2.3.1. Get Connection ID and Project IDΒΆ
In order to transfer the files you need the UUID of your Connection to JULICH.
To retrieve it navigate again to your Connections in the Settings
,
where you can click on
to view the details of the Connection.
How to get the Project ID:
go to
My Project
,click on
View details
go to
Settings
click on
General
. There you will see the Project ID that you can copy for use below.
Now you can transfer the file:
export MANTIK_PROJECT_ID=<UUID of the project you created by following this tutorial>
export MANTIK_UNICORE_AUTH_SERVER_URL="https://uftp.fz-juelich.de:9112/UFTP_Auth/rest/auth/JUDAC"
mantik unicore-file-service copy-file --connection-id="<Connection ID>" mlproject/wine-quality-executor.sif remote:<IMAGE PATH ON JUWELS>/wine-quality-executor.sif
The output will be
INFO:mantik.utils.mantik_api.client:A new data repository with ID: <DATA REPOSITORY ID> and name: <DEFAULT: REMOTE PATH> at <REMOTE LOCATION> has been created
File(path=<REMOTE PATH>, metadata=FileMeta(owner=<UNICORE USERNAME>, mode='Unknown', last_changed=<CURRENT TIME>, is_remote=True)
Note
In the above example we assume your current directory to be tutorials/wine-quality-estimator.
2.4. Check created Data RepositoryΒΆ
The previous command not only transferred a file to the remote location, it also created a reference to it on the Mantik platform.
If you navigate to Data
on the side menu, you should see a new Data Repository entry as a reference to the transferred file.