Containerizing Applications

We recommend to handle dependencies with Apptainer containers when running your code on remote infrastructure with Mantik.

This tutorial gives a brief instruction on working with containers, especially Docker and Apptainer (formerly known as Singularity) containers as well as their relation to one another.

Apptainer and Docker are both container runtime platforms that implement the OCI standard. While Docker is the most used platform, Apptainer has been developed explicitly for HPC usage.

One is however more likely to find suitable Docker images, e.g. on DockerHub or Dockerfiles in example projects than to find corresponding Apptainer images. In this tutorial we show how Apptainer images can be built from Docker images as well as from Apptainer definition files.

For further reference, see the Apptainer documentation.

Note

This tutorial is mostly also valid for Singularity. The original Singularity project was renamed to Apptainer (see their announcement).

To clear up the confusion between Singularity and Apptainer, it should be mentioned that there exist different Singularity projects:

  • Singularity: the original project.

  • Singularity or SingularityCE: a fork of the original project that is maintained by Sylabs which has diverged from the original concept.

  • Apptainer: the original project, but renamed to “Apptainer” to resolve the confusion between the original project and its fork; basically Singularity 3.9 onwards.

Apptainer 1.2 and Singularity are mostly compatible with each other. Simply replacing apptainer with singularity in the commands below should work when using Singularity.

For more information on the transition from Singularity to Apptainer see here.

Option 1: Build from local Docker image

In case, you have a Docker image available locally, or you executed

sudo docker pull <image>:<tag>

Apptainer images can be built from images that can be accessed via your Docker daemon:

sudo apptainer build <image_name.sif> docker-daemon://<docker-image-name:image-tag>

This option is especially useful if you develop your own Docker images.

Option 2: Build from remote Docker image

It is also possible to build Apptainer images directly from remote Docker images:

sudo apptainer build <image_name.sif> docker://<docker-image-name:image-tag>

Option 3: Building from definition files

The most flexible option to build Apptainer images is to write definition files (also called recipes).

As recipes are well-documented already, we will only cover the bare necessities for building from Docker images.

Headers and sections

Apptainer recipes are text files, commonly named recipe.def. They contain a header, i.e. the first lines of the file and sections. Note, the header is not part of any section.

Sections

Here, only the two most frequently used sections are described briefly. For detailed documentation, see here.

%files

The files section is used to copy files into the image. Paths are relative to the path from which apptainer build is executed.

Attention

Do not copy to user directories (e.g. /home). When running the image, the user directories are automatically mounted. It can lead to confusion between mounted and copied files. Good places to copy your files to are /opt, /app or /project.

%post

The post section contains steps to execute during the build. It is useful for installing additional software.

Example

An example recipe is given below:

Bootstrap: docker
From: python:3.7

%files
  README.md /opt/

%post
  pip install mantik

Known problems

Some options provided by Docker cannot be mapped to the Apptainer logic. For a detailed explanation see here.