Containerizing Applications¶
We recommend to handle dependencies with Apptainer containers when running your code on remote infrastructure with Mantik.
This tutorial gives a brief instruction on working with containers, especially Docker and Apptainer (formerly known as Singularity) containers as well as their relation to one another.
Apptainer and Docker are both container runtime platforms that implement the OCI standard. While Docker is the most used platform, Apptainer has been developed explicitly for HPC usage.
One is however more likely to find suitable Docker images, e.g. on DockerHub or Dockerfiles in example projects than to find corresponding Apptainer images. In this tutorial we show how Apptainer images can be built from Docker images as well as from Apptainer definition files.
For further reference, see the Apptainer documentation.
Note
This tutorial is mostly also valid for Singularity. The original Singularity project was renamed to Apptainer (see their announcement).
To clear up the confusion between Singularity and Apptainer, it should be mentioned that there exist different Singularity projects:
Singularity: the original project.
Singularity or SingularityCE: a fork of the original project that is maintained by Sylabs which has diverged from the original concept.
Apptainer: the original project, but renamed to “Apptainer” to resolve the confusion between the original project and its fork; basically Singularity 3.9 onwards.
Apptainer 1.2 and Singularity are mostly compatible with each other.
Simply replacing apptainer
with singularity
in the commands below should work when using Singularity.
For more information on the transition from Singularity to Apptainer see here.
Option 1: Build from local Docker image¶
In case, you have a Docker image available locally, or you executed
sudo docker pull <image>:<tag>
Apptainer images can be built from images that can be accessed via your Docker daemon:
sudo apptainer build <image_name.sif> docker-daemon://<docker-image-name:image-tag>
This option is especially useful if you develop your own Docker images.
Option 2: Build from remote Docker image¶
It is also possible to build Apptainer images directly from remote Docker images:
sudo apptainer build <image_name.sif> docker://<docker-image-name:image-tag>
Option 3: Building from definition files¶
The most flexible option to build Apptainer images is to write definition files (also called recipes).
As recipes are well-documented already, we will only cover the bare necessities for building from Docker images.
Headers and sections¶
Apptainer recipes are text files, commonly named recipe.def
. They contain a
header, i.e. the first lines of the file and
sections. Note, the header is not part of any section.
Header¶
The two relevant entries in the header for us are:
Bootstrap
: Defines from which platform the base images are pulled. Options of interest here aredocker
: pull from dockerhubdocker-daemon
: use a local Docker image
From
: Defines the base image to extend.
Bootstrap: docker
From: python:3.7
Bootstrap: docker-daemon
From: python:3.7
Sections¶
Here, only the two most frequently used sections are described briefly. For detailed documentation, see here.
%files
¶
The files
section is used to copy files into the image. Paths are relative
to the path from which apptainer build
is executed.
Attention
Do not copy to user directories (e.g. /home
).
When running the image, the user
directories are automatically mounted. It can lead to confusion between
mounted and copied files. Good places to copy your files to are
/opt
, /app
or /project
.
%post
¶
The post
section contains steps to execute during the build. It is useful for
installing additional software.
Example¶
An example recipe is given below:
Bootstrap: docker
From: python:3.7
%files
README.md /opt/
%post
pip install mantik
Known problems¶
Some options provided by Docker cannot be mapped to the Apptainer logic. For a detailed explanation see here.