How to contribute

  1. Contributing as a user

    1. Reporting bugs

    2. Suggesting improvements

  2. Code contributions

    1. Development and testing

      1. Setup and unit tests

      2. Test MLflow project

      3. Compute backend

      4. Docker Image

      5. Test CI locally

      6. Integration tests with UNICORE

      7. Create local runs

    2. Merge Requests

    3. Changelog

    4. Semantic versioning and tags

    5. Add or update documentation pages

  3. Contributors

Contributing as a user

Reporting bugs

This section guides you through submitting a bug report for mantik. Following these guidelines helps understand your report, reproduce the behavior, and find related reports.

Notes:

  • Before submitting a bug report, check that your issue does not already exist in the issue tracker.

  • Make sure your issue is really a bug, and is not a support request.

  • If you find a closed issue that seems like it is the same thing that you’re experiencing, open a new issue and include a link to the original issue in the body of your new one.

Submit your bugs in the issue tracker:

  1. Use the bug issue template. This can be selected from the dropdown menu in the Description field.

  2. Completely fill out the template.

  3. Save the issue.

Suggesting improvements

You may submit enhancement suggestion such as new features, improvements of existing functionality, or performance by submitting a ticket. Before submitting a new suggestion, check that your issue does not already exist in the issue tracker.

Code contributions

Development and testing

We use poetry (poetry>=1.2.0) for dependency management and packaging. Thus, an installation of poetry is required. A guide on how to install poetry can be found here.

The repo contains several different components (libraries):

  • mantik: Client for the Mantik API.

  • tokens: Verification of JWTs issued by the Mantik API.

  • mantik-compute-backend: Mantik Compute Backend and Fast API app.

The components have dependencies on one another as follows:

       |----> mantik <----|
       |         ^        |
       |         |        |
       |       tokens     |
       |         ^        |
       |         |        |
         compute backend

Setup and unit tests

  1. Install all project dependencies for development

    make install
    
  2. Install pre-commit hooks

    poetry run pre-commit install
    
  3. Run the tests

    poetry run pytest tests
    

Updating the poetry.lock file

To update the dependencies, run

make lock

in order to prevent updating of packages when locking use

make lock-no-update

Bumping the package version

All packages maintained in this repository are versioned equally. To simultaneously increase the version of all packages run

make bump-version version=<version>

where version should be without a v prefix, e.g. make bump-version=1.1.0

Bumping the MLflow version

mantik_compute_backend, mlflow_tracking_server, and mantik[mlflow] use the same MLflow version. That version can simultaneously be bumped using

make bump-mlflow-version version=<version>

where version should be the desired MLflow version.

This commands also runs updating of the lock files.

Test MLflow project

A minimal project is located at tests/resources/test-project. It simply prints the input to stdout.

Testing the project:

  1. Build the Docker image

    docker build -t mantik-test:latest -f tests/resources/test-project/Dockerfile .
    
  2. Build the Apptainer image

    sudo apptainer build \
      tests/resources/test-project/mantik-test.sif \
      tests/resources/test-project/recipe.def
    
  3. Run the project locally

    poetry run mlflow run tests/resources/test-project
    

    This should print test to stdout. The output can be changed by passing the print argument

    poetry run mlflow run tests/resources/test-project -P print="hello world"
    
  4. In tests/resources/test-project/compute-backend-config.json set the Apptainer.Path value to the local path of the Image.

  5. Set the required environment variables (see backend config section above).

  6. Run the project via the UnicoreBackend

    poetry run tests/resources/test-project/run.py
    

Note: When running the example via mlflow CLI, asynchronous execution is triggered. No job results can then be accessed.

Warning: When using a local Apptainer image, the image will be compressed and send to the backend. Here, during decompression a so far unspecified error may occur that crashes the submission (see this ticket). Please, use a remote image and remove any images from your project folder.

Compute backend

The compute backend API is a module located in mantik-compute-backend and depends on mantik. Its endpoint is /submit/<experiment_id>. Mlflow projects can be submitted via this endpoint.

The endpoint expects a zipped Mlflow project directory. Unicore backend configuration should reside inside the zipped directory, its name can be specified with the compute_backend_config_path http form, alongside UNICORE credentials, experiment id and mlflow parameters as serialized json.

Docker Image

A Dockerfile for the compute backend service is provided. To build the image, run

sudo docker build -t compute_backend_service:latest -f docker/mantik-compute-backend.Dockerfile .

To run the image locally, use

sudo docker run --env MLFLOW_TRACKING_URI=<tracking uri> --env LOGLEVEL=<desired log-level> --env LISTEN_PORT=<listen port> -p <host port>:<listen port> compute_backend_service:latest

The environment variable MLFLOW_TRACKING_URI must be provided, LOGLEVEL defaults to INFO and LISTEN_PORT to 80.

Test CI locally

Install gitlab-runner (see this example) on your machine. Usually, you you would need to register your gitlab-ci tokens, which is however not required as we want to run everything locally.

Therefore just executing the integration test in the ci can be done with a single command

gitlab-runner exec docker "<job name>"

Integration tests with UNICORE

For integration testing using a real UNICORE instance, we make use of the UNICORE Docker image provided by JSC. For more details see here.

To run the tests locally:

  1. Run UNICORE locally

    make up-integration
    
  2. Run the integration tests

    poetry run pytest tests/integration
    

Note: The UNICORE Docker container takes a while to process a job, hence tests can be rather slow (15-30 seconds). Thus, be careful with test cases.

Create local runs

For certain e2e tests or during development it might be necessary to create pre-existing runs via the CLI. These runs can be executed on your local machine. For this you need a mantik account that has management access to a project. This project needs to have an experiment, code repository and data repository configured. You can do all this via the mantik GUI. Furthermore you need to provide the URLs for the APIs of the mantik and mlflow endpoints that you want to test.

There already exist a tutorials repository that you can use for both data and code when configuring your setup via the GUI: https://gitlab.com/mantik-ai/tutorials

  1. Export the run-specific environment variables (find the UUIDs via the GUI)

    export MANTIK_USERNAME=<ACCOUNT_NAME>
    export MANTIK_PASSWORD=<ACCOUNT_PASSWORD>
    
    export MANTIK_PROJECT_ID=<PROJECT_UUID>
    export MANTIK_EXPERIMENT_REPOSITORY_ID=<EXPERIMENT_UUID>
    export MANTIK_CODE_REPOSITORY_ID=<CODE_REPOSITORY_UUID>
    export MANTIK_DATA_REPOSITORY_ID=<DATA_REPOSITORY_UUID>
    
    # IMPORTANT change these if you want to test a backend other than production, in this case it's dev2:
    export MANTIK_API_URL="https://api.dev2.cloud.mantik.ai"
    export MLFLOW_TRACKING_URI="https://api.dev2.cloud.mantik.ai/tracking"
    
  2. Initialize mantik token with your exported credentials

    eval $(mantik init)
    
  3. Start a local run (here we use the example wine-estimator tutorial project)

    mantik runs local "./wine-quality-estimator/mlproject/MLProject" \
      --name "Local: wine quality estimator" \
      --project-id $MANTIK_PROJECT_ID \
      --code-repository-id $MANTIK_CODE_REPOSITORY_ID \
      --branch main \
      --experiment-repository-id $MANTIK_EXPERIMENT_REPOSITORY_ID \
      --parameter alpha=0.1
    

Now the run should be displayed in the GUI under your project’s runs/submissions.

Merge requests

  • Describe your changes as accurately as possible. The merge request body should be kept up to date as it will usually form the base for the changelog entry.

  • Be sure that your pull request contains tests that cover the changed or added code. Tests are generally required for code be to be considered mergable, and code without passing tests will not be merged. Ideally, the test coverage should not decrease.

  • Ensure your merge request passes the pre-commit checks. Remember that you can run these tools locally.

  • If your changes warrant a documentation change, the merge request must also update the documentation.

Note: Make sure your branch is rebased against the latest base branch.

Changelog

The file CHANGELOG lists changes that contribute to each version of mantik. In the Unreleased section, changes that will be added in the next version are collected. Update the CHANGELOG with every merge request.

Semantic versioning and tags

Versioning should only be done by code maintainers.

The mantik python package is deployed to PyPI via our CI. The deployment is triggered by setting a tag on the repository. We use semantic versioning. The following steps are necessary for a version update:

  • Update CHANGELOG

  • Update version in pyproject.toml

  • Merge changes

  • Set a tag with the new version

Add or update documentation pages

All the documentation content is stored as .md files in the docs/source folder. This folder contains the structured content for the site, organized into subfolders based on topics.

To add new page:

Page is a part of the existing navigation group:

  1. Create a new .md file in the relevant folder based on the desired structure.

  2. Open the index.md file in that folder, where the toctree is defined. Add a reference to your new file. Example:

```{toctree}
   :titlesonly:

   new-page.md
```
  1. Every page should have one (and only one) first level heading otherwise the page content toc is not correctly displayed.

Page in a new navigation group:

  1. Create new folder in the docs/source directory.

  2. Create index.md file in this folder. The file will be the main entry of the navigation group and contain sub-menu.

  3. Create a new .md file for the sub-menu.

  4. Add a reference to your new file in the index.md in the current folder. Example:

// docs/source/new-topic/index.md

```{toctree}
   :titlesonly:

   new-page.md
```
  1. To make new navigation group appear in the left-side navigation add new toctree entry to the docs/source/index.md

All documents listed in toctree code block will appear as part of this menu group in the navigation. Make sure to reference the file path correctly in the index.md toctree for it to appear in the left-side navigation. Path to the file is relative to index.md. The navigation link takes the first title of the file as a name.

Note: For additional directives check sphinx documentation.

Types of content

Sub-menu:

```{toctree}

   sub-topic/menu-item1.md
   sub-topic/menu-item2.md
```

Assets required in the documentation like videos or pictures will be added to the folder docs/source/_static. The file can than be referenced via relative path as listed below. The video tag embeds the video docs/source/_static/mantik-api.mp4 from a document inside a subfolder of the docs/source e.g. docs/source/topic/chapter.md

Video:

Image:

Code:

def greet(name):
    print(f"Hello, {name}!")

Preview your changes

cd docs
make install // (only ones)
make html
python -m http.server --directory build/html

Open http://localhost:8000 in a browser

Contributors

Omar Ahmed, Elia Boscaini, Fabian Emmerich, Kristian Ehlert, Sara Grau, Britta Oberlack, Thomas Seidler, Jakub Jagielski, Nastassia Kundas, Jakob Fuhrmann, Kim Lyons