Getting started with itwinai

In this section, we will run you through the installation and give some instructions for the use of the itwinai framework for HPC and local systems.

User Installation

Requirements:

  • Linux environment. Windows and macOS were never tested.

Python virtual environment

Depending on your environment, there are different ways to select a specific python version.

πŸ’» Laptop or GPU node

If you are working on a laptop or on a simple on-prem setup, you could consider using pyenv. See the installation instructions. If you are using pyenv, make sure to read this.

🌐 HPC environment

In HPC systems it is more popular to load dependencies using Environment Modules or Lmod. Contact the system administrator to learn how to select the proper python modules.

On JSC, we activate the required modules in this way:

ml --force purge
ml Stages/2024 GCC OpenMPI CUDA/12 cuDNN MPI-settings/CUDA
ml Python CMake HDF5 PnetCDF libaio mpi4py

Install itwinai

Install itwinai and its dependencies using the following command, and follow the instructions:

# Create a python virtual environment and activate it
$ python -m venv ENV_NAME
$ source ENV_NAME/bin/activate

# Install itwinai inside the environment
(ENV_NAME) $ export ML_FRAMEWORK="pytorch" # or "tensorflow"
(ENV_NAME) $ curl -fsSL https://github.com/interTwin-eu/itwinai/raw/main/env-files/itwinai-installer.sh | bash

The ML_FRAMEWORK environment variable controls whether you are installing itwinai for PyTorch or TensorFlow.

Warning

itwinai depends on Horovod, which requires CMake>=1.13 and other packages. Make sure to have them installed in your environment before proceeding.

Developer Installation

If you are contributing to this repository, please continue below for more advanced instructions.

Warning

Branch protection rules are applied to all branches which names match this regex: [dm][ea][vi]* . When creating new branches, please avoid using names that match that regex.

Install itwinai environment

Regardless of how you loaded your environment, you can create the python virtual environments with the following commands. Once the correct Python version is loaded, create the virtual environments using our pre-make Makefile:

make torch-env # or make torch-env-cpu
make tensorflow-env # or make tensorflow-env-cpu

# Juelich supercomputer
make torch-gpu-jsc
make tf-gpu-jsc

TensorFlow

Installation:

# Install TensorFlow 2.13
make tensorflow-env

# Activate env
source .venv-tf/bin/activate

A CPU-only version is available at the target tensorflow-env-cpu.

PyTorch (+ Lightning)

Installation:

# Install PyTorch + lightning
make torch-env

# Activate env
source .venv-pytorch/bin/activate

A CPU-only version is available at the target torch-env-cpu.

Development environment

This is for developers only. To have it, update the installed itwinai package adding the dev extra:

pip install -e .[dev]

Test with pytest

Do this only if you are a developer wanting to test your code with pytest.

First, you need to create virtual environments both for torch and tensorflow. For instance, you can use:

make torch-env-cpu
make tensorflow-env-cpu

To select the name of the torch and tf environments you can set the following environment variables, which allow to run the tests in environments with custom names which are different from .venv-pytorch and .venv-tf.

export TORCH_ENV="my_torch_env"
export TF_ENV="my_tf_env"

Functional tests (marked with pytest.mark.functional) will be executed under /tmp/pytest location to guarantee they are run in a clean environment.

To run functional tests use:

pytest -v tests/ -m "functional"

To run all tests on itwinai package:

make test

Run tests in JSC virtual environments:

make test-jsc