Developer Installation
This guide provides step-by-step instructions for installing the itwinai library for
developers.
Cloning the Repository
When cloning the repository, you have to make sure to also clone the submodules. You can do both with the following command:
git clone [--recurse-submodules] git@github.com:interTwin-eu/itwinai.git
Where the --recurse-submodules is an optional flag that allows to pull also git submodules.
It is not generally needed.
Setting up the system dependencies
First of all, before installing itwinai and its Python dependencies let’s make sure that the system dependencies such as CUDA drivers, compilers, and MPI libraries, are correctly set up.
Supported OSs are Linux and macOS.
Warning
On high-performance computing (HPC) systems, you must load the appropriate modules before creating or activating your Python virtual environment to ensure compatibility with system libraries.
On HPC systems, it is common to manage dependencies using Environment Modules or Lmod. These tools allow you to dynamically load and unload software modules, such as compilers, CUDA drivers, and MPI libraries. If you are unsure which modules to load for your application, contact your system administrator or refer to your HPC system’s documentation for specific guidance.
Find below the modules you should load on the supercomputers where we tested itwinai, depending
on whether you want PyTorch or TensorFlow support:
Modules for the JUWELS system at Juelich Supercomputer (JSC):
ml --force purge
ml Stages/2025 GCC OpenMPI CUDA/12 cuDNN MPI-settings/CUDA
ml Python CMake HDF5 PnetCDF libaio mpi4py git
# Now you can create or active the python environment here
ml --force purge
ml Stages/2024 GCC/12.3.0 OpenMPI CUDA/12 MPI-settings/CUDA
ml Python/3.11 HDF5 PnetCDF libaio mpi4py CMake cuDNN/8.9.5.29-CUDA-12
# Now you can create or active the python environment here
Modules for Vega Supercomputer:
ml --force purge
ml CMake/3.29.3-GCCcore-13.3.0
ml mpi4py/3.1.5
ml OpenMPI/4.1.6-GCC-13.2.0
ml cuDNN/8.9.7.29-CUDA-12.3.0
ml CUDA/12.6.0
ml NCCL/2.22.3-GCCcore-13.3.0-CUDA-12.6.0
ml Python/3.12.3-GCCcore-13.3.0
# Now you can create or active the python environment here
ml --force purge
ml CMake/3.29.3-GCCcore-13.3.0
ml mpi4py/3.1.5
ml OpenMPI/4.1.6-GCC-13.2.0
ml cuDNN/8.9.7.29-CUDA-12.3.0
ml CUDA/12.6.0
ml NCCL/2.22.3-GCCcore-13.3.0-CUDA-12.6.0
ml Python/3.12.3-GCCcore-13.3.0
# Now you can create or active the python environment here
On LUMI, Python virtual environments are discouraged in favour of containers. Load the following modules before running commands in your AI containers:
ml --force purge ml LUMI partition/G module use /appl/local/containers/ai-modules module load singularity-AI-bindings
These modules are needed to bind into the container the correct software suite on LUMI. More info can be found here.
After using the commands above to load the modules, check which modules you loaded by running
the ml command in the terminal.
Creating a Python Virtual Environment
The suggested way of managing Python dependencies, including itwinai, is through Python virtual environments. Creating a virtual environment is allows to isolate dependencies and prevent conflicts with other Python projects.
Beware that some HPC centers advise against using Python virtual environments as they create a large amount of files, which can clog some distributed filesystems. In such situation, you should prefer using containers.
To manage python virtual environments we use UV, which can be installed from this page. Learn more on UV package manager from our UV tutorial
If you don’t already have a virtual environment, you can create one with the following command:
# Remember to load the software modules first (see section above)!
uv venv
# Alternatively to the command above, if you just want to use plain pip instead of UV
python -m venv .venv
Notice that a new directory called .venv is created to contain your virtual
environment. Now, you can start your virtual environment with the following command:
# Remember to load the software modules first (see section above)!
source .venv/bin/activate
Installing the itwinai Library as a Developer
In this project, we use uv as a project-wide package manager. Therefore, we suggest
that you skim through the uv tutorial before
continuing this tutorial.
Optional Dependencies (extras)
The itwinai library has numerous optional dependencies that can be activated when
installing with pip through extras:
dev: for developers, including libraries for running tests etc.torch: for installation with PyTorch support.tf: for installation with TensorFlow support.tf-cuda: for installation with TensorFlow support with GPU acceleration enabled.docs: for installation of packages required to build the docs locally.hpo: for installation of packages used for hyperparameter optimization (HPO).
You can at any point in time find (or update) the optional dependencies in the
pyproject.toml file in the root of the repository.
Installing the itwinai Library
As a developer, you will also install the library using pip (or uv pip if you
wish), but the main difference is that you need to install it as editable using the
-e flag. Another difference is that you also need the dev extra.
Note
When installing on HPC, it is sometimes an advantage to use the --no-cache-dir
option to avoid filling up your ~/.cache directory. Filling up this directory
will often lead to you use up your disk quota, especially in terms of inodes.
Below you can find complete commands for installation, depending on if you are installing
itwinai with or without GPU (CUDA) support and locally or on HPC:
uv sync --extra torch --extra dev
# Or alternatively, using pip
uv pip install -e ".[torch,dev]"
uv sync --extra torch --extra dev
# Or alternatively, using pip
uv pip install -e ".[torch,dev]" \
--extra-index-url https://download.pytorch.org/whl/cu121
Note: This is the same as Local (CUDA) but without using the cache directory.
uv sync --extra torch --extra dev
# Or alternatively, using pip
uv pip install -e ".[torch,dev]" \
--no-cache-dir \
--extra-index-url https://download.pytorch.org/whl/cu121
Note
If you want to use the Prov4ML logger, you need to install it explicitly since it is only available on GitHub:
For systems with Nvidia GPUs:
uv pip install "prov4ml[nvidia]@git+https://github.com/matbun/ProvML@v0.0.1"
For macOS:
uv pip install "prov4ml[apple]@git+https://github.com/matbun/ProvML@v0.0.1"
Installing Horovod and Microsoft DeepSpeed
If you also want to install Horovod and Microsoft DeepSpeed for distributed ML with
PyTorch, then make sure to install them after itwinai. You can choose if you
want to do this with or without GPU (CUDA) support:
uv pip install --no-cache-dir --no-build-isolation git+https://github.com/horovod/horovod.git
uv pip install --no-cache-dir --no-build-isolation deepspeed
curl -fsSL https://github.com/interTwin-eu/itwinai/raw/main/env-files/torch/install-horovod-deepspeed-cuda.sh | bash
Warning
Horovod requires CMake>=1.13 and
other packages
Make sure to have them installed in your environment before proceeding.
Warning
The installation of Horovod and DeepSpeed needs to be executed on a machine/node where GPUs are available. On some HPC systems, such as the JUWELS system on JSC, GPUs are not available on login nodes (the host you connect to when you SSH into the system), only on compute nodes. On the JUWELS system, run this command to install DeepSpeed and Horovod directly from the repository’s root:
curl -fsSL https://github.com/interTwin-eu/itwinai/raw/main/env-files/torch/horovod-deepspeed-JSC.slurm | sbatch