Tutorial for Using the uv Package Manager

uv is a Python package manager meant to act as a drop-in replacement for pip (and many more tools). In this project, we use it to manage our packages, similar to how poetry works. This is done using a lockfile called uv.lock.

The reasons for choosing uv over something like poetry are multiple:

uv tries to replace more than just poetry, but also pyenv, pip, virtualenv and more. Thus, we can use a single, unified tool for most of our Python needs, similar to cargo in Rust.
uv is much faster than other package managers, providing a speed boost of 10-100 times compared to pip.

uv as a Drop-In Replacement for pip

uv is a lot faster than pip, so we recommend installing packages from PyPI with uv pip install <package> instead of pip install <package>. Since uv works as a drop-in replacement for pip, you can use this feature to speedup any installation without changing the setup of your project.

uv as a Project-Wide Package Manager

If you wish to use the uv sync and/or uv lock commands, which is how you use uv to manage all your project packages, then note that these commands will only work with the directory called .venv in the project directory. This can occasionally be inconvenient, especially with an existing virtual environment, so we recommend using a symlink. If you need to switch between multiple virtual environments, you can update the symlink to point to the desired one.

Using Symlinks to Manage Multiple venvs

To create a symlink between your venv and the .venv directory, you can use the following command:

ln -s <path/to/your_venv> <path/to/.venv>

As an example, if I am in the itwinai/ folder and my venv is called envAI_juwels, the following will create the desired symlink:

ln -s envAI_juwels .venv

Installing from the uv.lock File

Warning

If uv creates your venv for you, the venv will not have pip installed. However, pip``is required to be able to run the installation scripts for ``Horovod and DeepSpeed, so we have included it as a dependency in the pyproject.toml.

To install from the uv.lock file into the .venv venv, you can do the following:

uv sync

Optional dependencies listed in the uv.lock file (e.g., torch, tf, and dev in itwinai) can be included with the –extra flag as shown below:

uv sync --extra torch --extra dev

These “extras” correspond to the optional dependencies in the pyproject.toml. In particular, if you are a developer you would use one of the following two commands. If you are on HPC with CUDA, you would use:

uv sync --no-cache --extra dev --extra torch --extra tf

If you are developing on your local computer, then you would use:

uv sync --extra torch --extra tf --extra dev

Updating the uv.lock File

To update the project’s uv.lock file with the dependencies of the project, you can use the command:

uv lock

This will create a uv.lock file if it doesn’t already exist, using the dependencies from the pyproject.toml.

Adding New Packages to the Project

To add a new package to the project (i.e. to the pyproject.toml file) with uv, you can use the following command:

uv add <package>

if you want to add a package to a specific optional dependency, then you can add the --optional <name-of-extra> flag:

uv add <package> --optional <extra>

As an example, if I wanted to add numpy as an optional dependency under the dependency group torch, then I would do the following:

uv add numpy --optional torch

This has the advantage that uv will run its dependency solver, meaning it will find suitable version constraints that fit with the other packages in the pyproject.toml.

Warning

This will add the package to your .venv venv, so make sure to have symlinked to this directory if you haven’t already.