itwinai.loggers

Abstraction layer for loggers.

class itwinai.loggers.LogMixin

Bases: object

abstract log(item: Any | List[Any], identifier: str | List[str], kind: str = 'metric', step: int | None = None, batch_idx: int | None = None, **kwargs) → None

Log item with identifier name of kind type at step time step.

Parameters:

• item (Union[Any, List[Any]]) – element to be logged (e.g., metric).

• identifier (Union[str, List[str]]) – unique identifier for the element to log(e.g., name of a metric).

• kind (str, optional) – type of the item to be logged. Must be one among the list of self.supported_types. Defaults to ‘metric’.

• step (Optional[int], optional) – logging step. Defaults to None.

• batch_idx (Optional[int], optional) – DataLoader batch counter (i.e., batch idx), if available. Defaults to None.

class itwinai.loggers.Logger(savedir: str = 'mllogs', log_freq: int | Literal['epoch', 'batch'] = 'epoch')

Bases: LogMixin

Base class for logger

Parameters:

• savedir (str, optional) – filesystem location where logs are stored. Defaults to ‘mllogs’.

• log_freq (Union[int, Literal['epoch', 'batch']], optional) –

  how often should the logger fulfill calls to the log() method:

  • When set to ‘epoch’, the logger logs only if batch_idx is not passed to the log method.

  • When an integer is given, the logger logs if batch_idx is a multiple of log_freq.

  • When set to 'batch', the logger logs always.

  Defaults to ‘epoch’.

supported_types: List[str]

Supported logging ‘kind’s.

savedir: str = None

Location on filesystem where to store data.

property log_freq: int | Literal['epoch', 'batch']

Get log_feq, namely how often should the logger fulfill or ignore calls to the log() method.

start_logging()

Start logging context.

Example:

>>> with my_logger.start_logging():
>>>     my_logger.log(123, 'value', kind='metric', step=0)

abstract create_logger_context()

Initialize logger.

abstract destroy_logger_context()

Destroy logger.

abstract save_hyperparameters(params: Dict[str, Any]) → None

Save hyperparameters.

Parameters:

params (Dict[str, Any]) – hyperparameters dictionary.

serialize(obj: Any, identifier: str) → str

Serializes object to disk and returns its path.

Parameters:

• obj (Any) – item to save.

• identifier (str) – identifier of the item to log (expected to be a path under self.savedir).

Returns:

local path of the serialized object to be logged.

Return type:

str

should_log(batch_idx: int | None = None) → bool

Determines whether the logger should fulfill or ignore calls to the log() method, depending on the log_freq property:

• When log_freq is set to ‘epoch’, the logger logs only if batch_idx is not passed to the log method.

• When log_freq is an integer is given, the logger logs if batch_idx is a multiple of log_freq.

• When log_freq is set to 'batch', the logger logs always.

Parameters:

batch_idx (Optional[int]) – the dataloader batch idx, if available. Defaults to None.

Returns:

True if the logger should log, False otherwise.

Return type:

bool

class itwinai.loggers.ConsoleLogger(savedir: str = 'mllogs', log_freq: int | Literal['epoch', 'batch'] = 'epoch')

Bases: Logger

Simplified logger.

Parameters:

• savedir (str, optional) – where to store artifacts. Defaults to ‘mllogs’.

• log_freq (Union[int, Literal['epoch', 'batch']], optional) – determines whether the logger should fulfill or ignore calls to the log() method. See Logger.should_log method for more details. Defaults to ‘epoch’.

create_logger_context()

Initialize logger.

destroy_logger_context()

Destroy logger. Do nothing.

save_hyperparameters(params: Dict[str, Any]) → None

Save hyperparameters. Do nothing.

Parameters:

params (Dict[str, Any]) – hyperparameters dictionary.

log(item: Any | List[Any], identifier: str | List[str], kind: str = 'metric', step: int | None = None, batch_idx: int | None = None, **kwargs) → None

Print metrics to stdout and save artifacts to the filesystem.

Parameters:

• item (Union[Any, List[Any]]) – element to be logged (e.g., metric).

• identifier (Union[str, List[str]]) – unique identifier for the element to log(e.g., name of a metric).

• kind (str, optional) – type of the item to be logged. Must be one among the list of self.supported_types. Defaults to ‘metric’.

• step (Optional[int], optional) – logging step. Defaults to None.

• batch_idx (Optional[int], optional) – DataLoader batch counter (i.e., batch idx), if available. Defaults to None.

• kwargs – keyword arguments to pass to the logger.

class itwinai.loggers.MLFlowLogger(savedir: str = 'mllogs', experiment_name: str = 'default_experiment', tracking_uri: str | None = None, run_description: str | None = None, log_freq: int | Literal['epoch', 'batch'] = 'epoch')

Bases: Logger

Abstraction around MLFlow logger.

Parameters:

• savedir (str, optional) – path on local filesystem where logs are stored. Defaults to ‘mllogs’.

• experiment_name (str, optional) – experiment name. Defaults to itwinai.loggers.BASE_EXP_NAME.

• tracking_uri (Optional[str], optional) – MLFLow tracking URI. Overrides savedir if given. Defaults to None.

• run_description (Optional[str], optional) – run description. Defaults to None.

• log_freq (Union[int, Literal['epoch', 'batch']], optional) – determines whether the logger should fulfill or ignore calls to the log() method. See Logger.should_log method for more details. Defaults to ‘epoch’.

active_run: mlflow.ActiveRun

Current MLFLow experiment’s run.

create_logger_context()

Initialize logger. Start MLFLow run.

destroy_logger_context()

Destroy logger. End current MLFlow run.

save_hyperparameters(params: Dict[str, Any]) → None

Save hyperparameters as MLFlow parameters.

Parameters:

params (Dict[str, Any]) – hyperparameters dictionary.

log(item: Any | List[Any], identifier: str | List[str], kind: str = 'metric', step: int | None = None, batch_idx: int | None = None, **kwargs) → None

Log with MLFlow.

Parameters:

• item (Union[Any, List[Any]]) – element to be logged (e.g., metric).

• identifier (Union[str, List[str]]) – unique identifier for the element to log(e.g., name of a metric).

• kind (str, optional) – type of the item to be logged. Must be one among the list of self.supported_types. Defaults to ‘metric’.

• step (Optional[int], optional) – logging step. Defaults to None.

• batch_idx (Optional[int], optional) – DataLoader batch counter (i.e., batch idx), if available. Defaults to None.

• kwargs – keyword arguments to pass to the logger.

class itwinai.loggers.WandBLogger(savedir: str = 'mllogs', project_name: str = 'default_experiment', log_freq: int | Literal['epoch', 'batch'] = 'epoch')

Bases: Logger

Abstraction around WandB logger.

Parameters:

• savedir (str, optional) – location on local filesystem where logs are stored. Defaults to ‘mllogs’.

• project_name (str, optional) – experiment name. Defaults to itwinai.loggers.BASE_EXP_NAME.

• log_freq (Union[int, Literal['epoch', 'batch']], optional) – determines whether the logger should fulfill or ignore calls to the log() method. See Logger.should_log method for more details. Defaults to ‘epoch’.

create_logger_context()

Initialize logger. Init WandB run.

destroy_logger_context()

Destroy logger.

save_hyperparameters(params: Dict[str, Any]) → None

Save hyperparameters.

Parameters:

params (Dict[str, Any]) – hyperparameters dictionary.

log(item: Any | List[Any], identifier: str | List[str], kind: str = 'metric', step: int | None = None, batch_idx: int | None = None, **kwargs) → None

Log with WandB. Wrapper of https://docs.wandb.ai/ref/python/log

Parameters:

• item (Union[Any, List[Any]]) – element to be logged (e.g., metric).

• identifier (Union[str, List[str]]) – unique identifier for the element to log(e.g., name of a metric).

• kind (str, optional) – type of the item to be logged. Must be one among the list of self.supported_types. Defaults to ‘metric’.

• step (Optional[int], optional) – ignored by WandBLogger.

• batch_idx (Optional[int], optional) – DataLoader batch counter (i.e., batch idx), if available. Defaults to None.

• kwargs – keyword arguments to pass to the logger.

class itwinai.loggers.TensorBoardLogger(savedir: str = 'mllogs', log_freq: int | Literal['epoch', 'batch'] = 'epoch', framework: Literal['tensorflow', 'pytorch'] = 'pytorch')

Bases: Logger

Abstraction around TensorBoard logger, both for PyTorch and TensorFlow.

Parameters:

• savedir (str, optional) – location on local filesystem where logs are stored. Defaults to ‘mllogs’.

• log_freq (Union[int, Literal['epoch', 'batch']], optional) – determines whether the logger should fulfill or ignore calls to the log() method. See Logger.should_log method for more details. Defaults to ‘epoch’.

• framework (Literal['tensorflow', 'pytorch'], optional) – whether to log PyTorch or TensorFlow ML data. Defaults to ‘pytorch’.

Raises:

ValueError – when framework is not recognized.

create_logger_context()

Initialize logger.

destroy_logger_context()

Destroy logger. Close SummaryWriter.

save_hyperparameters(params: Dict[str, Any]) → None

Save hyperparameters.

Parameters:

params (Dict[str, Any]) – hyperparameters dictionary.

log(item: Any | List[Any], identifier: str | List[str], kind: str = 'metric', step: int | None = None, batch_idx: int | None = None, **kwargs) → None

Log with Tensorboard.

Parameters:

• item (Union[Any, List[Any]]) – element to be logged (e.g., metric).

• identifier (Union[str, List[str]]) – unique identifier for the element to log(e.g., name of a metric).

• kind (str, optional) – type of the item to be logged. Must be one among the list of self.supported_types. Defaults to ‘metric’.

• step (Optional[int], optional) – logging step. Defaults to None.

• batch_idx (Optional[int], optional) – DataLoader batch counter (i.e., batch idx), if available. Defaults to None.

• kwargs – keyword arguments to pass to the logger.

class itwinai.loggers.LoggersCollection(loggers: List[Logger])

Bases: Logger

Wrapper of a set of loggers, allowing to use them simultaneously.

Parameters:

loggers (List[Logger]) – list of itwinai loggers.

should_log(batch_idx: int | None = None) → bool

Transparent method which delegates the Logger.should_log` to individual loggers. Always returns True.

Parameters:

• batch_idx (int, optional) – dataloader batch index.

• None. (Defaults to) –

Returns:

always True.

Return type:

bool

log(item: Any | List[Any], identifier: str | List[str], kind: str = 'metric', step: int | None = None, batch_idx: int | None = None, **kwargs) → None

Log on all loggers.

Parameters:

• item (Union[Any, List[Any]]) – element to be logged (e.g., metric).

• identifier (Union[str, List[str]]) – unique identifier for the element to log(e.g., name of a metric).

• kind (str, optional) – type of the item to be logged. Must be one among the list of self.supported_types. Defaults to ‘metric’.

• step (Optional[int], optional) – logging step. Defaults to None.

• batch_idx (Optional[int], optional) – DataLoader batch counter (i.e., batch idx), if available. Defaults to None.

• kwargs – keyword arguments to pass to the logger.

create_logger_context()

Initialize all loggers.

destroy_logger_context()

Destroy all loggers.

save_hyperparameters(params: Dict[str, Any]) → None

Save hyperparameters for all loggers.

Parameters:

params (Dict[str, Any]) – hyperparameters dictionary.

class itwinai.loggers.EpochTimeTracker(series_name: str, csv_file: str)

Bases: object

Profiler for epoch execution time used to support scaling tests. It uses CSV files to store, for each epoch, the name of the experiment, the number of compute nodes used, the epoch_id, and the execution time in seconds.

Parameters:

• series_name (str) – name of the experiment/job.

• csv_file (str) – path to CSV file to store experiments times.

add_epoch_time(epoch_idx: int, time: float) → None

Add row to the current experiment’s CSV file in append mode.

Parameters:

• epoch_idx (int) – epoch order idx.

• time (float) – epoch execution time (seconds).

save(csv_file: str | None = None) → None

Save data to a new CSV file.

Parameters:

csv_file (Optional[str], optional) – path to the CSV file. If not given, uses the one given in the constructor. Defaults to None.