secmlt.metrics package

Submodules

secmlt.metrics.classification module

Classification metrics for machine-learning models and for attack performance.

class secmlt.metrics.classification.Accuracy

Bases: object

Class for computing accuracy of a model on a dataset.

__call__(model: BaseModel, dataloader: torch.utils.data.DataLoader) → torch.Tensor

Compute the metric on a single attack run or a dataloader.

Parameters:

• model (BaseModel) – Model to use for prediction.

• dataloader (DataLoader) – A dataloader, can be the result of an attack or a generic test dataloader.

Returns:

The metric computed on the given dataloader.

Return type:

torch.Tensor

__init__() → None

Create Accuracy metric.

class secmlt.metrics.classification.AccuracyEnsemble

Bases: Accuracy

Robust accuracy of a model on multiple attack runs.

__call__(model: BaseModel, dataloaders: list[torch.utils.data.DataLoader]) → torch.Tensor

Compute the metric on an ensemble of attacks from their results.

Parameters:

• model (BaseModel) – Model to use for prediction.

• dataloaders (list[DataLoader]) – List of loaders returned from multiple attack runs.

Returns:

The metric computed across multiple attack runs.

Return type:

torch.Tensor

class secmlt.metrics.classification.AttackSuccessRate(y_target: float | torch.Tensor | None = None)

Bases: Accuracy

Single attack success rate from attack results.

__init__(y_target: float | torch.Tensor | None = None) → None

Create attack success rate metric.

Parameters:

y_target (float | torch.Tensor | None, optional) – Target label for the attack, None for untargeted, by default None

class secmlt.metrics.classification.EnsembleSuccessRate(y_target: float | torch.Tensor | None = None)

Bases: AccuracyEnsemble

Worst-case success rate of multiple attack runs.

__init__(y_target: float | torch.Tensor | None = None) → None

Create ensemble success rate metric.

Parameters:

y_target (float | torch.Tensor | None, optional) – Target label for the attack, None for untargeted,, by default None

secmlt.metrics.classification.accuracy(y_pred: torch.Tensor, y_true: torch.Tensor) → torch.Tensor

Compute the accuracy on a batch of predictions and targets.

Parameters:

• y_pred (torch.Tensor) – Predictions from the model.

• y_true (torch.Tensor) – Target labels.

Returns:

The percentage of predictions that match the targets.

Return type:

torch.Tensor

Module contents

Metrics to evaluate machine learning models and attacks.