PGD Attack with TensorBoard Tracking

2.2. PGD Attack with TensorBoard Tracking#

In this notebook, we will demonstrate how to use SecML-Torch to perform a Projected Gradient Descent (PGD) attack while tracking various metrics using TensorBoard integration.

We will:

  • Load visualization utilities and dependencies

  • Load the CIFAR-10 dataset and a pre-trained robust model

  • Configure a PGD attack with multiple tracking capabilities

  • Visualize the attack results and tracked metrics using TensorBoard

2.2.1. Import dependencies and load utils functions#

We install/load SecML‑Torch, RobustBench and TensorBoard (to visualize metrics). If this is your first run, packages and the CIFAR‑10 dataset may be downloaded, which requires internet access.

%%capture --no-stderr --no-stdout
try:
    import secmlt
except ImportError:
    %pip install git+https://github.com/pralab/secml-torch

try:
    import tensorboard
except ImportError:
    %pip install tensorboard

import torch
import torchvision
import torchvision.transforms as transforms
from torch.utils.data import DataLoader, Subset
from pathlib import Path

# SecML-Torch imports
from secmlt.models.pytorch.base_pytorch_nn import BasePyTorchClassifier
from secmlt.metrics.classification import Accuracy
from secmlt.adv.evasion.pgd import PGD
from secmlt.adv.evasion.perturbation_models import LpPerturbationModels
from secmlt.adv.backends import Backends

import warnings
warnings.filterwarnings("ignore")

warnings.simplefilter(action="ignore", category=FutureWarning)

# CIFAR-10 class names
cifar10_classes = [
    "plane",
    "car",
    "bird",
    "cat",
    "deer",
    "dog",
    "frog",
    "horse",
    "ship",
    "truck",
]

2.2.2. Set device and paths#

# Setup device and paths
device = "cuda" if torch.cuda.is_available() else "cpu"
dataset_path = "data/datasets/"
logs_path = "data/logs/pgd_tutorial"

# Create directories if they don't exist
Path(dataset_path).mkdir(parents=True, exist_ok=True)
Path(logs_path).mkdir(parents=True, exist_ok=True)

print(f"Using device: {device}")
print(f"Dataset path: {dataset_path}")
print(f"Logs will be saved to: {logs_path}")

Using device: cpu
Dataset path: data/datasets/
Logs will be saved to: data/logs/pgd_tutorial

2.2.3. Loading CIFAR-10 Dataset#

We’ll load the CIFAR-10 dataset and use a small subset for demonstration.

%%capture --no-stdout

# Load CIFAR-10 dataset
transform = transforms.Compose([transforms.ToTensor()])

test_dataset = torchvision.datasets.CIFAR10(
    root=dataset_path, train=False, download=True, transform=transform
)

num_samples = 20
batch_size = num_samples // 2
test_subset = Subset(test_dataset, list(range(num_samples)))
test_data_loader = DataLoader(test_subset, batch_size=batch_size, shuffle=False)

print(f"Loaded {len(test_subset)} samples from CIFAR-10 test set")

Loaded 20 samples from CIFAR-10 test set

2.2.4. Loading a Pre-trained Model#

We’ll load a pretrained model and evaluate its robustness.

net = torch.hub.load("chenyaofo/pytorch-cifar-models", "cifar10_resnet20", pretrained=True, trust_repo=True)

net = net.to(device)
net.eval()

# Wrap the model with SecML-Torch's BasePyTorchClassifier
model = BasePyTorchClassifier(net, preprocessing=transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010)))

2.2.5. Baseline Performance Evaluation#

Let’s evaluate the model’s performance on clean images. We’ll later use this information to compare performance degradation of the model, after computing adversarial examples through PGD attack.

# Test accuracy on clean examples
clean_accuracy = Accuracy()(model, test_data_loader)
print(
    f"Clean accuracy: {clean_accuracy.item():.4f} ({clean_accuracy.item() * 100:.2f}%)"
)

Clean accuracy: 0.9500 (95.00%)

2.2.6. Configuring PGD Attack with Tracking#

Here we import per-step trackers for loss, predictions, perturbation norm (L∞), and gradient norm, plus the TensorBoard tracker that aggregates these signals and logs them to logs_path for visualization. Then we configure PGD hyperparameters and attach the tracker so that every iteration is recorded and viewable in TensorBoard.

# Import tracking components
from secmlt.trackers import (
    LossTracker,
    PredictionTracker,
    PerturbationNormTracker,
    GradientNormTracker,
    TensorboardTracker,
)

print("Tracking components imported successfully")

Tracking components imported successfully

# Configure PGD attack parameters
epsilon = 8 / 255     # Maximum L∞ perturbation
num_steps = 10        # Number of PGD iterations
step_size = 4 / 255   # Step size per iteration
perturbation_model = LpPerturbationModels.LINF  # L∞ norm constraint

print(f"Attack configuration:")
print(f"  - Epsilon: {epsilon:.4f} ({epsilon * 255:.1f}/255)")
print(f"  - Number of steps: {num_steps}")
print(f"  - Step size: {step_size:.4f} ({step_size * 255:.1f}/255)")

Attack configuration:
  - Epsilon: 0.0314 (8.0/255)
  - Number of steps: 10
  - Step size: 0.0157 (4.0/255)

# Set up individual trackers
trackers = [
    LossTracker(),
    PredictionTracker(),
    PerturbationNormTracker("linf"),
    GradientNormTracker(),
]

# Set up TensorBoard tracking
tensorboard_tracker = TensorboardTracker(logs_path, trackers)

print(f"Configured {len(trackers)} trackers with TensorBoard logging")
print(f"TensorBoard logs cleared and will be saved to: {logs_path}")

Configured 4 trackers with TensorBoard logging
TensorBoard logs cleared and will be saved to: data/logs/pgd_tutorial

2.2.7. Executing the PGD Attack with Tracking#

We instantiate PGD with the chosen hyperparameters and attach the TensorBoard tracker, then run it on the test dataloader. While the attack iterates, the trackers record per-step loss, predictions, perturbation L∞ norms, and gradient norms; these are written to logs_path for inspection in TensorBoard. The call returns an adversarial dataset aligned with the input batches, which we use for evaluation and visualization.

# Create PGD attack with tracking
pgd_attack = PGD(
    perturbation_model=perturbation_model,
    epsilon=epsilon,
    num_steps=num_steps,
    step_size=step_size,
    random_start=False,
    backend=Backends.NATIVE,
    trackers=tensorboard_tracker,
)

print("PGD attack configured successfully")

PGD attack configured successfully

# Execute the attack
print("Executing PGD attack with tracking...")
pgd_native_adv_ds = pgd_attack(model, test_data_loader)
print("Attack completed!")

Executing PGD attack with tracking...

Attack completed!

2.2.8. Results Analysis#

Let’s analyze the results of our PGD attack. We expect here to see performance degradation in terms of accuracy, which is the effect of the untargeted PGD attack we just run on our samples.

# Test accuracy on adversarial examples
adversarial_accuracy = Accuracy()(model, pgd_native_adv_ds)

print("=== Attack Results ===")
print(
    f"Clean accuracy:        {clean_accuracy.item():.4f} ({clean_accuracy.item() * 100:.2f}%)"
)
print(
    f"Robust accuracy:  {adversarial_accuracy.item():.4f} ({adversarial_accuracy.item() * 100:.2f}%)"
)
print(
    f"Attack success rate:   {1 - adversarial_accuracy.item():.4f} ({(1 - adversarial_accuracy.item()) * 100:.2f}%)"
)

=== Attack Results ===
Clean accuracy:        0.9500 (95.00%)
Robust accuracy:  0.0000 (0.00%)
Attack success rate:   1.0000 (100.00%)

2.2.9. Accessing Raw Tracker Metrics#

Beyond TensorBoard, you can inspect trackers directly in code. Each tracker collects per-sample values at each PGD iteration and exposes a get() method that returns a tensor with shape [num_samples, num_iterations] (concatenated across batches). The example below uses LossTracker and PredictionTracker. We compute prediction flips, where 1 means a flip in the true label happened, which should match the display above.

# Inspect tracker histories
loss_hist = trackers[0].get()   # LossTracker: shape [N_samples, T_iters]
preds_hist = trackers[1].get()  # PredictionTracker: shape [N_samples, T_iters]

# Average loss over iterations
avg_loss = loss_hist.mean(dim=0) if loss_hist.numel() > 0 else None

# Prediction changes across iterations per sample
if preds_hist.numel() > 0:
    pred_flips_per_sample = (preds_hist[:, 1:] != preds_hist[:, :-1]).sum(dim=1)
    total_flips = int(pred_flips_per_sample.sum().item())
else:
    pred_flips_per_sample, total_flips = None, 0

print("Raw tracker summaries:")
if avg_loss is not None:
    print(f" - Final average loss (iter {avg_loss.numel()-1}): {avg_loss[-1].item():.4f}")
else:
    print(" - Loss history empty")
if pred_flips_per_sample is not None:
    print(f" - Total prediction flips across samples/iters: {total_flips}")
    print(f" - First 5 samples flips: {pred_flips_per_sample[:5].tolist()}")
else:
    print(" - Prediction history empty")

Raw tracker summaries:
 - Final average loss (iter 9): 32.6321
 - Total prediction flips across samples/iters: 19
 - First 5 samples flips: [1, 1, 1, 1, 1]

2.2.10. TensorBoard Visualization#

Now let’s launch TensorBoard to see all the tracked metrics in detail. TensorBoard provides the most comprehensive view of the attack progression with interactive plots. We can view it inline (uncomment line below print statement), or access to the webpage on localhost. We can see how the loss, norms values progress during the attack steps.

# Launch TensorBoard inline
print(
    "🚀 TensorBoard: http://localhost:6007 (open in your browser if not shown inline)\n"
)

# Uncomment this for inline visualization
# %load_ext tensorboard
# %tensorboard --logdir $logs_path --port 6007 --reload_interval 2

🚀 TensorBoard: http://localhost:6007 (open in your browser if not shown inline)

04-pgd-tensorboard-tracking.png

Contents