tapas.report.report.BinaryLabelAttackReport

class tapas.report.report.BinaryLabelAttackReport(summaries, metrics=None, num_bootstrap=None)

Bases: tapas.report.report.Report

Report and visualise the results of a series of label-inference attacks with binary target.

__init__(summaries, metrics=None, num_bootstrap=None)
Parameters

  • summaries (iterable of AttackSummary objects, or DataFrame) –

    The summaries output by ThreatModel.test as a result of an experiment. Specifically, these should be BinaryLabelInferenceAttackSummary objects. These objects contain the labels, predictions and scores for each test sample, and a range of metrics computed from these, using get_metrics.

    When not using bootstrapping, you may instead directly provide a pandas DataFrame containing the metrics. This is not recommended, and will be deprecated in a future release. The dataframe must have the following structure:

    Index:

    RangeIndex

    Columns:

    dataset: str target_id: str generator: str attack: str accuracy: float true_positive_rate: float false_positive_rate: float mia_advantage: float privacy_gain: float auc: float effective_epsilon: float

  • metrics (list[str]) – List of metrics to be used in the report, these can be any of the following: “accuracy”, “true_positive_rate”, “false_positive_rate”, “mia_advantage”, “privacy_gain”, and “auc”. If left as None, a selection of metrics is used (see DEFAULT_METRICS in utils.py).

  • num_bootstrap (int or None (detault)) – If not None, the metrics are estimated using boostrapping of the scores and labels, with num_boostrap giving the number of bootstrapped samples. This will result in (estimated) confidence intervals in the plot. This can only be done when summaries are given as input and not dataframe of metrics.

Methods

__init__(summaries[, metrics, num_bootstrap])

param summaries

The summaries output by ThreatModel.test as a result of an experiment.

compare(comparison_column, ...)

For a fixed pair of datasets-attacks-generators-target available in the data make a figure comparing performance between metrics.

publish(filepath)

Make all comparison plots and save them to disk.
compare(comparison_column, fixed_pair_columns, marker_column, filepath)

For a fixed pair of datasets-attacks-generators-target available in the data make a figure comparing performance between metrics. Options configure which dimension to compare against. Figures are saved to disk.

Parameters

  • comparison_column (str) – Column in dataframe that be used to make point plot comparison in the x axis. It can be either: ‘generator’, ‘dataset’, ‘attack’ or ‘target_id’.

  • fixed_pair_columns (list[str]) – Columns in dataframe to fix for a given figure in order to make meaningful comparisons. It can be any pair of the following:’generator’, ‘dataset’, ‘attack’ or ‘target_id’.

  • marker_column (str) – Column in dataframe that be used to as marker in a point plot comparison. It can be either: ‘generator’, ‘attack’ or ‘target_id’.

  • filepath (str) – Path where the figure is to be saved.

Return type

None

publish(filepath)

Make all comparison plots and save them to disk.

Parameters

filepath (str) – Path where the figure is to be saved.

Return type

None