tapas.report.report.MIAttackReport

class tapas.report.report.MIAttackReport(summaries, metrics=None, num_bootstrap=None)

Bases: tapas.report.report.BinaryLabelAttackReport

Report for a Membership Inference Attack.

__init__(summaries, metrics=None, num_bootstrap=None)
Parameters

  • summaries (iterable of AttackSummary objects, or DataFrame) –

    The summaries output by ThreatModel.test as a result of an experiment. Specifically, these should be BinaryLabelInferenceAttackSummary objects. These objects contain the labels, predictions and scores for each test sample, and a range of metrics computed from these, using get_metrics.

    When not using bootstrapping, you may instead directly provide a pandas DataFrame containing the metrics. This is not recommended, and will be deprecated in a future release. The dataframe must have the following structure:

    Index:

    RangeIndex

    Columns:

    dataset: str target_id: str generator: str attack: str accuracy: float true_positive_rate: float false_positive_rate: float mia_advantage: float privacy_gain: float auc: float effective_epsilon: float

  • metrics (list[str]) – List of metrics to be used in the report, these can be any of the following: “accuracy”, “true_positive_rate”, “false_positive_rate”, “mia_advantage”, “privacy_gain”, and “auc”. If left as None, a selection of metrics is used (see DEFAULT_METRICS in utils.py).

  • num_bootstrap (int or None (detault)) – If not None, the metrics are estimated using boostrapping of the scores and labels, with num_boostrap giving the number of bootstrapped samples. This will result in (estimated) confidence intervals in the plot. This can only be done when summaries are given as input and not dataframe of metrics.

Methods

__init__(summaries[, metrics, num_bootstrap])

param summaries

The summaries output by ThreatModel.test as a result of an experiment.

compare(comparison_column, ...)

For a fixed pair of datasets-attacks-generators-target available in the data make a figure comparing performance between metrics.

load_summary_statistics(attacks[, metrics])

Load attacks data, calculate summary statistics, merge into a single dataframe and initialise object.

publish(filepath)

Make all comparison plots and save them to disk.
compare(comparison_column, fixed_pair_columns, marker_column, filepath)

For a fixed pair of datasets-attacks-generators-target available in the data make a figure comparing performance between metrics. Options configure which dimension to compare against. Figures are saved to disk.

Parameters

  • comparison_column (str) – Column in dataframe that be used to make point plot comparison in the x axis. It can be either: ‘generator’, ‘dataset’, ‘attack’ or ‘target_id’.

  • fixed_pair_columns (list[str]) – Columns in dataframe to fix for a given figure in order to make meaningful comparisons. It can be any pair of the following:’generator’, ‘dataset’, ‘attack’ or ‘target_id’.

  • marker_column (str) – Column in dataframe that be used to as marker in a point plot comparison. It can be either: ‘generator’, ‘attack’ or ‘target_id’.

  • filepath (str) – Path where the figure is to be saved.

Return type

None

classmethod load_summary_statistics(attacks, metrics=None)

Load attacks data, calculate summary statistics, merge into a single dataframe and initialise object.

Parameters

  • attacks (list[dict]) –

    List of dictionaries with results of attacks. Each dictionary should contain the following keys:

    dict: labels: list[int]

    List with true labels of the target membership in the dataset.

    predictions: list[int]

    List with the predicted labels of the target membership in the dataset.

    generator_info: str

    Metadata with information about the method used to generate the dataset.

    attack_info: str

    Metadata with information about the attacked used to infer membership of the target on the dataset.

    dataset_info: str

    Metadata with information about the original raw dataset.

    target_id: str

    Metadata with information about the target record used on the attack.

  • metrics (list[str] List of metrics to be included in the report, these can be any of the following:) –

  • "accuracy" –

  • "true_positive_rate" –

  • "false_positive_rate" –

  • "mia_advantage" –

  • "privacy_gain" –

  • "auc". –

Return type

MIAReport class

publish(filepath)

Make all comparison plots and save them to disk.

Parameters

filepath (str) – Path where the figure is to be saved.

Return type

None