tapas.report.attack_summary.MIAttackSummary
- class tapas.report.attack_summary.MIAttackSummary(labels, predictions, scores=None, generator_info='', attack_info='', dataset_info='', target_id='')
Bases:
tapas.report.attack_summary.BinaryLabelInferenceAttackSummaryClass summarising main performance metrics of a membership inference attack.
- __init__(labels, predictions, scores=None, generator_info='', attack_info='', dataset_info='', target_id='')
- Parameters
labels (list[int]) – List with true labels of the target membership in the dataset.
predictions (list[int]) – List with the predicted labels of the target membership in the dataset.
scores (list[float]) – List with the scores related to each prediction.
generator_info (str) – Metadata with information about the method used to generate the dataset.
attack_info (str) – Metadata with information about the attacked used to infer membership of the target on the dataset.
dataset_info (str) – Metadata with information about the original raw dataset.
target_id (str) – Metadata with information about the target record used on the attack.
Methods
__init__(labels, predictions[, scores, ...])- param labels
List with true labels of the target membership in the dataset.
get_metric_filename([postfix])Returns the file name to which results should be saved.
Calculates all MIA relevant metrics and returns them as a dataframe.
write_metrics(output_path[, postfix])Write metrics to file.
Attributes
Accuracy of the attacks based on the rate of correct predictions.
Area under the Receiver Operating Characteristic curve (ROC).
Computes the "effective epsilon" of the attack, which is estimated as:
False positives based on rate of attacks where the target is incorrectly inferred as being in the sample.
MIA attack advantage as defined by Stadler et al.
Privacy gain as defined by Stadler et al.
True positives based on rate of attacks where the target is correctly inferred as being in the sample.
- property accuracy
Accuracy of the attacks based on the rate of correct predictions.
- Return type
float
- property auc
Area under the Receiver Operating Characteristic curve (ROC). If scores are not provided, this uses self.predictions as score.
- Return type
float
- property effective_epsilon
- Computes the “effective epsilon” of the attack, which is estimated as:
max_{tau in T*} log(max(TP_tau/FP_tau, (1-FP_tau)/(1-TP_tau))).
Where T* is defined as the set of statistically significant thresholds. This is arbitrarily set as T* = {t: count(D <= t) >= 10 ^ count(D > t) >= 10}.
If there is a threshold t in T* such that FP_t = 0 or TP_0 = 1, then the effective epsilon will be infinite.
If self.scores is None, this returns max(tp/fp, (1-tp)/(1-fp)).
For an analysis involving the statistical significance of this result, use TODO report class.
- Return type
float, potentially inf.
- property fp
False positives based on rate of attacks where the target is incorrectly inferred as being in the sample.
- Return type
float
- get_metric_filename(postfix='')
Returns the file name to which results should be saved.
- get_metrics()
Calculates all MIA relevant metrics and returns them as a dataframe.
- Returns
A dataframe with attack info and metrics. The dataframe has the following structure. Index:
RangeIndex
- Columns:
dataset: str target_id: str generator: str attack: str accuracy: float true_positive_rate: float false_positive_rate: float mia_advantage: float privacy_gain: float
- Return type
A dataframe
- property mia_advantage
MIA attack advantage as defined by Stadler et al.
- Return type
float
- property privacy_gain
Privacy gain as defined by Stadler et al.
- Return type
float
- property tp
True positives based on rate of attacks where the target is correctly inferred as being in the sample.
- Return type
float
- write_metrics(output_path, postfix='')
Write metrics to file.
- Parameters
output_path (str) – The prefix of the path where the metrics should be saved.
postfix (str) – An optional string to append to the filename