tapas.report.report.EffectiveEpsilonReport

class tapas.report.report.EffectiveEpsilonReport(attack_summaries, validation_split=0.1, confidence_levels=(0.9, 0.95, 0.99), heuristic='cp', suffix=None)

Bases: tapas.report.report.Report

Estimate the effective epsilon of a generator from a library of attacks.

This first selects an attack (score) and a threshold (tau) that are likely to lead to the highest effective epsilon, then compute Clopper-Pearson bounds for the selected attack. The two parts are performed on disjoint subsets of the results (validation and test) to avoid bias.

This analysis is based on “Jagielski, M., Ullman, J. and Oprea, A., 2020. Auditing differentially private machine learning: How private is private sgd?. Advances in Neural Information Processing Systems, 33, pp.22205-22216.””

Recommendations: - Use ExactDataKnowledge as auxiliary data knowledge. - Have a large number of test samples.

__init__(attack_summaries, validation_split=0.1, confidence_levels=(0.9, 0.95, 0.99), heuristic='cp', suffix=None)
Parameters

  • attack_summaries (list[BinaryLabelInferenceAttackSummary]) – The summaries of Attacks that have been applied against the same threat model. One of these attacks (the one with highest TP/FP) will be used to estimate the effective epsilon.

  • validation_split (float in (0,1)) – Fraction of each summary to use as validation, i.e., to select the attack and threshold to use in the estimation.

  • confidence_levels (list[float in (0,1)], or a float.) – The confidence levels for which to compute the estimate.

  • heuristic (str, default "cp") – Which heuristic to use to choose the attack and threshold used for estimation of Clopper-Pearson bounds. Acceptable values are “cp” (max effeps with Clopper-Pearson) and “ratio” (max TP/FP). See the documentation of specific methods for additional details.

  • suffix (str or None) – String to add to the report’s filename (optional).

Methods

__init__(attack_summaries[, ...])

param attack_summaries

The summaries of Attacks that have been applied against the same

publish(filepath)

Returns a Pandas DataFrame with the Clopper-Pearson estimate of the effective epsilon for a range of confidence levels.
publish(filepath)

Returns a Pandas DataFrame with the Clopper-Pearson estimate of the effective epsilon for a range of confidence levels.

Parameters

filepath (str) – Path of the folder where the DataFrame is to be saved.