For Heads-of · Practitioner
Differential Privacy
A mathematical framework for limiting what can be learned about any individual from an algorithm's output, by adding carefully calibrated noise so that the output barely changes whether or not that individual's data was included.
- differential privacy
- privacy
- noise
- statistical guarantees
Guarantee
The output of a differentially private algorithm is almost the same whether or not any one person's data is included, so it reveals little about that person. The strength of the guarantee is set by a parameter, epsilon: smaller means stronger privacy and noisier results.
Trade-off
The noise reduces accuracy, so there is always a trade-off between privacy and usefulness. Choosing epsilon is a governance decision, and an organisation claiming differential privacy should be able to say what value it used and why.
Application
Used in federated learning, in publishing aggregate statistics, and in training models on sensitive data. It is one of the few defences with a formal guarantee against attacks that extract training data, which is the sensitive information disclosure risk in the register.
Related Terms
Federated Learning
A machine learning approach in which a model is trained across many separate data sources without the data itself being gathered in one place.
Model Inversion
An attack that uses a model's outputs to reconstruct information about its training data, such as recovering a recognisable face from a facial recognition model, or sensitive attributes of the people in a dataset.
Synthetic Data
Artificially generated data, created by algorithms or models, used instead of or alongside real data in training and testing.
Training Data
The dataset used to train a machine learning model, from which the model learns patterns and relationships.
Transparency
The disclosure of information about an AI system to the people who need it: that AI is being used, what it does, what it is for and what risks it poses, in terms they can understand. For people whose personal data it uses, being informed is also a legal right.