For Heads-of · Practitioner
Membership Inference
Also known as: membership inference attack
An attack that works out whether a particular person's record was in a model's training data by examining how the model responds. Showing that someone's data was used can itself be a disclosure of personal data.
- privacy attack
- model inversion
- training data
- UK GDPR
- adversarial machine learning
How it works
Models often behave slightly differently on examples they were trained on: more confident, lower error. An attacker who can query the model compares its behaviour on a target record with its behaviour on similar records it has not seen. The more a model has memorised, the better the attack works, so overfitted models and models trained on small datasets are the most exposed.
Why it matters
If a model was trained on, say, patients with a particular condition or customers in arrears, learning that a person's record was included reveals something about them. Under UK GDPR that is a disclosure of personal data, and potentially of special category data. Model inversion, a related attack, goes further and reconstructs attributes of the training data.
Controls
Train with differential privacy where the data is sensitive, avoid overfitting, limit the detail the model returns (a decision or score rather than full probabilities), and rate-limit queries. Cover the risk in the data protection impact assessment for any model trained on personal data. The register entry is sensitive information disclosure; data minimisation and lawful basis review is the control.
Sources
Related Terms
Differential Privacy
A mathematical framework for limiting what can be learned about any individual from an algorithm's output, by adding carefully calibrated noise so that the output barely changes whether or not that individual's data was included.
Model Extraction
Reconstructing a model's behaviour or parameters by querying it repeatedly and learning from the answers, to copy its intellectual property or to build a surrogate for planning further attacks.
Model Inversion
An attack that uses a model's outputs to reconstruct information about its training data, such as recovering a recognisable face from a facial recognition model, or sensitive attributes of the people in a dataset.
Synthetic Data
Artificially generated data, created by algorithms or models, used instead of or alongside real data in training and testing.
Training Data
The dataset used to train a machine learning model, from which the model learns patterns and relationships.