For Heads-of · Practitioner
Model Inversion
Also known as: model inversion attack
An attack that uses a model's outputs to reconstruct information about its training data, such as recovering a recognisable face from a facial recognition model, or sensitive attributes of the people in a dataset.
- privacy attack
- training data
- reconstruction
- UK GDPR
- adversarial machine learning
How it works
An attacker queries the model, often many times, and uses its responses (especially confidence scores) to work backwards to inputs the model strongly associates with a given output. For a model trained on personal data, the reconstruction can reveal what a person looks like or attributes they never disclosed. It is related to membership inference, which asks only whether a record was in the training data.
Why it matters
If a model can be made to reveal its training data, exposing the model is a way of exposing that data. Under UK GDPR the risk belongs in the data protection impact assessment for any model trained on personal data, and it is one reason a trained model may itself need to be treated as containing personal data.
Controls
Limit what the model returns (labels rather than full confidence scores), rate-limit and monitor queries, train with differential privacy where the data is sensitive, and restrict access to models trained on personal data. The register entry is sensitive information disclosure; inference usage quotas and rate limiting is the control.
Sources
Related Terms
Differential Privacy
A mathematical framework for limiting what can be learned about any individual from an algorithm's output, by adding carefully calibrated noise so that the output barely changes whether or not that individual's data was included.
Membership Inference
An attack that works out whether a particular person's record was in a model's training data by examining how the model responds. Showing that someone's data was used can itself be a disclosure of personal data.
Model Extraction
Reconstructing a model's behaviour or parameters by querying it repeatedly and learning from the answers, to copy its intellectual property or to build a surrogate for planning further attacks.
Training Data
The dataset used to train a machine learning model, from which the model learns patterns and relationships.