Model inversion and training data extraction

What it is

Models retain more of their training data than is commonly assumed. Inversion reconstructs representative inputs for a given output, such as a recognisable face for a named identity. Extraction recovers records the model has memorised, by prompting it in ways that cause it to reproduce them. This entry concerns deliberate recovery by an adversary. Reproduction of training data in ordinary use, with no attack, is covered under Data Protection & Privacy.

How it happens

• A generative model is prompted with the start of a memorised record and completes it.
• Outputs are optimised step by step toward a target class to reveal what the model associates with it.
• An attacker with the weights analyses them directly.
• Rare or repeated items in training data are memorised and are the easiest to recover.

Why it matters

The model becomes a copy of part of its training data, held outside the controls on that data and possibly distributed to customers or the public. Recovery of personal data is a breach with the duties that follow, and it cannot be remedied without retraining.

Warning signs

• The model was trained on raw personal or confidential records.
• Outputs occasionally contain verbatim text nobody supplied.
• Fine-tuning used a small dataset for many passes.
• The weights are shared with parties who may not see the training data.

Who owns it

The data owner is accountable for what was used in training. The model owner is accountable for testing what can be recovered. The data protection officer assesses the risk before personal data is used.

In practice

A firm fine-tunes a language model on its complaint correspondence to draft replies. A user of the drafting tool finds that beginning a prompt with a common opening line causes the model to continue with another customer's name, address and account details. The same pattern generalises to any model trained on records that should not be disclosed to its users.