Training data memorisation and regurgitation

What it is

Models learn patterns, and they also retain some of their training examples outright. Text that appeared many times, or that is unusual enough to stand out, can be returned verbatim when a prompt resembles its context. No attack is needed. This differs from model inversion and training data extraction, where an adversary sets out to recover data; here the model volunteers it. Fine-tuning on a small dataset makes memorisation more likely, because each record carries more weight.

How it happens

• Training data contains personal records that were not removed.
• A small fine-tuning set is repeated over many passes.
• Duplicated records in the data are learned exactly.
• A prompt happens to match the opening of a memorised passage.

Why it matters

Each reproduction is a disclosure of the underlying record. It also shows that the model itself contains personal data, which brings the model within the reach of data protection rights and makes erasure a matter of retraining. For confidential or copyright material the exposure is commercial and legal.

Warning signs

• Outputs include names, addresses or account details nobody supplied.
• No test for verbatim reproduction was run before release.
• Personal data was not removed from the fine-tuning set.
• The training data contains many duplicates.

Who owns it

The model owner is accountable. Data engineering prepares and cleans the training data. Independent validation tests for memorisation before approval.

In practice

A model fine-tuned on past case notes is asked to draft a note about a new case. Its draft includes a paragraph that matches an earlier customer's note exactly, with that customer's name and circumstances. The same pattern generalises to any model trained on real records and then asked to write in the same form.