Membership and attribute inference

What it is

Models tend to behave slightly differently on data they were trained on. Membership inference uses that difference to tell whether a given record was in the training set. Where membership itself is sensitive, for example a model trained on patients with a condition or customers in arrears, the answer is a disclosure. Attribute inference goes further and uses the model to estimate an unknown characteristic of a person from what is known about them. Recovery of the records themselves is covered in model inversion and training data extraction.

How it happens

• The attacker submits a known record and examines the model's confidence.
• The model is overfitted, which widens the gap between seen and unseen data.
• Detailed outputs, such as probabilities, are returned to users.
• The training population is small or unusual.

Why it matters

Personal data is disclosed through a system that holds none in any visible form. The organisation may have treated the trained model as outside data protection controls; inference shows why that assumption can fail.

Warning signs

• Models trained on sensitive populations are exposed to outside users.
• No privacy testing appears in the validation record.
• Outputs include full confidence scores.
• Training and test performance differ widely.