Inaccurate personal data in outputs

What it is

Generative models produce plausible text, and plausible is not the same as true. When the subject is a named person, an invented detail is inaccurate personal data: a qualification never held, a conviction that did not happen, a debt that does not exist. Data protection law requires personal data to be accurate and gives people the right to have errors corrected. A model's output about a person falls within that duty when it is used or recorded as information about them. The general problem is covered in misinformation and confabulation.

How it happens

• A model asked about a person fills gaps in what it knows with invention.
• Two people with similar names are merged.
• A summary of a record adds or alters a detail.
• An inaccurate output is saved to the person's file and later treated as fact.

Why it matters

The person may be refused something, misdescribed to others or defamed, with no knowledge of the cause. Correcting a model is harder than correcting a database: there is no record to amend, and the same error may be generated again.

Warning signs

• Outputs about people are used without checking against the system of record.
• Generated summaries are stored on customer or staff files.
• Complaints allege statements the organisation cannot trace to a source.
• The system will answer questions about named individuals from memory alone.

Who owns it

The business owner of the use is accountable for the accuracy of what is relied on. Engineering constrains the system to verified sources. The data protection officer handles correction requests.

In practice

A recruiter asks an assistant to summarise a candidate's background from their name and employer. The summary attributes to the candidate a disciplinary finding that belongs to someone else in the same profession. The candidate is not shortlisted. The same pattern generalises to any use of a model's unverified statements about an individual.