Sensitive information disclosure

What it is

A model can say anything it has seen. Sensitive information reaches it in three ways: in training or fine-tuning data, in documents retrieved at answer time, and in the conversation itself. Once there, the access controls that governed the original data no longer apply; what limits disclosure is the model's behaviour and whatever filters surround it. This is the general entry. Verbatim reproduction of training records is covered in training data memorisation and regurgitation, deliberate extraction in model inversion and training data extraction, and leakage of instructions in hidden context exposure.

How it happens

• A retrieval index contains documents the person asking is not entitled to see.
• A model is fine-tuned on records containing personal data.
• One user's conversation or cached answer is shown to another.
• A summary or draft includes details drawn from an unrelated record.
• A prompt designed to extract data succeeds.

Why it matters

Disclosure of personal data is a breach of the security and confidentiality duties in data protection law, and may have to be reported. Confidential business information, once revealed, cannot be made secret again. Because the route is a conversation, exposure can be repeated many times before anyone notices.

Warning signs

• The retrieval index was built without applying the source systems' permissions.
• No filter inspects outputs for personal or classified data.
• Staff report seeing information about colleagues or other customers.
• Nobody can list what sensitive data the model was trained on.

Who owns it

The system owner is accountable. The data owner decides what the system may hold. Security implements access and output controls. The data protection officer advises and assesses any breach.

In practice

A firm connects an assistant to its shared drive so staff can ask questions of internal documents. The index includes a folder of salary reviews. An employee asks what the pay range is for their grade and receives named figures. The same pattern generalises to any system given broad access to data and then opened to a wider audience than the data was.