Risk · R-103 · Data Protection & Privacy
Personal data in prompts and context
Users or integrations put personal data into prompts, uploads or retrieved context without a lawful basis or appropriate safeguards.
- Severity
- High
- Audience
- Heads-of · Practitioner
- Frameworks
- Data minimisation · Data Privacy · LLM02 · Article 5
- Responsible AI domain
- Lifecycle stage
- Reviewed
- 2026-11-11
What it is
What goes into a model is processing, as much as what comes out. Staff paste customer records into a prompt to get a summary; an integration attaches a whole case file where one field was needed; a retrieval step pulls in documents about other people. Each sends personal data to the model, and usually to its provider, where it may be logged, retained or reviewed. The risk is that this happens by habit and design default, with no one having decided it was necessary and lawful. The broader governance failure of unapproved tools is covered in shadow AI use.
How it happens
- Staff copy records into an assistant to save time.
- An integration sends full records because that was simplest to build.
- Retrieval returns documents containing third parties' details.
- Uploaded files contain more than the user realised.
Why it matters
Data protection law requires that personal data be limited to what is necessary and kept secure. Data sent to a provider is outside the organisation's direct control, and special category data sent without a condition for processing is a serious breach.
Warning signs
- No guidance tells staff what may be entered into which tools.
- Prompts are not inspected for personal data.
- Integrations pass whole records to the model.
- The provider's retention and review terms are unknown.
Who owns it
The system owner is accountable for what the system sends. The data owner sets what may be shared. Security operates inspection. Line managers are accountable for their teams' practice.
In practice
A complaints handler pastes a customer's full correspondence, including medical evidence, into a general assistant to draft a reply. The provider's terms allow human review of conversations for safety. The same pattern generalises to any tool that makes sharing personal data the quickest way to get work done.
Mitigating controls
Primary controls
- Prompt and context data loss prevention Preventive
- Inspects prompts, uploads and retrieved context and blocks or redacts personal data. Prompts, uploads and retrieved context are inspected before they reach a model, and blocked, redacted or flagged where they contain personal or restricted data.
Supporting controls
- AI acceptable use policy Directive
- Tells staff what may be entered into which tools. A policy approved by top management states what AI use is permitted, what is prohibited, what every system must pass through, who is accountable, and where to take a case the policy does not settle.
- Personal data redaction and pseudonymisation Preventive
- Removes identifiers before text reaches the model. Personal data is removed, masked or pseudonymised before data is used for training, evaluation or logging, unless it is needed and the need is recorded, and the result is tested.
- Role-based AI training Directive
- Staff understand why the restriction exists. Reviewers, developers, risk and assurance staff, approvers and front-line teams receive training specific to their AI responsibilities, with competence assessed and completion tracked.
Framework and clause references
| Framework | Clause | Title |
|---|---|---|
| Guidance on AI and data protection | Data minimisation | Data minimisation and accuracy Minimisation in AI use. |
| NIST Generative AI Profile (NIST AI 600-1) | Data Privacy | Data Privacy The data privacy risk. |
| OWASP Top 10 for LLM Applications | LLM02 | Sensitive Information Disclosure A route to disclosure. |
| UK General Data Protection Regulation | Article 5 | Principles relating to processing of personal data Data minimisation and purpose limitation. |
Sources
- Guidance on AI and data protection UK ICO · guidance
- NIST Generative AI Profile NIST · framework
- OWASP Top 10 for LLM Applications OWASP GenAI Security Project · framework
Related terms
How to cite
Axiom Verity (2026). Personal data in prompts and context. Reference Library, R-103. https://axiom-verity.com/risks/personal-data-in-prompts-and-context. Reviewed 2026-11-11.
Found an error or a better source? Report a correction.