Misleading AI-generated communications

What it is

Organisations now use generative models to write letters, emails, product descriptions, marketing and chat replies. Communications to customers have to be fair, clear and not misleading, and in regulated sectors particular content is required: risk warnings, key terms, the right to complain. A model produces fluent, confident text that may state a feature the product lacks, omit a condition, or stress benefits over risks. Each instance is small. The model writes thousands. The general accuracy problem is covered in misinformation and confabulation; this entry concerns the conduct consequence.

How it happens

• Text is generated from a prompt and sent without review.
• The model is asked to make copy more persuasive.
• Required wording is dropped when content is shortened or personalised.
• Product terms changed and the source the model draws on did not.

Why it matters

A customer who relies on a misleading statement may hold the organisation to it or claim redress. In financial promotion the rules are strict, including on who may approve the content. Personalisation makes the problem harder to see, since no two customers receive the same text.

Warning signs

• Generated communications are not sampled against content standards.
• Marketing prompts ask for urgency or persuasion.
• Complaints quote statements the organisation cannot find in its templates.
• Mandatory wording is missing from some outputs.

Who owns it

The owner of the communication, usually marketing or operations, is accountable. Compliance sets and tests content standards. Whoever approves financial promotions approves those written by AI.

In practice

A savings provider uses a model to personalise renewal emails. For some customers the model describes a rate as guaranteed when it is variable. The same pattern generalises to any customer communication produced faster than it can be read by someone who knows the rules.