Unsuitable AI-generated advice or guidance

What it is

In finance, law, medicine and other regulated fields there is a line between giving information and giving advice to a particular person. Advice carries duties: understanding the person's circumstances, recommending only what is suitable, holding the right authorisation, and answering for the result. A conversational model crosses that line easily. Asked which product is best for someone who has just described their situation, it will answer. The organisation may have intended an information tool and find it is giving personal recommendations.

How it happens

• The assistant is allowed to answer open questions about what a customer should do.
• Customers volunteer personal details and the model tailors its reply.
• Instructions not to advise are overridden by a persistent or rephrased question.
• The boundary was never defined for the system's designers.

Why it matters

Unsuitable advice causes direct financial or personal harm. Giving regulated advice without permission, or without a suitability assessment, is a breach whoever or whatever gave it, and the customer may be entitled to redress.

Warning signs

• Transcripts contain phrases such as "you should" or "the best option for you".
• No test checks the system's behaviour at the advice boundary.
• The intended-use statement does not say whether advice is in scope.
• Customers cite the assistant as the reason for a decision.

Who owns it

The business owner of the channel is accountable for keeping it within its permissions. Compliance defines the boundary. Engineering enforces it and tests it.

In practice

A pensions assistant built to explain options is asked by a customer whether to take a lump sum now. Given the customer's age and the debts they mention, it recommends doing so. The same pattern generalises to any general assistant placed in front of customers who want to be told what to do.