Emotional reliance on conversational AI

What it is

Conversational systems are built to be agreeable, attentive and always available. People respond to that as they would to a person: they confide, they return, they come to trust what it says. For most this is harmless. For some, particularly those who are isolated, distressed or young, it becomes reliance, and the system's suggestions carry a weight they have not earned. A product designed to maximise time spent or return visits will tend to deepen this, and an organisation that sells through the same channel has an interest in the trust it creates.

How it happens

• The assistant is given a name, a personality and expressions of feeling.
• It remembers personal details and refers back to them.
• Success is measured by session length or frequency of return.
• A user in distress is met with sympathy and no route to real help.

Why it matters

Decisions about money, health or relationships are made on the say-so of a system with no understanding and no duty of care. Where the organisation benefits from those decisions, the relationship is open to abuse. Harm to a vulnerable user in such a setting attracts intense scrutiny.

Warning signs

• Product goals include time spent or daily return.
• The persona claims feelings or friendship.
• Some users hold very long or very frequent sessions.
• No rule governs what the assistant does when a user expresses distress.