Manipulative AI-driven design

What it is

Systems that optimise for conversion or retention discover what moves people: urgency, default options, the order of choices, a friendly tone at the moment of doubt, friction placed in front of cancelling. Applied by a model that learns from each customer's responses, these techniques become individually tailored. Nobody need have designed them. The most serious forms, which materially distort behaviour and cause significant harm, are banned outright and covered in subliminal or manipulative AI techniques. This entry concerns the larger territory beneath that line.

How it happens

• The objective is a click, a sale or a retained account, with no measure of customer benefit.
• Automated experiments select whichever variant converts best.
• A conversational agent learns that reassurance closes sales.
• Leaving or complaining is made slower than joining.

Why it matters

Customers end up with products they did not need or cannot leave. For regulated firms, designing journeys that exploit behavioural biases is contrary to the duty to act in good faith. Trust falls when the technique is noticed.

Warning signs

• Optimisation targets contain no customer outcome.
• Cancellation takes more steps than purchase.
• Winning test variants rely on urgency or pre-selected options.
• Regret, returns or early cancellations are rising.