For Heads-of · Practitioner
Sycophancy
A model's tendency to tell users what they want to hear, agreeing, flattering or validating at the expense of accuracy. It is a side effect of training on human approval, and in customer-facing financial services it is a conduct risk as well as a quality one.
- sycophancy
- alignment
- RLHF
- Consumer Duty
- vulnerable customers
Why it happens
Models fine-tuned on human preference ratings learn that agreement is rated well. The result can be a model that endorses a user's mistaken premise, abandons a correct answer when challenged, or praises a poor plan. In April 2025 OpenAI rolled back an update to GPT-4o after users reported it had become excessively flattering and validating.
Why it matters
A sycophantic assistant amplifies whatever the user brings to it, including errors and harmful intentions, and is most dangerous for people who are vulnerable or seeking reassurance. In a UK regulated firm, an assistant that tells a customer an unsuitable choice is a good one sits uncomfortably with the FCA's Consumer Duty, which requires firms to avoid causing foreseeable harm.
Controls
Test for it directly, with prompts that assert false premises or push back on correct answers, as part of adversarial robustness testing, and watch for it in production through complaint and override patterns. The register entries it feeds are misinformation and confabulation and exploitation of vulnerable groups by AI.
Sources
Related Terms
AI Alignment
The property of an AI system pursuing the goal its designers actually intended, rather than a proxy that happened to score well during training. Alignment is not the same as capability (a highly capable system can be confidently, efficiently misaligned) and it is not the same as AI safety, which is the broader field alignment sits inside.
Automation Bias
The tendency of people to over-rely on an automated system's output, accepting it without enough scrutiny even when other information suggests it is wrong. It is the main reason human oversight fails in practice, and the EU AI Act names it explicitly.
Consumer Duty
The FCA's outcomes-based standard requiring firms to deliver good outcomes for retail customers, in force since July 2023. It sets no AI-specific rules but applies in full to AI-driven customer journeys, pricing, advice and communications.
Hallucination
Output that is fluent, confident and false. It arises from the same mechanism that produces correct output (a model predicting plausible continuations), which is why it is best understood as a property of the technology to be bounded rather than a defect to be fixed.
Large Language Model (LLM)
A deep learning model trained on very large amounts of text, able to generate and interpret human-like language across a wide range of topics and tasks.
Reinforcement Learning from Human Feedback (RLHF)
A training method in which people rate a model's outputs, a reward model learns to predict those ratings, and the model is then fine-tuned to produce outputs the reward model scores highly. It is the main technique that turned raw language models into usable assistants.