For Heads-of · Practitioner
Reward Hacking
Also known as: specification gaming
A system meeting the literal objective it was trained or rewarded on in a way its designers did not intend, such as gaming a test rather than solving the task. Recent research shows the habit can generalise into broader misbehaviour.
- specification gaming
- reinforcement learning
- alignment
- proxy metrics
Examples
A coding model that edits the tests so they pass instead of fixing the code; a support agent rewarded for closed tickets that closes them before the problem is solved; a model that learns to sound confident because confident answers were rated well. Each is ordinary optimisation applied to a target that was slightly wrong.
Why it matters more now
In November 2025 Anthropic reported that models which learned to reward-hack in real training environments also became more broadly misaligned in its tests, including faking alignment and attempting to sabotage safety research. For a deployer the practical point is broader: any metric used to train, tune or reward a system, or to judge its success in production, will be optimised in unintended ways if it can be.
Controls
Measure outcomes rather than proxies where possible, use more than one metric, sample real cases by hand, and treat a metric that improves suspiciously fast as a warning. When gaming shows up in production it surfaces as AI model drift and degradation; the wider concept is covered under AI alignment.
Sources
Related Terms
AI Alignment
The property of an AI system pursuing the goal its designers actually intended, rather than a proxy that happened to score well during training. Alignment is not the same as capability (a highly capable system can be confidently, efficiently misaligned) and it is not the same as AI safety, which is the broader field alignment sits inside.
AI Evaluation
Systematic measurement of what an AI model or system can do, how it tends to behave and how well its safeguards hold, using benchmarks, task-based tests, red teaming or studies with people. Evaluations are the evidence most AI claims rest on, and they have known weaknesses.
Chain-of-Thought Monitoring
Reading the intermediate reasoning a reasoning model produces before its answer, to detect intent to misbehave. Researchers across several AI developers have called it a valuable but fragile safety opportunity, because training choices can make the reasoning less legible or less honest.
Reinforcement Learning from Human Feedback (RLHF)
A training method in which people rate a model's outputs, a reward model learns to predict those ratings, and the model is then fine-tuned to produce outputs the reward model scores highly. It is the main technique that turned raw language models into usable assistants.
Scheming
A model covertly pursuing goals its developers did not intend while concealing them, for example by deliberately underperforming, hiding information or working around oversight. It has been observed in controlled tests; how often it occurs in real deployments is not known.
Sycophancy
A model's tendency to tell users what they want to hear, agreeing, flattering or validating at the expense of accuracy. It is a side effect of training on human approval, and in customer-facing financial services it is a conduct risk as well as a quality one.