For Heads-of · Practitioner
Reinforcement Learning from Human Feedback (RLHF)
A training method in which people rate a model's outputs, a reward model learns to predict those ratings, and the model is then fine-tuned to produce outputs the reward model scores highly. It is the main technique that turned raw language models into usable assistants.
- RLHF
- reinforcement learning
- alignment
- Constitutional AI
- training
How it works
People compare pairs of model responses and pick the better one. A separate reward model is trained on those preferences, and the language model is then optimised against it with reinforcement learning. The method comes from 2017 research and was applied at scale to language models in OpenAI's 2022 InstructGPT work. Variants replace some human feedback with AI feedback, such as Anthropic's Constitutional AI, which trains against a written set of principles.
Its side effects
Because the model learns to produce what raters prefer, it can learn to please rather than to be right. Sycophancy and confident-sounding errors are known consequences, and optimising hard against an imperfect reward model invites reward hacking. The raters' own preferences and biases also shape the result.
Why it matters to governance
RLHF is where much of a model's behaviour, including its refusals and its tone, is set, and that behaviour can shift between versions as the training changes. That is one reason to re-evaluate on every model update; the register entry for behaviour that shifts unnoticed is AI model drift and degradation.
Sources
Related Terms
AI Alignment
The property of an AI system pursuing the goal its designers actually intended, rather than a proxy that happened to score well during training. Alignment is not the same as capability (a highly capable system can be confidently, efficiently misaligned) and it is not the same as AI safety, which is the broader field alignment sits inside.
Fine-Tuning
The process of adapting a pre-trained model to a specific task or domain by training it further on a smaller, task-specific dataset.
Large Language Model (LLM)
A deep learning model trained on very large amounts of text, able to generate and interpret human-like language across a wide range of topics and tasks.
Reward Hacking
A system meeting the literal objective it was trained or rewarded on in a way its designers did not intend, such as gaming a test rather than solving the task. Recent research shows the habit can generalise into broader misbehaviour.
Sycophancy
A model's tendency to tell users what they want to hear, agreeing, flattering or validating at the expense of accuracy. It is a side effect of training on human approval, and in customer-facing financial services it is a conduct risk as well as a quality one.