For Heads-of · Practitioner
Emergent Capabilities
Abilities that appear in larger AI models but were not explicitly trained for and were not evident in smaller versions. Whether they genuinely arrive suddenly with scale, or only look sudden because of how they are measured, is disputed.
- emergent capabilities
- scaling
- evaluation
- model behaviour
Examples
In-context learning (learning a task from examples in the prompt without further training), multi-step reasoning, and translation between languages rarely paired in the training data.
A disputed claim
Early research reported abilities appearing abruptly once models crossed a scale threshold. Later work (Schaeffer, Miranda and Koyejo, 2023) showed that many of these jumps disappear when performance is measured on a continuous scale rather than as pass or fail, suggesting the ability improved gradually and the metric made it look sudden. Both sides agree that a larger model can do things its predecessor visibly could not.
Why it matters for governance
Whether emergence is real or an artefact of measurement, the practical point holds: a new model version may have capabilities, and failure modes, that testing of the previous version did not reveal. Every model upgrade, including a vendor's silent one, needs its own evaluation rather than inheriting the last one's results; skipping it leaves untested AI failure scenarios.
Sources
Related Terms
AI Alignment
The property of an AI system pursuing the goal its designers actually intended, rather than a proxy that happened to score well during training. Alignment is not the same as capability (a highly capable system can be confidently, efficiently misaligned) and it is not the same as AI safety, which is the broader field alignment sits inside.
AI Evaluation
Systematic measurement of what an AI model or system can do, how it tends to behave and how well its safeguards hold, using benchmarks, task-based tests, red teaming or studies with people. Evaluations are the evidence most AI claims rest on, and they have known weaknesses.
AI Safety
The field of research and practice concerned with making AI systems behave as intended, avoid unintended harm and stay under meaningful human control. In 2025 the UK and US government institutes renamed themselves around security, which narrowed the official focus without removing the questions an organisation deploying AI has to answer.
Large Language Model (LLM)
A deep learning model trained on very large amounts of text, able to generate and interpret human-like language across a wide range of topics and tasks.
Testing AI Systems
Systematic evaluation of AI models and systems to verify that they perform as intended, identify failure modes and assess risks.
Related risks and controls
Untested AI failure scenarios
A capability nobody tested for is a failure scenario nobody wrote down.
Excessive agency in autonomous agents
The setting where an unanticipated capability does the most damage: an agent with the tools to act on it.
Adversarial robustness testing
Where capabilities the vendor did not advertise get found before deployment rather than after.