In one paragraph
An organisation can hold a ratified AI policy, a functioning governance forum, a populated risk register and an ISO/IEC 42001 gap analysis, and still have no idea what changed in its production LLM systems last week. That is the characteristic failure of LLM governance. It is structural, and no amount of effort fixes it. Conventional governance assumes a system whose behaviour changes when somebody deliberately changes it, through a process you can gate. An LLM-based system has at least four independently mutable components: the model version, the prompt, the retrieval corpus and the tool permissions. Each moves on a different cadence, each is owned by a different person, and typically only one of them sits inside anything resembling change control. Govern the policy layer while those four drift underneath it and you get documentation that is accurate about intentions and silent about reality. The operational question is narrower and harder than the policy one. What is the unit of change, who can make it, and how would you know afterwards?
Why LLM governance is not AI governance with different words
Five properties make the difference. Each breaks a control that works elsewhere.
The unit of change is the assembly, and not the model. What a system does is determined by the model version, the system prompt, the contents of the retrieval corpus, and what tools it is allowed to call. Change any one and behaviour changes. Most organisations version the code, sometimes version the prompt, rarely version the corpus, and almost never treat a tool-permission change as a release. The AI lifecycle controls gap is usually this gap specifically.
You do not control the model. It updates on the provider's schedule, and a version you validated can be deprecated with a notice period shorter than your change cycle. That is a governance inversion and not merely a supplier-management inconvenience: the most behaviour-determining component of your system sits outside your change control by design.
Evaluation is not testing. Conventional software testing gives a pass or a fail. LLM evaluation gives a score against a reference set, so somebody has to decide what score is acceptable. That decision is a risk appetite decision, made in most organisations by whoever built the evaluation. Without a threshold agreed in advance, an evaluation suite measures without governing.
Non-determinism breaks the audit trail. "Reproduce the failure" is the first step of most incident processes and frequently impossible here. Governance has to shift from reproducibility to observability. If you cannot re-run it, you must have recorded it.
Agents change the object of governance from output to action. Once a system calls tools, the governed thing is what it did and no longer what it said. That is why excessive agency rose to LLM03 in OWASP's 2026 revision, its largest upward move.
Six disciplines that constitute LLM governance in practice
Inventory assemblies, not systems. Almost every organisation has an AI inventory listing systems: "customer service assistant", "document triage". That is a starting point and not a governable unit, because two deployments of the same system with different tool permissions are different risk objects.
A governable inventory record names the model and version, the prompt version, the retrieval corpora in scope, the tool permissions, the deployment context, and the accountable owner. When any one of those changes, the record changes.
The uncomfortable corollary is that the estate is larger than the inventory. LLM capability arrives embedded in software an organisation already owns: the summarisation feature in the CRM, the drafting assistant in the office suite. None of it came through a procurement process that flagged it as AI. Reconciling the inventory against actual spend and actual data flows usually finds systems nobody had registered, and that reconciliation is a governance control in its own right.
Version everything that changes behaviour. If a change can alter what the system does, it is a release. That means prompts under source control with review, retrieval corpora with a recorded state you can pin an incident to, tool permission changes raised as changes, and model versions recorded per deployment instead of assumed to be "latest".
The test is simple and most organisations fail it. For any output produced last month, can you reconstruct the exact assembly that produced it? If not, incidents cannot be investigated, regulators cannot be answered, and improvements cannot be attributed.
Treat evaluation sets as controlled assets. An evaluation set is the closest thing an LLM system has to a specification, and it deserves the handling: version-controlled, reviewed when it changes, and protected from the failure mode where it quietly gets easier over time because failing cases were removed instead of fixed.
Two properties separate a governance-grade evaluation from a development one. It has thresholds agreed in advance by somebody accountable, and not scores interpreted after the fact. And it covers the failure modes the organisation has decided it cares about, which is a different set from the ones that are easy to measure. Untested AI failure scenarios is largely a description of the gap between those two sets.
Gate deployment on something. The gate is where governance either exists or does not. A defensible one asks four questions and refuses to proceed without answers. What changed in the assembly. What the evaluation says against the agreed threshold. Whether the tool permissions changed and who approved it. Who is accountable for this deployment.
None of that requires a heavyweight process. It requires that the questions get asked every time, including for a prompt change a developer considers trivial. A prompt change is a behaviour change, and treating it as configuration is the commonest route to an ungoverned production system.
Monitor behaviour, not just availability. Conventional monitoring answers whether the system is up. LLM monitoring has to answer whether it is still doing what it did, which means tracking evaluation scores against the production distribution and not the test set, refusal and escalation rates, tool-call patterns, and output characteristics that shift before anybody complains.
Model drift in this context includes drift you did not cause. The provider updated the model, or the corpus grew, and behaviour moved without a change record. Monitoring and logging is the control, and its LLM-specific requirement is capturing the assembled context and the tool calls, and not only the request and the response.
Tier autonomy, and escalate oversight with it. The distinction that matters most for agentic systems is how much a system can do without a human able to intervene. Tier it: informational, advisory, automated, automated-and-consequential. Attach escalating oversight obligations to each tier. A recurring argument becomes a lookup.
The frameworks are visibly still catching up here, and it is worth knowing that. NIST's Generative AI Profile is built around content risks. The Cloud Security Alliance published a community draft Agentic Profile on 27 March 2026 addressing action-consequence mapping, tool-use authorisation, delegation chains and autonomy tiering, and it notes that an official NIST AI Agent Interoperability Profile is expected in Q4 2026.
Firms deploying agents today are ahead of the standards. Waiting is not the governance answer. Write down the autonomy tiering you are operating to, so that when the standard lands you are mapping and not starting.
Where the frameworks actually help
ISO/IEC 42001 supplies the management-system scaffolding: policy, objectives, risk criteria, impact assessment, internal audit, management review. It tells you what functions must exist. It says nothing about what to do with a prompt change, and it is not meant to.
NIST AI 600-1, the Generative AI Profile, published July 2024, is the closest thing to an LLM-specific risk taxonomy from a standards body. Twelve risk categories including confabulation, information integrity, information security, human-AI configuration, and value chain and component integration, each mapped back to AI RMF subcategories with suggested actions. It works well as a completeness check on a risk register, and most registers miss at least three of the twelve.
The OWASP GenAI LLM Top 10 is the threat-side counterpart, and the better starting point for control design.
None of them is an operating model. The gap between "the standard says maintain an AI risk register" and "who approves a prompt change on a Tuesday" is where the work in this article sits, and no framework will close it for you.
The organisational question underneath all of it
LLM governance fails organisationally more often than technically, and in a recognisable pattern. The policy sits with risk or compliance. The assembly sits with engineering. Neither believes the gap is theirs.
The ownership and accountability framework is the control. The specific thing it must resolve is not "who owns AI governance", because that question always has an answer, usually a committee. It is who approves a change to a production assembly, and whether that person can decline. If engineering makes the change and governance is informed afterwards, the programme is documentation.
Two supporting pieces. An acceptable use policy naming what may not be built at all is worth more than one describing how to build carefully, because it is the only control operating before effort is sunk. And unclear ownership tends to be the root cause behind several other register entries at once, so it is worth checking whether your top three risks share it.
A starting position
For an organisation with a policy and no operating model, in order:
- Reconcile the inventory against reality, meaning spend, data flows, and embedded features in software you already own. Expect to find systems nobody registered.
- Pick your three highest-consequence assemblies and write down, for each, the model version, prompt version, corpora and tool permissions in production today. If that takes more than an hour per system, that is the finding.
- Put prompts under source control, if they are not already. Cheapest meaningful control available.
- Agree one evaluation threshold with somebody accountable, for one system. The precedent matters more than the number.
- Define the deployment gate's four questions and apply them to the next change, however small.
- Write down your autonomy tiering before the standards make you.
The governance pillar covers the policy layer this sits underneath. The engineering section covers the production concerns it sits alongside. The join between them is where most programmes have nothing at all.