Governance decides what should happen. Engineering is where it actually does.
Governance, Compliance, and Security set intent: what's approved, what's required, what's defended against. Engineering is the first-line discipline that turns that intent into a system that behaves the way it's supposed to — architecture, integration, evaluation, monitoring, and incident response for AI systems actually running in production, not the ones still on a slide.
Last reviewed: 2026-07-29
What belongs on this page
Architecture and integration patterns
Where the instruction/data boundary sits, how tool access is scoped, and what a system is actually permitted to do before anyone asks whether it should.
Evaluation and production monitoring
Pre-deployment evaluation is a gate, not a milestone; production monitoring watches drift and output quality, not only uptime and latency.
Lifecycle and incident response
A rollback plan and an AI-specific incident runbook, defined before go-live, not improvised the first time a model behaves the way generic IT incident response wasn't built to handle.
Six control points, from design to retirement.
A working AI system's lifecycle, not a request-time flow — see Security for how a single request gets defended. This is where the system itself gets designed, evaluated, shipped, watched, and eventually retired.
| Stage | Control point | What good looks like | Evidence |
|---|---|---|---|
| Architecture & integration | Instruction/data boundary design | Untrusted content (retrieved documents, tool output, user input) is structurally separated from system instructions at design time, not filtered as an afterthought. | Architecture decision records, data-flow diagrams, tool-permission scoping |
| Pre-deployment evaluation | Golden-set testing and red-team sign-off | A defined evaluation set, adversarial testing against it, and an explicit go/no-go decision — before the system reaches production, not during its first incident. | Evaluation reports, red-team sign-off, documented go/no-go decision |
| Human-in-the-loop operating design | Approval gates built into the pipeline | Checkpoints for irreversible or high-consequence actions sit structurally in the system, with a named owner and an escalation path, not just a policy describing intent. | HITL design documentation, approval-gate configuration, escalation matrix |
| Production monitoring | Drift and output-quality telemetry | Model drift, cost, latency, and a sampled measure of output quality are all tracked, with alert thresholds defined before the first alert fires. | Telemetry dashboards, drift alerts, output-quality sampling logs |
| Incident response | AI-specific runbook and kill-switch | A rollback path and containment procedure specific to AI failure modes — not the generic IT incident process applied to a system it was never designed for. | AI incident runbook, rollback-drill records, post-incident reviews |
| Lifecycle & retirement | Version and model deprecation | Retiring a model or agent version is a defined procedure — including what happens to its data, logs, and any standing permissions — not a silent removal. | Deprecation register, retirement checklist, data-retention sign-off |
What a delivery review expects to see.
Evidence that the lifecycle above was actually followed, not general assurances about engineering discipline.
- Architecture decision records showing the instruction/data boundary design, not just the finished diagram.
- A rollback plan and kill-switch defined before go-live, not improvised during an incident.
- Production monitoring that samples output quality, not only uptime and latency.
- An AI-specific incident runbook, distinct from generic IT incident response.
Long-form analysis on this
Articles that work through the practice rather than catalogue it.
Deterministic Work Belongs Either Side of the Model Call
There is a reflex that sends the whole problem to the model, and a correction that tries to take work away from it. Both fail, and working through a real standard shows exactly where each one breaks. The deterministic work is real and it belongs on either side of the call rather than in place of it.
Model Drift, Performance Regression and Versioning: Telling Them Apart
Three terms, three different things, and conflating them means monitoring the wrong one. Drift is the world moving. Regression is a measurement getting worse. Versioning is the artefact changing. For most LLM deployments the dominant cause of behaviour change is none of the classic ones. It is the model provider shipping an update you were not told about.
LLM Governance: What It Actually Means Operationally
Most AI governance programmes produce a policy, a committee and a register, then discover none of it touches the thing that actually changes: the assembly of model version, prompt, retrieval corpus and tool permissions that determines what the system does. LLM governance is an operations problem wearing a policy costume.
Prompt Injection and System Prompt Hardening: What Actually Works
System prompt hardening is worth doing and will not save you. The achievable goal is not a model that cannot be tricked, since natural language offers no way to build that. It is a system where being tricked does not matter much. OWASP's 2026 revision makes the same point: the biggest mover was not the attack, it was what the model was allowed to do afterwards.
Risks and controls in this domain.
Pulled live from the Risk Library and Control Library — every entry tagged Engineering in the full Risk & Control spine, not a hand-maintained duplicate.
8 risks
- Adversarial evasion attacks high
- Adversarial testing coverage gap medium
- AI model drift and degradation high
- AI not mapped to important business services high
- Model backdoors and trojans critical
- Model extraction and inference-API abuse high
- No impact tolerance for AI-dependent services high
- Untested AI failure scenarios medium
Frequently asked questions
What is the instruction/data boundary in an AI system?
The structural separation between system instructions and untrusted content — retrieved documents, tool output, user input. Designed at architecture time, not filtered as an afterthought, it's the single control point most AI security incidents trace back to skipping.
What should pre-deployment evaluation for an AI system include?
A defined evaluation set, adversarial (red-team) testing against it, and an explicit, documented go/no-go decision — completed before the system reaches production, not discovered as a gap during its first incident.
Where should human-in-the-loop checkpoints sit in an AI system?
Structurally in the pipeline, for irreversible or high-consequence actions specifically, with a named owner and a defined escalation path — not described only in a policy document that the running system has no mechanism to enforce.
What should AI production monitoring track beyond uptime?
Model drift, cost, latency, and a sampled measure of output quality, each with alert thresholds defined before the first alert fires — uptime and latency alone miss the failure modes specific to AI systems, like gradual quality degradation.
Why does AI incident response need its own runbook?
Because generic IT incident response wasn't built for AI-specific failure modes — a rollback path and containment procedure for a misbehaving model or agent is different in kind from restarting a service, and needs a runbook, drills, and a kill-switch defined ahead of time.
This connects directly to one of the areas Andrew advises on.
Third-Party & Operational Resilience Risk is one of five areas of expertise on the services page — the risk lens on exactly the lifecycle and monitoring discipline this page describes, grounded in SS1/21 and SS2/21.