Governance decides what should happen. Engineering is where it actually does.
Governance, Compliance, and Security set intent: what's approved, what's required, what's defended against. Engineering is the first-line discipline that turns that intent into a system that behaves the way it's supposed to — architecture, integration, evaluation, monitoring, and incident response for AI systems actually running in production, not the ones still on a slide.
Last reviewed: 2026-07-29
What belongs on this page
Architecture and integration patterns
Where the instruction/data boundary sits, how tool access is scoped, and what a system is actually permitted to do before anyone asks whether it should.
Evaluation and production monitoring
Pre-deployment evaluation is a gate, not a milestone; production monitoring watches drift and output quality, not only uptime and latency.
Lifecycle and incident response
A rollback plan and an AI-specific incident runbook, defined before go-live, not improvised the first time a model behaves the way generic IT incident response wasn't built to handle.
Six control points, from design to retirement.
A working AI system's lifecycle, not a request-time flow — see Security for how a single request gets defended. This is where the system itself gets designed, evaluated, shipped, watched, and eventually retired.
| Stage | Control point | What good looks like | Evidence |
|---|---|---|---|
| Architecture & integration | Instruction/data boundary design | Untrusted content (retrieved documents, tool output, user input) is structurally separated from system instructions at design time, not filtered as an afterthought. | Architecture decision records, data-flow diagrams, tool-permission scoping |
| Pre-deployment evaluation | Golden-set testing and red-team sign-off | A defined evaluation set, adversarial testing against it, and an explicit go/no-go decision — before the system reaches production, not during its first incident. | Evaluation reports, red-team sign-off, documented go/no-go decision |
| Human-in-the-loop operating design | Approval gates built into the pipeline | Checkpoints for irreversible or high-consequence actions sit structurally in the system, with a named owner and an escalation path, not just a policy describing intent. | HITL design documentation, approval-gate configuration, escalation matrix |
| Production monitoring | Drift and output-quality telemetry | Model drift, cost, latency, and a sampled measure of output quality are all tracked, with alert thresholds defined before the first alert fires. | Telemetry dashboards, drift alerts, output-quality sampling logs |
| Incident response | AI-specific runbook and kill-switch | A rollback path and containment procedure specific to AI failure modes — not the generic IT incident process applied to a system it was never designed for. | AI incident runbook, rollback-drill records, post-incident reviews |
| Lifecycle & retirement | Version and model deprecation | Retiring a model or agent version is a defined procedure — including what happens to its data, logs, and any standing permissions — not a silent removal. | Deprecation register, retirement checklist, data-retention sign-off |
What a delivery review expects to see.
Evidence that the lifecycle above was actually followed, not general assurances about engineering discipline.
- Architecture decision records showing the instruction/data boundary design, not just the finished diagram.
- A rollback plan and kill-switch defined before go-live, not improvised during an incident.
- Production monitoring that samples output quality, not only uptime and latency.
- An AI-specific incident runbook, distinct from generic IT incident response.
Risks and controls in this domain.
Pulled live from the Risk Library and Control Library — every entry tagged Engineering in the full Risk & Control spine, not a hand-maintained duplicate.
8 risks
- Adversarial evasion attacks high
- Adversarial testing coverage gap medium
- AI model drift and degradation high
- AI not mapped to important business services high
- Model backdoors and trojans critical
- Model extraction and inference-API abuse high
- No impact tolerance for AI-dependent services high
- Untested AI failure scenarios medium
Frequently asked questions
What is the instruction/data boundary in an AI system?
The structural separation between system instructions and untrusted content — retrieved documents, tool output, user input. Designed at architecture time, not filtered as an afterthought, it's the single control point most AI security incidents trace back to skipping.
What should pre-deployment evaluation for an AI system include?
A defined evaluation set, adversarial (red-team) testing against it, and an explicit, documented go/no-go decision — completed before the system reaches production, not discovered as a gap during its first incident.
Where should human-in-the-loop checkpoints sit in an AI system?
Structurally in the pipeline, for irreversible or high-consequence actions specifically, with a named owner and a defined escalation path — not described only in a policy document that the running system has no mechanism to enforce.
What should AI production monitoring track beyond uptime?
Model drift, cost, latency, and a sampled measure of output quality, each with alert thresholds defined before the first alert fires — uptime and latency alone miss the failure modes specific to AI systems, like gradual quality degradation.
Why does AI incident response need its own runbook?
Because generic IT incident response wasn't built for AI-specific failure modes — a rollback path and containment procedure for a misbehaving model or agent is different in kind from restarting a service, and needs a runbook, drills, and a kill-switch defined ahead of time.
This connects directly to one of the areas Andrew advises on.
Third-Party & Operational Resilience Risk is one of five areas of expertise on the services page — the risk lens on exactly the lifecycle and monitoring discipline this page describes, grounded in SS1/21 and SS2/21.