An AI incident is not limited to a system outage. It can be a plausible but harmful output, unauthorised tool use, confidential-data exposure, uncontrolled cost, repeated business actions or a quality change that silently affects customers.
A useful plan lets the team detect, contain, investigate and recover without waiting for the agent to explain itself.
Define what counts as an AI incident
Create severity levels based on impact: data exposure, unauthorised action, corrupted records, customer harm, regulatory risk, financial loss, unavailable process and loss of monitoring.
Include near misses. A blocked attempt to use an excessive permission is evidence that a control was needed, not a non-event.
Prepare ownership and contact paths
Name an incident lead, technical owner, business owner, security or privacy contact, communications owner and vendor contact. Define who can stop the system and who approves restart.
Keep the runbook available outside the affected AI workflow. Store service identities, dependencies, data flows, escalation paths and rollback instructions.
Detect with business and technical signals
Use production agent monitoring for trace gaps, unauthorised tools, repeated writes, schema failures, cost spikes, latency changes, quality drops and unusual human overrides.
Reports from users and operators need one intake path and a correlation ID. Preserve the original report and time.
Contain before diagnosing
Stop the harmful capability, not necessarily the entire company. Revoke a tool, switch to read-only, queue cases for people, lower a limit or route to a deterministic fallback.
Use an external kill switch. Preserve evidence before changing configurations when that does not prolong harm.
- Acknowledge and assign severity.
- Stop or restrict affected actions.
- Protect accounts, tokens and data.
- Record the last known good configuration.
- Notify required internal owners.
Preserve evidence and assess impact
Save traces, logs, model and prompt versions, tool definitions, permissions, inputs, outputs, approvals, API responses and downstream changes. Restrict access to incident evidence.
Identify which records, users, time period and decisions were affected. Search for the same pattern beyond the first reported case.
Recover through a controlled path
Restore data from authoritative sources, rotate credentials when necessary, deploy a verified configuration and reconcile downstream systems. Use a limited canary or shadow mode before full restart.
Repeat the relevant pre-launch evaluations. Restart requires evidence that the cause is controlled, monitoring works and rollback remains available.
Learn without erasing accountability
Separate triggering event, root cause, control failure and impact. Assign corrective actions with owners and dates. Add the incident pattern to regression tests.
NIST SP 800-61r3 aligns incident response with Detect, Respond and Recover while preparation comes from broader Govern, Identify and Protect activities.
Communicate facts, not speculation
State what is known, affected scope, containment, customer action and next update time. Coordinate contractual, privacy and regulatory notifications with qualified professionals.
Do not let a generated summary become the sole incident record. Human owners approve material communication.
A small incident that improved the process
During a large multilingual API transfer, I found a corrupted UTF-8 character after the server had accepted the draft. I treated it as a data-integrity incident: publication stopped, both affected records stayed in draft, and the stored fields were preserved for comparison.
I changed the transfer method, resent complete bodies, fetched the records again and required zero character differences across all 21 fields before proceeding. The public pages were checked after publication.
The corrective action became a permanent regression rule. The value of the incident was not the manual fix; it was turning the failure into an automated control.
Questions and answers
When should the kill switch be used?
When continued action can increase harm or evidence shows a critical control is unreliable. Define authority and thresholds before an incident.
Should all AI be shut down?
Not automatically. Contain the affected capability and dependencies while preserving safe operations when possible.
What evidence matters most?
Execution traces, versions, permissions, tool calls, inputs, outputs, approvals, API responses and downstream changes.
Who approves restart?
A named business and technical owner according to severity, with security, privacy or legal input where relevant.
What makes recovery complete?
Affected state is reconciled, the cause is controlled, tests pass, monitoring is active, communications are handled and corrective actions are tracked.
Operational worksheet
Before approval, put every control into a worksheet with an owner, evidence, threshold and next review date. A statement such as “monitor quality” is not actionable. “Operations owner reviews the material-correction rate each Monday and pauses the workflow above the agreed threshold” can be tested.
Link every control to a business consequence. If a schema failure can corrupt a customer record, specify containment and reconciliation. If a cost alert only indicates harmless seasonal volume, define the context that prevents false escalation.
Keep versions of prompts, schemas, policies, tools and external dependencies. A result without its configuration cannot be reproduced. Record exceptions and their expiry dates so that temporary workarounds do not become permanent invisible policy.
Finally, rehearse the human path. The named owner should be able to find the evidence, stop or restrict the workflow, move cases to fallback and explain the current state without asking the AI system to diagnose itself.