How to Write an AI Incident Response Plan Before You Need It
An AI incident response plan you write after the failure is a postmortem. Here is how to build the runbook that contains the damage while it is happening.
Write your AI incident response plan on a calm afternoon, not during the fire. An incident is a model that produced a harmful output, leaked data, made an unauthorized decision, or degraded in a way your customers noticed before you did. The plan is the difference between a two-hour containment and a two-week trust collapse. Most teams have a security incident plan and assume it covers AI. It does not. AI failures are probabilistic, hard to reproduce, and often invisible in your normal error logs. You need a runbook built for that shape of failure.
What goes in an AI incident response plan?
Five things, in order: detect, triage, contain, communicate, learn. Each needs an owner and a defined trigger before the incident, because in the moment nobody reads a wiki.
Detection is the part people skip. A crashed server pages you. A model that quietly starts giving wrong answers does not. You need signals: output quality monitors, user-flag rates, confidence-distribution shifts, and a way for a human to raise a flag. Pair this with a monitoring agent that catches silent workflow failures, because the worst AI incidents are the ones no dashboard turns red for.
Triage means deciding severity fast. Is this one bad answer or a systemic regression. Is data exposed. Is a regulated decision affected. Write the severity levels down now, with examples, so the on-call person is matching a symptom to a level instead of inventing policy under stress.
How to contain an AI incident fast
Containment is where your architecture pays off or fails you. If you cannot turn the model off, you cannot contain anything. So the first requirement is a kill switch for every AI agent and a documented path to it. Not a code deploy. A flag someone can flip in under a minute.
The second requirement is rollback. If you pinned the model version in production instead of floating on the latest, you can revert to a known-good state instead of debugging a moving target. Floating versions turn every incident into a mystery.
The third is a fallback path. The best containment is not "system down." It is falling back to deterministic behavior when the AI is unsure or to a human review queue. Degraded but safe beats offline, and it beats confidently wrong.
Who does what when the AI fails
Assign roles before you need them. In a small company one person may wear several hats, but the hats still need names.
- Incident commander. Owns the response, makes the containment call, and is the single decision point. Not a committee.
- Communicator. Owns messaging to customers, support, and if needed regulators. Never let engineers freelance customer comms during an incident.
- Investigator. Reconstructs what happened from the audit trail. This only works if you already have data lineage tracing every output to its source.
- Scribe. Timestamps every action. The timeline is your defense later and your postmortem material.
Settle in advance the question of who is accountable when the AI is wrong. During an incident is the worst time to discover nobody owns the outcome.
The runbook itself
Keep it short and executable. A page, not a binder. For each severity level: the trigger, the first three actions, who to page, and what to tell customers. Include the exact commands or console clicks to hit the kill switch and to roll back. Include the contact for legal and for your biggest affected customers. Rehearse it. Run a tabletop drill twice a year where you inject a fake incident and time yourselves. The first drill will be embarrassing. That is the point of doing it in private.
The reason this matters beyond safety is trust. Buyers who have watched vendors fumble an outage will ask how you handle incidents, and a real plan is a sales asset. My venture Girard AI treats the incident runbook as part of the product, not a side document, because the customers who care most are the ones worth having. Once the fire is out, the real work is rebuilding trust after the failure, and a clean, well-documented response is what makes that recovery possible instead of terminal.