If you run anything “AI-powered” in production, you already know the truth: fancy models do not exempt you from ugly outages. They just create new and more confusing ones.

Maybe your chatbot starts hallucinating refunds. Maybe your claims triage LLM silently routes everything to “manual review” and your backlog explodes. Or maybe, like the recent Cloudflare incident that knocked ChatGPT and other services offline, an upstream provider change suddenly takes your AI stack down with it.Cloudflare resolves outage that impacted thousands, ChatGPT, X and more

In traditional security, you would never ship a product without an incident response plan. But many teams are rolling out ChatGPT, Claude, Gemini, or bespoke models with no clear answer to the basic question: “What do we do when this thing misbehaves?”

This is where AI incident response comes in. It is not just “normal incident response but with a robot.” It is a structured way to detect, contain, and learn from failures that have model behavior, data, and humans entangled in messy ways. The good news: you can lean on existing frameworks like NIST’s AI Risk Management Framework (AI RMF) and incident playbooks, and adapt what already works for security and model risk.NIST AI RMF Core and incident response

Below is a practical, vendor-agnostic guide to what you should do when your AI systems fail — and how to be ready before they do.

What counts as an “AI incident”?

You cannot respond to incidents you do not define.

Regulators and standards bodies increasingly treat AI incidents as any event where an AI system causes, or could reasonably cause, harm, significant service impact, or a breach of policy or law. The NIST AI RMF explicitly calls out “AI incidents, risky emergent behavior, and adversarial events” as things you need monitoring, escalation, and response mechanisms for.NIST AI RMF Playbook

For most organizations, that means you treat as incidents when:

  • The AI is unavailable or degraded (e.g., your ChatGPT- or Gemini-powered workflow is failing 30% of calls).
  • The AI produces harmful or policy-violating outputs (biased decisions, toxic content, leaking confidential data).
  • The AI is compromised or misused (prompt injection, jailbreaking, agents executing unintended actions).
  • The AI behaves in a way you cannot explain or control that impacts users or business processes.

Think of AI incidents as a superset of model bugs, outages, and safety failures. The key is that they demand a coordinated response, not just a quiet model tweak.

Borrow a spine: frameworks that already exist

You do not have to invent an AI incident response philosophy from scratch. A lot of the backbone already exists:

  • NIST AI RMF and Playbook – NIST’s framework gives you high-level “Govern, Map, Measure, Manage” functions. The Playbook goes further with concrete actions around post-deployment monitoring, incident capture, response, and recovery.NIST AI RMF Playbook PDF
  • NIST incident response process – The classic security lifecycle (Prepare → Detect → Analyze → Contain → Eradicate → Recover → Post-incident) maps surprisingly well to AI failures.NIST incident response phases overview
  • Model risk management – Banking and insurance regulators have published detailed guidance on model risk governance, including incident-style controls for model failures and misuse.US Supervisory Guidance on Model Risk Management

You can treat these like Lego: use NIST’s incident phases as your skeleton, overlay AI-specific controls from the AI RMF, and plug in your organization’s existing security and operational playbooks.

Step 1: Prepare before anything breaks

When an AI system fails, you will not have time to negotiate basics like “Who owns this thing?” Preparation is where most teams are currently weakest.

At a minimum, you should:

  • Inventory AI systems and owners
    For every AI system or agent (internal or vendor), document:

    • Business purpose and criticality.
    • Model provider (e.g., OpenAI, Anthropic, Google, self-hosted).
    • Data it can access or generate.
    • System and incident owner (a real name, not just “the platform team”).

  • Define AI-specific severity levels
    Severity should consider not only uptime, but also:

    • Potential harm to individuals (wrong medical advice, unfair denials).
    • Regulatory exposure (personal data, biometric data, credit decisions).
    • Downstream blast radius (does this model feed other systems?).

    Many organizations now maintain a two-axis taxonomy: one for technical impact (like a normal outage), and one for human/ethical impact (harm, bias, rights).Example AI incident response playbook

  • Create an AI incident runbook
    Start by duplicating your existing major-incident template, then sprinkle in AI-specific extras:

    • How to “kill switch” an AI feature (disable a workflow, switch to a backup provider like Claude or Gemini, or fall back to manual).
    • How to revoke access for non-human identities (API keys, service accounts, agents).NIST workshop talk: IR preparation for AI systems
    • How to snapshot logs, prompts, and outputs for later forensic analysis.
    • Who talks to customers, legal, and regulators if personal data or safety is involved.

This prep step is the most boring, but it is what determines whether tomorrow’s incident becomes a contained hiccup or a front-page problem.

Step 2: Detect faster and take reports seriously

The hardest part of AI incident response is knowing when something is wrong. Traditional monitoring (CPU, latency, 500 errors) will catch outages like the OpenAI GPU network issue that caused elevated error rates across ChatGPT and the API.OpenAI incident write-up: elevated error rates

But that will not catch “the model suddenly started approving high-risk payments.”

You need layered detection:

  • Technical signals

    • Provider status pages (OpenAI, Anthropic, Google, Microsoft).
    • Timeouts, elevated error rates, input/output token anomalies.
    • Sudden shifts in usage patterns (e.g., agents spamming external APIs).

  • Behavioral and quality signals

    • Sampling or automated evaluation of model outputs.
    • Guardrail systems that flag policy violations (toxic content, PII leaks).
    • Human-in-the-loop reviewers for high-impact decisions.

  • User and employee reports

    • A clear way for people to say “this AI behaved badly” (within the app, in Slack, or a simple form).
    • A commitment to treat those reports like security reports: triage, log, and close the loop, not “we will look into it.”

If you are using AI agents that can act (change configs, send emails, modify infrastructure), detection also needs controls on what they can do and alerts when they step outside expected patterns.

Step 3: Analyze the incident like both an engineer and a risk manager

Once you have detected something, your first question is: “Is this incident-class, or just a bug?”

Early analysis should answer:

  1. Is there active harm or high risk?
    If yes, jump straight to containment. You can do deep root cause analysis later.

  2. Is the failure systemic or localized?

    • One user vs. many users.
    • One prompt pattern vs. all inputs.
    • One provider or region vs. your whole stack.

  3. Is this a model issue, data issue, integration issue, or upstream provider issue?

    • Model behavior: hallucinations, bias, jailbreaking.
    • Data: wrong or stale training/in-context data.
    • Integration: incorrect prompts, mis-specified tools, bad business logic.
    • Infrastructure: the Cloudflare-style network or provider outage that just happens to hit your AI stack.Cloudflare outage post-mortem

Here, basic forensics matter more than clever prompts. Capture logs, prompts, outputs, and any relevant context. If you are using tools like ChatGPT, Claude, or Gemini behind the scenes, you may also need to pull vendor logs or status history.

Step 4: Contain quickly, then communicate

Containment is where you protect people and your business, even before you completely understand the “why.”

Typical AI-specific containment moves include:

  • Disable or degrade the AI feature

    • Turn off the problematic feature.
    • Force a fallback path: e.g., route users to human support or a non-AI flow.
    • Switch to a backup provider if you have multi-model routing.

  • Reduce the AI’s blast radius

    • Restrict what the AI can see (sensitive data, tools, systems).
    • Disable high-risk tools for agents (like code execution, email sending, or payment APIs) until you understand what went wrong.

  • Communicate with the right people

    • Internally: incident channel, on-call responders, leadership for severe cases.
    • Externally: status page, customer announcements if users are impacted.
    • Regulators or data protection authorities if personal data or high-stakes decisions are implicated, in line with guidance like the UK ICO’s AI and data protection resources.ICO guidance on AI and data protection

The rule of thumb: be at least as transparent about AI failures as you are about normal outages. Users will forgive downtime more easily than they will forgive silent, unexplained AI behavior that affects them personally.

Step 5: Eradicate the root cause and harden the system

Once the fire is out, you need to remove the underlying issue and prevent easy recurrence.

Depending on the incident, that might mean:

  • Fixing prompts and guardrails

    • Tightening system prompts and policies.
    • Strengthening filters and safety layers.
    • Adding evaluations for edge cases that surfaced during the incident.

  • Adjusting data and access

    • Correcting or removing bad reference data.
    • Reducing which tables, indices, or tools the AI can touch.
    • Rotating API keys and revoking tokens if any compromise is suspected.

  • Changing architecture

    • Adding a second model to cross-check critical outputs.
    • Inserting human review for high-risk actions (approving loans, sending legal emails).
    • Adding circuit breakers or feature flags so you can disable AI slices quickly next time.

Think of eradication as refactoring your AI system for resilience, not just making the current error disappear.

Step 6: Recover, learn, and update your playbook

Recovery is not just bringing the AI back online. It is also about rebuilding trust and increasing your maturity.

After the incident:

  • Restore safely

    • Re-enable features in stages, with extra monitoring.
    • Validate with internal stakeholders and, where possible, a small canary group of users.

  • Run an AI-aware post-mortem Ask and document:

    • What did the system do, and who did it affect?
    • How quickly did we detect and contain it?
    • Which decisions needed humans in the loop that were not there?
    • How will we change prompts, tests, architecture, and governance?

  • Feed back into governance The NIST AI RMF explicitly treats incidents as inputs to governance and continuous improvement.NIST AI RMF Core and incident response
    Use the post-mortem to:

    • Update your AI risk register and controls.
    • Adjust your incident severity definitions if they did not match reality.
    • Add concrete “play” entries to your AI incident playbook, so the next on-call responder has something better than a blank page.

Done right, every AI incident makes your overall stack — models, people, and processes — meaningfully stronger.

Putting it all together: get ready before the next failure

AI outages and misbehaviors are not speculative; they are now routine. Public incidents affecting tools like ChatGPT and other LLM-based services have shown how quickly a configuration change or model quirk can cascade across thousands of users and dependent apps.Impactful outages including AI services

If you are using ChatGPT, Claude, Gemini, or any other AI system in a production workflow, you should assume:

  • It will go down.
  • It will misbehave.
  • It will surprise you at the worst possible time.

To be ready, your concrete next steps are:

  1. Draft a one-page AI incident response addendum to your existing incident plan this week. Name owners for each AI system, define what counts as an AI incident for you, and document at least one kill switch or fallback per system.
  2. Set up minimal AI incident detection: alerts on provider outages and error rates, plus a simple path for employees and users to report weird AI behavior that automatically opens a ticket.
  3. Run a tabletop exercise next month: simulate “our LLM starts leaking confidential data” or “our AI agent misconfigures a cloud resource,” walk through your response, and refine your playbook based on what breaks.

If you treat AI incidents as first-class citizens in your operational and risk world, you can ship ambitious AI features without turning every outage or hallucination into a crisis.