Gary Cheung · 6 October 2025 · 5-minute read
At a glance
- The risk: A warning can arrive while consequential AI decisions continue because nobody has tested who can intervene.
- The governance implication: Rehearse authority, evidence, fallback capacity, and communication before deployment and material change.
- The leadership takeaway: Test whether the team can make and execute a defensible pause decision while facts remain incomplete.
When an AI-assisted lending process starts rejecting applicants unexpectedly, the immediate business decision is whether to continue, restrict its use, or pause it. Waiting for a complete diagnosis may allow further harm; stopping without a workable fallback may create a different customer and operational exposure.
An AI incident response tabletop exercise should rehearse that trade-off before a real incident. Its value is in exposing whether the accountable business owner, technical team, risk function, and legal advisers can turn an uncertain signal into an authorized action.
Rehearse the decision that cannot wait
Monitoring tells a team that something may be wrong. An AI escalation playbook describes a response. Neither establishes that the people involved can use it when the owner is unavailable, evidence conflicts, or stopping affects customers.
NIST AI RMF 1.0 addresses responsibility for disengaging unsuitable AI systems and documented incident response and communication. It is voluntary U.S. guidance, released on 26 January 2023; NIST currently identifies a revision in progress. It does not impose a universal reporting deadline.
My governance judgment is that the rehearsal should precede consequential use and be revisited after material changes to the model, workflow, vendor, or decision authority. A policy approved after launch cannot resolve an authority gap during the first incident.
Build the scenario around exposure and incomplete evidence
Illustrative scenario: a lender receives complaints about unusual application rejections following an input-data change. Aggregate performance looks stable, while outcomes for one applicant group have shifted. The cause and affected population are still uncertain.
Give participants successive updates: a vendor disputes the concern, more applications enter the queue, and manual review capacity is limited. Ask what they would decide at each point and what information would change that decision.
A generic accuracy threshold such as 90% is insufficient. The trigger should reflect the decision, affected customers, error consequences, and time available to contain harm. Test the cost of continuing alongside the cost and capacity of the fallback.
Zillow’s 2021 SEC disclosure records a $407.9 million inventory write-down for the year ended 31 December 2021 and describes pricing unpredictability and operational constraints. It does not establish an AI-only failure or prove that rehearsal would have prevented the loss. The useful exercise question is who may halt further commitments when the evidence supporting them weakens.
Separate authority to act from authority to advise
Assign the business owner responsibility for the outcome, a technical lead for investigation and containment, risk for independent challenge, and legal or compliance for assessing applicable obligations. Define a deputy and an urgent containment mandate so absence does not leave the process running by default.
Rehearse three linked decisions: restrict or pause use; manage affected customers and fallback operations; authorize restart. Restart should require evidence appropriate to the exposure, an accountable decision, and a monitoring plan. Retraining alone is not a recovery decision.
Use clearly marked exercise channels and simulated communications. Test who receives the same facts, who approves external statements, and how jurisdiction-specific notification requirements are assessed. Do not send exercise messages to actual customers or regulators.
Turn the debrief into evidence of control
NIST SP 800-84, published in September 2006, provides general IT exercise guidance. A tabletop is discussion-based; it can reveal gaps in roles and plans but does not prove that a technical shutdown or recovery mechanism works.
Record the signal received, information available, decision made, decision-maker, elapsed time, and unresolved disagreement. Separate an authority failure from a missing data feed, unavailable fallback, or technical control failure; each needs a different remedy.
Give every material gap an owner, due date, and retest criterion. Where safe and appropriate, separately test the actual containment and recovery mechanisms. Attendance and a completed scenario are weak evidence if the same pause decision would still be blocked tomorrow.
Questions for Leaders
- Who can restrict or pause this AI-assisted decision when the accountable owner is unavailable, and who can authorize restart?
- What evidence is sufficient to act before the cause is fully understood, and how quickly could customers be harmed?
- Can the fallback handle the workload, and which exercise gaps must close before launch or expanded use?
Closing Thought
The purpose of rehearsal is to discover where accountability stops working under pressure. A useful exercise leaves leaders with clearer authority, executable response options, and evidence of what still needs fixing before the next consequential decision.
Sources and Further Reading
- NIST AI RMF 1.0 — voluntary guidance, released 26 January 2023; revision in progress. Core: MANAGE 2.4, 4.1 and 4.3.
- NIST SP 800-84 — final general IT test, training and exercise guidance, September 2006; not an AI-specific legal obligation.
- Zillow Group 2021 Form 10-K: Restructuring and Zillow Offers Wind Down — year ended 31 December 2021.
- Building an AI Escalation Playbook and AI Model Monitoring 101.
About Gary Cheung
Gary Cheung writes about AI governance, information risk, and accountable technology decisions.
To discuss an AI governance challenge, share the decision context here.
