Gary Cheung · Published 26 August 2025 · 4 min read
At a glance
- The risk: A completed checklist can conceal changing business exposure and unclear decision ownership.
- The governance implication: Monitoring needs a named owner, escalation triggers, and authority to change or stop the process.
- The leadership takeaway: Test whether people can raise concerns and whether those concerns change operating decisions.
Consider an illustrative lending decision: an AI output helps determine which customers receive credit. Passing validation checks matters, but the business also needs to decide whether it can keep relying on that output as customers, data, and workflows change. Who owns the outcome, and who can challenge or stop the process when evidence calls that reliance into question?
An AI risk management culture connects control evidence to those decisions. Checklists provide useful discipline. Resilience also requires people to recognize concerns, escalate them, and act before exposure grows.
A passed checklist does not settle the operating decision
A checklist records whether a control was performed against specified criteria. It cannot, by itself, establish that those criteria still fit the business use. A model can remain technically unchanged while the customer population, decision threshold, or downstream process changes.
The accountability gap appears when validation, compliance, and business teams each assume that another function owns continued use. A completed review then becomes permission by default, even when nobody has explicitly accepted the changed exposure.
NIST’s U.S. AI Risk Management Framework 1.0, released on 26 January 2023, is voluntary guidance rather than a binding compliance standard. It supports risk management across design, development, use, and evaluation. My view is that leaders should treat review findings as inputs to an accountable operating decision, rather than a substitute for one.
Zillow shows why the business exposure matters
Zillow’s 2021 exit from its Offers home-buying business illustrates how forecasting risk can combine with market, operational, and execution risk. Its 2021 Form 10-K reported a $407.9 million inventory write-down for the year ended 31 December 2021 and described home-price unpredictability, capacity constraints, and operational challenges.
The filing does not establish that an AI model alone caused the losses or that a checklist cadence caused the outcome. It should not be presented as proof of either claim.
The governance lesson I draw is narrower: the owner of a consequential business use needs timely signals about its assumptions and exposure, plus authority to pause or change it. Model performance and business resilience are related, but they are not interchangeable.
Make escalation usable before it is needed
Risk conversations need to connect technical teams, risk specialists, and the business owner. Staff should know which anomalies warrant investigation, where to raise them, and who must respond. A reporting channel has limited value if concerns repeatedly stall without a decision.
For the illustrative lender, a rise in disputed decisions or a material change in the applicant population could trigger review. The trigger should identify an owner, response time, and available actions, such as additional human review or temporary restriction of the affected use.
Monitoring frequency should reflect consequence, speed of change, and the time available to intervene. Human oversight needs to be available when judgment is required; a universal quarterly review or continuous manual checking will not suit every use.
A risk register can connect each signal to an owner and escalation route. The human oversight layer then needs sufficient information and authority to act.
Let changed conditions reopen the decision
Checklist thinking asks whether a review was completed. Risk culture thinking also asks what has changed since approval, whether the original limits still hold, and what evidence would justify continuing, restricting, or stopping the use.
Incidents and near misses should feed back into controls, training, and operating decisions. Record what was learned, who accepted any residual risk, and whether the revised control actually addresses the failure. Learning remains incomplete when the same concern recurs without changed behavior.
Leaders should assess whether alerts reach someone with authority, whether challenges receive timely decisions, and whether teams can use a practical fallback. Completion rates alone do not show this. A risk-aware culture cannot prevent every failure; it improves the chance that emerging problems become visible and receive an accountable response.
Leadership Questions
- Who owns the business outcome, and who can stop or change the AI-enabled process when evidence challenges its assumptions?
- Do monitoring triggers reach a named owner with the information, authority, and response time needed to act?
- Which recent incident or near miss changed a control, an approval condition, or an operating decision?
Closing Thought
Resilience depends on how quickly an organization can recognize changing risk, make it visible, and respond with accountable judgment. Checklists support that discipline when people use the evidence to make and revisit decisions.
Sources and Further Reading
- NIST: AI Risk Management Framework 1.0 — voluntary U.S. guidance released 26 January 2023; NIST identifies a revision in progress.
- Zillow Group 2021 Form 10-K: restructuring and related activities note — inventory write-down and wind-down context for the year ended 31 December 2021.
- AI Risk Registers: What to Track, Measure, and Escalate
- The Human Oversight Layer: Why AI Governance Needs More Than Just Automation
About Gary Cheung
Gary Cheung writes about AI governance, information risk, and accountable technology decisions. Discuss an AI governance challenge.
