By Gary Cheung · 4-minute read
At a glance
- The risk: A narrow or paper-based assurance review can create confidence without testing how an AI system behaves in its real context.
- The governance implication: Assurance should examine data, performance, robustness, vendors, documentation, and meaningful human intervention in proportion to impact.
- The leadership takeaway: Require evidence that controls work in practice, identify remaining uncertainty, and name who can act when results fall outside expectations.
Before approving or continuing to rely on AI in a consequential workflow, leaders need a defensible answer to a practical question: what evidence shows that this system is fit for this decision, in this operating context?
AI assurance should make uncertainty visible before the organisation accepts exposure. A policy, vendor statement, or successful demonstration may describe intended controls; it does not establish how the system performs with actual data, users, exceptions, and consequences.
Risk insight: Assurance fails when it produces confidence without identifying the decision owner, the limits of the evidence, and who can act when those limits are reached.
Assurance Must Match the Decision
Start with the business decision, the people affected, and the consequence of error. Testing should reflect the operating population, intended use, and material failure modes rather than a generic model score.
Evidence should show data suitability, performance across relevant groups and conditions, known limitations, and the thresholds the business will accept. Where a disparity or performance gap remains, the decision owner should know whether it requires mitigation, restricted use, additional review, or rejection.
Assurance is not a promise that bias or error has disappeared. It is a documented judgment about what was tested, what remains uncertain, and whether the residual exposure is acceptable.
Explainability Must Support Challenge
Explainability is useful when it enables a person to understand, question, and investigate a consequential output. The required information will differ for an operator, an accountable executive, an affected person, and a regulator.
Regulation (EU) 2024/1689, adopted on 13 June 2024, sets out binding requirements for high-risk AI systems concerning technical documentation, information for deployers, human oversight, accuracy, robustness, and cybersecurity. It does not prescribe one universal explanation method for every AI system. Applicability depends on classification, role, jurisdiction, and the relevant implementation provisions.
Assurance should therefore test whether the available explanation supports the real challenge process: can a reviewer identify limitations, trace relevant evidence, dispute an outcome, and reach someone authorised to act?
Robustness Must Cover Real Operating Conditions
Assurance should test behaviour outside the clean, expected path. Relevant scenarios may include unusual inputs, changed data, malicious manipulation, degraded dependencies, capacity pressure, unavailable human reviewers, and failed integrations.
The evidence should connect each scenario to a business consequence and a response. Leaders need to know which failures are detected, when the workflow pauses, whether a safe fallback exists, and how service is restored without silently accepting poor decisions.
A model can pass a technical test and still be unsuitable for deployment if the surrounding process cannot contain failure. Robustness is therefore a property of the decision system, not only the model.
Vendor Evidence Must Be Usable
Using a third-party model does not remove the deploying organisation’s need to understand its own exposure. The precise duties vary by jurisdiction, sector, contractual role, and use case.
Assurance should establish what the vendor can evidence and what the organisation can independently validate. Contracts and operating arrangements should address material model or data changes, incidents, access to relevant documentation, monitoring responsibilities, remediation, and exit or fallback options.
The December 2023 NAIC model bulletin offers a useful insurance-sector reference in the United States, but it is a model bulletin rather than a single nationwide rule; applicability depends on adoption and the relevant state authority.
Human Oversight Must Be Tested
A human checkpoint is not effective simply because it appears in a process diagram. Reviewers need the context, time, competence, and authority to challenge an output and change what happens next.
Assurance should observe the control in a realistic scenario. Can the reviewer recognise uncertainty, obtain supporting evidence, disagree without impractical escalation, pause the workflow, and use a safe fallback? It should also test whether workloads or incentives make approval routine.
Define the signals that trigger renewed assurance: material vendor changes, drift, incidents, customer complaints, expansion to a new population or purpose, or control failure. Name the person who decides whether use can continue.
Related: The Human Oversight Layer.
Leadership Questions
- What evidence shows the system works as intended in its actual operating context?
- Who can challenge, pause, or override it, and do they have enough information and authority?
- What vendor change, performance signal, or incident triggers a fresh assurance review?
Closing Thought
Assurance is useful when it gives decision-makers evidence they can act on: what was tested, what remains uncertain, and who is accountable for follow-up.
Sources and Further Reading
- NIST AI Risk Management Framework — voluntary guidance released in January 2023.
- NIST AI Resource Center — resources for AI testing, evaluation, verification, and validation.
- European Commission AI Act Explorer — Regulation (EU) 2024/1689, adopted 13 June 2024; check current amendments and applicable dates.
- NAIC Model Bulletin on the Use of AI by Insurers — approved December 2023.
- Beyond1n0: AI risk registers and escalation
About the author: Gary Cheung writes about AI governance, information risk, and accountable technology decisions. Discuss an AI decision.
