Guardrails Are Not Governance: Why Most AI Deployments Still Fail at the Basics

Gary Cheung · 6 December 2025 · 5-minute read

A business preparing to let generative AI draft customer responses must decide whether technical safeguards make that use reliable enough—not merely whether a content filter is switched on. The consequential decision is what the organization will allow the system to influence, whose data it may use, and who can intervene when the safeguards are insufficient.

Guardrails can reduce exposure. They cannot define the business purpose, assign accountability, approve residual risk, or decide when continued use is no longer justified. Those are governance decisions. The distinction matters because an AI system can remain inside its technical boundary while the surrounding business process creates customer, operational, legal, or reputational consequences.

“Guardrail” is a broad label. It can describe system instructions, input and output filters, deterministic validation, least-privilege permissions, rate limits, blocked actions, or a requirement for human approval. Some controls prevent an action, some detect a condition, and some limit the consequence after a failure.

These controls matter. OWASP’s 2025 prompt-injection guidance recommends constrained behaviour, validated output formats, filtering, least privilege, human approval for high-risk actions, segregation of untrusted content, and adversarial testing. It also says foolproof prevention of prompt injection is unclear.

The governance question is therefore not whether guardrails work in general. It is whether the selected controls address the failure modes of this use case, under realistic operating conditions, with evidence that their limitations are understood.

A content filter may block prohibited language exactly as designed while the organization still uses the system for an unapproved purpose. A permission boundary may stop one unauthorized action while staff over-rely on a confident but inaccurate recommendation. A human-review step may exist on paper while reviewers lack time, information, or authority to challenge the output.

NIST’s voluntary AI Risk Management Framework separates governance from individual technical controls. Its Govern function calls for clear roles, ongoing review, accountability structures, executive responsibility for AI-risk decisions, and attention to third-party systems across the lifecycle.

Editorial judgment: a guardrail should be treated as a control claim, not a declaration of safety. Leaders still need to decide what reliance is permitted, which residual exposure is acceptable, and who owns the business consequence if the system behaves within its technical limits but the decision is wrong.

Built-in protections from a model or cloud provider can be valuable. They may also be designed for a different risk tolerance, configuration, user population, or product boundary. The customer organization controls—or chooses—not to control the surrounding data, integrations, prompts, permissions, workflow, human review, and downstream action.

NIST AI 600-1, published 26 July 2024 as a voluntary Generative AI companion to AI RMF 1.0, identifies risks including confabulation, data privacy, human over-reliance, information integrity, and third-party component integration. Output restrictions address only part of that exposure.

The accountable owner therefore cannot rely on “the vendor has safety features” as the control conclusion. The organization needs evidence about which safeguards are enabled, what they cover, how they are tested, what changes the vendor can make, and what contingency applies if the service or control fails.

Each material guardrail should connect to an operating decision:

• Control objective: What specific event or consequence is it intended to prevent, detect, or limit?
• Owner and authority: Who maintains it, who reviews evidence, and who may override, restrict, or stop the use case?
• Test and limitation: How is effectiveness challenged, including bypass attempts, changing inputs, and failure of connected components?
• Response: What happens when the control triggers, degrades, or produces uncertain evidence?

Illustrative scenario: an insurer uses generative AI to summarize complaints and draft responses. A content filter blocks abusive language, but it does not determine whether a complaint was routed correctly or whether the draft omits a material customer issue. Governance could keep the draft advisory, sample outcomes against source records, define escalation thresholds, and give the business owner authority to pause automated routing. This is an example of control design, not a reported client engagement.

• Which consequential business decision does each guardrail support, and what failure does it actually control?
• Who can challenge, override, restrict, or pause the AI-supported process when evidence is weak or conditions change?
• What testing and operating evidence would justify continued use—and what would require reapproval?

Guardrails and governance are not competitors. Guardrails constrain parts of the technical system; governance determines the purpose, authority, evidence, accountability, and response around the business decision. The practical test is not whether the AI has safety features. It is whether the organization knows what those features cannot decide—and has named someone with the authority to decide instead.

• NIST — AI Risk Management Framework 1.0 Core (released 26 January 2023; voluntary; revision in progress).
• NIST AI 600-1 — Generative AI Profile (published 26 July 2024; voluntary cross-sector companion to AI RMF 1.0).
• OWASP GenAI Security Project — LLM01:2025 Prompt Injection (technical risk and mitigation guidance; not a legal requirement).
• Building an AI Escalation Playbook
• AI Incident Response: How to Run a Tabletop Exercise

Gary Cheung writes about AI governance, information risk, and accountable technology decisions. To discuss an AI governance challenge, share the decision context through the AI Decision Review.