Gary Cheung · 20 October 2025 · 5-minute read
At a glance
- The risk: Information can move into prompts, retrieval stores, logs, outputs, or connected actions without an owner seeing the whole path.
- The governance implication: Approve data use for the complete AI workflow, not only the model or user interface.
- The leadership takeaway: Decide which information may enter each service, who may approve exceptions, and who can contain an exposure.
When a team wants to connect a generative AI assistant to customer records, the consequential decision is not simply whether the model is secure. Leaders must decide which information the workflow may retrieve, transform, retain, and expose—and whether the business benefit justifies that path.
AI amplifies information risk because one request can cross user input, retrieval systems, model providers, logs, plugins, outputs, and downstream decisions. A control applied at only one point may leave the rest of the path ungoverned.
The control boundary is the workflow, not the model
It is tempting to treat the approved AI product as the boundary. In practice, the exposure depends on how the product is configured and connected: what a user enters, which sources it retrieves, what the provider retains, who can review logs, and where an output is copied or acted upon.
NIST’s Generative AI Profile, NIST AI 600-1, describes privacy risks including the use of personal data in training, model leakage or inference of sensitive information, and harms created by inaccurate sensitive inferences. It was published on 26 July 2024 as voluntary, cross-sector guidance—not as a legal requirement.
The governance implication is broader than “do not paste confidential data.” The business owner needs an end-to-end view of the information path, including third parties and fallback processes, before authorizing consequential use.
Different exposure mechanisms need different controls
Direct submission. Employees may enter non-public information into an external service. Cisco’s 2024 Data Privacy Benchmark Study surveyed 2,600 security and privacy professionals across 12 countries; 48% of respondents said their organizations had entered non-public company information into generative AI tools. This is a survey result, not a measure of all employees or organizations.
Connected retrieval and storage. Enterprise assistants can retrieve from document repositories, vector databases, APIs, and conversation history. Access that is valid for one user or purpose can become inappropriate when the tool combines sources or exposes results to another audience.
Generated output. NIST notes that generative models may leak, generate, or infer sensitive information. Research has also demonstrated training-data extraction under specific model and attack conditions. That evidence establishes a credible risk mechanism; it does not mean every prompt is retained for training or that every service will reproduce submitted data.
These mechanisms require different responses: input controls, least-privilege retrieval, retention settings, output testing, contractual review, monitoring, and incident handling. One generic “AI DLP” control should not be assumed to cover them all.
Approved tools still require accountable decisions
An enterprise contract or privacy setting may reduce exposure, but it does not decide whether a particular use of customer, employee, source-code, or regulated information is appropriate. Nor does blocking public tools resolve the demand that created shadow use.
Set a decision matrix by information class, service, purpose, and action. An approved combination should identify the business owner, permitted users, retention or reuse conditions, connected sources, output destination, and required human review. Treat exceptions as decisions with an approver and expiry date—not permanent workarounds.
OWASP’s 2025 guidance on sensitive information disclosure recommends measures across data sanitization, access control, output validation, and user guidance. It is security guidance rather than regulation; organizations still need to map measures to their own data, jurisdictions, and contractual duties.
Design containment before granting access
Monitoring should answer business questions: which unapproved tools are being used; what classes of information are submitted; which repositories an assistant can reach; whether outputs reveal restricted content; and whether the response team can revoke access or isolate the affected integration.
Define what happens when exposure is suspected. The owner should be able to restrict the workflow, preserve relevant evidence, assess affected information and jurisdictions, involve privacy, legal, security, or compliance as appropriate, and decide whether any notification duty applies. Not every policy violation is a reportable breach, and no single notification timeline applies everywhere.
My information-risk judgment is that access should not expand faster than the organization’s ability to trace and contain the information path. If the business cannot identify the owner or disable the connection, the decision is not ready for production.
Questions for Leaders
- Which information classes may this AI workflow retrieve, transform, retain, or disclose—and for what approved purpose?
- Who approves an exception, and what evidence must support that decision?
- Can the accountable owner contain the workflow and determine the affected information quickly enough to limit harm?
Closing Thought
AI does not make familiar information-risk principles obsolete. It makes weak ownership and incomplete data boundaries visible faster. The practical test is whether leaders can explain the full information path, authorize it deliberately, and stop it when the assumptions no longer hold.
Sources and Further Reading
- NIST AI 600-1: Generative Artificial Intelligence Profile — voluntary cross-sector guidance published 26 July 2024.
- Cisco 2024 Data Privacy Benchmark Study — survey scope and generative-AI data-use findings.
- OWASP LLM02:2025 Sensitive Information Disclosure — security guidance, not regulation.
- Extracting Training Data from Large Language Models — 2020 research paper demonstrating extraction under studied conditions.
- Shadow AI: Where Is the Line? and Building an AI Escalation Playbook.
About Gary Cheung
Gary Cheung writes about AI governance, information risk, and accountable technology decisions.
To discuss an AI governance challenge, share the decision context here.
