At a glance
- The risk: AI can make impersonation and social engineering more convincing, while broad claims about autonomous AI attacks can misdirect security priorities.
- The governance implication: Treat AI as a factor that can amplify existing attack paths; keep core identity, verification, and response controls central.
- The leadership takeaway: Test high-impact payment and access workflows against voice, video, and text impersonation, and require independent verification for sensitive actions.
Attackers can use generative tools to create persuasive text, voice, or images, but the presence of AI does not make an attack autonomous. Many incidents still depend on familiar methods: stolen credentials, social engineering, urgency, and weak approval controls. Leaders need a grounded view of where AI changes the threat and where established defenses remain decisive.
The most important governance question is not whether attackers use AI in every campaign. It is whether the organization can verify identity and intent when convincing synthetic content reaches people and processes that can move money, change access, or disclose sensitive information.
How AI can strengthen social engineering
Generative AI can help attackers draft or translate messages, adapt wording to a target, and produce synthetic voice or visual material. These tools can lower effort or improve plausibility. They do not remove the need to obtain information, reach a target, exploit a process, or persuade someone to act.
A widely reported 2019 case involved a fraudster using a synthetic voice to impersonate a parent-company executive. A UK energy company’s chief executive was persuaded to transfer €220,000. In 2024, Hong Kong police investigated a reported US$25 million fraud in which an employee was deceived during a video meeting featuring multiple deepfake participants. These cases demonstrate impersonation risk; they do not establish that AI alone caused the loss. The control failures involved verification and payment authorization.
Separate observed threats from forecasts
Security reporting has described growing experimentation with generative AI, while also distinguishing that activity from mature, large-scale AI-enabled operations. IBM’s 2024 X-Force reporting did not substantiate the claim that 60% of phishing emails were AI-generated. Treat precise prevalence claims carefully: methods differ, attribution is difficult, and a polished message cannot reliably reveal how it was written.
Claims about attackers using reinforcement learning to tune malware or machine learning to continuously rewrite malicious code should also be tied to verifiable evidence. For risk planning, distinguish demonstrated incidents, observed experimentation, and plausible future scenarios. That keeps investment decisions proportional to what is known.
Defenses still depend on trusted process
AI-assisted content can make familiar controls more important. Use out-of-band verification for payment changes and urgent requests. Apply strong authentication, least privilege, separation of duties, and dual approval to sensitive actions. Make reporting easy and ensure staff know that voice, video, or caller ID does not prove identity.
Detection tools can help identify suspicious patterns, but they can generate false positives or miss unfamiliar behavior. Keep human decision authority clear for consequential actions. Exercise incident response with realistic impersonation scenarios, including who can freeze a transfer, reset an account, preserve evidence, and notify affected parties.
Govern the exposure, not the headline
Executives should connect AI-enabled threat scenarios to existing business processes and control owners. Prioritize workflows with irreversible or high-value outcomes: payments, privileged access, account recovery, executive communications, and release of sensitive information. Confirm that safeguards work under time pressure and when normal communication channels may be compromised.
For AI services used by defenders, assess their own data access, retention, security, failure modes, and human oversight. Maintain a clear record of where AI is used in security operations and which response actions are automated. The same governance discipline applies to tools used by defenders and services that attackers may target.
Leadership questions
- Which actions could an impersonation trigger, and what independent verification is required before they occur?
- Can staff pause a high-impact action when identity or context is uncertain, without being penalized for delay?
- Which AI-related threat scenarios have been exercised, and who owns the response and recovery decisions?
Closing thought
AI can increase the reach and realism of deception. The durable defense is a process that does not confuse convincing content with verified authority—and gives people clear permission and means to stop an uncertain action.
Further reading
- IBM X-Force Threat Intelligence Index 2024
- Associated Press reporting on the Hong Kong deepfake fraud
- Le Monde reporting on the 2019 synthetic-voice fraud
- CISA JCDC AI Cybersecurity Collaboration Playbook
- Beyond1n0: Secure AI Deployment
About Gary Cheung
Gary writes about information risk, AI governance, and executive accountability. For a discussion about an AI decision or its governance, contact Beyond1n0.
