Phishing in the Age of AI: Deepfakes, Voice Cloning and Tailored Lures

For years, security awareness training taught people to look for the signs of a phishing email: spelling mistakes, awkward phrasing, generic greetings, suspicious urgency. Those signs were never reliable, and they are becoming even less so.

Generative AI lets attackers produce polished, personalized messages in any language, quickly and cheaply. Voice cloning can imitate a known person from a short recording. Deepfake video can put a familiar face on a stranger. Social engineering is not new, but its tools have improved dramatically.

What is changing

Fluent, tailored messages

AI tools can research a target from public sources and write convincing messages that reference real projects, colleagues and events. Security researchers and AI providers have published reports describing threat actors using AI to draft phishing content and translate lures.

Voice and video impersonation

Voice cloning has been used in fraud schemes where callers imitate executives or family members. In a widely reported 2024 case, an employee of a multinational firm in Hong Kong transferred a large sum after a video conference in which the other participants, including a senior executive, were reportedly deepfakes.

Multi-channel attacks

Attackers no longer rely on email alone. Text messages, phone calls, messaging apps, social media and collaboration platforms are all used, often in combination, such as an email followed by a "confirming" phone call.

Help desk targeting

Some groups call IT help desks while impersonating employees, asking to reset passwords or enroll new multifactor devices. CISA and FBI have described these tactics in advisories on the group known as Scattered Spider.

MFA relay phishing

Phishing kits that proxy real sign-in pages can capture passwords and session cookies in real time, defeating many traditional multifactor methods.

Why "spot the fake" is no longer enough

If messages are well written, voices sound right and faces look familiar, asking people to detect fakes becomes an unreliable last line of defense. Awareness still matters, but controls must assume that some people will be deceived.

Controls that work regardless of how convincing the lure is

Phishing-resistant multifactor authentication

FIDO2 security keys and platform passkeys bind authentication to the legitimate website, so credentials entered on a fake site cannot be replayed. This defeats relay phishing kits that capture one-time codes.

Verification procedures for sensitive requests

Payments, changes to bank details, credential resets and requests for sensitive data should require verification through a separate, known channel, such as calling back on a number from the directory, not one provided in the request. Make it policy, so employees feel empowered to verify even when a request appears to come from a senior leader.

Help desk identity verification

Strengthen how the help desk verifies identity before resetting passwords or MFA: callbacks to known numbers, manager approval, or in-person verification for privileged accounts.

Email and domain protections

Implement SPF, DKIM and DMARC to reduce spoofing of your domains, and use filtering that analyzes links and attachments.

Reduce what attackers can learn

Review what public sources reveal about your organization's structure, processes and people, and limit unnecessary detail.

Updating awareness training

Training should evolve from "spot the typo" to:

• Recognizing pressure tactics: urgency, secrecy and requests to bypass normal process
• Verifying through a second channel, always, for sensitive requests
• Knowing that voices and video can be faked
• Reporting quickly, without fear of blame, when something seems off
• Practicing with realistic simulations across email, phone and messaging

Frequently asked questions

Can we detect deepfakes technically? Detection tools exist and are improving, but they are not reliable enough to be the primary defense. Process controls, such as verification through known channels, are more dependable.

Should we still run phishing simulations? Yes, but use them to build reporting habits and test processes, not to shame people. Include phone and text simulations where appropriate.

What about our executives' voices and videos online? Public recordings can be used for cloning. Reducing exposure helps somewhat, but verification procedures that do not depend on recognizing a voice are more effective.

What to do this week

• Check your DMARC policy and move toward enforcement if it is still set to monitor only.
• Write down, and communicate, the verification rule for payments and credential resets.
• Test whether your help desk would reset MFA for an unverified caller.
• Make sure reporting a suspicious message is one click, and that reporters get a thank-you.

How CDT can help

CDT's social engineering assessments test your organization with realistic email, phone and in-person scenarios, and our cyber training helps people recognize and report modern social engineering. Penetration testing evaluates the technical controls that limit the damage when someone is deceived.

Sources

• CISA and FBI advisory AA23-320A on Scattered Spider
• CISA, Phishing Guidance: Stopping the Attack Cycle at Phase One