Adversarial Use of AI: What's Real and What's Hype

Few topics in cybersecurity generate as much speculation as artificial intelligence. Some claims suggest AI has made traditional defenses obsolete; others dismiss the threat entirely. The truth, based on the growing body of public threat reporting, is more practical: AI is making many existing attacks faster, cheaper and more convincing, while truly novel AI-driven attack capabilities are emerging more gradually.

Because this area is changing quickly, defenders should rely on current, evidence-based reporting rather than headlines.

What public reporting shows

Since 2024, several AI providers and threat intelligence teams, including Google's Threat Intelligence Group, OpenAI and Anthropic, have published reports describing how threat actors, including state-sponsored groups, attempted to use their AI services. Common themes include:

Productivity for existing tasks

Actors used AI to research targets and organizations, summarize public information about vulnerabilities, troubleshoot code, write scripts and translate content. In many documented cases, AI made operators faster rather than giving them fundamentally new abilities.

Phishing and influence content

Generating fluent, tailored messages in multiple languages is one of the clearest uses. Influence operations have also used AI to produce content at scale.

Assistance with malware and tooling

Reports describe actors seeking help developing and debugging malicious code. Some more recent reporting describes AI being used more extensively across stages of an operation, including by less-skilled actors, which suggests the gap between assistance and automation is narrowing.

Deepfakes in fraud

Voice cloning and deepfake video have been used in real fraud cases. In one widely reported 2024 case, an employee transferred a large sum after a video call in which the other participants, including a senior executive, were reportedly deepfakes.

What remains overstated

• Fully autonomous attacks with no skilled human involvement are not what most public evidence shows, although providers have reported attempts to automate larger portions of operations.
• AI inventing entirely new classes of attack is not well supported by current public reporting; most documented use applies known techniques.
• AI making defenses irrelevant is not supported. The fundamentals, including patching, strong authentication, least privilege and monitoring, still stop most attacks, however they were planned.
• Every attack being "AI-powered" is a marketing claim, not an intelligence finding.

These conclusions may change. Organizations should revisit them as new reports are published.

What changes for defenders

Social engineering gets harder to spot

Because AI removes many classic warning signs, controls that do not depend on people spotting fakes become more important: phishing-resistant multifactor authentication, and verification of sensitive requests through known channels.

Speed increases

If adversaries can research, write and adapt faster, the window between vulnerability disclosure and exploitation, and between initial access and impact, may shrink. Rapid patching and early detection matter more.

Your own AI becomes a target

Organizations deploying AI systems create new attack surfaces, such as prompt injection, data poisoning and model theft, that adversaries can exploit.

Defenders can use AI too

AI can help defenders triage alerts, summarize investigations, analyze malware and write detection logic, when used carefully, with human oversight and an understanding of its limits.

Practical steps

1. Strengthen identity: move to phishing-resistant MFA, especially for administrators and executives.
2. Formalize verification: require out-of-band confirmation for payments, credential resets and sensitive data requests.
3. Patch faster, prioritizing known exploited vulnerabilities in internet-facing systems.
4. Update awareness training to cover AI-generated lures and deepfakes.
5. Secure your own AI deployments with least privilege, input and output controls, and adversarial testing.
6. Follow the reporting: track threat reports from AI providers and government agencies, and adjust.

Frequently asked questions

Should we block employees from using AI tools? Blanket bans often push use underground. Clear policies, approved tools, data handling rules and training are usually more effective.

Can AI detect AI-generated phishing? Detection tools help, but attackers adapt. Controls that remain effective regardless of how convincing a message is are more dependable.

How do we keep up? Assign someone to track reporting from AI providers, CISA and threat intelligence sources, and review your assumptions periodically.

What to do this week

• Confirm that payment and banking-detail changes require call-back verification to a known number.
• Brief finance, HR and executive assistants on voice cloning and deepfake video fraud.
• List the AI tools employees use, and publish clear rules for sensitive data.
• Identify any AI system in your environment that can take actions, and check what it can access.

How CDT can help

CDT's AI/ML cybersecurity services test and secure your AI systems, our social engineering assessments test resilience to AI-assisted lures, and our training prepares people for modern social engineering.

Sources

• Google Threat Intelligence Group, Adversarial Misuse of Generative AI
• OpenAI, Disrupting malicious uses of AI
• Anthropic, Threat Intelligence Report