Last week, I attended a DC604 presentation on phishing and how artificial intelligence is changing the threat landscape. The session highlighted how AI is making phishing attacks more convincing, easier to execute, and significantly more scalable than ever before. Four key changes stand out:
1. AI has removed traditional phishing red flags
Historically, phishing emails and fake websites often contained obvious warning signs such as poor grammar, spelling mistakes, and generic messaging. AI has largely eliminated these indicators. Today’s phishing lures are professionally written, contextually relevant, and tailored to specific targets using publicly available information from sources such as LinkedIn, Facebook, and corporate websites.
2. AI lowers the barrier to entry
Phishing toolkits such as GoPhish and Evilginx have existed for years. However, AI has dramatically reduced the technical expertise required to conduct phishing attacks and shortened the time to value for attackers.
3. Deepfake technology is eroding trust
AI is accelerating the rise of deepfake scams. Fraudsters can now easily alter faces and voices during video calls or generate convincing audio recordings that impersonate trusted individuals. As a result, it is becoming increasingly difficult to verify the identity of someone you’ve never met in person.
4. AI enables fraudsters to operate at scale
Fraudsters can automate phishing campaigns, generate personalized messages, and engage victims in realistic conversations with minimal human intervention. This increased efficiency allows attackers to reach far more victims while maintaining a high degree of personalization.
Recommended Controls
Individuals and organizations should adapt their defenses to this evolving threat landscape:
User Controls
- Implement phishing-resistant multi-factor authentication (MFA), such as passkeys or hardware security keys.
- Verify unusual requests through a secondary trusted communication channel.
- Limit publicly available information that could be leveraged for highly targeted attacks.
- Be cautious when interacting with unsolicited emails, messages, or video calls, even when they appear legitimate.
- Maintain healthy skepticism when dealing with urgent requests involving credentials, financial transactions, or sensitive information.
Company Controls
- Train employees to recognize social engineering tactics rather than relying solely on spotting spelling or grammar mistakes.
- Establish verification procedures for financial transactions, account changes, and sensitive requests.
- Deploy email security controls and monitor for suspicious domains that impersonate your organization.
AI Platform Controls
- Detect and restrict requests that facilitate phishing, impersonation, credential theft, social engineering, or fraud.
- Implement safeguards against generating highly personalized phishing content using scraped or personally identifiable information.
- Apply enhanced monitoring and risk controls to accounts exhibiting suspicious behavior, such as high-volume message generation or automated campaign activity.
- Limit the ability to generate realistic voice and video impersonations of real individuals without appropriate consent and verification mechanisms.
- Invest in research to identify and mitigate jailbreak techniques that bypass safety controls.
- Support industry standards for content provenance, such as Google’s SynthID and C2PA (Coalition for Content Provenance and Authenticity), to help users determine whether digital content was AI-generated or manipulated.
- Share threat intelligence, abuse indicators, and emerging attack techniques with industry partners and researchers.
- Conduct regular red-team exercises to identify how models could be misused for phishing, scams, impersonation, and other forms of fraud.
As AI continues to evolve, trust can no longer be based solely on appearance, writing quality, or even a familiar voice. The responsibility for combating AI-enabled phishing is shared. Individuals must strengthen their verification practices, organizations must improve security controls and employee awareness, and AI platform providers must continue investing in safeguards that make it harder for threat actors to weaponize these technologies at scale.