GPT bot
Image: Kindel Media via Pexels

By criminals, for criminals: AI tool easily generates ‘remarkably persuasive’ fraud emails

An artificial intelligence tool promoted on underground forums shows how AI can help refine cybercrime operations, researchers say.

The WormGPT software is offered “as a blackhat alternative” to commercial AI tools like ChatGPT, according to analysts at email security company SlashNext.

The researchers used WormGPT to generate the kind of content that could be part of a business email compromise (BEC) scam, in which criminals defraud companies through bogus messages that request payments, especially wire transfers.

“In one experiment, we instructed WormGPT to generate an email intended to pressure an unsuspecting account manager into paying a fraudulent invoice,” SlashNext says. “The results were unsettling. WormGPT produced an email that was not only remarkably persuasive but also strategically cunning, showcasing its potential for sophisticated phishing and BEC attacks.”

The researchers’ comments echo recent warnings from government officials and security experts about the huge cybercrime risks of generative AI.

An effective BEC attack requires communication that doesn’t raise suspicions. An attacker who isn’t fluent in the recipient’s language might need some help creating a persuasive email and polishing it up. Commercial AI tools typically block that kind of activity.

To generate authentic-looking text, WormGPT uses a version of the open-source large language model (LLM) known as GPT-J that has been customized “specifically for malicious activities,” the researchers say.

“In summary, it’s similar to ChatGPT but has no ethical boundaries or limitations,” SlashNext says.

Researchers from another company, Mithril Security, recently showed how they sneakily distributed a modified open-source AI tool that was trained to provide disinformation.

Get more insights with the
Recorded Future
Intelligence Cloud.
Learn more.
What is Threat Intelligence
No previous article
No new articles
Joe Warminsky

Joe Warminsky

is the news editor for Recorded Future News. He has more than 25 years experience as an editor and writer in the Washington, D.C., area. Most recently he helped lead CyberScoop for more than five years. Prior to that, he was a digital editor at WAMU 88.5, the NPR affiliate in Washington, and he spent more than a decade editing coverage of Congress for CQ Roll Call.