By criminals, for criminals: AI tool easily generates ‘remarkably persuasive’ fraud emails

An artificial intelligence tool promoted on underground forums shows how AI can help refine cybercrime operations, researchers say.

The WormGPT software is offered “as a blackhat alternative” to commercial AI tools like ChatGPT, according to analysts at email security company SlashNext.

The researchers used WormGPT to generate the kind of content that could be part of a business email compromise (BEC) scam, in which criminals defraud companies through bogus messages that request payments, especially wire transfers.

“In one experiment, we instructed WormGPT to generate an email intended to pressure an unsuspecting account manager into paying a fraudulent invoice,” SlashNext says. “The results were unsettling. WormGPT produced an email that was not only remarkably persuasive but also strategically cunning, showcasing its potential for sophisticated phishing and BEC attacks.”

The researchers’ comments echo recent warnings from government officials and security experts about the huge cybercrime risks of generative AI.

An effective BEC attack requires communication that doesn’t raise suspicions. An attacker who isn’t fluent in the recipient’s language might need some help creating a persuasive email and polishing it up. Commercial AI tools typically block that kind of activity.

To generate authentic-looking text, WormGPT uses a version of the open-source large language model (LLM) known as GPT-J that has been customized “specifically for malicious activities,” the researchers say.

“In summary, it’s similar to ChatGPT but has no ethical boundaries or limitations,” SlashNext says.

Researchers from another company, Mithril Security, recently showed how they sneakily distributed a modified open-source AI tool that was trained to provide disinformation.