Salesforce-owned Heroku faces backlash over handling of stolen user credentials

Popular cloud platform Heroku revealed this week that it forcibly reset some user passwords in response to a security breach, setting off a wave of criticism over how the Salesforce-owned company responded to the incident. 

On May 4, Heroku sent emails to some users indicating that their passwords had been changed but did not explain why. 

“A subset of Heroku customers will receive email notifications directly from Salesforce Incident Alerts ([email protected]) regarding our continuous efforts to enhance security,” Heroku said on its status page, only providing a link to a support page for questions. 

Until May 5, the support page provided no information about why passwords were being changed, setting off criticism from users confused by the company's messaging.

"This is turning into a complete train wreck and a case study on how not to communicate with your customers," one Heroku user wrote on the Y Combinator message board.

Heroku's emails to users alluded to a security incident last month in which an attacker abused stolen OAuth user tokens issued to Heroku and software testing platform Travis-CI. The attacker downloaded data from dozens of organizations, including the JavaScript package manager npm, Heroku said.

The breach potentially has wide-ranging implications because the hackers used their access in Heroku to pivot into private repositories on GitHub, which owns npm.

GitHub confirmed the incident in several blog posts. Mike Hanley, GitHub's chief security officer, said the attacker "may be mining the downloaded private repository contents, to which the stolen OAuth token had access, for secrets that could be used to pivot into other infrastructure."

OAuth provides many of the biggest tech companies with ways to provide secure access to server resources. OAuth access tokens are the thing that applications use to make API requests on behalf of a user.

GitHub has uncovered evidence that an attacker abused stolen OAuth user tokens issued to two third-party OAuth integrators, Heroku and Travis-CI. Read more about the impact to GitHub, npm, and our users. https://t.co/eB7IJfJfh1

— GitHub Security (@GitHubSecurity) April 15, 2022

By May 5, Heroku released a longer explanation for the situation, noting the backlash from confused customers. 

According to the statement, Heroku worked with GitHub, threat intelligence vendors and even law enforcement to investigate what happened in April. 

“On April 7, 2022, a threat actor obtained access to a Heroku database and downloaded stored customer GitHub integration OAuth tokens. Access to the environment was gained by leveraging a compromised token for a Heroku machine account,” the company explained. 

“According to GitHub, the threat actor began enumerating metadata about customer repositories with the downloaded OAuth tokens on April 8, 2022. On April 9, 2022, the attacker downloaded a subset of the Heroku private GitHub repositories from GitHub, containing some Heroku source code.”

GitHub discovered the activity on April 12 and told Salesforce about what was happening on April 13. 

By April 16, Heroku said it revoked all GitHub integration OAuth tokens, preventing customers from deploying apps from GitHub through the Heroku Dashboard or via automation. The statement notes that before they re-enable that functionality, the company wants to “ensure the integration is secure.”

Adding to the confusion around the situation, Heroku admitted the investigation revealed that the same compromised token was used by the hacker to gain access to a user account database. The hacker stole the hashed and salted passwords for customers’ user accounts.

Salesforce said this separate incident was why it was “ensuring all Heroku user passwords are reset and potentially affected credentials are refreshed.”

The company added that it rotated internal Heroku credentials and put additional detections in place but is still in the process of investigating the source of the token compromise.

Salesforce did not respond to questions about how many user accounts were affected, only directing The Record to the statement on the status page. 

“Nothing is more important to us than the security of customer data. We value transparency and, without compromising our ongoing investigation, wanted to share additional details that may facilitate a deeper understanding for our customers of this issue,” a Salesforce spokesperson told The Record.  

“We continue to work diligently in response to this incident. If we determine that any customer is affected, we will update them with further guidance without undue delay.”

The situation caused widespread confusion among users of Heroku, which boasts of facilitating the creation of more than 13 million apps that handle 60 billion requests every day. The cloud platform allows developers to build, deliver, monitor and scale apps.

https://t.co/g2o1FRj5w6

This appears to be the worst-case scenario: compromise of Heroku’s own core database. I’m afraid this is going to continue to get worse. I won’t be surprised if env vars got popped too.

— jacob is on mastodon: @[email protected] (@jacobian) May 5, 2022

If you're on Heroku:

Rotate all config var creds.
Change password.
Rotate API Key and 2FA.
Audit as much as you can of Heroku releases and deploys.

It's not gonna be fun, but do it.

— Craig Kerstiens (@craigkerstiens) May 5, 2022

Dozens of users took to social media to complain about the company’s response. Corey O’Connor, director of products at SaaS security company DoControl, told The Record that Heroku’s handling of the incident left much to be desired. 

“This was not an effective way to deal with this situation at all. It’s a clear case that better breach response planning should’ve been considered, and now Heroku is trying to fly the plane and build it at the same time in regards to damage control,” O’Connor explained. “Credential management (i.e. the OAuth tokens) was a big driver in this attack, and is coincidentally a part of the security recommendations from both GitHub and Heroku.”

Other experts said Heroku was in a no-win situation and handled it as best as it could. Casey Bisson, head of developer relations at code security firm BluBracket, said the incident was serious because the nature of the breach means attackers had access to multiple classes of information stored in different locations.

“It’s a bad situation that raises many questions about Heroku data management practices leading up to the breach. However, I think they deserve credit for their handling of it once it was disclosed to them,” Bisson said. 

Keeper Security CTO Craig Lurey added that the attack is the latest in a series of high-profile incidents related to malicious actors' theft of "infrastructure secrets," the machine-to-machine credentials that give one system access to another one. 

Lurey noted that some source code was stolen and that the investigation is ongoing, meaning it is unlikely to know the extent of the damage for a while.

“We’ve found an increased trend in hackers trying to target privileged accounts in various build and deployment systems. This makes sense, as these are the accounts with access to your company's crown jewels,” Lurey said. “Unfortunately, it often takes companies to be the victims of an incident like this to take action.”