IT teams scramble to recover from CrowdStrike incident as officials warn of ‘risks of consolidation'

Fallout from massive technology outages caused by the cybersecurity firm CrowdStrike continued throughout Friday as people around the world navigated canceled flights, paralyzed workspaces and downed 911 systems.

White House cybersecurity leader Anne Neuberger said her morning began at 4 a.m. with a call from the Situation Room about the outages — which affected millions of Windows computers and was sourced back to a faulty software update issued by CrowdStrike.

At the Aspen Security Forum on Friday, Neuberger explained that an interagency task force was quickly put together to assess the impact on the U.S. government and critical services before going sector by sector to analyze potential downstream ramifications. 

“What's the impact to power in the country? To hospitals in the country? To 911 systems, the national suicide hotline. We continue to do those calls until we have a good picture now of where we are,” she said, adding that she called CrowdStrike CEO George Kurtz and was reassured that it was not a malicious attack. She spoke to several of her counterparts around the world to offer U.S. assistance as well. 

The outages affected dozens of federal, state and city entities alongside airlines, hospitals, public transportation and TV broadcasters.

A senior administration official said President Joe Biden is receiving updates on the outage throughout the day, adding the White House is in “regular contact with CrowdStrike” and is tracking progress in remediating affected systems. The White House has been convening with agencies to assess the impact.

“At this time, our understanding is that flight operations have resumed across the country, although some congestion remains, and 911 centers are able to receive and process calls,” the official said. “We are assessing impact to local hospitals, surface transportation systems, and law enforcement closely and will provide further updates as we learn more.”

Both Neuberger and U.S. Secretary of State Antony Blinken spoke at length at the conference about what they felt was the source of the disaster: the consolidation of technology among a handful of companies.

“The irony of this morning is that a major international cyber security company was impacted,” she said. 

“We need to really think about our digital resilience, not just in the systems we run, but in the globally connected security systems. The risks of consolidation, how we deal with that consolidation, and how we ensure that if an incident does occur, it can be contained and we can recover quickly.”

Blinken said it is imperative that organizations globally build resilience and redundancy back into critical systems but added that the world cannot be reliant “on any single point of failure.”

“We've been doing that with supply chains across the world, building coalitions of countries to make sure that we're coordinating on supply chains,” Blinken added. “Build new ones, and making sure that if we see a problem, we can address it immediately.”

Several recent incidents involving sector leaders like Microsoft, UnitedHealth Group and Snowflake have validated concerns about the consolidation of the technology industry around a small group of companies. 

Lina Khan, the chair of the Federal Trade Commission, said incidents like the one on Friday “reveal how concentration can create fragile systems.”

“​​Concentrating production can concentrate risk, so that a single natural disaster or disruption has cascading effects,” she said. 

In the trenches

The painstaking process of rebooting devices began early on Friday as IT staff came into work. Several cybersecurity companies said they were seeing hundreds of thousands of devices impacted worldwide. 

CrowdStrike has sent out a fix for the issue, which affects Windows 10 and later systems, but IT employees who spoke to Recorded Future News said the recovery process has to be done manually.

The typical process involves booting a computer into safe mode, deleting the faulty file, and then restarting. 

“Multiple reboots should not normally be necessary unless there are additional underlying problems,” explained Mike Walters, president of cybersecurity firm Action1. “Sometimes, if the system hasn't been rebooted in years, the reboot can cause specific problems at startup. Other problems can arise with databases that may not work after startup due to an incorrect reboot.”

An employee at a New York hedge fund who spoke on condition of anonymity to more freely discuss the situation said each reboot process took about 10 to 15 minutes but varied greatly based on the type of computer and operating system. 

He typically rebooted the computer twice before manually removing the faulty file. As of Friday afternoon, he had fixed 50 computers. His team needed to fix thousands of affected devices. 

“It's something that happens a lot actually, an update breaking something. It’s very common. It just never happens on this scale,” he said. 

Nadir Izrael, cofounder of the cybersecurity firm Armis, said the recovery is “painstaking” because it has to be done manually and cannot just be transmitted over the web. In some cases, devices have to be rebooted 12 to 15 times.

Several experts said small organizations likely recovered already but large companies with thousands of devices will probably need time over the next week to recover fully. 

The issue will be exacerbated for organizations that have machines in remote areas, according to former National Security Agency hacker Jake Williams. He added that in order to be rebooted, some devices will require a recovery key for BitLocker — an encryption feature included with Microsoft Windows — and many organizations have not kept track of their keys. 

AttackIQ’s Andrew Costis explained that the typical workaround will not work because BitLocker recovery keys are likely stored on devices that were also impacted, meaning the keys “may be unrecoverable.”

“This may require either a full system rebuild or a restore from the last known good backup,” he said. 

Multiple cybersecurity experts said cybercriminals will exploit the situation. ColorTokens CISO Agnidipta Sarkar said bad actors are preparing to attack people as they recover “because they would be most vulnerable then.”

CrowdStrike published a blog explaining that threat actors are sending emails posing as the company’s customer support and have impersonated employees of the company in phone calls. 

Hackers have also called and emailed customers pretending to be researchers and experts selling remediation tools as well, according to CrowdStrike. Kurtz, CrowdStrike’s CEO, warned of potential attacks and urged customers to “ensure that you’re engaging with official CrowdStrike representatives.” 

Critical Start’s Callie Guenther said others are using the crisis to trick users into divulging sensitive information or installing malware. 

The Cybersecurity and Infrastructure Security Agency also said it has observed threat actors “taking advantage of this incident for phishing and other malicious activity.” Experts are already seeing an explosion in fake domains spoofing CrowdStrike and the company provided a list of fake domains they have encountered. 

“When systems fail and chaos ensues, it creates ideal conditions for criminals to prey on the unique opportunity,” said Trustwave CISO Kory Daniels. “History has shown us that these moments of disruption are often accompanied by a surge in criminal behavior.”