Dallas: Royal ransomware gang infiltrated networks weeks before striking

Hackers began surveillance of the city of Dallas’ networks weeks before carrying out a devastating ransomware attack in May, according to a recent report on the incident.

The 31-page After-Action Report, published last week, outlines what happened before, during and after the ransomware attack crippled critical systems used by the city’s police, firefighters, hospitals and government officials. As the ninth-largest city in the country, Dallas was a “a logical choice for bad actors wishing to initiate and prosecute” an attack, the experts said.

The city operates more than 860 applications and has about 200 IT workers within the Dallas Department of Information & Technology Services (ITS).

The hackers — part of the Royal ransomware gang — first infiltrated government systems on April 7 and immediately began surveillance operations. They used a government service account to pivot into the city’s infrastructure and deploy remote management tools.

From April 7 to May 2, the hackers exfiltrated nearly 1.17 terabytes of data and prepared themselves to deploy the ransomware, which they did the following morning.

“Using its previously deployed beacons, Royal began moving through the City’s network and encrypting an apparently prioritized list of servers using legitimate Microsoft system administrative tools,” they explained.

“City attack mitigation efforts began immediately upon the detection of Royal’s ransomware attack. To thwart Royal and slow its progress, City Server Support and Security teams began taking high- priority services and service supporting servers offline. As this was done, City service restoration identification activities began.”

The city noted officials focused on restoring critical systems like the Public Safety Computer-Aided Dispatch, which was brought down during the attack and caused police and ambulances to go to the wrong location multiple times for days.

Officials also focused on 311 services and city-facing communication websites as the first systems that needed to be restored.

In addition to internal and external cybersecurity assistance, the city called on federal law enforcement agencies like the FBI and Cybersecurity and Infrastructure Security Agency (CISA) to help recover from the incident.

In total, the Dallas City Council approved a budget of $8.5 million for restoration efforts and city officials said it is likely they will not need additional funds. That budget covers the cost of outside cybersecurity services as well as breach notification services for the thousands of people who had information exposed due to the attack.

The city’s IT team dedicated nearly 40,000 hours to dealing with the ransomware attack.

2 a.m. on May 3

Part of the report focuses on the Royal ransomware gang, which they said is composed of “experienced cyber operators” believed to have previously belonged to the now-defunct Conti ransomware gang.

The gang initially went by Zeon before adopting the Royal moniker in September 2022. They are not a ransomware-as-a-service gang like their peers, instead keeping their coding and infrastructure private.

The report notes that the hackers initially used the BlackCat/AlphV ransomware during attacks before shifting to their own custom ransomware.

The group deployed its ransomware on Dallas systems at 2 a.m. the morning of May 3 and continued encrypting systems until 6 a.m. the next day. By 8:30 a.m. on May 3, the city had put its incident response plan into use — contacting the mayor’s office, city council officials and others.

From then on, the city’s ITS team instituted a 24/7 rotating schedule to reconstruct the network and contain the damage. The effort was split into teams, with different segments focused on server/system recovery, asset retrieval, elimination of malware and reimagining of affected systems.

The city was able to restore the first systems by May 8, and by May 11 another batch of systems were fully restored. Once critical systems were back, teams focused on restoring regular services like water billing, warrant processing and other critical city payment services.

Overall, 230 servers were damaged by the attack. More than 100 servers were retired permanently because they were either outdated, unsupported by newer systems or deemed non-essential.

“The cumulative count of 1,398 endpoint devices went through reconstruction directly due to the effects of the Royal ransomware infection,” they said.

The attack revealed that the city — like many others — has dozens of systems that need to be modernized. The compromises made for usability in exchange for security “may pose challenges for securing the environment.”

Dozens of systems were never updated or are running software that is no longer supported.

“While they may provide short-term benefits, they can lead to risk. In terms of cybersecurity, technical debt can potentially aid the success of cyber events by virtue of inadequate built-in security measures in newer systems and unremedied vulnerabilities,” they said.

“It is recommended that City leadership participate in ongoing prioritization of technical services so that technical debt is eliminated or focused to low priority City applications and services.”

The report notes that the city’s budget for cybersecurity has increased from 2.5% of the total IT budget to now almost 10% of the budget at $7.8 million, not including the $8.5 million designated for the ransomware recovery. The city’s security team has grown from 18 employees in 2020 to 35 in 2023.

They noted that in addition to their own spending, they coordinate with CISA on penetration tests, the most recent of which was conducted two months before the ransomware attack. CISA did not respond to requests for comment about this penetration exercise.

The city said both internal and external cybersecurity experts have deemed their response to the attack “quite aggressive” and lauded themselves for their ability to discover and address the attack.

“Though there was an initial delay to identifying and understanding that an attack against the City was underway, City leadership was able to turn a large number of resources toward the challenge in a very short period of time,” they said.

“The recovery endeavor successfully attained a restoration rate exceeding 90 percent within an 18-day period. It is important to note that this swift advancement was achieved despite the necessity to rebuild over 230 servers and 1,168 workstations.”

When asked whether they agreed with Dallas’ assessment of their own work, several cybersecurity experts said dealing with attacks of this magnitude are incredibly complex and “good” responses vary greatly.

Optiv’s Nick Hyatt said that in a perfect world, a good response to an attack like this would involve pre-emptive detection of attackers, minimal downtime, minimal disruption to services, very little negative public attention and a thorough understanding of what went wrong and how to resolve those issues to reduce disruption in the future.

But in reality, organizations often don't know they're under attack until it's too late, he explained.

“Recovery at that point is just standard disaster recovery. A big lesson learned from disaster recovery is where your security gaps lie, and what needs to be implemented to reduce impact in the future,” he said.

Still attractive targets

Ransomware attacks on cities as large as Dallas or Oakland have become rarer in recent years as governments step up their cybersecurity protections and groups target smaller governments with fewer resources. New Orleans, Atlanta and Baltimore dealt with major attacks in 2018 and 2019. Tulsa also reported an attack by the Conti ransomware group in 2021.

Atlanta was forced to spend more than $9.5 million recovering from the incident and Baltimore reportedly spent $19 million dealing with their attack.

The ransomware incident in Dallas, a city of 1.3 million people, was one of several affecting cities both big and small across the U.S. this year.

Just weeks before the Dallas attack, the City of Oakland's networks were severely damaged by a wide-ranging ransomware attack that hampered city services for weeks and leaked troves of sensitive data about city residents and government officials onto the internet.

Since the attack on Dallas, several other municipalities have faced their own ransomware attacks, including the 200,000-resident city of Augusta, Georgia.

Keeper Security CEO Darren Guccione said that it is easy for outsiders to blame victims for a ransomware attack, but in his view the city of Dallas did handle certain measures of the response according to best practices.

“A cyberattack of this scale, against an entity of this nature, is almost guaranteed to have tangible effects,” he explained.

“The key for Dallas, as well as other cities, is to learn from this incident – both the positive and the negative aspects of it – and strengthen their defenses against future attacks accordingly.”