A CISO describes the emotional impact of ransomware attacks

In the cybersecurity industry, a lot of attention is given to the attackers. Figuring out who is behind an incident, what tools they’re using, and who they might target next can help organizations prevent future breaches. But often very little notice is given to the defenders—the front-line security personnel who work around-the-clock to remediate incidents and recover corporate systems.

Jason Lewkowicz talked to The Record recently about the challenges of working through one of these attacks, which included 20-hour days, skipped meals, and intense periods of stress. “The feeling of, ‘What have I gotten myself into?’ was overwhelming, without question,” he said. The following conversation has been lightly edited for clarity:

The Record: You’ve been with your company for only one year, correct?

Jason Lewkowicz: It’s been exactly one year. I joined the day after the announcement of our security breach.

TR: That first day must have been intense.

JL: My hiring had nothing to do with the security breach. They wanted to bring in new external talent from an organization that knew what good looked like to help drive their security program to the next level. As luck would have it, I had been the Managing Director responsible for incident response at my previous company for 17 years, so this was an area that I was very fluent in. I was aware of the Maze group and what they did and how they operated. 

My joining was a bit of a blessing and a curse. I was not looking forward to joining a company in the midst of an incident, the timing was terrible. And I was joining in April, when organizations were rapidly changing their business models to deal with the pandemic. So I had that plus the breach and it was just an unprecedented level of stress —emotionally tolling to say the least.

TR: I’m guessing when you agreed to take the job you were thinking you would spend the first few weeks or months getting to know the team and figuring out your priorities. Can you walk me through what it was like for everything to change so rapidly?

JL: The big mind shift for me was that I put aside the initial 30-60-90 day approach that you would take joining a company, which would be meeting the teams, understanding their processes, looking at their data, what have they been saying to the board, what commitments they have, what are their program plans. All normal new-joiner stuff was put to the side. And it was a mind shift from being a business operator and planner to being a firefighter.

I’m joining a company where I have no brand and no relationships to fall on. A lot of their systems in the moment are down and offline… The feeling of, “What have I gotten myself into?" was overwhelming, without question.”

Thankfully, one of the pieces which I drilled into my previous team was continuous cyber exercises, sometimes with live fire, sometimes just tabletop. The ability for me to switch back into that mindset was relatively easy because there was a lot of muscle memory built. The big challenge I faced was not having fluency with their incident response plan, and not having any established relationships, and who the right people are to contact. A lot of their systems in the moment are down and offline. I’m joining a company where I have no brand and no relationships to fall on. A lot of their systems in the moment are down and offline… The feeling of, “What have I gotten myself into?” was overwhelming, without question.

TR: How did you deal with the pressure? Was it working around the clock, with daytime firefighting and nighttime onboarding?

JL: Nope, it was entirely spent firefighting. I would say the first three days I probably worked 20 hours a day. It was very taxing. And one of the things which is extremely important when managing incidents is ensuring your team is taking breaks, eating food, and getting rest. And that was a challenge to deal with, because in a cyber crisis there is this expectation of recover, restore, get back online—all the things, so that we can continue delivering business. There’s no, “It’s totally fine. Let your team go and have a nap.” The business pressures and concern of clients is continuously escalating.

Any organization that gets impacted with a security breach today… many of your contracts say you have to make it known to your customers. And many want to disconnect until they have the all clear—and they don’t want you giving them the all clear, they want it from a third party, which introduces its own challenges.

When you factor in the volume of clients that we have, the other big challenge is every customer wanting to be on the phone with the CISO. It’s my second day, and I’m getting asked: “What are you guys doing about this? What’s your program plans? What are your SLAs? What type of tech and tooling do you have? How do you know it’s working properly?”

I don’t have any of those answers, so that’s also extremely overwhelming. And as far as how did I deal with it? There isn’t time to really think about it because everything is moving so fast. But there’s definitely a lesson learned, I think, for anybody who’s going through this type of thing, which is communications. You must have a communications plan that’s robust and you have to communicate in a way that doesn’t back yourself into a corner.

Most often clients want to know immediately, was any of our data impacted? And the truth is, rarely is that known at the time a breach is discovered. So I think that’s a very big hurdle, because as the CISO, when an incident takes place, everybody’s looking to you. It’s this weird hybrid role that’s often at times kind of just as important as your CFO, COO, CEO, where everything’s resting on you, everyone’s looking at you for direction, all parties expect something faster than humanly possible to deliver. And it’s overwhelming.

TR: Did you have access to the person who formerly held your role? 

JL: I did—my current company, has a Chief Security Officer who has responsibility for corporate, delivery, and physical security. So he was acting in both roles and obviously the CSO post my joining. I of course had access to him.

As the CISO, when an incident takes place, everybody’s looking to you. It’s this weird hybrid role that’s often at times kind of just as important as your CFO, COO, CEO, where everything’s resting on you, everyone’s looking at you for direction, the clients are expecting something faster than humanly possible to deliver. And it’s overwhelming.”

TR: Had you experienced anything like this in your previous role, or was it totally new?

JL: Every organization goes through what I’ll call fire drills, where some zero-day vulnerability is released and there is the same type of reaction mentality. With those, at the organizations I worked with before my current employer, they ended up being non-events. But for 72 to 196 hours, we were working around the clock ensuring that there wasn’t a compromise, that we had compensating controls in place, that we were validated and that we didn’t have breaches on any of the systems that were vulnerable or exposed at the time. Between 2016 and 2020, every organization likely went through five to seven of those “holy crap” moments over the years. The challenge, of course, at a large organization is that your footprint is massive. And oftentimes organizations are trying to do as much as they can with as little as they can, and they’ll have third party surge support. But when you have a NotPetya event, your surge support is likely spread thin because everybody is having that same type of event. That’s one of the challenges that you have with the internet and malware.

TR: At what point would you say that you got a breath of air?

JL: It was probably all the way until the time that we had claimed containment, and then it slowly started moving back around the time of eradication. So I would say from April 20, my join date, until the middle of June, it was continuous burn. More than 1,100 clients all want to talk to us repeatedly—and I get it. Our clients and our people are our assets, so no clients, no revenue, no company. Of course we want to do all we can to get on the phone with them. And in most cases, what they’re looking for is how can we educate them so this type of thing doesn’t happen to them. They all understand security breaches happen—but what lessons are there in this for them. My challenge being there is only one of me and there’s only so many hours in a day. Most clients don’t want to talk to the people you have educated on the topic—they want the chief. 

The demand is hard. My health suffered. I’m a pretty physically fit and active guy. Part of what makes me high performing is that I work out to de-stress. Not having that, I felt like I was carrying a lot of negative energy and I was far more short with my children and my spouse. That was very challenging for the family. They were great in terms of supporting me. They know what I do. They’re familiar with what I call “the rodeos.” But I didn’t feel like I was present enough for my family. And there was no work life balance—it was all work. So I didn’t have a stress outlet, I wasn’t getting enough sleep, I wasn’t drinking enough water. The recharge, I think, is a huge component that often gets overlooked. And one of the things I was trying to pivot the team to was if you need to work 12 hours, work 12 hours, but do it in 4 hour chunks and go for a walk or eat something healthy. Do whatever you need, but step away so that your brain has the ability to relax, because when you’re on these calls and it’s just peppered with questions, whether it’s with the board, the CEO, or a client, it’s a huge stressor.

TR: And this was all taking place from your house, right?

JL: Entirely remote. Everything was closed, everything was on lockdown. We were able to do all of our response efforts remotely—there was no physical war room, which is common during an incident. Everyone will convene in one or multiple locations, where you’ve got telebridges and video conferencing and whatnot set up… that wasn’t afforded to us because of the pandemic.

TR: Those war rooms are really important, because you get to at least see your coworkers and support them—or at the very least get them pizza.

JL: Absolutely. I have the ability to play parent and foster them through because a lot of people don’t have experience going through something like this and don’t realize just how taxing it is on you. For a lot of our teams, there is a feeling of guilt or responsibility. What could I have done differently to prevent this? Was this my fault? Am I working hard enough to recover things? Especially thinking back to the beginning of a pandemic, you’ve got customers who are working on the front lines to find vaccines. You had a lot of people feeling defeat, remorse, guilt, and there’s a whole component to it where I think the emotional support that needs to be provided to the team is probably just as equal to the technical aptitude that’s needed to navigate out of one of these events. As the incident commander, spending time with your team and making sure that they’re resting, recharging, and you’re asking them the emotional questions like, are you OK or is there anything that I can do for you is huge.

For a lot of our teams, there is a feeling of guilt or responsibility. What could I have done differently to prevent this? Was this my fault? Am I working hard enough to recover things? Especially thinking back to the beginning of a pandemic, you’ve got customers who are working on the front lines to find vaccines. You had a lot of people feeling defeat..."

TR: What initiatives helped bring people together?

JL: One of the big challenges was not having any internal established relationships. I’m the new guy, and any time you have a new boss coming into an organization, there’s anxiety about what change this individual is going to bring. What impact is it going to have on me? All the team has that stress where they’re also trying to overperform to make a great impression on me. And I would often take a moment to tell a joke—let’s try and do something funny to bring some humanity to the conversation and not be so serious. And I would oftentimes remind them I understand we are going through a crisis. This is a war right now. And the fact that you haven’t eaten or slept or exercised or, in many cases, showered, I’m not so focused on those things. We’ll have time to talk about what changes and things that we want to do once we get our handle on all this stuff. But for now, just keep doing what you’re doing.

The incident ended up helping the team have confidence in me, build trust in me, knowing that I genuinely did have their back. And it ended up being a great time to join an organization—you can really see the performance of a team and deficiencies of a team in a crisis. You can see where you’re weakest, because it’s live fire. It becomes very apparent where things need to be improved, which was fantastic for me—the 30-60-90 I got to do in like three days.

TR: What kind of feedback have you gotten from your team since the incident?

JL: The main feedback that I received from the team during the incident and up until now has just been appreciation. They appreciated the fact that I knew what to do and how to provide them guidance without being negative. I gave them the leeway to make mistakes and figure things out so that it was a learning experience. It wasn’t just someone telling them what to do, which I think they found to be very helpful. The other piece, too, is air cover. There wasn’t a, “Yes, we’ll get that done even though the timeline is unrealistic.” I believe very much in setting expectations and managing them, and sometimes the expectations that you set out will end up being not liked. And so with every inquiry that was made of the team, I would say here’s how long it’s going to take to do, this is as fast as we can do it even if you bring in new people, even if you bring in new technology. We have reached maximum velocity and we’re not going to be able to go any faster, and if you ask us to do it faster, you run the risk of things breaking. So, I believe, there was a significant amount of appreciation for the fact that I was not setting them up for failure.

TR: Can you give a sense of the scale of the organization? How many people have some sort of responsibility for cybersecurity?

JL: The enterprise has roughly 300,000 employees. My headcount for corporate security is appropriately proportioned for our size. But as far as people who are involved, everybody was rolling up their sleeves. Everybody had some level of responsibility. Threat vulnerability management teams who are looking at our perimeter hygiene were out there continuously scanning, making sure our infrastructure and applications were secure.

One of the big concerns for organizations when they’re breached is an immediate follow-on breach. People know you’re down, so they say let’s pile on and go after that organization. There was a DEFCON 5 for a few months and it really hasn’t shifted. We’ve maintained the same level of scanning and offensive security, looking for holes and vulnerabilities. Where are we insecure? Where do we identify any deficiencies? There were lots of tuning exercises and talking with intelligence providers and really fine-tuning what intelligence we’re getting, what was adding value, what wasn’t adding value. It was all about building a taxonomy and a way to measure a lot of the programs that we were putting in place so that we could articulate where we were, to where we are to where we’re going, which I think is a very important component in managing a team.

The different threat actors out there all specialize in different things—they have specific techniques they use or different weapons they like to use, and they will also go after specific organizations. So to really understand at a base level who would be interested in the professional services space, what are the techniques that they use, how do we measure resiliency against those techniques… Across the entire company, people were busy. They’re still VERY busy. 

TR: What other silver linings do you think came from the incident?

JL: We definitely took a step forward in terms of how we operationalize and how we measure ourselves—that was a big, big one. We are more proactive and we’re moving to a more proactive model in terms of where we want to be and what are the steps that we’re going to take to get there. 

Have you actually walked through your incident response plan and done a tabletop—and not a theatrical tabletop that some companies do where everything’s perfect and shows how polished you are, but one where everybody falls over and makes mistakes and where you appoint a junior guy to be the commander.”

As far as investment from the board and from our CEO and our executive committee, they were always invested in cyber security. There was no question that they knew about what the cyber landscape looked like because my predecessor was providing that information back to them on a regular basis. But through this event, there was a very focused microscope lens on what we’re doing and how we’re doing it. And it’s created a very different outcome in terms of why we do things. There’s a huge focus now on security and data protection, and it’s something that we’re trying to build into the fabric of our company. The more awareness that we can bring to the type of threats and the protection and handling of data, the better off will be—because security needs to start with employees.

And I think a lot of this resonates with clients, because we have a really clear narrative explaining why we’re secure. These are the things we have done, this is how we measure that we’ve done them successfully, here’s the benchmark score showing that we’ve accomplished certain levels of hygiene.

TR: What final advice would you give to CISOs and companies that are trying to prepare themselves for these types of incidents?

JL: I think questions that CISOs should be asking are what data do you have that you care about, and where is it? Have you actually walked through your incident response plan and done a tabletop—and not a theatrical tabletop that some companies do where everything’s perfect and shows how polished you are, but one where everybody falls over and makes mistakes and where you appoint a junior guy to be the commander. I think there’s so much value there that would speed up how an organization responds to incidents.

Ask questions like have you ever tried to restore from backups? How long does it take to restore them? If all 40 of your applications that you marked critical get knocked down, in which order would you prioritize turning them back on? There needs to be a focus on these pieces before an incident so that when you experience one it becomes muscle memory and people just know what to do. I’m not seeing enough time spent there, and I make that statement based on the continuous interactions I have with my peers. It’s a time suck to do those things—it’s not easy.

Anybody who becomes a CISO in the next five years will eventually find themselves in the seat as incident commander, whether it’s a small thing where you find a random breach or something like a ransomware attack. And these are great lessons for dealing with it.