‘You get respect for owning what happened’ – SolarWinds’ CISO on the legacy and lessons of Sunburst

Security chief counts new build system and greater intel sharing among positive legacies of watershed cyber-attack

From the infamous NotPetya campaign to the CCleaner backdoor, watershed infosec moments are not infrequent, but the SolarWinds supply chain attack stands out as particularly consequential.

The so-called ‘Sunburst’ attack, which leveraged a backdoor implanted in a software update for IT performance monitoring system Orion, gave attackers access to 18,000 SolarWinds customers in 2020. Among them were Microsoft, NASA, security firm FireEye, and the US justice and state departments.

A hugely stressful experience for those affected, not least the man tasked with leading SolarWinds’ incident response. However, as the Texas-headquartered company’s chief information security officer (CISO) Tim Brown tells The Daily Swig, the aftermath has not been without salutary benefits.

He believes the advantages have accrued not just to SolarWinds itself – through a pivot to an organization-wide ‘secure by design’ paradigm – but attitudes to cyber-resilience in general, too.

“Many people believed that a nation-state attack of this level – being very patient, very stealthy, very quiet, very mission-centric – was [like something] out of a movie, that this was not real,” says Brown, who joined SolarWinds in 2017 as vice president of security.

But the attack, which was attributed to Russian state-linked hacking group APT29 (aka Cozy Bear or Nobelium), was all too real – and many organizations were ready to heed the lessons.

“This is what you need to be defending against – and not just from nation states,” says Brown. “The cybercriminal types, the ransomware types, are looking at [Sunburst] saying, ‘well, if I invest $5 million, I get $50 million – that’s a good return on my investment’.

“CISOs around the world thanked me for helping with their budget, because their boards asked: ‘Could this happen to us’?

And of course, the CISO says: ‘Yes, we need to invest here, here, and here’. So we actually injected a lot of security into the overall ecosystem.”

The incident has galvanized the US government, too. A flurry of initiatives, including an overhaul of government software procurement practices and a mobilization plan for securing the software supply chain, have drawn cautious approval from the likes of Randori’s Aaron Portnoy and Sonatype’s Brian Fox.

Further large-scale supply chain attacks against Codecov and Kaseya VSA, and the vulnerability in near-ubiquitous open source library Log4j, have only served to keep complacency at bay.

‘Human side’

Brown, who has previously been chief technology officer (CTO) at Dell, says that with the technical details of Sunburst now “well understood”, people now “want to know what was it like from the inside, for me personally – what’s it really like behind the scenes?”

Tim Brown

Arguably this interest in the “human side” partly reflects an anxiety that no organization, no matter how secure, is entirely immune from such sophisticated, targeted, and stealthy attacks.

Catch up with the latest software supply chain attack news

“It’s not an easy job,” says Brown. “I think we have a duty not to hide, a duty to explain risk in business terms, and elevate our risk posture to that. That helps us manage our own stress and manage risk for the company, but also just put it into the right context for everyone.”

Vendors like SolarWinds, whose network and infrastructure monitoring tools often require privileged access to sensitive data, can afford to have only a small risk appetite.

“What people need to understand is where they fit within critical streams, critical infrastructure, and supply chains,” says Brown. “Based on that, you can define your risk tolerance.

“So if your component is fenced off at the bottom of the ocean and takes an input and an output, and that’s all you do, the level of risk that you’re facing is pretty minimal, right? But if you’re sitting in the middle of a nuclear power plant, then your risk ends up being much higher.”

Ephemeral build environments

SolarWinds’ post-Sunburst revamp “has engineering implications, security implications, process and procedure implications”, says Brown. “Think of it as an umbrella to put security inside of our thought process early as possible in the process for everything that we do.”

SolarWinds has, among other things, given most employees YubiKeys and now has three security operations centers (SOCs) to gain visibility across the environment.

Central to the overhaul is a new software development process unveiled in June, underpinned by four principles. These include ephemeral build environments that self-destruct after completing specific tasks, meaning “you don’t have something static that can be attacked”.

SolarWinds has overhauled its software development process according to secure-by-design principles

The second pillar is deterministically constructed products from which by-products will always have identical, secure components.

Brown explains that two identical builds are, ordinarily, not binarily compatible because of divergent times, but SolarWinds has achieved deterministic builds that can be compared binarily.

The third pillar is around creating multiple build pipelines – a fast developer build, security build, and slower validation build – in parallel.

“I compare the results for assurance that nothing changed in that pipeline or associated supply chain,” Brown explains. “No one person has access to all three. We would need collusion between multiple people to affect the build system.”

The final pillar involves tracking software build steps for complete traceability and permanent proof of record.

Sharing is caring

SolarWinds is open sourcing components of its new build system in the professed spirit of sharing best practices.

“Our adversaries collaborate well, they have no problem sharing,” he notes, but says Sunburst has encouraged both more government-sponsored sharing and “grassroots sharing between private and private [entities]”.

This notably includes sharing insights on what attackers are “doing to go after certain industries”, he says, citing the contributions of industry-specific ISACs approvingly.

DON’T FORGET TO READ ‘Endemic’ Log4j bug to persist in the wild for at least a decade, US government warns

Microsoft, Fire Eye’s Mandiant, and numerous security researchers “pushed the envelope of what gets shared” following Sunburst.

He also credits the US Cybersecurity and Infrastructure Security Agency (CISA) with “amplifying the truth without ulterior motives”, while the UK’s NCSC “helped people understand the risks that they face and what they should do.”

‘Owning what happened’

Brown’s own experience of talking to thousands of customers in the wake of the attack has, meanwhile, advertised the importance of an empathetic approach to customer engagement.

“It’s important to be open and transparent and answer their questions,” he says.

“We had a pretty bad few months, but our customers did too. It was Christmas and thousands of customers and IT departments around the world had to figure out if it was affecting them. So never forget your customers, never forget their pain.”

Empathy plus transparency have underpinned SolarWinds’ reputational recovery, according to Brown.

“No one had ever really been as open, transparent, and forthcoming from an incident,” he claims. “One of the biggest lessons we can show the industry is that you get respect from owning what happened and then recover. You don’t need to hide.”

Nevertheless, media coverage was unavoidably damaging in the short to medium term, he concedes. “For the first few months, in general the CISOs at larger companies and governments were like, ‘This can happen to anybody, we understand [this was a] sophisticated actor. Thank you for owning [the situation]’.

“The press took a lot longer,” he adds. “Bad news tends to sell. But if your guiding light is helping the customers, then eventually it turns around, to the point of people recognizing that you are doing a good response.”

As well as being “open, honest, and humble” in the wake of incidents, concludes Brown, vendors must not overlook the importance of building “great products that customers love” – otherwise it’s easy for them to switch products, “no matter what you do”.

RELATED One in five data breaches due to software supply chain compromise, IBM report warns