Cloudflare sheds more light on Thanksgiving security breach in which tokens, source code accessed by suspected spies

Cloudflare has just detailed how suspected government spies gained access to its internal Atlassian installation using credentials stolen via a security breach at Okta in October.

In a write-up on Thursday, CEO Matthew Prince, CTO John Graham-Cumming, and CISO Grant Bourzikas said the Atlassian intrusion was detected all the way back on Thanksgiving Day, November 23, 2023, and that the trespassers were ejected the following day.

The October Okta security breach involved more than 130 customers of that IT access management biz, in which snoops swiped data from Okta in hope of drilling further into those organizations. Cloudflare was among those affected, as it was in 2022 as a result of a separate Okta intrusion.

Cloudflare acknowledged in October it was caught up in Okta's latest security meltdown, and is now disclosing more details about what happened.

The intruders – likely agents of a nation state, according to Prince et al – obtained one service token and three service account credentials through that 2023 Okta compromise. At the time, Okta indicated that information stolen from its customer support systems was pretty benign, and could be used in things like phishing or social engineering attacks. It turns out that session tokens, granting access into networks of the likes of Cloudflare, were taken from Okta's systems.

"One was a Moveworks service token that granted remote access into our Atlassian system," said Prince, Graham-Cumming, and Bourzikas on that note.

"The second credential was a service account used by the SaaS-based Smartsheet application that had administrative access to our Atlassian Jira instance, the third account was a Bitbucket service account which was used to access our source code management system, and the fourth was an AWS environment that had no access to the global network and no customer or sensitive data."

Because Cloudflare incorrectly believed those tokens were unused, it failed to rotate them. So the thief or thieves were able to use them to gain access to Cloudflare's systems.

From November 14, 2023 through November 17, 2023, the intruders appear to have been probing Cloudflare's systems, doing reconnaissance through its Confluence-based internal wiki, and its Jira bug database.

Further accesses were detected on November 20 and 21, following by the establishment of a persistence presence in the cloud corp's Atlassasian server via ScriptRunner for Jira. Having administrative access to Jira via the Smartsheet service, the snoops were able to install the Sliver Adversary Emulation Framework, a common tool for command-and-control connectivity and backdoor access.

The intruders also gained access to Cloudflare's Bitbucket source code management system, but efforts to access a console server linked to a not-yet-active datacenter in São Paulo, Brazil failed.

The intruders, according to the cloud giant, scoured the biz's wiki for information about remote access, secrets, and tokens. Also of interest were 36 Jira tickets, out of more than two million, that focused on vulnerability management, secret rotation, multi-factor authentication bypass, network access, and even the biz's response to the Okta incident.

This attack was performed by a nation-state attacker with the goal of obtaining persistent and widespread access to Cloudflare’s global network

The spies' interest in secrets was also evident in the 120 Bitbucket code repositories viewed out of a total of almost 12,000. Some 76 of the 120 were downloaded to the Atlassian server. While Cloudflare is uncertain whether these were exfiltrated, it's treating them as such. These repos were mostly related to the way backups work, global network configuration and management, identity, remote access, and Terraform and Kubernetes. A few contained encrypted secrets and those were immediately rotated even though they were strongly encrypted, according to the US CDN giant.

"Even though we understand the operational impact of the incident to be extremely limited, we took this incident very seriously because a threat actor had used stolen credentials to get access to our Atlassian server and accessed some documentation and a limited amount of source code," said Prince et al.

"Based on our collaboration with colleagues in the industry and government, we believe that this attack was performed by a nation-state attacker with the goal of obtaining persistent and widespread access to Cloudflare’s global network."

Cloudflare managed to expel the attackers by November 24, 2023, and set about assessing the damage and investigating what happened. Three days later, a company-wide remediation effort dubbed "Code Red" became the focus of much of its technical staff. And this project was assisted by external security firm Crowdstrike, which carried out an independent assessment of the cyber-assault.

Code Red concluded on January 5, 2024, but according to Prince, Graham-Cumming, and Bourzikas "work continues across the company around credential management, software hardening, vulnerability management, additional alerting, and more." ®