Microsoft: Big Cryptomining Attacks Hit Kubeflow

Microsoft has spotted a new, widespread, ongoing attack targeting Kubernetes clusters running Kubeflow instances, in order to plant malicious TensorFlow pods that are used to mine for cryptocurrency.

The Kubeflow open-source project is a popular framework for running machine learning (ML) tasks in Kubernetes, while TensorFlow is an end-to-end, open-source ML platform.

Given that the attack is still active, any new Kubernetes clusters that run Kubeflow could be compromised, according to Microsoft.

On Tuesday, Microsoft security researchers warned that toward the end of May, they saw a spike in deployments of TensorFlow pods on Kubernetes clusters – pods that are running legitimate TensorFlow images from the official Docker Hub account. But a closer look at the entry point of the pods revealed that their purpose is to mine cryptocurrency.

Yossi Weizman, senior security research software engineer at Microsoft’s Azure Security Center, said in a post on Tuesday that the “burst” of these malicious TensorFlow deployments was “simultaneous,” indicating that the attackers initially scanned the clusters, kept a list of potential targets, and then pulled the trigger on all of them at once.

Weizman explained that the attackers used two separate images: The first is the latest version of TensorFlow (tensorflow/tensorflow:latest) and the second is the latest version with GPU support (tensorflow/tensorflow:latest-gpu). The use of TensorFlow images in the cluster “makes a lot of sense,” Weizman said, given that “if the images in the cluster are monitored, usage of [a] legitimate image can prevent attackers from being discovered.”

Another reason why the attackers’ choice is understandable is that the TensorFlow image they chose is a convenient way to run GPU tasks using CUDA, which “allows the attacker to maximize the mining gains from the host,” he said. CUDA is a toolkit created by NVIDIA, used to develop, optimize and deploy GPU-accelerated apps.

Similar to Last Year’s Cryptomining Attack

The newly discovered attack is similar to a cryptocurrency mining attack that Microsoft reported last June. That earlier campaign also targeted Kubeflow workloads, exploiting misconfigured dashboards to launch a widespread XMRIG Monero-mining campaign. Fast-forward a year, and this recently discovered cryptomining pulls a similar move, using exposed Kubeflow interfaces for running cryptocurrency mining containers.

The latest campaign adds some tweaks: As Weizman described it, this time around, the attackers abused the access to the Kubeflow centralized dashboard in order to create a new pipeline.

As Weizman detailed in the post, Kubeflow Pipelines is a platform for deploying ML pipelines, based on Argo Workflow, which an open-source, container-native workflow engine for orchestrating parallel jobs on Kubernetes. Pipeline entails a series of steps, each one of them as an independent container, that together form a ML workflow. The image of the container that runs in each step is determined in the pipeline configuration, he said.

Access to the pipeline’s user interface is key in this attack: Once attackers gain access to that dashboard, they can create a new cluster in the pipeline. In this case, that means containers that run TensorFlow images that set up cryptocurrentcy mining.

All of the malicious pods were set up with the same pattern: “sequential-pipeline-{random pattern}”. That name is originated in the “generateName” field of the Argo Workflow object that’s used for creating the pipeline, Weizman said.

At least two pods were deployed on each cluster: One for CPU mining, and the other for GPU mining. The GPU container used the open-source Ethminer to mine Ethereum, while the CPU miner used the aforementioned open-source XMRIG Monero miner.

As part of the ongoing attack flow, the attackers are using a reconnaissance container – also run from a TensorFlow pod – to scoop up information about the environment, such as GPU and CPU details, in preparation for mining.

What to Do to Avoid a Cyberattack

Microsoft advised that those who run Kubeflow should make sure they’ve locked down the centralized dashboard so it’s not insecurely exposed to the internet. If Kubeflow has to be exposed to the internet, make sure it requires authentication.

Microsoft gave the example of Kubeflow being used to support OpenID Connect (OIDC) using Azure Active Directory for Azure deployments. In order to get all the pods running in the cluster in JSON format, run: kubectl get pods –all-namespaces -o json …

… and search for containers that run TensorFlow images. If they exist, inspect the entry point of those containers, Microsoft advised.

Cryptomining: Lead Weights That Bog Down the Cloud

Yaniv Bar-Dayan, CEO and co-founder at Vulcan Cyber, said that cryptomining is nothing to shrug off, given the hit it puts on cloud resources.

“While cryptomining campaigns can seem innocuous, they put excess strain on cloud resources, inflict additional cloud and utility costs on attacked parties, shorten the lifespan of IT devices and cause unnecessary business disruption,” he told Threatpost via email on Thursday.

These campaigns also highlight organizations’ risk of exposure, he noted – in other words, if an attacker can pull off a cryptomining attack, they can pull off even worse. “If an attacker can launch a cryptomining campaign on an organization’s infrastructure, it’s likely that they can launch ransomware as well or gain access to data, intellectual property, personnel files and other at-risk assets that can damage a business if breached,” he continued.

Bar-Dayan said that Vulcan Cyber recommends taking the appropriate steps for defense. For example, ensure proper configurations, and, echoing Microsoft, make sure that systems aren’t exposed to the open internet. Also, make sure the right identity and access controls are in place.

Download our exclusive FREE Threatpost Insider eBook, “2021: The Evolution of Ransomware,” to help hone your cyber-defense strategies against this growing scourge. We go beyond the status quo to uncover what’s next for ransomware and the related emerging risks. Get the whole story and DOWNLOAD the eBook now – on us!