Crypto-mining campaign targets Kubeflow installs on a large scale

Microsoft uncovered a malicious campaign targeting Kubeflow workloads to deploy TensorFlow pods that are used to mine cryptocurrency.

Microsoft researchers uncovered a malicious campaign targeting Kubeflow workloads to deploy TensorFlow pods that are used to mine for cryptocurrency.

The campaign was first spotted at the end of May, experts noticed the deployment of TensorFlow pods at scale on multiple Kubernetes clusters. The pods ran legitimate TensorFlow images to mine cryptocurrency. Kubeflow allows deploying machine learning (ML) workflows on Kubernetes.

“The burst of deployments on the various clusters was simultaneous. This indicates that the attackers scanned those clusters in advance and maintained a list of potential targets, which were later attacked at the same time.” reads the blog post published by Microsoft.

Attackers used two different images, the first is the latest version of TensorFlow (tensorflow/tensorflow:latest) and the second is the latest version with GPU support (tensorflow/tensorflow:latest-gpu). The TensorFlow images employed in the attacks allow running GPU tasks using CUDA, which allows the attacker to optimize the mining operations.

Threat actors abused the access to the Kubeflow centralized dashboard to create a new ML pipeline using the Kubeflow Pipelines platform.

“In this attack, the attackers abused the access to the Kubeflow centralized dashboard in order to create a new pipeline. Kubeflow Pipelines is a platform for deploying ML pipelines, based on Argo Workflow. Pipeline is a series of steps, each one of them is an independent container, and together they form a ML workflow. The image of the container that run in each step is determine in the pipeline configuration.” continues the post.

The malicious pods employed in the attacks were composed using the same pattern,
“sequential-pipeline-{random pattern}”.

Attackers deployed at least two pods for each cluster, one for CPU mining, and the other for GPU mining. The containers used open-source miners from GitHub, Ethminer and XMRIG.

Ethminer was used to mine using the GPU container, while XMRIG leverages the CPU for the mining activities.

The attackers also deployed a reconnaissance container to gather information on the targeted environment (i.e. GCP, CPU) prior to start the mining activity.

Microsoft recommends administrators secure their centralized dashboards and check for containers that run TensorFlow images and inspect them for any sign of abuse.

The researchers warn that the campaign is still active.

Microsoft is urging admins to check for containers that run TensorFlow images and inspect them for malicious activity.

Follow me on Twitter: @securityaffairs and Facebook

Pierluigi Paganini

(SecurityAffairs – hacking, Kubeflow)