Hackers exploit Ray framework flaw to breach servers, hijack resources

A new hacking campaign dubbed "ShadowRay" targets an unpatched vulnerability in Ray, a popular open-source AI framework, to hijack computing power and leak sensitive data from thousands of companies.

According to a report by application security firm Oligo, these attacks have been underway since at least September 5, 2023, targeting education, cryptocurrency, biopharma, and other sectors.

Ray is an open-source framework developed by Anyscale that is used to scale AI and Python applications across a cluster of machines for distributed computational workloads.

The framework boasts over 30,500 stars on GitHub, and it is used by many organizations worldwide, including Amazon, Spotify, LinkedIn, Instacart, Netflix, Uber, and OpenAI, that use it for training ChatGPT.

Active exploitation underway

In November 2023, Anyscale disclosed five Ray vulnerabilities, fixing four tracked as CVE-2023-6019, CVE-2023-6020, CVE-2023-6021, and CVE-2023-48023.

However, the fifth bug, a critical remote code execution flaw tracked as CVE-2023-48022, was not fixed because, according to them, its lack of authentication was a long-standing design decision.

"The remaining CVE (CVE-2023-48022) - that Ray does not have authentication built in - is a long-standing design decision based on how Ray's security boundaries are drawn and consistent with Ray deployment best practices, though we intend to offer authentication in a future version as part of a defense-in-depth strategy," reads the AnyScale security advisory.

Specifically, Anyscale stated that the flaw is exploitable only in deployments that violated the recommendations in the project's documentation to limit Ray's use in a strictly controlled network environment.

Furthermore, Anyscale does not believe these flaws are vulnerabilities but instead bugs, as their platform is designed to execute code as a distributed execution framework.

Regardless of how the flaw is classified, its lack of authentication has created an opportunity for hackers who have been exploiting the CVE-2023-48022 bug in unsecured environments.

"Because CVE-2023-48022 was disputed, many development teams (and most static scanning tools) are not aware that this vulnerability should concern them," reads Oligo's report.

"Some of them might have missed this documentation section of Ray, while some of them are unaware of this feature."

"[We...] have observed instances of CVE-2023-48022 being actively exploited in the wild, making the disputed CVE a "shadow vulnerability"—a CVE that doesn't show up in static scans but can still lead to breaches and significant losses."

Oligo discovered that hundreds of publicly exposed Ray servers were compromised via CVE-2023-48022, allowing attackers to access sensitive information, including AI models, environment variables, production database credentials, and cloud environment access tokens.

Exposed secrets
Source: Oligo

In some observed cases, the attackers used their access to powerful graphics cards in machine learning training to conduct cryptocurrency (Monero) mining operations.

Multiple XMRig miners running on a compromised server
Source: Oligo

Others utilize reverse shells to gain persistence in compromised environments, executing arbitrary code through Python pseudo-terminals.

Reverse shells in jobs history
Source: Oligo

The researchers also investigated base64 encoded payloads, revealing attempts to escalate privileges on compromised machines using open-source scripts that are not detected by any AV engines on VirusTotal.

Defending against ShadowRay

Following these discoveries, Oligo says they alerted many companies that were breached using the Ray bug and provided assistance with remediation.

To secure Ray deployments, it's crucial to operate within a secured environment by enforcing firewall rules, adding authorization to the Ray Dashboard port, and continuously monitoring for anomalies.

Additionally, avoid using default settings like binding to 0.0.0.0 and leverage tools that enhance the clusters' security stance.