CI/CD servers readily breached by abusing  SCM webhooks, researchers find

Webhook, line, and sinker

Cloud-based source code management (SCM) platforms support integration with self-hosted CI/CD solutions through webhooks, which is great for DevOps automation.

However, the benefits can come with security trade-offs.

According to new findings from researchers at Cider, malicious actors can abuse webhooks to access internal resources, run remote code execution (RCE), and possibly obtain reverse shell access.

Webhook IP ranges

Software-as-a-service (SaaS) SCM systems provide an IP range for their webhooks. Organizations must open their networks to these IP ranges to enable integration between the SCM and their self-hosted CI/CD systems.

“We knew the combination of a SaaS source control management system and a self-managed CI with the webhook service IP range allowed towards the CI is a common architecture, and we wanted to check our possibilities there,” Omer Gil, head of research at Cider, told The Daily Swig.

Catch up on the latest DevSecOps news

Attackers can use webhooks to get past an organization’s firewalls. But SCM webhooks have strict limits, and there is very little room to make modifications to webhook requests.

However, the researchers discovered that with the right changes, they could get beyond the limited endpoints available to SCM webhooks.

Accessing CI/CD endpoints

On the CI/CD side, the researchers ran their experiments on Jenkins, an open-source DevOps server.

“We chose Jenkins since it’s self-hosted and commonly used, but [our findings] can be applied to any system that is accessible from the SCM, like artifact registries for example,” Gil said.

On the SCM side, they tested both GitHub and GitLab. While webhooks have been designed to trigger specific CI endpoints, they could modify requests to direct them to other endpoints that return user data or the console output of pipelines. Nevertheless, limits remain.

“Webhooks are sent as a POST request, which limits the options against the target service, since endpoints used to retrieve data usually only accept the GET parameter,” Gil said. “While it’s not possible to fire a GET request through GitHub, in GitLab it’s a different case, since if the POST request is responded with a redirection, the GitLab webhook service will follow it with sending the GET request.”

Exploiting webhooks

Using GitLab, the researchers were able to use webhooks to combine POST and GET requests to access internal resources. Interestingly, some Jenkins resources are accessible without authentication.

“By default, some resources can be accessed anonymously. Having said that, it’s not very common for an organization to leave it as is – but some do allow anonymous access,” Gil said.

In case authentication was required, the researchers found that they could direct webhooks to the login endpoint and conduct brute-force password attacks against the CI/CD platform. Once authenticated, they obtained a session cookie that could be used to access other resources.

If the Jenkins instance had a vulnerable plugin, the webhook mechanism could exploit it. In the proof-of-concept video above, the researchers show that they could force a vulnerable Jenkins server to download a malicious JAR file, run it on the server, and launch a reverse shell endpoint for the attacker.

This finding is a reminder of the risks created when CI/CD servers are partially open to the internet.

“A hermetic solution is to deny inbound traffic from the SCM webhook service, but it usually comes with engineering costs,” Gil said. “Some countermeasures can be taken, like setting a secure authentication mechanism in the CI, patching, and making sure all actions in the server are saved in the logs.”

RECOMMENDED ‘Security teams often fight against developers taking control’ of AppSec: Tanya Janca on the drive to DevSecOps adoption