CI/CD Methodology
Last updated
Last updated
https://cloud.hacktricks.xyz/pentesting-ci-cd/pentesting-ci-cd-methodology
Ansible Tower / AWX / Automation controller Security
Dev environments have become a major part of today’s attack surface. And within them, the most lucrative assets are the systems responsible for CI and CD — those that build, test, and deploy code — and typically possess the secrets and access to the most critical assets of the organization. So it’s only natural that attackers are continuously on the lookout for novel ways to gain access to these systems.
VCS stands for Version Control System, this systems allows developers to manage their source code. The most common one is git and you will usually find companies using it in one of the following platforms:
Github
Gitlab
Bitbucket
Gitea
Cloud providers (they offer their own VCS platforms)
Pipelines allow developers to automate the execution of code (for building, testing, deploying... purposes) after certain actions occurs: A push, a PR, cron... They are terrible useful to automate all the steps from development to production.
However, these systems need to be executed somewhere and usually with privileged credentials to deploy code.
Even if some VCS platforms allow to create pipelines for this section we are going to analyze only potential attacks to the control of the source code.
Platforms that contains the source code of your project contains sensitive information and people need to be very careful with the permissions granted inside this platform. These are some common problems across VCS platforms that attacker could abuse:
Leaks: If your code contains leaks in the commits and the attacker can access the repo (because it's public or because he has access), he could discover the leaks.
Access: If an attacker can access to an account inside the VCS platform he could gain more visibility and permissions.
Register: Some platforms will just allow external users to create an account.
SSO: Some platforms won't allow users to register, but will allow anyone to access with a valid SSO (so an attacker could use his github account to enter for example).
Credentials: Username+Pwd, personal tokens, ssh keys, Oauth tokens, cookies... there are several kind of tokens a user could steal to access in some way a repo.
Webhooks: VCS platforms allow to generate webhooks. If they are not protected with non visible secrets an attacker could abuse them.
If no secret is in place, the attacker could abuse the webhook of the third party platform
If the secret is in the URL, the same happens and the attacker also have the secret
Code compromise: If a malicious actor has some kind of write access over the repos, he could try to inject malicious code. In order to be successful he might need to bypass branch protections. These actions can be performed with different goals in mid:
Compromise the main branch to compromise production.
Compromise the main (or other branches) to compromise developers machines (as they usually execute test, terraform or other things inside the repo in their machines).
Compromise the pipeline (check next section)
The most common way to define a pipeline, is by using a CI configuration file hosted in the repository the pipeline builds. This file describes the order of executed jobs, conditions that affect the flow, and build environment settings. These files typically have a consistent name and format, for example — Jenkinsfile (Jenkins), .gitlab-ci.yml (GitLab), .circleci/config.yml (CircleCI), and the GitHub Actions YAML files located under .github/workflows. When triggered, the pipeline job pulls the code from the selected source (e.g. commit / branch), and runs the commands specified in the CI configuration file against that code.
Therefore the ultimate goal of the attacker is to somehow compromise those configuration files or the commands they execute.
The Poisoned Pipeline Execution (PPE) vector abuses permissions against an SCM repository, in a way that causes a CI pipeline to execute malicious commands. Users that have permissions to manipulate the CI configuration files, or other files which the CI pipeline job relies on, can modify them to contain malicious commands, ultimately “poisoning” the CI pipeline executing these commands.
For a malicious actor to be successful performing a PPE attack he needs to be able to:
Have write access to the VCS platform, as usually pipelines are triggered when a push or a pull request is performed. (Check the VCS pentesting methodology for a summary of ways to get access).
Note that sometimes an external PR count as "write access".
Even if he has write permissions, he needs to be sure he can modify the CI config file or other files the config is relying on.
For this, he might need to be able to bypass branch protections.
There are 3 PPE flavours:
D-PPE: A Direct PPE attack occurs when the actor modifies the CI config file that is going to be executed.
I-DDE: An Indirect PPE attack occurs when the actor modifies a file the CI config file that is going to be executed relays on (like a make file or a terraform config).
Public PPE or 3PE: In some cases the pipelines can be triggered by users that doesn't have write access in the repo (and that might not even be part of the org) because they can send a PR.
3PE Command Injection: Usually, CI/CD pipelines will set environment variables with information about the PR. If that value can be controlled by an attacker (like the title of the PR) and is used in a dangerous place (like executing sh commands), an attacker might inject commands in there.
Knowing the 3 flavours to poison a pipeline, lets check what an attacker could obtain after a successful exploitation:
Secrets: As it was mentioned previously, pipelines require privileges for their jobs (retrieve the code, build it, deploy it...) and this privileges are usually granted in secrets. These secrets are usually accessible via env variables or files inside the system. Therefore an attacker will always try to exfiltrate as much secrets as possible.
Depending on the pipeline platform the attacker might need to specify the secrets in the config. This means that is the attacker cannot modify the CI configuration pipeline (I-PPE for example), he could only exfiltrate the secrets that pipeline has.
Computation: The code is executed somewhere, depending on where is executed an attacker might be able to pivot further.
On-Premises: If the pipelines are executed on premises, an attacker might end in an internal network with access to more resources.
Cloud: The attacker could access other machines in the cloud but also could exfiltrate IAM roles/service accounts tokens from it to obtain further access inside the cloud.
Platforms machine: Sometimes the jobs will be execute inside the pipelines platform machines, which usually are inside a cloud with no more access.
Select it: Sometimes the pipelines platform will have configured several machines and if you can modify the CI configuration file you can indicate where you want to run the malicious code. In this situation, an attacker will probably run a reverse shell on each possible machine to try to exploit it further.
Compromise production: If you ware inside the pipeline and the final version is built and deployed from it, you could compromise the code that is going to end running in production.
***Chain-bench is an open-source tool for auditing your software supply chain stack for security compliance based on a new CIS Software Supply Chain benchmark. The auditing focuses on the entire SDLC process, where it can reveal risks from code time into deploy time.
Check this interesting article about the top 10 CI/CD risks according to Cider: https://www.cidersecurity.io/top-10-cicd-security-risks/
On each platform that you can run locally you will find how to launch it locally so you can configure it as you want to test it
Gitea + Jenkins lab: https://github.com/cider-security-research/cicd-goat
***Checkov: Checkov is a static code analysis tool for infrastructure-as-code.
