Reverse Engineering and Exploiting Builds in the Cloud

## Overview

In the presentation titled *Reverse Engineering and Exploiting Builds in the Cloud*, Etienne, Chris, and Matthias from the Heroku platform security team examine security architectures, multi-tenancy pitfalls, and misconfigurations in cloud-hosted Continuous Integration / Continuous Deployment (CI/CD) environments.

Rather than focusing on conventional container breakout techniques (such as exposed Docker sockets or excessive Linux capabilities), the speakers analyze how attackers leverage intentional build features—where arbitrary code execution is fundamentally expected—to target the underlying cloud infrastructure, control planes, and internal services.

---

## Multi-Tenancy Architecture Patterns in Cloud CI/CD

The speakers identify the essential components of cloud CI/CD platforms: compute nodes running tests and supporting services (e.g., mock databases), build-time subnets, CI controlling infrastructure, and the cloud control plane. They highlight three primary multi-tenancy models:

* **Container-Level Isolation (Single VM):** The most cost-effective approach, where multiple tenants share a single VM host and network, isolated only by container boundaries. An isolation failure or network access directly exposes neighboring tenant builds. * **VM-Level Isolation (Shared Network):** Each tenant is separated on dedicated virtual machines, but those VMs reside within a shared network. Network misconfigurations can still allow lateral interaction across tenant builds. * **Fully Dedicated Resources:** The most isolated and costly architecture, where a tenant receives dedicated VMs and an isolated network, preventing cross-tenant access.

The authors emphasize that because CI/CD systems accept configuration files (e.g., YAML) or Dockerfiles, achieving remote code execution inside build environments is trivial and expected. Security boundaries depend entirely on how strictly the runtime network and host resources are isolated.

---

## Network Bypasses via the Dockerfile `ADD` Directive

While providers attempt to lock down build network egress—specifically restricting access to internal metadata services—the mechanics of intermediate Docker build containers introduce architectural blind spots:

* **Intermediate Container Isolation:** Standard commands run via `RUN` directives (e.g., executing `curl` or `wget`) run inside intermediate build containers with their own virtual networks. In hardened environments like Google Cloud Build, attempts to query internal cloud metadata services via `curl` time out due to network-level filtering. * **The `ADD` Directive Bypass:** Unlike `RUN`, the Docker `ADD` directive executes directly in the context of the host's Docker daemon. As a result, it does not share the restrictive virtual network rules imposed on intermediate containers. * **Mechanism:** Specifying a remote URL in an `ADD` directive forces the Docker daemon running on the host to fetch the remote resource directly over the host network stack, bypassing intermediate container egress restrictions and accessing services normally restricted from tenant workloads.

---

## Case Study: Google Cloud Build Exploitation and Privilege Escalation

The speakers demonstrate this bypass in Google Cloud Build:

1. **Metadata Harvesting:** While standard `curl` commands timed out when attempting to reach the cloud metadata service, replacing them with `ADD` directives allowed the retrieval of sensitive attributes, including the instance secret key and IP address. 2. **Internal Pivoting:** By chaining multiple `ADD` directives, the researchers extracted additional credentials, including TLS private and public certificates. 3. **Internal Service Access:** Using these recovered certificates, they queried an internal Cloud Build logging service that records Docker execution commands—an internal endpoint normally inaccessible to build users. 4. **Token Extraction and Scope Abuse:** Inspecting the retrieved Docker logs revealed an OAuth 2.0 access token used by the Docker daemon to authenticate to the Google Container Registry (GCR). Decoding the token revealed broad scopes extending beyond GCR to Google Cloud Platform (GCP), source tools, and Cloud KMS.

---

## Practical Impact and Takeaways

* **Multi-Tenant Exposure:** In multi-tenant environments where untrusted users are only permitted to submit Dockerfiles under the assumption that metadata and internal systems are blocked, the `ADD` directive breaks that isolation assumption, enabling full cloud environment access and resource provisioning. * **Real-World Abuse:** The researchers noted that similar bypasses historically affected platforms like GitHub Actions prior to architectural updates, allowing attackers to circumvent fraud controls and obtain unauthorized compute resources.