Breaking Out of The AI Cage: Pwning AI Providers with NVIDIA Vulnerabilities

Overview: Cloud security researchers Hilik Ben-Sasson and Andres Riancho from Wiz Research present a critical infrastructure vulnerability in the NVIDIA Container Toolkit at Black Hat. As organizations race to deploy GPU-accelerated artificial intelligence pipelines, the NVIDIA Container Toolkit has become a foundational component enabling GPU passthrough within Docker and Kubernetes environments. The researchers detail how a Time-of-Check to Time-of-Use (TOCTOU) file mounting flaw allowed unprivileged container workloads to escape isolation, compromise host nodes, and harvest proprietary AI models and credentials across multi-tenant AI cloud providers.

Key Topics: - Architecture of modern AI cloud infrastructure and NVIDIA Container Toolkit GPU virtualization - Discovery and mechanics of a container escape TOCTOU vulnerability during driver volume mounting - Weaponizing container images with symbolic link chains to mount the host root filesystem - Multi-tenant cloud compromise: extracting Kubelet credentials, tenant secrets, and proprietary AI models

Technical Details: The researchers investigated the mechanism by which the NVIDIA Container Toolkit exposes host GPU hardware to containers. When launching an accelerated container, the toolkit executes a hook that inspects container file paths and dynamically mounts NVIDIA driver libraries and device nodes (such as CUDA binaries in `/usr/lib`) into the container. The vulnerability lies in a TOCTOU flaw: the toolkit executes an initial security validation pass on destination directories, but during subsequent mount execution calls, it operates on a filesystem structure that can be dynamically modified by the container's layout. Because mounts modify path resolutions on the fly, the initial security assertions become completely invalidated.

To weaponize this vulnerability into a reliable container escape, the authors constructed a custom Dockerfile with a multi-stage symbolic link structure. They linked `/usr/lib` to a staging directory (`compat`) containing pre-created folders (`libnvidia-6` and `libnvidia-7`). During startup, the toolkit's security checks approved both folders. However, when the toolkit performed the first mount operation against the symbolic link, it overwrote `compat`'s internal path hierarchy. When the second mount operation executed against the dynamically created link, the toolkit traversed out of the container boundary and mounted the host's root filesystem directly inside the container. Testing this exploit on a major AI cloud provider (DigitalOcean Paperspace), the researchers gained full read/write access to the underlying Kubernetes node. They extracted Kubelet credentials from `/etc/kubernetes`, accessed cluster-wide management APIs spanning over 700 shared nodes, dumped neighboring tenant containers, extracted image pull secrets, and exfiltrated private customer AI models and proprietary source code.

Key Takeaways: AI cloud providers must not rely solely on standard container runtimes to isolate untrusted user workloads sharing physical GPU nodes. Multi-tenant AI platforms should deploy hardened hypervisor-level virtualization (such as Kata Containers or Firecracker microVMs) to isolate untrusted model execution. Security teams must ensure that container runtime hooks and hardware passthrough toolkits perform atomic directory validations using secure file descriptors rather than resolving mutable path strings susceptible to symlink race conditions.

(AI summary — Gemini Flash)