LF Live Webinar: Building and Maintaining Your Own Secure Container OS

### Overview In this Linux Foundation Live Webinar, Curtis Rissi discusses Bottlerocket OS, an open-source, minimal, and secure Linux-based operating system purpose-built by AWS for running containerized workloads. The presentation covers the security architecture of Bottlerocket, how its custom build system works to generate target variants and disk images, how atomic updates are performed, and mechanisms for extending host capabilities safely using SELinux permissions, host/bootstrap containers, and eBPF.

### Key Topics - **Design & Security Architecture**: Bottlerocket enforces security by default using SELinux in enforcing mode to strictly isolate workloads from the host. It employs a read-only root filesystem backed by dm-verity, which checks block-level hashes and blocks I/O operations if unauthorized tampering occurs. Configuration directories like `/etc` operate as a stateless filesystem populated at boot via helper tools. The OS excludes package managers, interpreters, and direct login shells, minimizing attack surface. - **Variants & Custom Builds**: Bottlerocket acts as a container host OS builder. Users define "variants" specifying software packages, configurations, and disk layouts. While AWS primarily supports Amazon EKS and Amazon ECS (in preview during the talk), variants can theoretically target bare metal, other clouds, or architectures such as ARM64 (e.g., AWS Graviton). - **Atomic Updates via TUF & Updog**: Updates are distributed via The Update Framework (TUF) repositories using cryptographic verification. Bottlerocket applies whole-image atomic updates to a secondary passive partition using an internal tool called Updog, which swaps boot priorities and automatically rolls back if a boot failure is detected. - **Extensibility & Privilege Separation**: The OS isolates container runtimes into two separate environments: one for scheduled workloads (e.g., Kubernetes pods or ECS tasks) and another for administrative host containers. Workloads run by default in restricted SELinux domains (`container_t`), but can be elevated to `control_t` or `super_t` when necessary. Curtis also highlights non-orchestrated host containers, bootstrap containers (for pre-service initialization), and kernel extensions via eBPF as preferred extensibility models.

### Technical Details - **Build Pipeline**: First-party tooling is built using Rust and Cargo. The build system employs Cargo as a dependency orchestrator, RPM spec files for defining package sources, and the Bottlerocket SDK inside Docker containers. Source dependencies are downloaded and compiled locally into cached RPMs. - **Compilation Tools**: Each package leverages `build.rs` and an internal tool called `buildsys` to trigger Docker-based compilation. The final variant disk image is assembled using `rpm2image`, which can then be converted into cloud formats like an Amazon Machine Image (AMI). - **Kernel & eBPF**: Bottlerocket includes eBPF support in its Linux kernel, allowing tools such as Tigera Calico to run networking and packet-filtering acceleration as a Kubernetes DaemonSet without requiring write access to the host filesystem. Principal engineer Ben confirmed in the chat that the kernel is compiled with `kheaders`, and kernel development files are accessible under `/usr/src/kernels` for kernel module (`k-mod`) workflows.

### Takeaways Bottlerocket is currently in production for Amazon EKS and in preview for Amazon ECS. It eliminates the traditional undifferentiated heavy lifting of hardening general-purpose distributions for container hosting. Community members can build customized variants, contribute pull requests, and file feature requests via GitHub (`bottlerocket-os`).