BazelCon 2019 Day 2: Building Self Driving Cars with Bazel + Q&A

### Overview At BazelCon 2019, Michael and Nico, software engineers at Cruise, presented their journey of adopting and optimizing Bazel for building autonomous vehicle software. Developing for self-driving cars in San Francisco involves massive, complex C++ codebases, heterogeneous CI environments, static/dynamic analysis (such as sanitizers), and hardware-specific requirements like GPUs for machine learning. Despite a tenfold increase in engineering staff and a fivefold increase in CPU time to compile everything over two years, a small team of four engineers successfully reduced build times by roughly sixfold.

### Key Topics - **Migration from CMake to Bazel**: How Cruise transitioned ~200 engineers, thousands of C++ and Python files, and hundreds of CMake files over six months without disrupting ongoing developer velocity. - **Observability and Profiling**: Using built-in Bazel profiling, custom wrapper scripts (`tools/bazel`), BigQuery, and Google Cloud Storage to capture detailed action telemetry and track percentiles across developers. - **Remote Caching & Compression**: Moving from a single workstation cache to an on-prem server and GCS, while adding Zstandard (zstd) compression to cut cached build times and expand storage capacity. - **Remote Build Execution (RBE)**: Incrementally migrating builds and tests to Google Cloud RBE with dynamic scheduling and local fallback, resolving non-hermetic and flaky tests, and integrating cloud GPUs.

### Technical Details - **Incremental Migration via Tooling**: Cruise developed an 800-line Python script that parsed CMake abstract syntax trees and generated Bazel `BUILD` files on the fly. Both CMake and Bazel ran in parallel for months, enabling seamless fallback until all 275 CMake files were removed after late 2017. - **Telemetry Pipeline**: Cruise wrapped Bazel commands to collect profile execution traces, command-line arguments, hostnames, CPU cores, and OS versions on every run. Profiles were written to GCS and ingested into BigQuery alongside cache hit/miss data, allowing engineering teams to proactively assist developers experiencing 90th-percentile build latencies. - **Cache Compression**: Enabling Zstandard compression for remote cache blobs made fully cached builds 40% faster and allowed storing six times more data in the on-prem cache, sustaining developers working on long-lived branches. - **Remote Execution Setup**: Using Google RBE with Docker containerization eliminated local CPU bottlenecks (stepping beyond 28-core workstation limits). Cruise enabled dynamic scheduling to run tasks locally and remotely in parallel, preventing regressions on small builds. Testing migration took roughly four months and required converting non-hermetic tests, replacing exclusive serial tests with parallel containerized runs, and executing tests up to 100,000 times to pinpoint rare flakes.

### Takeaways - Focus on developer workstation build speeds first before perfecting CI or correctness to secure early developer buy-in. - Resist the urge to clean up technical debt during a build system migration; address tech debt after completing the cutover. - Caching alone only benefits small incremental changes; scaling large builds requires distributed remote execution. - Remote test execution is significantly harder than remote compilation due to non-hermetic dependencies and flakiness, but containerized isolation unlocks massive parallelism and reliable test caching.