Lessons from a Large JVM Monorepo

## Overview Yanos from Airbnb's developer platform team shares lessons from managing a massive JVM monorepo with over 30 million lines of code across Java, Scala, and Kotlin. The codebase contains 8,000 third-party library versions, supports 1,000+ deployable artifacts, and has 1,000+ contributing engineers. The talk presents "postcards" (bite-sized lessons) organized by technical challenge rather than deep dives.

## Key Technical Challenges and Solutions

### Cross-Platform Builds and Universal Binaries Basel calculates different cache hashes for Linux and macOS JDKs despite bytecode being platform-independent, preventing cache sharing across machines. Airbnb implemented universal binaries (fat binaries with wrapper scripts) to dispatch to correct platform binaries. Trade-off is 4x size increase (e.g., JDK 200MB → 800MB), but negligible in practice. Implementation requires overwriting runtime and bootstrap runtime toolchains, replacing jar/single_jar tools with universal binaries, and handling Linux arm architecture manually since pre-built binaries aren't available.

### Protoc Compiler Contamination C++ protoc compiler unexpectedly appears in builds despite Java-only codebase, sneaking in as transitive dependencies even when using Thrift instead of gRPC. Solution: use incompatible_enable_proto_toolchain_resolution flag to provide custom Proto toolchain. Note: some open source rule sets (grpc_java, grpc_kotlin) haven't yet adopted this flag.

### Remote Execution (RBE) and Persistent Workers RBE runs 2x slower without persistent workers. JVM's warmup overhead is significant, especially with small packages. Airbnb uses experimental_remotework_mark_tool_inputs flag to maintain persistent worker state. Dedicated pools for Java and Kotlin prevent worker invalidation from unrelated actions; upgrade pools handle worker key changes. High-priority pools optimize for interactive builds with different queueing costs. Example: Open Rewrite integration reduced runtime from indefinite to 15 minutes via RBE.

### Determinism Checks Run builds on different hosts with cache disabled to catch non-determinism. Issues identified: timestamps in JARs (fixed with single_jar normalized timestamps), absolute paths in Kotlin .jps files, and Scala generator non-determinism (fixed by disabling diagnostics).

### Inner Loop Development Strategy Minimize Bazel in local developer workflows. Airbnb delegates to IntelliJ's incremental compiler via custom Bazel plugin fork, achieving up to 30x faster builds than pure Bazel (from 30 seconds to 1 second). Developers still sync to keep Bazel models current. Aligns with finding that newer MacBooks provide best local performance.

### Build File Auto-Generation Package-level build files maximize parallelism but create maintenance burden. Custom generator (similar to Gazelle) infers dependencies from imports with wildcard import ban. IDE plugin auto-updates build files on import changes and provides repository-wide symbol search via RDB backend.

### Multi-Version Java Support Handle Java 8, 17, and 21 simultaneously. Source compilation: use highest-version JDK (21) with --release flag. Default to Java 8, override per target. Test runtime: embed Java binary and set JAVA_BIN. Use aspects to validate deployables don't mix incompatible bytecode versions—critical since data pipelines hold repository on Java 8.

### Kotlin and Scala Modifications Custom rule set changes: fix Lombok annotation processor in mixed Java/Kotlin mode, replace stdlib with custom Maven dependencies, increase compiler heap. Direct-to-bytecode generation for GraphQL schemas (40% critical path reduction from 10M+ generated lines) using Kotlin metadata and JavaAssist.

### Test Infrastructure Custom test runner supporting JUnit 4 and 5 with per-class timeouts and resource metrics. Run tests at package level to amortize JVM JIT overhead; two cores typically halves execution time. Flaky test management: maintain config file mapping targets to disabled test IDs. Disable (not ignore) on CI; enable on periodic runs. Migrating from Gradle to Bazel reduced flaky tests 5x. Scala tests only support class-level disabling, not individual methods.

### Miscellaneous Best Practices Mark Lombok-generated code for Error Prone, avoid host system dependencies, enable watch_fs for 1-2 second improvements, hard-link cache to save space (critical for per-version repos), use downloader config to prevent known incident patterns.

## Key Takeaways Bazel excels at large monorepo parallelism but demands significant customization. Cross-platform consistency, persistent workers for RBE, and inner-loop developer experience require deliberate investment. Auto-generation and custom tooling dramatically reduce maintenance. Supporting multiple Java versions needs careful graph management. Determinism checks and flaky test management are essential at scale.