A Developer's Diary: The Trials and Triumphs of Maintaining Devboards in AOSP

### Overview In this Embedded Linux Conference (ELC) session, Amit Pundir, a senior engineer at Linaro with over a decade of experience in Android Open Source Project (AOSP) bring-up and maintenance, discusses the realities, challenges, and benefits of maintaining development boards within AOSP. Linaro acts as an upstream-first organization supporting Arm architecture, actively maintaining reference boards like the Qualcomm DragonBoard 845c and Robotics Board RB5. These reference platforms allow AOSP and upstream developers to smoke-test changes, benchmark features, and identify regressions early across hardware platforms.

### Key Topics - **AOSP vs. Android Releases**: AOSP serves as an active, moving development trunk where new features land first, making it prone to regressions and lacking vendor-specific optimizations. In contrast, formal Android releases represent stabilized code snapshots. - **Keeping Up with AOSP Churn**: Managing frequent synchronization issues, build system migrations (from `Android.mk` to `Android.bp` blueprints and now Bazel), and evolving framework interfaces (such as migrating from HIDL to AIDL). - **Kernel, Boot, and GKI Pain Points**: Handling Generic Kernel Image (GKI) requirements, bootloader limitations, boot image header iterations, and strict Kernel Module ABI (KMI) symbol lists. - **Upstream Project Alignment**: Navigating integration and build breakages with upstream Linux, Mesa, and Freedreno graphics drivers, alongside AOSP policy changes regarding proprietary binary firmware blobs.

### Technical Details - **Build & Framework Shifts**: Upstream AOSP framework changes frequently necessitate updates to device manifests and compatibility matrices (FCM). Pundir notes that tracking Cuttlefish reference configs is often the most practical way to adapt to changes like the HIDL-to-AIDL transition. Migrating from `Android.mk` to `Android.bp` improved namespace management but added friction when trying to pass board-specific flags without breaking non-namespaced upstream projects (e.g., `miniGBM`). Furthermore, AOSP is moving toward Bazel for more hermetic builds and reduced incremental build times. - **GKI and Boot Failures**: GKI introduces strict symbol lists for out-of-tree drivers to constrain the attack surface; changes during unfrozen development cycles can drop vendor-required symbols. Signed protected kernel modules prevent vendors from substituting custom stacks. In one instance transitioning from GKI 1.0 to 2.0, the removal of the user-space firmware loader config broke dependent drivers, requiring Linaro to package firmware binaries into the ramdisk. - **Upstream Linux & SELinux Breakages**: Upstream Linux does not treat Device Tree node names or sysfs paths as stable ABIs (such as changing `/sys/devices/platform/soc` to `soc0`). Because AOSP enforces strict SELinux access controls tied to specific filesystem paths, these upstream renames trigger SELinux denials that break boot in enforcing mode. - **Firmware Blob Policies & Graphics Stacks**: AOSP enacted a policy banning vendor binary firmware blobs from being hosted directly in AOSP if they cannot be rebuilt, prompting Linaro to host firmware externally and provide download scripts. For graphics, using the open-source Freedreno and Mesa stacks extends board longevity but brings regular build failures, host build tool version mismatches, LLVM compatibility issues, and complex SELinux policy adjustments.

### Takeaways Maintaining devboards in AOSP is an ongoing cycle of tracking upstream churn, resolving regressions, and aligning vendor hardware with evolving Android requirements. Development boards remain vital reference platforms for device manufacturers and test beds for upstream stability, provided developers choose boards with active communities, upstream support, and comprehensive documentation.