What's Missing in Embedded Build Systems

## Overview

Agostanakapella, an embedded software architect and Buildroot maintainer at Mines, presents an analysis of gaps in current embedded Linux build systems (Yocto, OpenEmbedded, Buildroot). He argues that while these systems are highly configurable and provide necessary tools, developers still perform redundant work implementing common embedded system features. The talk advocates for standardized, integrated solutions within build systems while preserving developer flexibility.

## Key Topics Identified

The speaker highlights four major missing capabilities in embedded build systems:

**1. Field Updates (OTA)** Developers must distinguish between manufacturing images (complete, with bootloader and A/B partitions) and update images (partition-specific, in formats like cpio archives for SW-update). While tools like SW-update, RAUC, and others exist, integration gaps remain. U-Boot environment handling for A/B selection is fragile and complicates verified boot. Bootloader and TSA updates are often overlooked in initial designs, requiring later fielding.

**2. Persistence and Factory Reset** A/B updates require a separate persistent partition distinct from root filesystems. Multiple approaches exist (overlay on root, limited-scope mounts, read-only /usr with writable /etc and /var) but lack standardization. Factory reset must reformat persistent storage while preserving device-specific data. First-boot population via systemd-tmpfiles has limitations (cannot remove files on updates). FileSystem checks should auto-reformat corrupted persistent partitions.

**3. Manufacturing and Provisioning** Device-specific data (serial numbers, MAC addresses, passwords, SSH keys) requires separate storage that persists across factory resets. Manufacturing databases exist but no common communication interface. Weak WIC format covers some flash scenarios but lacks general solutions. Provisioning typically requires reinvention per project.

**4. Verified Boot** The speaker acknowledges limited personal experience but notes high complexity. UEFI Secure Boot and ARM Trusted Firmware provide some standardization at early boot stages, but root filesystem verification (dm-verity) and key management lack common tooling integration. No single standardized approach for derivatives exists; implementations are scattered across bootloaders and kernels.

## Technical Gaps and Constraints

The speaker identifies cross-cutting concerns: debug tools (like Valgrind) create dependency chains affecting production images; configuration fragments for extensions should work alongside defaults; overlay images for post-deployment customization need recipe integration. Current systems force developers to choose between freedom and ease-of-use rather than providing both.

## Recommended Solutions

- Build systems should generate minimal default images with A partition, B partition, persistent partition, and device provisioning partition out-of-the-box - Standardize persistence approaches (e.g., converge on read-only root, writable /etc, /var, /opt) - Provide common tooling and integration points in build systems while allowing customization via plugins, hooks, or config fragments - Focus integration in core build system layers (Yocto's openembedded-core, Buildroot) rather than external layers to maximize collaboration - Require upstream support from bootloaders, kernel, and update tools

## Takeaways

Developers currently assemble solutions from disparate tools rather than using integrated, standardized workflows. Build systems are ideally positioned to drive standardization and serve as collaboration hubs. The speaker references Leonard Pöttering's similar observations and draws parallels to how field update tools matured a decade ago—suggesting embedded provisioning standards may similarly improve over time if coordinated efforts begin now. Sensible defaults combined with extension points can balance simplicity and flexibility.