Lessons Learned: Migrating a Production Platform to Yocto

### Overview In this presentation, Mitch Gatons shares his practical experience and lessons learned from migrating a production edge device platform from Debian to the Yocto Project over an eight-to-nine-month period as a solo engineer. Driven by supply chain disruptions that prevented his company (Enview) from sourcing legacy edge hardware, the team needed to support an ambiguous set of hardware architectures across their fleet of multi-family apartment building automation devices. While Debian had served as the foundation for nearly a decade of development, Yocto provided architecture flexibility, streamlined full-system updates, and significantly reduced hardware resource constraints.

### Key Topics - **Debian vs. Yocto Approaches:** In a Debian-based workflow, OS flashing is separated from application deployment; cross-compiled packages must be generated for each target architecture, uploaded to a package repository, and pulled down onto edge devices via tools like Ansible or Puppet. In contrast, Yocto's Bitbake tool builds a complete, monolithic system image with the root filesystem and applications baked in, enabling deterministic deployment and simpler device provisioning. - **Avoiding Migration Pitfalls & Shortcuts:** Gatons explored and rejected multiple tempting shortcuts. Using an on-target package manager inside Yocto maintains an undesirable separation between application and system layers. Unpacking prebuilt Debian packages inside Bitbake recipes quickly becomes unmanageable due to deep dependency trees. Containerizing applications via tools like Docker introduced unacceptable system overhead and bloat while preventing full-system integration testing. - **Layered Architecture:** Gatons leveraged Yocto's layer model—spanning base Poky and OpenEmbedded layers, Board Support Package (BSP) layers (e.g., Raspberry Pi or Sunxi), custom board layers, and application layers—to swap configurations and target separate hardware boards cleanly.

### Technical Details - **Bitbake Tasks:** Gatons highlighted three critical recipe tasks: - `do_configure`: Used `quilt` within makefile targets via `oe_runmake` to apply shared patch sets to third-party SDKs, ensuring patch compatibility between Debian legacy builds and Yocto. - `do_compile`: Identified as the most difficult task. Application makefiles often contained hardcoded environment variables conflicting with Bitbake's GCC cross-compilation flags. In-tree Yocto patches were used for makefile adjustments, and runtime testing was essential to catch immediate segmentation faults that compilation alone did not reveal. - `do_install`: Made straightforward by using existing Debian package `install` files as blueprints for mapping binary, configuration, and service file destinations. - **OTA Updates & Provisioning:** The team implemented full-system over-the-air (OTA) updates using RAUC with an A/B partition scheme, which required early bootloader scripting. Device provisioning was reduced to flashing the system image and running a self-destructing first-boot setup script for cryptographic keys and certificates. - **Developer Workflows:** Gatons recommended `devtool` to allow application developers to deploy, test, and cleanly remove development binaries on target hardware in seconds. He also utilized `AUTOREV` for internal application recipes (excluding third-party recipes) alongside frequent nightly or weekly builds to catch upstream breakage early.

### Takeaways - **Make the System Boot First:** Avoid rushing to build applications before validating hardware, kernel, bootloader, and update scheme requirements, as hardware changes will invalidate compilation settings. - **Leverage Existing Debian Assets:** Legacy Debian packaging files, configuration paths, and makefile logic serve as detailed blueprints for Bitbake recipes. - **Involve Application Teams Early:** Shifting from package-based continuous delivery to full-image firmware artifacts requires developer buy-in; tools like `devtool` help mitigate friction when changing developer workflows.