Software Updates for Connected Devices: Key Considerations

## Overview Stein Steinberg, a systems and security management software engineer with 7 years of experience and a cryptography background, discusses the critical considerations for updating software on connected embedded devices. He works on Mender, an open-source over-the-air update tool for Linux that integrates with Yocto. The talk draws insights from interviews with approximately 100 device developers and 30 in-depth conversations about update practices.

## Key Topics

**Motivation for Updates** Devices require robust update mechanisms to address security vulnerabilities (exemplified by Linux kernel CVEs showing critical and high-severity issues), fix bugs discovered post-deployment, and roll out new features. Real-world incidents like the Fiat Chrysler hack demonstrate the dangers of unpatched connected devices.

**Current Industry State** A survey revealed that most companies write their own updaters from scratch rather than adopting generic tools—though this appears to be changing. About half of developers preferred image-based updates for their atomic properties and consistency between test and production devices, while the other half favored package-based approaches for faster installation and easier integration with existing build systems. Developers typically spend three to six months building homegrown updaters and deploy updates approximately six times annually.

**Embedded Environment Challenges** Embedded devices present unique constraints absent in cloud or server contexts: physical remediation is expensive, lifespans extend 5–10 years, power is unreliable (battery depletion during updates), network connectivity is inconsistent (3G tunnels, intermittent coverage), and bandwidth is severely constrained. Critically, bandwidth will not follow Moore's Law improvements seen in storage or CPU; a $30 embedded device and a $700 smartphone serve different use cases and require fundamentally different bandwidth budgets.

**Core Requirements for Embedded Updaters** Developers identified eight primary criteria: 1. **Robustness and security** despite power loss and network failures 2. **Integration with existing workflows** to avoid prohibitive disruption 3. **Low bandwidth consumption** 4. **Minimal downtime** during updates 5. **Atomic installation** (fully applied or not at all; no partial states visible to applications) 6. **Consistency** between test and production devices 7. **Sanity checks** (ensuring the device can receive future updates at minimum; extensible for custom health checks) 8. **Authenticity** via cryptographic signatures or MACs

Additional considerations include extensibility for custom pre- and post-install scripts, support for standalone and managed deployments, ease of adoption, quality assurance, and clear documentation.

**Installation Strategies** The speaker outlined four approaches: - **Runtime installation** (package manager in userspace): low bandwidth, quick downtime, but weaker atomicity - **Maintenance mode** (recovery OS): atomic, good integration, but long downtime and high bandwidth - **Symmetric/dual A/B filesystems**: fully atomic and consistent, low downtime, but requires double storage and bootloader support - **Proxy-based** (for ultra-constrained devices): gateway manages updates to sensors/lights over Bluetooth or low-power protocols

**Server-Side Remote Deployment** Moving beyond USB stick updates, centralized servers enable device grouping (risk-averse vs. bleeding-edge customers), phased rollout campaigns (1%, 5%, 10% of fleet), and deployment reporting for monitoring success or failure.

## Takeaways 1. Evaluate existing open-source update tools before building from scratch to avoid maintenance costs and replication of solved problems 2. Bandwidth constraints in embedded networks are permanent; design accordingly 3. Wireless update mechanisms without proper authentication (signatures) are a real security liability 4. Atomic updates and consistent testing/production environments are critical for reliability 5. Integration friction with existing build workflows is a major showstopper; solutions must be easy to adopt 6. Remote deployment servers, while not required, are essential for managing scale as device counts grow 7. Security in over-the-air updates requires end-to-end verification, regardless of transport security