Farming Together

# Farming Together: A Board Farm Ecosystem Discussion

## Overview This is a bird-of-a-feather session at a technology conference focused on board farms—hardware and software infrastructure that enables automated testing of embedded devices and kernels in continuous integration pipelines. The speaker shares perspectives on the current state of farm deployments, challenges the community faces, and identifies opportunities for standardization and collaboration.

## What is a Farm? A farm consists of hardware, reference boards, and control systems that allow automated deployment and testing of software. Farms integrate with continuous integration tools (like Jenkins) to automatically build artifacts, deploy them to boards, run tests, and feed results back into the pipeline. The speaker emphasizes that while the ideal farm is cleanly organized with rack mounts and structured cabling, the reality across organizations is often chaotic—sprawling collections of boards, cables, and hardware managed ad-hoc.

## Technical Architecture and Implementation Farms typically connect boards via USB and Ethernet to a farm machine (typically a Linux desktop or server). Users SSH into the farm machine and use proprietary or open-source scripts to control boards. Key capabilities discussed include: - Power control via USB relays to turn boards on/off - SD MUX hardware to change SD card contents for boards that boot from SD - HDMI receiver cards for remote monitoring - Container-based workspaces to isolate user environments - Workspace abstractions using TFTP, NFS, and other services

Examples include farms with 6–8 boards controlled by BeagleBone Black, larger deployments with 80+ boards in kernel CI, and even configurations with ~120 boards for customer testing.

## Challenges and Pain Points The session identifies several persistent challenges: - **USB Reliability:** USB hubs, serial adapters, and similar devices frequently disappear or reset with different major/minor numbers, disrupting long-running tests. Participants noted cheaper USB-to-serial cables are particularly unreliable; FTDI cables are more reliable but expensive. - **SD MUX Limitations:** Custom SD MUX solutions work but lack reliability. Wi-Fi SD cards have emerged as a more reliable alternative, allowing firmware updates over Wi-Fi. - **Intermittent Failures:** Tests written using tools like Tickle/Expect often fail unpredictably in automated environments but pass when run manually, making debugging difficult. - **Maintenance and Scaling:** Keeping farms organized and operational as they grow requires significant manual effort, especially when using containers (OpenVPN was noted as a problematic choice in hindsight). - **Hardware Diversity:** No standard design for farms exists; each organization builds custom solutions, leading to duplicated effort and siloed knowledge.

## Ecosystem and Use Cases Several projects and organizations were mentioned: - **Kernel CI:** Community project that builds and tests kernel releases across world-wide farms, automatically detecting regressions. Kevin Hillman introduced himself as a founding member with ~80 boards contributing. - **OSADL:** Real-time patch set testing using cyclic tests. - **Lava (Linaro):** Open-source test automation framework sitting between CI systems and hardware. - **TTC:** Long-standing test lab management solution. - **Fuego:** Test framework on top of TTC. - **eBFarm:** Custom farm solution. - **Chromium OS:** Specialized board with personalized connectors for device control. - **TIC (Test Automation Controller):** Open PCB-based board for controlling network USB and other interfaces.

## The Three-Layer Problem The speaker identifies a critical architectural gap: farms have three layers: 1. **Top (CI/DevOps):** Jenkins, Lava, proprietary test frameworks—mature and well-supported. 2. **Bottom (Hardware):** PDUs, USB relays, SD MUX, probes—fragmented but some collaboration emerging. 3. **Middle (Hardware Abstraction):** Bridges between hardware and CI—this is where the biggest gap exists. Each organization writes custom code to interface with specific hardware, with no standardized API or middleware.

## Solutions and Collaboration Initiatives Participants and speaker proposed: - A wiki to centralize knowledge about farms, hardware, and software (already being set up on helix). - A mailing list (Linux Foundation to set up "board-farms" list at lists.linux-foundation.org). - Standardized hardware abstraction layer with a defined API, inspired by libvirt's approach of supporting multiple backend drivers. - Open-source hardware designs: Kieran (or similar name) demonstrated a custom PCB for board control using relays and a BeagleBone Black, with Kicad files shared on GitHub and ordering through Eisler. - "Lab in a Box" concept: a scaled-down lab in a single PC with multiple small boards, all powered by the PC supply and controlled via Linaro's BeLiebrelAcmeCape, demonstrating an accessible entry point. - Wi-Fi SD cards as a more reliable alternative to traditional SD MUX.

## Takeaways - Farms are increasingly important for maintaining quality in open-source and commercial development, enabling detection of regressions across diverse hardware configurations. - The community is currently operating in silos, each solving the same problems independently. - Standardization and knowledge sharing—through a wiki, mailing list, and collaborative API—would dramatically reduce duplicated effort. - The middle hardware abstraction layer is the key to making farms more accessible and scalable. - Next steps: build community connections, share hardware designs and software solutions, and define a common interface specification.