Adding Coredumps to Your Debugging Toolkit

## Overview

Eric Johnson, a firmware solutions engineer at Memfault, presents the core dump subsystem in Zephyr RTOS as a powerful addition to the debugging toolkit. Core dumps capture device state at the moment of a fault or assertion failure, providing deeper diagnostic information than logging and panic messages alone. Unlike static panic messages, core dumps enable detailed post-mortem analysis of crashes.

## Key Components

The core dump subsystem consists of:

- **Front-end**: Formats memory regions into core dump data - **Back-end**: Handles I/O (logging, flash, or custom storage) - **Data sources**: Architecture-specific registers, thread stacks, and device-specific memory regions defined via device tree

Zephyr supports multiple backends—logging (streams to console), flash (stores in partitions), and custom implementations. Data sources capture RAM regions, thread kernel structures, and can include arbitrary memory regions through device tree configuration or runtime definition.

## Triggering Mechanism

Core dumps are triggered through assertions (via `__assert` macros calling `assert_post_action`) and fault handlers. The call stack differs: assertions show `assert_post_action` at frame zero, while faults place the fault site itself at frame zero.

## Host Analysis Tooling

The analysis workflow uses Zephyr's GDB server implementation:

1. **Log converter**: Transforms ASCII text output to binary format 2. **ELF parser**: Loads symbol information from the application build to correlate program counter values to functions 3. **Log parser**: Reconstructs memory regions from captured data 4. **GDB server**: Implements the GDB remote protocol, allowing clients to query memory regions 5. **GDB client with Python extensions**: Standard GDB debugger enhanced with custom Python scripts

The presenter emphasizes using the GDB Python-enabled version (with `-py` suffix) and recommends virtual environments with the GDB Bundle package to manage dependencies and auto-discover Python extensions.

## Demonstration

The demo showed: - Converting logging output to binary using Zephyr's log parser - Starting the GDB server with symbol file and core dump binary - Connecting the GDB client with custom initialization scripts - Using standard GDB commands (backtrace) and custom Python scripts (GPIO peripheral parsing, thread state inspection)

## Future Improvements

Johnson identified several areas for development: - **ISR safety**: The flash backend has semaphore issues in fault contexts (where interrupts are disabled), making it unsuitable currently - **Flexible memory regions**: Allow applications to override which memory regions are captured - **Bug submission**: Encourage users to include core dumps with issue reports for better triage - **Flash simulation backend**: Leverage Zephyr's flash simulator to store dumps in retained RAM - **West extensions**: Automate manual conversion and GDB server startup steps

Memfault, Johnson's employer, builds production core dump infrastructure, and the talk emphasizes this capability as critical for embedded device diagnostics. The presenter recommends exploring the GDB Bundle and custom scripting possibilities to extend debugging capabilities beyond traditional step debugging.