Mentorship Session: An Introduction to Linux Tracing and its Concepts

### Overview This Linux Foundation mentorship session introduces Linux tracing concepts, history, and tooling. Presented by Elena Zannoni, the session explores how tracing evolved from traditional debugging mechanisms like ptrace and profiling toward modern dynamic and static tracing infrastructures designed to inspect running systems with minimal runtime perturbation. The talk outlines foundational kernel instrumentation mechanisms—including kprobes, uprobes, static tracepoints, and tracefs—and examines how higher-level tools such as ftrace, perf, eBPF (BCC and bpftrace), and a re-architected DTrace build upon these building blocks.

### Key Topics - **Evolution from Debugging to Tracing**: Traditional debugging via ptrace relies on breakpoints and process interception, stopping execution interactively and perturbing program flow. Profiling samples hardware performance monitoring unit (PMU) counters statistically. Tracing bridges these approaches by collecting, filtering, and aggregating event data in real time without halting production workflows. - **Kernel Instrumentation Mechanisms**: Static tracepoints are carefully reviewed, predefined hooks maintained directly in kernel subsystem code (configured via macros like `DEFINE_EVENT` and `TRACE_EVENT`). Dynamic tracing mechanisms, such as kprobes (for kernel space) and uprobes (for user space), dynamically insert breakpoint or jump instructions at specific instruction offsets, with return probe variants (`kretprobes` and `uretprobes`) monitoring function exit values. - **Kernel Tracing Interfaces**: `tracefs` (typically mounted under `/sys/kernel/tracing` or `/sys/kernel/debug/tracing`) acts as the pseudo-filesystem interface controlling kernel tracing buffers, filter rules, enabled events, and dynamic probe definitions. - **Tool Ecosystem**: The presentation surveys tools including ftrace (function and function graph tracers), perf (hardware/software performance counters, PMU metrics, and custom probes), the BPF ecosystem (eBPF in-kernel verifier, maps, helper functions, and BCC tool wrappers), DTrace on Linux, bpftrace, trace-cmd, KernelShark, and SystemTap.

### Technical Details - **ftrace and tracefs Interaction**: Users configure tracing dynamically by echoing values into control files such as `current_tracer`, `tracing_on`, `kprobe_events`, and `trace`. Available static events (such as `alarmtimer` or syscall enter events) expose granular control directories containing `enable`, `filter`, and `format` files. - **perf Framework**: Originally built around performance counters (`perf_events`), perf provides commands like `perf stat`, `perf record`, and `perf probe`. With kernel debug info installed, `perf probe` maps function arguments (e.g., `do_writepages`) directly into dynamic tracepoints that log into shared perf ring buffers. - **eBPF Infrastructure**: BPF executes sandboxed, JIT-compiled bytecode inside the kernel validated by a strict in-kernel verifier to prevent loops and unsafe operations. BPF programs interact with tracing contexts (register sets, format strings) and pass structured data between kernel and user space via BPF maps and multiplexed `bpf()` system calls. - **DTrace on Linux**: Re-implemented by Oracle to reduce kernel patches, modern DTrace translates high-level D-language clauses into BPF bytecode using GCC and attaches them to kprobes, uprobes, or tracepoints via trampolines, outputting data to perf ring buffers for aggregation and histogram rendering. - **Toolchain & Debug Info Enhancements**: The speaker highlights ongoing work in the GCC and binutils toolchain to natively support BPF target compilation, BPF Type Format (BTF), Compile Once - Run Everywhere (CO-RE), and Compact C Type Format (CTF) as lightweight alternatives to DWARF for production environments.

### Takeaways Linux tracing relies on a cohesive, layered architecture where diverse user-space tools leverage common kernel instrumentation facilities (`tracefs`, kprobes, uprobes, tracepoints, and perf ring buffers). While perf excels at hardware profiling and ftrace offers direct kernel tracing via pseudo-files, modern dynamic tracing increasingly converges on eBPF as a programmable execution engine. Tool selection often depends on operational constraints—such as requiring root privileges, script complexity, production overhead, and whether compact debugging formats like CTF or BTF are supported.