eBPF for Observability: Hard Truths and Practical Wins

### Overview In this session, Kenta Tada (member of the CNCF End User Technical Advisory Board and organizer of Cloud Native Community Japan and BPF Japan) and co-host Yan explore the practical benefits, internal mechanisms, and production challenges of adopting eBPF for observability and kernel customization. While eBPF has rapidly evolved from simple tracing into a strategic infrastructure platform, the presenters emphasize the growing gap between rapid kernel feature development and developer education, as well as the technical friction encountered when running eBPF in production.

### Key Topics - **Evolving eBPF Ecosystem**: Moving beyond basic observability into areas like scheduling (`sched_ext`), hardware waiting via sleepable BPF contexts versus non-sleepable softirq BPF timers, Windows support, AI workload visualization with flame graphs, and network bottleneck mitigation. - **Custom Scheduling with sched_ext**: Using BPF programs and structs to implement custom process schedulers, resolving issues like scheduling overhead and 'thundering herd' wake-up behaviors. - **The Knowledge Gap and Education**: The mismatch between fast-moving APIs and sparse documentation, leading to developer friction and an influx of questions on Stack Overflow. Initiatives to address this include community-driven education and comments submitted to the JASA and Linux Foundation Japan embedded Linux developer certification draft. - **Production Challenges and Hard Truths**: Compatibility issues, kernel version drift, verifier behavior differences, stability of tracepoints versus kfuncs, and security/supply chain considerations. - **Interactive Discussion on Tracing & Tooling**: Community debates comparing Ftrace, DTrace, and eBPF-based tooling (BCC, bpftrace, retsnoop), observing eBPF programs themselves, and why uprobes remain more widely used than USDT.

### Technical Details - **Kernel Mechanisms**: The speakers highlight the trade-offs between BPF mechanisms, such as sleepable contexts suited for waiting on slow hardware responses (e.g., human interface devices) versus BPF timers operating in softirq contexts tailored for low-latency tasks. - **Observability of eBPF Itself**: Audience discussion highlights tools such as `bpf-top` (maintained by Netflix engineers) for monitoring CPU resource usage, runtime benchmarks, and average execution time of BPF programs. - **API Stability (Tracepoints vs. Kfuncs vs. BPF Helpers)**: While BPF helpers are strictly stable, kfuncs remain highly flexible for kernel developers but unstable across versions. Discussion explores establishing a 'stable-ish' tier of kfuncs. Tracepoint stability in Linux was clarified: while tracepoints like `sched_switch` and `sched_waking` are treated as de facto stable by tools, raw tracepoints and tracepoint arguments can drift across kernel versions, forcing user tools to use BTF or struct offsets as workarounds. - **Verifier Debugging & USDT**: Debugging complex verifier errors was addressed by mentioning projects like BPF Verifier Visualizer (`bpf-verifier`). The low adoption of USDT compared to uprobes was attributed to packaging overhead, sparse Linux documentation, and historical build dependencies on SystemTap headers, despite recent improvements like Rust's `usdt` crate.

### Takeaways - eBPF offers unprecedented kernel programmability, but successful adoption requires deep knowledge of low-level kernel subsystems (scheduling, memory, networking). - Tool maintainers face real maintenance burdens due to kernel version drift, shifting tracepoint arguments, and unstable kfuncs. - Community training, structured certification paths, and better tooling (like verifier visualization and runtime resource monitors like `bpf-top`) are critical to bridge the gap between kernel developers and application/embedded platform engineers.