eBPF Warfare - Detecting Kernel & eBPF Rootkits with Tracee

### Overview In this talk, Idan Revivo and Asaf Eitani from Aqua Security Research present methods for detecting Linux rootkits across user space, kernel space, and eBPF using their open-source security tool, Tracee. The speakers examine the attack flows of modern rootkits, break down how malicious actors hide processes, files, and network activity, and explain why eBPF provides the low-overhead, hard-to-evade, and safe instrumentation required to capture forensic artifacts and identify runtime threats.

### Key Topics - **Linux Tracing Criteria**: Why traditional user space tracing and kernel modules fall short compared to eBPF, which runs sandboxed verification in-kernel to avoid crashes and system instability. - **Tracee Architecture**: An open-source, eBPF-based security monitoring tool providing behavioral rule detection and in-memory artifact capturing (such as executed files, kernel modules, eBPF bytecode, and per-process network packets). - **User Space Rootkits (LD_PRELOAD)**: How user space rootkits intercept system library calls (such as `readdir` in `libc`) to filter out artifacts, exemplified by HiddenWasp. - **Kernel Module Rootkits**: Analysis of ~30 rootkits revealing a consistent attack pattern: module loading, hiding from kernel lists, hook placement, establishing communications, and privileged command execution. - **eBPF-Based Rootkits**: Emerging rootkits that misuse eBPF program loading and specific BPF helper functions to manipulate user space memory, fake return codes, and evade detection.

### Technical Details - **Detecting LD_PRELOAD Rootkits**: Tracee hooks dynamic symbol loading to alert when shared objects export symbols overriding standard `libc` functions (e.g., `readdir`) outside the genuine system library. - **Kernel Module Loading & Forensics**: Attackers load modules without touching disk via `init_module` (passing an in-memory buffer) versus `finit_module`. Tracee places a kprobe on `do_init_module` to detect loading and intercept/capture the ELF module buffer directly from memory (demonstrated using Diamorphine). - **Kernel Module Hiding**: Rootkits detach themselves from the kernel's doubly linked list queried by `lsmod`. Tracee performs periodic cross-checks across three kernel data structures containing module records; discrepancies trigger a `hidden_kernel_module` event. - **Hook Placement & Detection**: - *Syscall Table Hooking*: Rootkits alter syscall pointers. Because valid kernel functions lie between `_stext` and `_etext`, Tracee checks whether syscall table entries point outside this range (generating `hooked_syscalls` alerts). - *File & Sequence Operation Hooking*: Malware (e.g., Drovorub / APT28, demonstrated via `phide`) modifies kernel file operation pointers like `iterate_shared` or sequence operations to conceal files, processes, and network connections. Tracee detects these modifications using `hooked_proc_fops`. - **Stealth Command Execution**: While security tools monitor `execve`, kernel rootkits (such as Reptile) invoke `call_usermodehelper` to spawn user space processes. Tracee hooks `sched_process_exec` and checks the `invoked_from_kernel` flag to identify executions initiated from kernel space. - **eBPF Rootkit Mechanics**: Emerging BPF rootkits load via `bpf` syscalls and hide by hooking the `bpf` syscall to alter listing outputs (as seen in Guillaume's `ebpfkit`). They alter execution using dangerous helpers: - `bpf_probe_write_user`: Modifies user space memory on syscall return (demonstrated in `bad BPF` altering `read` responses). - `bpf_override_return`: Fakes success or failure error codes. - `bpf_send_signal`: Terminates monitoring processes (e.g., `ls` or `lsmod`). - `skb_store_bytes` / `skb_pull_data`: Alters network packet data. Tracee hooks the `bpf` syscall to extract attached probes, target symbols, and helpers flagged by the kernel verifier, while saving the bytecode for disassembly.

### Takeaways Defenders can monitor, block, or disable features to neutralize rootkits. Tracee offers out-of-the-box behavioral rules and artifact capture for user space, kernel, and eBPF threats. Organizations can also deploy `ebpfkit-monitor` to block untrusted eBPF programs, leverage kernel lockdown to prevent direct kernel memory modification, and use BTF / BTFhub to run modern eBPF tracing across various Linux distributions.