Building a flexible hypervisor-level debugger by Mathieu Tarral (@mtarral)

### Overview In this presentation at Insomniac, Mathieu Tarral discusses how hypervisors can be leveraged as flexible, out-of-band debuggers using Virtual Machine Introspection (VMI). Traditional in-guest debuggers suffer from observer effects, anti-debugging mechanisms, privilege clashes with ring-0 malware, and interference with operating system protections like Windows PatchGuard and Protected Media Path. Debugging from the hypervisor allows security analysts to inspect untouched guest virtual machines transparently across platforms without modifying the target system or installing in-guest agents.

### Key Topics - **Motivations and Concrete Use Cases**: Hypervisor-level debugging circumvents in-guest anti-debug techniques, enables kernel analysis without disabling security controls (such as PatchGuard), provides visibility into isolated environments (like Windows Virtual Secure Mode), and allows inspection of unikernels and early BIOS boot sequences (e.g., via BIOS interrupt `0x19`). - **History and Ecosystem**: Prior efforts have spanned built-in debug stubs (e.g., QEMU, VMware), bare-metal hyperjacking/USB boot debuggers (HyperDBG, VerDBG, RomuFlex, VerOSDBG), emulation tools (Pyrebox), and full-virtualization introspection platforms (VMI DBG, RVM I, Winbagility/Sandboxity, and radare2-based approaches). - **The Semantic Gap & Guest Awareness**: Hypervisors natively see only raw hardware state, registers, and memory addresses. Bridging the semantic gap requires intimate guest knowledge; tools like Rekall generate JSON profiles containing kernel structures, constants, and function offsets to translate raw offsets into meaningful processes, threads, and symbols. - **PyVMI DBG Architecture**: Written in Python, PyVMI DBG serves as a GDB stub built on top of LibVMI to decouple front-end debuggers (GDB, radare2, IDA) from underlying hypervisors (such as Xen and KVM). It supports both raw mode (treating the whole VM as a single target where VCPUs map to threads) and guest-aware mode (inspecting specific processes like `cmd.exe`, enumerating threads, and handling break-ins).

### Technical Details Tarral examines the design and evolution of "hyper breakpoints"—breakpoints executed via the virtualization layer: - **Software Hyper Breakpoints (INT3)**: Overwriting guest memory with `0xCC` and trapping INT3 events. While fast, it is easily detected by memory scans and unsafe across multiple VCPUs unless all cores are paused to prevent race conditions during opcode restoration. - **Hardware Hyper Breakpoints**: Using hardware debug registers (`DR0`–`DR3`) and handling INT1. These are per-VCPU safe and fast, with stealth achieved by intercepting `MOV to DR` instructions; however, they are restricted to four registers. - **Page Hyper Breakpoints**: Modifying Extended Page Tables (EPT) to trigger EPT violations on read, write, or execute accesses. While fine-grained, this incurs heavy VM-exit overhead and suffers from multi-VCPU race conditions when permissions are relaxed. - **Advanced Soft Hyper Breakpoints (EPT Shadow Views)**: Splitting memory views using processor EPT mechanisms (such as Xen's `altp2m` API, utilized by Drakvuf). A secondary EPT view points to a duplicated physical page containing the breakpoint set as execute-only, while the original page remains read-write. By switching the EPT pointer (`EPTP`) per VCPU upon EPT violations, the debugger achieves stealth, high performance, and multi-VCPU safety without triggering race conditions.

### Takeaways Tarral advocates treating virtual machines not as black boxes, but as transparent process trees. To establish a unified debugging framework across hypervisors, Tarral calls for hypervisor vendors to expose standardized VMI APIs, encourages community contribution to LibVMI (including cross-platform CMake builds, performance optimizations via `/dev/shm`, driver support, and exploring rewrites in Rust for memory safety), and invites developers to expand PyVMI DBG with additional stubs (such as LLDB and KD).