Integrated Trace' Using Virtio-Trace for a Virtualization Environment

### Overview In this presentation, Yoshihiro Yonekawa discusses the development of "Integrated Trace," a system designed to merge kernel tracing data from multiple virtual machine guests and their host into chronological order. Because virtual guests and hosts share underlying hardware (CPUs, memory, disk, network controllers), performance anomalies on a guest—such as latency spikes in real-time applications detected using tools like `cyclictest`—frequently stem from contention with host processes. Analyzing and resolving these performance issues necessitates merging trace data across both guests and the host using tools like `trace-cmd` and Linux `ftrace`.

### Key Topics - **Motivation and Performance Contention**: Virtualized guests often encounter delayed wakeups (e.g., wake-up latency jumping from ~0.3 ms to ~3.5 ms in `cyclictest`). Merging host and guest trace data and visualizing it using tools like TimeDoctor revealed delays caused by host processes (such as a flush process taking 3.3 ms). - **First Attempt and Lessons Learned**: The initial implementation used a guest trace agent communicating via `virtio-serial` using `splice()` to avoid memory copying, combined with a 9P network filesystem (9pfs) server to expose guest `debugfs` trace metadata to `trace-cmd`. However, this design caused severe overhead due to excessive small packet exchanges across the 9P client/server during basic file inspection (like `ls`), resulting in high memory copying and frequent guest context switches. - **Revised Architecture**: The updated architecture leverages `trace-cmd` directly on both guest and host rather than a custom trace agent, focusing on four primary enhancements.

### Technical Details The revised `trace-cmd` architecture implements four key improvements: 1. **Server and Client for Virtualization via Virtio-Trace**: Rather than relying on high-overhead IP networking to transfer trace data, `trace-cmd` uses `virtio-serial` with `splice()`. This reduces CPU usage on the guest from approximately 2.9% (using network transport) down to 1.7%. 2. **Structured Message Platform**: Replaced unstructured string-based network messages with structured binary messages to support complex tracing events, snapshot buffers, and strict backward compatibility between new and old clients and servers (implementing eight specific message types, including connection, probe, initialization, debugfs info, finish, and error). 3. **Agent Feature for Clients**: Instead of requiring an operator to manually log into each guest to record traces in a blocking mode, an agent mode allows the `trace-cmd` client on the guest to wait for control messages from the server and trigger tracing actions dynamically. 4. **Chronological Trace Merging**: Traditional `trace-cmd` trace merging relied on NTP synchronization and wall-clock timestamps via the `date` option, which is inadequate for sorting fine-grained guest/host context switches and lacks cross-guest clock guarantees in multi-tenant environments. The system instead uses the x86 Time Stamp Counter (TSC) alongside a kernel event (`kvm_write_tsc_offset`, added to KVM) to track guest TSC offsets relative to the host, enabling accurate chronological trace alignment across multiple `ftrace` buffers.

### Takeaways Merging host and guest `ftrace` logs is essential for diagnosing root-cause performance degradation in virtual environments. By combining `virtio-serial` zero-copy splicing, structured client-server protocols, agent-driven collection, and hardware TSC offset alignment, `trace-cmd` provides low-overhead, synchronized tracing across virtualized systems.