Virtual Versus Physical: Virtio Changes for New Hardware

### Overview Michael Tsirkin, a Distinguished Engineer at Red Hat and chair of the Virtio Technical Committee, discusses adaptations to the Virtio specification to better accommodate physical hardware implementations. Historically designed as a software-first interface for virtual machines, Virtio is increasingly implemented directly in hardware (such as smart NICs and hardware offload cards) by cloud vendors looking to reduce host CPU overhead. Because Virtio devices expose standard PCI interfaces to guest VMs, existing guest drivers work out-of-the-box, but hardware implementations face unique latency and architectural constraints.

### Key Topics - **Virtqueue Layout Evolution**: Transitioning from legacy split virtqueues to packed virtqueues to minimize PCIe bus round-trips and transaction overhead. - **In-Order Processing and Batching**: Optimizing descriptor handling on PCIe by acknowledging batches of buffers in order, reducing device write transactions. - **Page Fault Support in Hardware**: Debunking the notion that page faults are solely detrimental to performance by illustrating their necessity for NUMA placement, memory overcommit, transparent huge pages, and live migration. - **Hardware Device Live Migration**: Addressing challenges with pre-copy and post-copy live migration across heterogeneous hardware by calculating a least common denominator for device capabilities.

### Technical Details - **Packed Virtqueues**: The original split virtqueue format requires multiple dependent PCIe memory reads and writes, adding latency. Packed rings consolidate descriptors into a single ring structure with flags indicating available/used states. Enhancements allow drivers to pass the offset of the last available descriptor during notifications, allowing single-read and single-write processing cycles. - **Mitigating Receive Stalls with Out-of-Order Buffers**: Page faults on network receive rings risk stalling the ring, quickly overflowing device on-board memory and causing dropped packets. Virtio addresses this by using Virtio block–style out-of-order buffer completions: when a buffer faults, the hardware processes incoming packets using subsequent valid buffers. A newly proposed per-descriptor flag allows combining this fault resilience with in-order batching once faults clear. - **Support for Advanced Memory Features**: Supporting hardware page faults via PCIe Page Request Interface (PRI) or device-specific physical function reporting enables hypervisors (such as Firecracker or QEMU) to perform memory overcommit and transparent huge page defragmentation without pinning all guest RAM. - **Post-Copy Live Migration**: Hardware devices typically pin all guest memory, preventing post-copy migration where the VM starts at the destination before all memory transfers. With page fault support, missing pages trigger faults handled via mechanisms like userfaultfd to fetch pages on demand from the source. - **Cluster Compatibility**: Different hardware revisions or vendors support differing queue counts, interrupts, and offloads. To enable migration across heterogeneous nodes, cluster tooling must query devices and establish the least common denominator feature set before VM launch. - **Descriptor Batching Optimizations**: The Virtio TC is evaluating mechanisms to avoid sparse gaps when writing used descriptors, enabling continuous writes and wrap-around detection to signal up to a full ring of completed descriptors in a single PCIe write.

### Takeaways Adapting Virtio for hardware bridges software-defined flexibility with bare-metal performance. Through packed rings, refined in-order signaling flags, hardware-assisted page faulting, and compatibility tooling, Virtio enables high-throughput device offloading without sacrificing key virtualization benefits like live migration and memory overcommit.