Support for Fast and Reliable VMM Live Upgrades in Libvirt

### Overview Upgrading hypervisors and Virtual Machine Monitors (VMM) like QEMU typically requires migrating virtual machines off a host, applying upgrades, and migrating VMs back across every host in a cluster. This legacy external migration workflow creates extended maintenance windows and introduces critical operational challenges, including non-deterministic migration failures, guest CPU throttling during dirty page convergence, and network contention with shared storage services. Engineers from Nutanix present an alternative approach: local live migration in libvirt using file descriptor (FD) transfer to update QEMU binaries in place with near-zero downtime and without copying memory.

### Key Topics - **Problems with Traditional VMM Upgrades:** Traditional host evacuation suffers from non-deterministic migration timeouts, adverse guest performance caused by vCPU throttling for convergence, and network saturation affecting other cluster services. - **Local Live Migration Workflow:** Instead of transferring state over the network, libvirt spawns a new QEMU process running the upgraded binary on the same host and transfers state locally before terminating the old process. - **Eliminating Memory Copy via FD Transfer:** Memory backends are passed directly between QEMU instances using UNIX domain socket SCM_RIGHTS and an intermediate libvirt coordination step, completely skipping dirty logging and iterative memory copy loops. - **State Disambiguation and Backwards Compatibility:** Addressing libvirt's assumption of unique VM names and UUIDs per host through numeric domain IDs, symlink transitions, and shadow remote hash tables.

### Technical Details To implement local live migration without altering the machine ABI, the authors addressed three main architectural challenges: 1. **FD Transfer Mechanism:** A new QEMU QMP command, `fetch-backing-fd`, queries the memory backend file descriptors from the source QEMU process in the migration `begin` phase using `SCM_RIGHTS`. During the `prepare` phase, libvirt forwards these descriptors to the target QEMU process. The target process mounts the existing memory via the provided FDs rather than allocating new memory, enabling all pages to be marked as skippable during the `perform` phase. This avoids iterative memory copying and guest vCPU throttling. 2. **Path and Monitor Disambiguation:** File paths tied to VM UUIDs (such as `/monitor.sock`) collide when two instances run concurrently. The implementation appends the numeric libvirt domain ID (e.g., `domain-<ID>-<UUID>/monitor.sock`) and maintains a symlink at the legacy path. When migration completes in the `confirm` phase and the source VM terminates, libvirt switches the symlink to the destination instance. 3. **Libvirt Migration Phases and Hash Tables:** Libvirt's internal hash tables index VMs by UUID and name. The authors implemented a shadow `remote` hash table containing a duplicated object (`VM_1#`) to service the `prepare` and `finish` phases on the local host. After confirmation, the destination entry is moved into the primary `source` hash table, preserving uninterrupted functionality for queries like `virsh list`.

### Results and Takeaways In evaluation benchmarks testing an 8 GB to 32 GB VM running an intensive write workload dirtying memory at 8 GB/s (pathological dirty rate): - **External migration** took from 20 seconds up to over a minute, with downtime between 300 ms and 400 ms, often risking non-convergence. - **Iterative local migration** took up to 45 seconds, with 150 ms to 200 ms downtime. - **FD transfer local migration** achieved consistent migration times of 800 ms to 850 ms and guest downtimes under 50 ms, completely independent of memory size or write workload. Guest throughput penalty was measured at zero.

Future work includes extending the FD transfer mechanism to block devices, character devices, and pass-through devices, as well as upstreaming the latest patches to QEMU and libvirt.