Sharing IOMMU PageTables with TDP in KVM

### Overview Presented by Julian from Intel Corporation and co-prepared with Lu Baolu, Ken, and Kevin, this presentation details the design and implementation of sharing Two-Dimensional Paging (TDP) structures from KVM with I/O Page Tables (IOPT) used by the IOMMU. Currently, when virtual IOMMU (vIOMMU) is disabled, the IOMMU performs Guest Physical Address (GPA) to Host Physical Address (HPA) translation via IOPT, while the CPU performs the same translation using TDP. Because these two sets of page tables duplicate translation data, sharing KVM's TDP as the IOPT eliminates memory overhead, establishes unified page table management for dirty-bit tracking and page fault handling, and can reduce redundant TDP generation.

### Prerequisites and Negotiation To share page tables, three prerequisites must be met: 1. **Same Address Space:** Sharing is currently supported only for L1 GPA to HPA translations. QEMU must verify that TDP is enabled in KVM and ensure the vCPU model does not use nested paging features (EPT or NPT). On the IOMMU side, vIOMMU must either be absent or operating in nested mode where the second-level page table maps GPA to HPA (shadow mode is unsupported). Additionally, because System Management Mode (SMM) on x86 uses a separate address space, non-SMM EPT structures must be preserved when entering SMM. 2. **Compatible Page Table Format:** Formats must be defined and negotiated (e.g., `format_ept_level_4` or `format_npt_level_5`). QEMU queries KVM for its format and confirms compatibility with the IOMMU architecture. For instance, Intel IOMMU using only first-level page tables is incompatible with EPT format, but second-level 4-level paging succeeds. 3. **Non-Conflicting Page Table Content:** When I/O page faults (IOPF) are unsupported, guest frame numbers (GFNs) in user memory slots must be present and pinned for DMA. KVM private memory slots (Local APIC, TSS, and identity page tables) are safe to map in shared IOPT because DMA writes do not hit remapping or are reserved in E820. Permissions align with slot access (read-only vs. read/write), while execute bits are currently ignored by the IOMMU. If IOPF is not supported, live migration write protection must be disabled.

### Technical Details & Pinning Mechanics Sharing requires multiple communication interfaces between QEMU, KVM, and IOMMU, including sharing requests with notification callbacks, pinning/unpinning APIs, IOPF forwarding, and root update notifications accompanied by IOTLB invalidation.

When IOPF is absent, DMA targets require long-term pinning (`pin_user_pages`) alongside page table entry pre-population. Entries must not undergo zapping or PFN updates, and page table modifications (e.g., huge page splitting) require atomic updates from non-zero to non-zero values, which requires substantial modifications to KVM. Two pinning models were evaluated: a QEMU user API in KVM (bypassing IOMMU map/unmap) or an in-kernel interface called by the IOMMU (retaining DMA map calls but facilitating metadata passing such as snoop bit configuration).

When vCPU0 changes its MMU root, the root count of the old page is decremented via `kvm_mmu_unload`, the new root is loaded via `kvm_mmu_load`, and if non-SMM, its root count is incremented for IOMMU sharing. KVM pre-populates pinned ranges, notifies the IOMMU to switch its IOPT root and flush the IOTLB, and releases the previous root.

### Performance and Takeaways Early, unoptimized boot testing showed baseline boot time without sharing at 29 seconds. With page table sharing and huge pages enabled, boot time increased to 32 seconds (and took longer with huge pages disabled). The extra latency was driven by 132 pre-population cycles and IOTLB flushes (~1 second each). Future work focuses on reducing the root update frequency, handling snoop bits, unifying dirty page tracking, supporting nested virtualization, and enabling huge pages for peer-to-peer (P2P) DMA.