The Forefront of the Development for NVDIMM on Linux Kernel

### Overview In this talk, Yasunori Goto and Ruan Xia from Fujitsu present the forefront of NVDIMM (Non-Volatile Dual In-line Memory Module) persistent memory development on the Linux kernel. NVDIMM combines characteristics of storage and system memory: it is byte-addressable and fast enough to bypass traditional page caches and block I/O layers, yet retains data consistency across reboots. The Linux kernel provides several modes to interface with NVDIMM, including storage access, Device DAX, and Filesystem DAX (FS-DAX). However, FS-DAX has remained in an experimental status due to critical corner cases and architecture limitations, notably its incompatibility with copy-on-write (CoW) features like reflink in filesystems like XFS. The speakers detail the engineering work required to resolve these obstacles so that FS-DAX can shed its experimental label.

### Key Topics - **NVDIMM Access Interfaces:** Bypassing traditional I/O stacks via Direct Access (DAX) modes, user-space libraries in PMDK (`libpmem`, `libpmemobj`, `libpmem2`, and RDMA-focused `librpma`), and standard utilities like `ndctl` and `daxctl`. - **Metadata and Synchronization Corner Cases:** Addressing metadata update delays caused by CPU cache flushes bypassing filesystem operations, using `MAP_SYNC` flags, handling truncate races, and leveraging On-Demand Paging (ODP) for DMA/RDMA transfers. - **Per-Inode DAX Configuration:** Implementing fine-grained DAX toggling per inode (`-o dax=inode` and `xfs_io -c 'chattr +/-x'`) to allow performance tuning and avoid page cache eviction race conditions. - **Enabling Reflink and Deduplication on FS-DAX:** Extending the `iomap` interface to support copy-on-write without page cache, adding extent remapping, and providing DAX-aware direct memory comparison (`dax_compare`) for deduplication. - **NVDIMM Reverse Mapping under Reflink:** Handling hardware-level memory failure when multiple files share a single physical NVDIMM page via reverse mapping.

### Technical Details To enable reflink and CoW in FS-DAX without the conventional page cache intermediary, the kernel's `iomap` framework had to be significantly enhanced: - **`iomap` Enhancement for Copy-on-Write:** A new `iomap` type and an auxiliary structure (`source_map`) were introduced to pass source extent metadata during `iomap_begin`. In both regular write and page fault (`mmap`) paths, direct access (`dax_direct_access`) translates block offsets into physical memory addresses, allowing direct memory copies (`memcpy`) of source extent data to newly allocated extents alongside new user data before updating mappings. - **Iterative Reverse Mapping Solutions:** When an NVDIMM page suffers a hardware failure, the kernel triggers memory failure handling to find associated process virtual memory areas (VMAs) and terminate them. Because reflink maps multiple files to a single physical page, standard 1:1 page-to-file reverse mapping breaks down: 1. *Idea 1 (DAX rmap rb-tree):* An red-black tree attached to `page->private` was tested to track multiple file mappings per page. This caused significant memory overhead and could not handle metadata allocations. 2. *Idea 2 (Storage Lost Interface):* Leveraged XFS's internal reverse-mapping B-tree (`rmapbt`) via a filesystem-level `storage_lost` callback that queried extents to generate an owner list. 3. *Idea 3 (Zero-Overhead Immediate Invocation & Driver Interface):* The intermediate owner list overhead was eliminated by calling the memory failure handler directly per owner as the filesystem iterates through `rmapbt`. This was generalized across NVDIMM modes via a driver-level `memory_failure` interface.

### Takeaways The integration of reflink and deduplication with FS-DAX, along with the elimination of experimental per-page rmap overhead, paves the way to lifting the experimental status of FS-DAX in the Linux kernel. Remaining engineering challenges include resolving filesystem identification when NVDIMM sits behind mapped devices (such as LVM) and providing fallback mechanisms for filesystems or devices that lack reverse-mapping B-tree structures (such as XFS real-time devices).