Flexible orders anon pages

### Overview This session focuses on introducing flexible order anonymous folios (pages) in the Linux kernel. Historically, anonymous memory folios only supported order 0 (standard base pages, typically 4KB) or PMD order (transparent huge pages, e.g., 2MB on x86). However, order 0 pages suffer from poor scalability due to heavy page fault overhead, excessively long LRU (least recently used) lists on systems with multiple gigabytes of memory, and substantial TLB flush costs involving inter-processor interrupts (IPIs) on x86. While larger base page sizes (like 64KB) can improve performance, they suffer from high internal fragmentation, require linker and user-space memory allocator adjustments (or entirely recompiled user spaces), and demand separate kernel binaries. In contrast, medium-sized anonymous folios provide a middle ground that user space can consume transparently while taking advantage of modern CPU hardware TLB coalescing (e.g., order 3 on AMD, order 2 Hardware Page Aggregation, and order 4 contiguous bit on ARM64).

### Key Topics - **Allocation Policy and Alignment:** Deciding how to identify suitable VMAs (Virtual Memory Areas) and determine folio allocation sizes without creating an NP-hard bin-packing problem. Presenters and attendees discussed simplifying the allocation by enforcing strict logical, physical, and virtual alignment to folio size, or by allocating the largest possible folio that avoids overlaps. - **Fallback Mechanisms and Page Allocator Support:** Addressing memory pressure behaviors. Instead of looping down through every order from PMD order to order 0, a suggested improvement is an API extension where the kernel page allocator directly returns the highest available order up to a requested limit without forcing direct reclaim. - **Concurrency and Overlaps:** Handling concurrent page faults across threads in adjacent virtual memory areas where multiple threads attempt to allocate overlapping large folios. The proposed handling checks page table entries (PTEs) under lock and bails out/frees the duplicate allocation if an entry is already mapped. - **Reference and Map Counting:** Reviewing whether to adapt existing page table mapping schemes or develop a new accounting scheme for intermediate-sized folios, noting that compound map counts require dedicated handling. - **Kernel Statistics and Visibility:** Debating how medium-sized folios should be exposed in `/proc/meminfo` (e.g., whether to keep `AnonHugePages` reserved strictly for PMD-sized pages via `folio_test_pmd_mappable` and use `smaps`, tracepoints, or histograms for intermediate orders). - **Reclaim, Compaction, and Splitting:** Managing memory reclaim using access bits per PTE to detect underutilization and internal fragmentation; adjusting memory compaction so compound folios smaller than a page block are inspected rather than blindly skipped; coordinating collapsing (khugepaged) so it does not conflict with reclaim; and handling splits upon partial `munmap` or memory failure.

### Technical Details - **Architecture-Specific Coalescing:** ARM64 supports transparent Hardware Page Aggregation (HPA) at order 2 and an explicit contiguous bit hint in the PTE at order 4 (64KB block). A noted tradeoff with the ARM64 contiguous bit is that it maintains a single access and dirty bit for the whole 64KB block instead of individual bits per 4KB page. - **Page Cache Parallels:** Matthew Wilcox shared insights from page cache multi-order folios, demonstrating that alignment assumptions turn potential bin-packing problems into simple linear or binary search lookups against existing folios. - **Splitting and Partial Unmapping:** Folio splitting is necessary when a process partially unmaps a range covered by a large folio or during memory reclaim when low utilization is detected. This applies regardless of whether the mapping is PMD-level or PTE-level.

### Takeaways Flexible orders for anonymous folios represent a critical optimization for modern hardware, bridging the gap between 4KB pages and 2MB huge pages to optimize TLB efficiency and shorten LRU lists. Realizing this in the Linux kernel requires cross-subsystem coordination, including updates to the page allocator fallback logic, compaction algorithms, memory statistics reporting, and core memory management operations such as `madvise` and folio splitting.