HW counters for hold/cold pages

### Overview This presentation explores using dedicated hardware access counters to optimize memory management components in the Linux kernel. The primary speaker presents experimental findings on PowerPC (PPC64) hardware across three main areas: Multi-Generational LRU (mglru), Transparent Huge Pages (THP), and tiered memory page promotion. A co-presenter also discusses leveraging access profiling mechanisms for page promotion in tiered memory setups.

### Key Topics - **Multi-Generational LRU (mglru) Integration**: Using hardware counters to estimate the relative hotness of pages and classify them into generations during reclaim sorting, potentially bypassing reverse mapping (rmap) walks and saving page flag bits. - **Transparent Huge Pages (THP) Utilization**: Detecting hidden cold pages inside 2MB huge pages via hardware counters configurable at smaller page sizes (e.g., 64KB on PPC64) and utilizing metrics like the Gini index to split underutilized huge pages. - **Tiered Memory Page Promotion**: Evaluating relative page access frequencies between slow memory nodes (such as Intel Optane) and DRAM nodes to avoid counterproductive migrations. - **DAMON (Data Access Monitor) Operations**: Discussing whether hardware counters should be integrated directly into core kernel mechanisms (like an asynchronous promotion daemon) or exposed via DAMON operations modules.

### Technical Details - **mglru Experiments**: Instead of absolute access counts, mglru requires relative hotness estimations across pages within an LRU vector. The speaker tested simple approximations based on minimum and maximum access counts as well as compute-intensive k-means clustering. While completely dropping rmap walks led to a 6% performance improvement in memcached, it also triggered a significant spike in refaults, and complex classification markedly increased lock contention on `lruvec`. - **THP Results**: In microbenchmarks simulating uneven access inside 2MB pages, using hardware counters allowed the system to detect cold sub-pages and split huge pages. Cold pages were sent to swap without subsequent swap-ins, reducing test execution time from approximately 700–800 seconds down to 70 seconds. - **Hardware Characteristics**: Unlike performance monitoring units (PMU) or sampling mechanisms like PEBS (which risk collisions with userspace profiling tools), PPC64 provides dedicated physical frame number (PFN) indexed 64-bit hardware counters in a dedicated memory region. These counters increment on accesses and decay over time.

### Takeaways While optimizing mglru reclaim using hardware counters introduces locking contention and classification challenges, THP utilization shows significant, clear performance benefits by identifying sparse, hidden cold pages. For tiered memory page promotion, discussions remain ongoing on whether to introduce dedicated kernel threads (such as `kpromote_d`) or integrate hardware counter operations into DAMON.