Linux Kernel Device-Exclusive Handling Vulnerability with THP and Hugetlb

A vulnerability in the Linux kernel's memory management related to device-exclusive entries and Transparent Huge Pages (THP) has been identified. This issue arises because the device-exclusive handling does not function correctly with THP, particularly when hugetlb is enabled. The vulnerability can disrupt the migration, swapout, and split handling of memory folios that have device-exclusive Page Table Entries (PTEs). As a result, these folios cannot be swapped out, leading to a persistent indication of high memory usage, and can cause infinite loops when attempting to offline certain memory blocks.

Impact

The vulnerability can cause memory management issues, preventing proper migration and swapping of pages, which can lead to increased memory usage and potential memory exhaustion.

Reproduction

The vulnerability can be reproduced by allocating a large amount of memory (up to 1 GiB) and making the pages device-exclusive on a kernel with the HMM testing feature enabled. Once the pages are device-exclusive, they cannot be swapped out, causing memory management tools to report incorrect memory usage. Attempting to offline certain memory blocks can also result in infinite loops, due to failed migration of pages that should be movable.