Linux Kernel File Descriptor Table Memory Allocation Vulnerability

A vulnerability in the Linux kernel's file handling can lead to excessive memory usage. When the 'nr_open' limit is set high (such as 1,073,741,816), processes can inadvertently cause memory allocations that exceed the maximum integer size. This issue triggers warnings in the kernel's memory management code and can disrupt normal system operations. The problem arises because the kernel's file descriptor allocation functions do not properly account for high limits, leading to impractical memory requests that can exceed 8 gigabytes.

Impact

This vulnerability can cause a denial-of-service condition by exhausting system memory, leading to failed operations and potential system instability.

Reproduction

To reproduce this vulnerability, set the 'nr_open' limit to a high value, such as 1,073,741,816. Then, run a program that tests file descriptor limits by attempting to use a file descriptor near this limit. The program should be configured to trigger the allocation warning by exceeding the maximum allowed size.

Remediation

The vulnerability has been addressed in a patch that adds a check to the file descriptor allocation function. This check ensures that the requested allocation size does not exceed the maximum integer limit, preventing the excessive memory usage that caused the issue.