LlamaIndex Uncontrolled Memory Consumption Vulnerability in SimpleDirectoryReader Component

A resource management vulnerability has been identified in the SimpleDirectoryReader component of LlamaIndex, specifically in version 0.12.23. The issue arises from the improper application of user-defined file limits, which are enforced only after all files in a directory have been loaded into memory. This flaw can lead to excessive memory usage and increased CPU consumption, particularly in environments with limited resources. The vulnerability can be exploited by providing directories with a large number of files, causing memory exhaustion and degraded performance.

Impact

The vulnerability causes significant memory inefficiencies and poses a denial-of-service risk by allowing the processing of large directories to exhaust system resources, leading to unresponsiveness, especially in multi-tenant environments.

Reproduction

The vulnerability can be reproduced by creating a directory with a large number of files, such as three million empty files, and then using the SimpleDirectoryReader to process the files while setting the num_files_limit parameter to a low value, like ten. This setup will cause the memory usage to spike over one gigabyte before the limit is applied, demonstrating the uncontrolled memory consumption issue.

Remediation

Users can upgrade to LlamaIndex version 0.12.41 or later, where this vulnerability has been fixed.