Linux Kernel IB/Mlx5 Deadlock Vulnerability in Memory Registration Deregistration

A potential deadlock vulnerability has been identified in the Linux kernel's IB/mlx5 component, specifically during the deregistration of memory registrations. The issue arises when the 'kzalloc' function is called while holding the 'umem_mutex' or any other lock acquired under 'umem_mutex'. This creates a conflict because 'kzalloc' can initiate a file system reclaim process, which may invoke a memory notifier function that, in turn, tries to reacquire the 'umem_mutex', leading to a deadlock. The vulnerability can also be triggered when 'kzalloc' is called from a cache management function that accesses the same mutex, creating a similar deadlock scenario.

Impact

Exploitation of this vulnerability can lead to a deadlock situation, where the system becomes unresponsive due to circular lock dependencies.

Reproduction

The vulnerability can be reproduced by deregistering a memory region while holding the 'umem_mutex', which can be done through the 'mlx5_ib_dereg_mr' function. This process can be initiated by revoking a memory registration, which triggers the 'umem_mutex' lock. Simultaneously, the 'mlx5_mkey_cache_init' function can be called, which also requires locking a different mutex. When 'kzalloc' is invoked during this process, it triggers the file system reclaim, leading to the deadlock as the memory notifier tries to reacquire the 'umem_mutex'.

Remediation

The vulnerability has been addressed by modifying the 'mlx5_ib_dereg_mr' function to release the 'umem_mutex' before acquiring the 'rb_lock', ensuring that the mutex is not held during memory allocations that could trigger the reclaim process.