Linux Kernel rhashtable Deadlock Vulnerability

A potential deadlock vulnerability in the Linux kernel's rhashtable implementation has been addressed. The issue arose from a circular locking dependency involving nested locks between the rhashtable bucket, rq lock, and dsq lock. The original design could trigger a lockdep warning due to this deadlock scenario. By moving the hash table growth check and work scheduling outside of the rhashtable lock, the vulnerability was resolved, breaking the deadlock chain. This modification enhances rhashtable's flexibility by removing restrictive locking that previously constrained its use in scheduler and workqueue contexts.

Impact

The vulnerability could lead to a deadlock situation, causing processes to hang indefinitely, waiting for locks to be released.