libsixel Heap Buffer Overflow Vulnerability Due to Signed Integer Overflow in Allocation Size Calculation

A heap buffer overflow vulnerability has been identified in libsixel versions 1.4.4 through 1.8.7-r1. This issue arises from a signed integer overflow in the 'sixel_encode_highcolor' function, where the allocation size for pixel buffers is incorrectly calculated. The public 'sixel_encode' function only checks that width and height are greater than zero, without imposing any upper limits. As a result, if a pixel buffer is requested with dimensions that, when multiplied, exceed the maximum value for an integer, the allocation size wraps around to a smaller value. This allows the 'malloc' function to succeed in allocating a buffer that is insufficient for the encoder's needs, leading to a buffer overflow as the encoder writes past the allocated memory.

Impact

Exploitation of this vulnerability causes a heap buffer overflow in the 'sixel_encode_highcolor' function. The encoder writes controlled data into a buffer that is smaller than expected, creating a risk of heap corruption. This vulnerability can be exploited to manipulate the heap in a way that could lead to arbitrary code execution, depending on the specific circumstances and environment.

Reproduction

The vulnerability can be reproduced by calling the 'sixel_encode' function with a width and height that, when multiplied together, exceed the maximum integer value. This can be done by setting both dimensions to 50000, which results in a product of 2.5 billion, exceeding the integer limit. After allocating a pixel buffer with the wrapped size, the 'sixel_encode' function is called, triggering the heap buffer overflow.

Remediation

Users can upgrade to libsixel version 1.8.7-r2, where this vulnerability has been fixed.