OISM libcoap Integer Signedness Error in TLS Verification Callback Leading to Denial-of-Service

A denial-of-service vulnerability has been identified in OISM libcoap version 4.3.5. The issue arises from an integer signedness error in the TLS verification callback function, tls_verify_call_back(), located in src/coap_openssl.c. Remote attackers can exploit this vulnerability by sending a crafted TLS certificate that causes the OpenSSL function i2d_X509() to return -1. Libcoap improperly handles this error, using the negative return value as a size parameter for memory allocation, which can lead to a memory allocation failure.

Impact

Exploitation of this vulnerability causes a memory allocation error, where the library attempts to allocate an excessively large amount of memory based on the negative return value from the OpenSSL function i2d_X509(). This error can be observed in the application's AddressSanitizer log, indicating that the requested allocation size exceeds the maximum supported limit.

Reproduction

The vulnerability can be reproduced by using libcoap 4.3.5 with OpenSSL support enabled. When the library verifies a TLS certificate, the OpenSSL function i2d_X509() will return -1, simulating a failure in the certificate processing. Libcoap will then attempt to allocate memory using this invalid return value, leading to a memory allocation error.

Remediation

Users can update to the latest version of OISM libcoap, where this vulnerability has been fixed. Instructions for updating can be found in the libcoap repository.