MLflow Environment Variable Resolution Vulnerability in AI Gateway Secrets Allowing Credential Exfiltration

A vulnerability exists in MLflow versions prior to 3.11.0, where AI Gateway secrets can resolve environment variable references. This flaw allows low-privileged authenticated users in basic-auth deployments, or unauthenticated users in default deployments without basic-auth, to exfiltrate sensitive server-side environment credentials to an attacker-controlled endpoint. The issue stems from the 'api_key' field in gateway secrets accepting '$ENV_VAR' references, which are resolved at runtime and sent in authentication headers to the specified 'api_base'. Exploitation could lead to the leakage of critical credentials, such as cloud artifact credentials, potentially causing artifact poisoning and cross-boundary code execution in downstream environments.

Impact

Exploitation of this vulnerability allows for the unauthorized exfiltration of sensitive environment variables, including cloud credentials and MLflow server secrets, to an attacker-controlled endpoint. In cases where leaked cloud credentials provide write access to artifact storage, this could lead to maliciously poisoning model artifacts and executing arbitrary commands in environments that load the compromised models.

Reproduction

The vulnerability can be reproduced by creating a gateway secret that includes a reference to an environment variable containing sensitive information, such as a cloud credential. Once the secret is created, it can be used in a model definition and endpoint, which, when invoked, will trigger the resolution of the environment variable reference. The resolved value is then sent to the specified 'api_base' endpoint, allowing for the exfiltration of the sensitive information.

Remediation

Users can update to MLflow version 3.11.0 or later, where this vulnerability has been fixed.