SaltStack authorization bypass

Product SaltStack Salt
Severity High
CVE Reference CVE-2020-11651 CVE-2020-11652
Type RCE

Background

F-Secure consultants discovered a number of vulnerabilities in the "Salt" management framework by the company SaltStack. The open source Salt project (https://github.com/saltstack/salt) is at the heart of SaltStack's (the company) product offerings but is also very popular as a configuration tool to manage servers in datacenters and cloud environments.

Salt is used to monitor and update the state of servers. Each server runs an agent called a "minion" which connects to a "master", a Salt installation that collects state reports from minions and publishes update messages that minions can act on. Typically, such messages are updates to the configuration of a selection of servers, but they can also be used to run the same command in parallel over multiple, even all, managed systems asynchronously. 

The default communication protocol in salt is ZeroMQ. The master exposes two ZeroMQ instances, one called the "request server" where minions can connect to report their status (or the output of commands) and one called the "publish server" where the master publishes messages that the minions can connect and subscribe to. For more information, see https://docs.saltstack.com/en/getstarted/system/communication.html.

Description

The vulnerabilities described in this advisory allow an attacker who can connect to the "request server" port to bypass all authentication and authorization controls and publish arbitrary control messages, read and write files anywhere on the "master" server filesystem and steal the secret key used to authenticate to the master as root. The impact is full remote command execution as root on both the master and all minions that connect to it.

The vulnerabilities, allocated CVE ids CVE-2020-11651 CVE-2020-11652, are of two different classes. One being authentication bypass where functionality was unintentionally exposed to unauthenticated network clients, the other being directory traversal where untrusted input (i.e. parameters in network requests) was not sanitized correctly allowing unconstrained access to the entire filesystem of the master server.

Remediation

SaltStack engineers patched these vulnerabilities in release 3000.2 and users of Salt are encouraged to make sure that their installs are configured to automatically pull updates from SaltStacks repository server, see https://repo.saltstack.com/ for more information. A patch release for the previous major release version is also available, with version number 2019.2.4.

Adding network security controls that restrict access to the salt master (ports 4505 and 4506 being the defaults) to known minions, or at least block the wider Internet, would also be prudent as the authentication and authorization controls provided by Salt are not currently robust enough to be exposed to hostile networks.

Detection

A scan revealed over 6,000 instances of this service exposed to the public Internet. Getting all of these installs updated may prove a challenge as we expect that not all have been configured to automatically update the salt software packages.

To aid in detecting attacks against vulnerable salt masters, the following information is provided.

Exploitation of the authentication vulnerabilities will result in the ASCII strings "_prep_auth_info" or "_send_pub"  appearing in data sent to the request server port (default 4506). These strings should not appear in normal, benign, traffic.

Published messages to minions are called "jobs" and will be saved on the master (default path /var/cache/salt/master/jobs/). These saved jobs can be audited for malicious content or job ids ("jids") that look out of the ordinary. Lack of suspicious jobs should not be interpreted as absence of exploitation however.

No reliable log evidence has been identified.

Technical Details

Authentication bypass vulnerabilities (CVE-2020-11651)

The ClearFuncs class processes unauthenticated requests and unintentionally exposes the _send_pub() method, which can be used to queue messages directly on the master publish server. Such messages can be used to trigger minions to run arbitrary commands as root.

The ClearFuncs class also exposes the method _prep_auth_info(), which returns the "root key" used to authenticate commands from the local root user on the master server. This "root key" can then be used to remotely call administrative commands on the master server. This unintentional exposure provides a remote un-authenticated attacker with root-equivalent access to the salt master.

Directory traversal vulnerabilities (CVE-2020-11652)

The wheel module contains commands used to read and write files under specific directory paths. The inputs to these functions are concatenated with the target directory and the resulting path is not canonicalized, leading to an escape of the intended path restriction.

The get_token() method of the salt.tokens.localfs class (which is exposed to unauthenticated requests by the ClearFuncs class) fails to sanitize the token input parameter which is then used as a filename, allowing insertion of ".." path elements and thus reading of files outside of the intended directory. The only restriction is that the file has to be deserializable by salt.payload.Serial.loads().

Exploitation

We expect that any competent hacker will be able to create 100% reliable exploits for these issues in under 24 hours. Due to reliability and simplicity of exploitation, F-Secure will not be providing proof-of-concept exploit code as this would only harm any users who are slow to patch. In this case, we will leave exploitation as an exercise for the reader.

Disclosure timeline

2020-03-12 The GPG key for the SaltStack security team published on saltstack.com had expired in 2018 and a request for an updated key was sent.
2020-03-16 Repeated request for an updated GPG key resulted in publication of a re-signed key to the security contact page. Full vulnerability report sent to SaltStack security team. 
2020-03-20 Request for confirmation of receipt was sent to the SaltStack security team. 
2020-03-24 The SaltStack security team confirm receipt of the vulnerability report and that they are reviewing it.
2020-04-07 SaltStack asks if F-Secure has requested CVE's for the reported vulnerabilities and F-Secure replies in the negative, recommending that SaltStack proceed to contact Mitre in order to reserve CVE IDs.
2020-04-15 F-Secure informs SaltStack that an Internet wide scan turned up over 6,000 publicly exposed salt masters and expresses concern that these will be at risk of compromise when the vulnerabilities are disclosed. Also, F-Secure requests information on SaltStacks plan for distributing fixes.
2020-04-16 SaltStack informs F-Secure that fixes are being tested and planned for release "early next week". F-Secure reiterates concerns over the number of salt masters exposed to the public Internet and requests information about how SaltStack plans to communicate this release to their customers.  
2020-04-18 SaltStack informs F-Secure of their communication plans and requests the list of identified IP-addresses that expose a salt master to the public Internet as well as suggestions for alternative approaches to disclosure.
2020-04-20 F-Secure provides IP-address list and suggests a multiple-phase communication strategy as an alternative.
2020-04-23 SaltStack publishes advance notice to their users urging them not to expose salt masters to the Internet and to prepare to apply the patch once it is published on the 29th (https://github.com/saltstack/community/blob/master/doc/Community-Message.pdf).
2020-04-27  F-Secure requests information from SaltStack about the CVE ids allocated for the vulnerabilities.
 2020-04-29 F-Secure reiterates request for CVE ids to SaltStack.
 2020-04-29 SaltStack responds with the allocated CVE identifiers.
 2020-04-29 SaltStack publishes version 3000.2 and 2019.2.4 addressing these issues.
 2020-04-30   F-Secure publishes advisory.