March 13, 2019

A Saga of Code Executions on Zimbra

Zimbra is well known for its signature email product, Zimbra Collaboration Suite. Putting client-side vulnerabilities aside, Zimbra seems to have very little security history in the past. Its last critical bug was a Local File Disclosure back in 2013.

Recently with several new findings, it has been known that at least one potential Remote Code Execution exists in all versions of Zimbra. Specifically,

- Pre-Auth RCE on Zimbra <8.5.

- Pre-Auth RCE on Zimbra from 8.5 to 8.7.11.

- Auth'd RCE on Zimbra 8.8.11 and below with an additional condition that Zimbra uses Memcached. More on that in the next section.

Breaking Zimbra part 1


1. The XXE cavalry - CVE-2016-9924, CVE-2018-20160, CVE-2019-9670

Zimbra uses a large amount of XML handling for both its internal and external operations. With great XML usage comes great XXE vulnerabilities.

Back in 2016, another researcher discovered CVE-2016-9924 with the bug locating in SoapEngine.chooseFaultProtocolFromBadXml(), which happens on the parsing of invalid XML requests. This code is used in all Zimbra instances version below 8.5. Note however, as there's no way to extract the output to the HTTP response, an out-of-band extraction method is required in exploiting it.

For more recent versions, CVE-2019-9670 works flawlessly where the XXE lies in the handling of Autodiscover requests. This can be applied on Zimbra from 8.5 to 8.7.11. And for the sake of completeness, CVE-2018-20160 is an XXE in the handling of XMPP protocol and an additional bug along CVE-2019-9670 is a prevention bypass in the sanitizing of XHTML documents which also leads to XXE, however they both require some additional conditions to trigger. These all allow direct file extraction through response.

It's worth to mention that exploiting out-of-band XXE on recent Java just got a lot harder due to a patch in the core FtpClient which makes it reject all FTP commands containing newline. This doesn't affect the exploits for the vulnerabilities mentioned above, but it did make some of my previous efforts to chain XXE with other bugs in vain.
sun.net.ftp.impl.FtpClient

On installation, Zimbra sets up a global admin for its internal SOAP communications, with the username 'zimbra' and a randomly generated password. These information are always stored in a local file named localconfig.xml. As such, a file-read vulnerability like XXE could potentially be catastrophic to Zimbra, since it allows an attacker to acquire the login information of a user with all the admin rights. This has been demonstrated as the case in a CVE-2013-7091 LFI exploit where under certain conditions, one could use such credentials to gain RCE.

However things have never been that easy. Zimbra manages user privileges via tokens, and it sets up an application model such that an admin token can only be granted to requests coming to the admin port, which by default is 7071. The aforementioned LFI exploit conveniently assumes we already have access to that port. But how often do you see the weirdo 7071 open to public?

2. SSRF to the rescue - CVE-2019-9621

If you can't access the port from public, let the application do it for you. The code at ProxyServlet.doProxy() does exactly what its name says, it proxies a request to another designated location. What's more, this servlet is available on the normal webapp and therefore accessible from public. Sweet! However the code has an additional protection, it checks whether the proxied target matches a set of predefined whitelisted domains. That is, unless the request is from an admin. Sounds right, an admin should be able to do what he wants.

(Un)Fortunately, the admin checks are flawed. First thing it checks is whether the request comes from port 7071. However it uses ServletRequest.getServerPort() to fetch the incoming port. This method returns a tainted input controllable by an attacker, which is the part after ':' in the Host header. What's more, after that the check for the admin token happens only if it is fetched from a parameter, meanwhile we can totally send a token via cookie! In short, if we send a request with 'foo:7071' Host header and a valid token in cookie, we can proxy a request to arbitrary targets that is otherwise only accessible to admins.

3. Pre-Auth RCE from public port

ProxyServlet still needs a valid token though, so how does this fit in a preauth RCE chain? Turns out Zimbra has a 'hidden' feature that can help us generate a normal user token under the special global 'zimbra' account. When we modify an ordinary SOAP AuthRequest which looks like this:
...<account by="name">an</account>...
into this:
...<account by="adminName">zimbra</account>...
Zimbra will then lookup all the admin accounts and proceed to check the password. This is actually quite surprising because Zimbra admins and users naturally reside in two different LDAP branches. A normal AuthRequest should only touch the normal user branch, never the other. If the application wants a token for an admin, it already has port 7071 for that.

Note that while this little trick could give us a token for the 'zimbra' user, this token doesn't have any of the admin flag in it as it's not coming from port 7071. This is when ProxyServlet jumps in, which will help us to proxy another admin AuthRequest to port 7071 and obtain a global admin token.

Now that we've got everything we need. The flow is to read the config file via XXE, generate a low-priv token through a normal AuthRequest, proxy an admin AuthRequest to the local admin port via ProxyServlet and finally, use the global admin token to upload a webshell via the ClientUploader extension.

Breaking Zimbra part 2


Zimbra has its own implementation of IMAP protocol, where it keeps a cache of the recently logged-in mailbox folders so that it doesn't have to load all the metadata from scratch next time. Zimbra serializes a user's mailbox folders to the cache on logging out and deserializes it when the same user logs in again.

It has three ways to maintain a cache: Memcached(network-based input), EhCache(memory-based) and file-based. If one fails, it tries the next in list. Of all of those, we can only hope to manipulate Memcached, and this is the condition of the exploit: Zimbra has to use Memcached as its caching mechanism. Even though Memcached is prioritized over the others, (un)fortunately on a single-server instance, the LDAP key zimbraMemcachedClientServerList isn't auto-populated, so Zimbra wouldn't know where the service is and will fail over to Ehcache. This is probably a bug in Zimbra itself, as Memcached service is up and running by default and that way it wouldn't have any data in it. On a multi-server install however, setting this key is expected as only Memcached can work accross many servers.

To check whether your Zimbra install is vulnerable, invoke this command on every node in the cluster and check if it returns a value:
$ zmprov gs `zmhostname` zimbraMemcachedClientServerList

This was assigned CVE-2019-6980. The deserialization process happens at ImapMemcachedSerializer.deserialize() and triggers on ImapHandler.doSELECT() i.e. when a user invoking an IMAP SELECT command. The IMAP port in most cases is publicly accessible, so we can safely assume the trigger of this exploit.

To bring this to RCE, one still needs to find a suitable gadget to form a chain. The twist is, none of the current public chains (ysoserial) works on Zimbra.

1. Making of a gadget

Of all the gadgets available, MozillaRhino1 particularly stands out as all classes in the chain are available on Zimbra's classpath. This chain is based on Rhino library version 1.7R2. Zimbra uses the lib yuicompressor version 2.4.2 for js compression, and yuicompressor is bundled with Rhino 1.6R7. The unfortunate thing is there's an internal bug in 1.6R7 that would break the MozillaRhino1 chain before it ever reaches code execution, so we're out of luck. The good thing is, thanks to the effort in attempting to get the original chain to work and to the blog post detailing the MozillaRhino1 chain [1], we learnt a lot about Rhino's internals and on our way to pop another gadget.

There are two main points. First, the class NativeJavaObject on deserialization will store all members of an object's class. Members refer to all elements that define a class such as variables and methods. In Rhino context, it also detects when there's a getter or setter member and if so, it declares and includes the corresponding bean as an additonal member of this class. Second, a call to NativeJavaObject.get() will search those members for a matching bean name and if one is found, invoke that bean's getter. These match the nature of one of the native 'gadget helpers' - TemplatesImpl.getOutputProperties(). Essentially if we can pass in the name 'outputProperties' in NativeJavaObject.get(), Rhino will invoke TemplatesImpl.getOutputProperties() which will eventually lead to the construction of a malicious class from our predefined bytecodes. Searching for a place that we can control the passed-in member name leads to the discovery of JavaAdapter.getObjectFunctionNames() (Thanks to the valuable help from @matthias_kaiser) and it's directly accessible from NativeJavaObject.readObject().

The chain is now available in ysoserial's payload storage under the name MozillaRhino2. It works all the way to the latest version (with some tweaks) and has some additional improvement over MozillaRhino1. One interesting thing I found while reading Matt's blog post is that OpenJDK 1.7.x always bundles with rhino as its scripting engine, which essentially means that these rhino gadgets may very well work natively on OpenJDK7 and below.

This discovery escalates the bug from a Memcached Injection into a Code Execution. To exploit it, query into the Memcached service, pop out any 'zmImap' key, replace its value with the serialized object from ysoserial and next time the corresponding user logins via IMAP, the deserialization will trigger.

2. Smuggling from HTTP to Memcached

RCE from port 11211 sounds fun, but less so practical. So again, we turn to SSRF for help. The idea is to use the HTTP request from SSRF to inject our defined data in Memcached. To accomplish this, first we need to control a field in the HTTP request that allows the injection of newlines (CRLF). This is because a CRLF in Memcached will denote the end of a command and allow us to start a new arbitrary command after that. Second, since we're pushing raw objects into Memcached, our controlled input also needs to be able to carry binary data.

Zimbra has quite a few SSRFs in itself, however there's only one place that suffices both conditions, and it happens to be the all-powerful ProxyServlet earlier.

For a successful smuggle from HTTP to Memcached protocol, you should see something like above under the hood. It has exactly 6 ERROR and 1 STORED, correlating to 6 lines of HTTP headers and our payload, which also means our payload was successfully injected.

3. RCE from public port

That said, things are different when we use SSRF to inject to Memcached. In this situation we could only inject data into the cache, not pop data out because HTTP protocol cannot parse Memcached response. So we have no idea what our targeted Memcached entry's key looks like, and we need to know the exact key to be able replace its value with our malicious payload.

Fortunately, the Memcached key for Zimbra Imap follows a structure that we can construct ourselves. It follows the pattern
zmImap:<accountId>:<folderNo>:<modseq>:<uidvalidity>
with:
- accountId fetched from hex-decoding any login token
- folderNo the constant '2' if we target the user's Inbox folder
- modseq and uidvalidity obtained via IMAP as shown below

Now we have everything we need. Putting it together, the chain would be as follows:
- Get a user credentials
- Construct a Memcached key for that user following the above instructions
- Generate a ysoserial payload from the gadget MozillaRhino2, use it as the Memcached entry value.
- Inject the payload to Memcached via the SSRF. In the end, our payload should look like:
"set zmImap:61e0594d-dda9-4274-87d8-a2912470a35e:2:162:1 2048 3600 <size_of_object>" + "\r\n" + <object> + "\r\n"
- Login again via IMAP. Upon selecting the Inbox folder, the payload will get deserialized, followed by the RCE gadget.

The patches


Zimbra issued quite a number of patches, of which the most important are to fix XXEs and arbitrary deserialization. However the fix is only available for 8.7.11 and 8.8.x. If you happen to use an earlier version of Zimbra, consider upgrading to one of their supported version.

As a workaround, blocking public requests going to '/service/proxy*' would most likely break the RCE chains. Unfortunately there's none that I can think of that could block all the XXEs without also breaking some of Zimbra features.

Edit 30/04: Including a more specific workaround for the Autodiscover XXE and ProxyServlet SSRF which seem to be actively exploited. Locate {zimbra_home}/mailboxd/etc/service.web.xml.in, find the servlet tags named ProxyServlet and AutoDiscoverServlet, remove %%zimbraMailPort%% and %%zimbraMailSSLPort%% in both allowed.ports param and restart Zimbra. This would prevent public access to the affected components. Other than that, this thread provides some useful information suggested by savvy Zimbra user on how to clean an infected instance.

22 comments:

  1. Hi! In my simulation environment, Zimbra's version is Zimbra 8.7.11_GA_1854. I tried to use the CVE-2018-20160 to trigger the XXE vulnerability, but it failed. The CVE-2018-20160 vulnerability can display content in an HTTP response, but I have tried it many times and found it to be unsuccessful. Do you know how to trigger the CVE-2018-20160 vulnerability?

    ReplyDelete
  2. How does ssrf upload webshell through clientuploader? I tried it many times and it failed. Because ProxyServlet cookie will not be added to the request, may I ask how you do it?

    ReplyDelete
  3. hi,I am a Chinese,can you send me a link about how to exploit the CVE-2019-9621,I`ve been searching for a long time

    ReplyDelete
  4. Hi,

    Sorry for the ignorance and kudos for what it looks like a really outstanding job. For what I could understand, the use of imap is needed for doing some harm. Is not using imap service a good precaution to avoid the risk of being vulnerable? At least while we plan the timeframe for the update?

    Also, excecuting this "zmprov gs `zmhostname` zimbraMemcachedClientServerList" and getting nothing as return, does it means that we have not yet been attacked? Or only that we are not yet vulnerable?

    Thank you very much for your help!

    Thank you!

    ReplyDelete
    Replies
    1. Hi, sorry for the late reply. That command output does mean you are not vulnerable to the deserialization RCE, so blocking IMAP wouldn't make much sense.

      Delete
  5. Imagine my surprise when I saw 5 random chars account with an admin one in one of my clients server. LOL, Wtf??
    Lucky for me, he left many prints through its journey into that server, and I noticed him.
    But he was able to upload a java web shell, wget some Bash stuff and tried to execute it to gain root access. Patched and cut him off just in time.
    However logs analysis (particularly XXE in Autodiscover.xml) brought me here.
    A very nice job. Work like this helps to build a better and more secure internet.
    Thanks

    ReplyDelete
  6. Would be awesome to test that via a python script just to see if our environment is vulnerable. Didn't saw any PoC scripts yet :/

    ReplyDelete
  7. Can we just upgrade mailbox server from 8.7.1 to 8.7.11 and then patch 10 without upgrading LDAP, proxy ,mta

    ReplyDelete
  8. Can we change the ownership of /jetty/webapps/zimbra to other system accounts and only allow zimbra for read and execute without write permission so that we can prevent hacker on uploading files

    ReplyDelete
    Replies
    1. You should do it on the /webapps directory, not just /zimbra

      Delete
  9. When it comes to blocking '/service/proxy': I decided to block HTTP(S) completely and only allow access through an HTTP proxy with HTTP authentication. So, to use my webmail, I have to go to the proxy and enter a browser password first. I had to exlude some paths to make CalDAV and CardDAV work.

    See details here: https://forums.zimbra.org/viewtopic.php?f=15&t=65932&start=30#p289864.

    ReplyDelete
    Replies
    1. That seems to only add an additional layer of authentication and doesn't really block /service/proxy. For the workaround I'd block ProxyServlet from public access altogether. To do that, change the allowed.ports parameter to only 7070 in the servlet tag named ProxyServlet in {zimbra_home}/mailboxd/etc/service.web.xml.in and restart the instance. You could also disable AutoDiscoverServlet there for the XXE.

      Delete
    2. This comment has been removed by the author.

      Delete
    3. Is it mean to change the below line
      from
      \\\\%%zimbraMailPort%%, %%zimbraMailSSLPort%%, 7070\\\
      \\\\7070\\\

      Delete
    4. Yes. I also updated the blog post to include this workaround.

      Delete
  10. Hi, I tried in my lab to use metasploit zimbra XXE module, with zimbra 8.6.0 1194 on centos7.
    The exploit can't upload java file, but can create the user.I want to try this vulnerability in my lab, Someone can give me some suggestion?

    [*] Executing payload on /downloads/LrLEjpQrXJbHDOq.jsp
    [*] Server stopped.
    [!] This exploit may require manual cleanup of '$(find /opt/zimbra/ -regex '.*downloads/.*LrLEjpQrXJbHDOq.jsp' -type f)' on the target
    [!] This exploit may require manual cleanup of '$(find /opt/zimbra/ -regex '.*downloads/.*LrLEjpQrXJbHDOq.*1StreamConnector.class' -type f)' on the target

    ReplyDelete
  11. I have checked that the mailbox server is not listening on port 7070 and only the admin port 7071. What is the meaning of 7070 port in the remaining configuration?
    How to total disable the autodiscover servlet?

    ReplyDelete
  12. Hi,

    Thanks for the detailed announcement.

    We would like to know if Zimbra's MTA vulnerable? From what we can see on your blog post, HTTP+IMAP are vulnerable and mainly any service proxied by Zimbra' Nginx. We have an internal server where all the ports are restricted beside tcp port 25 in order to receive emails. The service that listens to this port is Postfix. Is there ANY chances we could have been vulnerable?

    Thanks!

    ReplyDelete
    Replies
    1. AFAIK Postfix has nothing to do with the other services so you should be fine.

      Delete
  13. Thanks for the answer. How about LMTP ?

    ReplyDelete