Investigating and Securing Systems
Article Information |
---|
This article applies to the following ZCS versions. |
Investigating and Securing Exploited Systems
Credit
Thank you to the multiple customers/partners who have helped with this investigation and provided info and methods listed here.
Background
Some sites have seen issues where attackers have achieved "admin" level access in ZCS, and from there have had trojan Zimlets installed which run bitcoin mining operations. In many of these cases, the vector of admin-level access has occurred via the Critical Security Vulnerability fixed in February 2013 through these versions:
- 8.0.2 Patch 1
- 8.0.3
- 7.2.2 Patch 1
- 7.2.3
For more information on the nature of the related vulnerabilities, please review these resources:
Public Announcements by Zimbra
- Critical Security Patches posted for 8.0.X/7.2.X - https://www.zimbra.com/forums/announcements/68390-critical-security-patches-posted-8-0-x-7-2-x.html
- Critical Security Vulnerability Addressed in 7.2.6/8.0.6 Maintenance Releases - https://www.zimbra.com/forums/announcements/67336-critical-security-vulnerability-addressed-7-2-6-8-0-6-maintenance-releases.html
- Security Guidance for reported "0day Exploit" - https://www.zimbra.com/forums/announcements/67236-security-guidance-reported-0day-exploit.html
- 8.0.2 Patch Release Notes: http://files2.zimbra.com/website/docs/8.0/ZCS_Patch_8_0_2_r1.pdf (Feb 2013)
- 7.2.2 Patch Release Notes: http://files.zimbra.com/website/docs/7.2/ZCS_Patch_Release_Notes_7_2_2_r3.pdf (Feb 20113)
For customers running versions earlier than this or who have had systems illegally accessed, the following action plan is available.
Investigating Exploited Systems
1. Check for recently modified Zimlets Note: adjust the mtime and ctime values back to around Dec 06, which is the first report we've seen:
# find /opt/zimbra/zimlets-deployed -mtime -35 -o -ctime -35
or
# find /opt/zimbra/zimlets-deployed -newerct 20131206 -ls # find /opt/zimbra/zimlets-deployed -newermt 20131206 -ls
Note: your site may vary, but please check any modified Zimlets carefully back to Dec 06. A more thorough search might look for any changed file since a certain date:
# find /opt/zimbra \( -path /opt/zimbra/backup -o -path /opt/zimbra/store -o -path /opt/zimbra/db/data \) \ -prune -o -newermt 20140127 -ls > /var/tmp/changes.out
2. Identify any jsp files that may have been installed:
Examples:
$ ls -lR com_zimbra_example_simplejspaction -rw-r--r--@ 1 648 Dec 26 07:25 cmd.jsp -rw-r--r--@ 1 481 Dec 26 07:25 com_zimbra_example_simplejspaction.xml -rw-r--r--@ 1 14 Dec 26 07:25 jspfile.jsp
$ ls -lR com_zimbra_example_simplejspaction2 -rw-r--r--@ 1 483 Jan 1 06:07 com_zimbra_example_simplejspaction2.xml -rw-r--r--@ 1 14 Jan 1 06:07 jspfile.jsp -rw-r--r--@ 1 648 Jan 1 06:07 xd.jsp
$ ls -lR com_zimbra_example_simplejspaction -rw-r-----@ 1 481 Dec 23 13:21 com_zimbra_example_simplejspaction.xml -rw-r-----@ 1 14 Dec 23 13:21 jspfile.jsp -rw-r-----@ 1 74614 Dec 23 13:21 meep.jsp
$ ls -lR com_zimbra_data zim.jsp
3. Disabling this Zimlet (or any other like it):
$ zmzimletctl undeploy com_zimbra_example_simplejspaction $ zmprov fc zimlet
4. Locate, backup and remove JSPs that use getRuntime:
# find /opt/zimbra -iname \*.jsp -exec grep -i getRuntime {} \;
5. Locate, backup and remove any files in /var/tmp and /tmp
# find /var/tmp /tmp -iname miner\* # find /var/tmp /tmp -mtime -35
Examples:
-rw-r--r--@ 1 592464 Jul 10 2013 kpoll -rw-r--r--@ 1 120 Jan 8 03:58 ksys.cfg
6. Check top for any high CPU processes
In addition to the above minerd processes, some sites with ssh open to the world may find that an attacker has used a password to gain ssh access. At one site, ssh access led to having a perl processing running at the following location, also mining bitcoins:
/opt/zimbra/log/wit.pl.1
It is highly recommended to allow only the necessary service ports open to untrusted networks or the Internet. The required service ports include the following, typically:
All other ports should be closed to untrusted networks.
7. Verify zimbra RPMs (RHEL/CentOS only) for modified files:
# for Z in $(rpm -qa | grep ^zimbra); do echo "Checking $Z"; rpm -q --verify --nouser --nogroup --nomode --nomtime $Z; done
8. Check and remove any rogue admin accounts:
Creation and control of one or more admin accounts by the attacker (since Dec 06, 2013). Find all admin accounts with this ldapsearch:
- be sure to replace "ldap.example.com" with the hostname of your LDAP server*
$ source ~/bin/zmshutil; zmsetvars; ldapsearch -x -LLL -b "" -H ldap://ldap.example.com:389 -D "$zimbra_ldap_userdn" -w "$zimbra_ldap_password" '(zimbraIsAdminAccount=TRUE)' dn
And all DomainAdmin accounts with this:
$ source ~/bin/zmshutil; zmsetvars; ldapsearch -x -LLL -b "" -H ldap://ldap.example.com:389 -D "$zimbra_ldap_userdn" -w "$zimbra_ldap_password" '(zimbraIsDomainAdminAccount=TRUE)' dn
9. Search mailbox.log for possible exploit:
$ grep -B 3 PARSE_ERROR /opt/zimbra/log/mailbox.log | grep '/service/soap:' $ grep -B 3 PARSE_ERROR /opt/zimbra/log/mailbox.log | grep '/service/admin/soap:'
Securing Systems
Important steps for protecting these systems:
1. Upgrade to 7.2.6 or 8.0.6
2. Patch if on an older version - Zimbra has provided patches for the following versions:
- 8.0.5, 8.0.4, 8.0.3
- 7.2.5, 7.2.4, 7.2.3, 7.2.2
All Patches and Release Notes for these versions are available on the Archives download page:
3. Workaround to block Bug 80338 exploit if on an older version:
Bug 80338 Workaround
For Bug 80338, there is an existing workaround using nginx: Add the below 3 lines to
/opt/zimbra/conf/nginx/templates/nginx.conf.web.[http|https].default.template
and then run the following:
/zmproxyconfgen; zmproxyctl restart
if ($request_uri ~ "\.\.") { return 404; } if ($request_uri ~ "\%2[eE]\%2[eE]") { return 404; }
if ($request_uri ~ "\.%2[eE]") { return 404; }
if ($request_uri ~ "%2[eE]\.") { return 404; }
4. Make sure that all ports to the ZCS system/platoform are blocked except those explicitly required for external services, as listed here:
5. If possible, block the admin port (port 7071 by default) from the Internet.
6. After securing via the above, change the relevant zmlocalconfig passwords:
You should change all of these on all nodes, and they must match for each password on all Zimbra nodes. However, not every password listed here must be the same, they can each differ as they are used by different processes. It is very important to make sure the first one succeeds first:
Change the zimbra_ldap_userdn password:
$ zmldappasswd newpassword
Test that it worked:
$ zmlocalconfig -s zimbra_ldap_password
Then proceed with the rest:
$ zmldappasswd -a newpassword $ zmldappasswd -b newpassword $ zmldappasswd -n newpassword $ zmldappasswd -p newpassword $ zmldappasswd -r newpassword
After running on all nodes, confirm that all nodes have the correct passwords in zmlocalconfig:
$ zmlocalconfig -s ldap_amavis_password $ zmlocalconfig -s ldap_bes_searcher_password $ zmlocalconfig -s ldap_nginx_password $ zmlocalconfig -s ldap_postfix_password $ zmlocalconfig -s ldap_root_passwd
7. If there is any reason to think files may have been accessed, then updating both SSL certificates and ssh keys should also be performed:
SSL Certificates:
SSH Auth Keys: