Troubleshooting High CPU load average in ZCS 5.0.x as compared to 4.5.x
In order to troubleshoot the high CPU usage observed in ZCS ver 5.0.x as compared to 4.5.x follow these steps: (Please refer to Bug# 23539 for more information: http://bugzilla.zimbra.com/show_bug.cgi?id=23539)
Collect the following information:
Make this change ASAP before you observe High CPU load average:
1) Please add the following java options (-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps) using this command:
$ zmlocalconfig -e mailboxd_java_options="`zmlocalconfig -m nokey mailboxd_java_options` -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"
Note: You'll need to do a zmcontrol stop/start after making the above config change.
Get the output of these commands during high CPU usage:
2) Obtain some thread dumps in order to troubleshoot the problem (and before you restart):
$ /opt/zimbra/libexec/zmmailboxdmgr threaddump # run this command 4 times (30 seconds apart) $ cat /proc/<jetty_pid>/task/*/stat > jetty-task-stat.$(date +%Y%m%d-%H%M%S) $ pstack <jetty_pid> # where <jetty_pid> is the pid of the jetty process from "ps -ef | grep jetty".
3) Generate charts for the day that the problem occurred:
$ zmstat-chart -s /opt/zimbra/zmstat/<YYYY-MM-DD> -d <output-dir>
Then, scrutinize the following:
a) generated charts b) /opt/zimbra/log/zmmailboxd.out, c) the data in /opt/zimbra/zmstat covering the period of the high CPU load, d) the jetty-task-stat files from above and e) the output from pstack.
4) Other important files to observe(or obtain) at this point:
a) /opt/zimbra/conf/zmstat-chart.xml (if the file doesn't exist, you can obtain the output of zmstat-chart-config), b) /opt/zimbra/log/mailbox.log, c) /opt/zimbra/conf/my.cnf, d) /opt/zimbra/conf/slapd.conf, e) /opt/zimbra/openldap-data/DB_CONFIG, f) /opt/zimbra/zmstat/<mention the date here - when you observe high CPU load>/*. (You can zip or tar them altogether, if you are transferring them somewhere)
5) For further information obtain the output of the following commands,
a) /opt/zimbra/sleepycat/bin/db_stat -m -h /opt/zimbra/openldap-data | head -n 11 b) /opt/zimbra/sleepycat/bin/db_stat -c -h /opt/zimbra/openldap-data | head -n 12 c) grep dn: /opt/zimbra/backup/sessions/<recent_session>/ldap.bak | wc -l d) du -c -h /opt/zimbra/openldap-data/*.bdb e) cat /etc/fstab
Temporary Workaround:
Restart Zimbra:
a) zmcontrol stop b) zmcontrol start