Troubleshooting High CPU load average in ZCS 5.0.x as compared to 4.5.x

Admin Article

Article Information

This article applies to the following ZCS versions.

ZCS 5.0 Article ZCS 5.0

In order to troubleshoot the high CPU usage observed in ZCS ver 5.0.x as compared to 4.5.x follow these steps: (Please refer to Bug# 23539 for more information:

Collect the following information:

Make this change ASAP before you observe High CPU load average:

1) Please add the following java options (-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps) using this command:

$ zmlocalconfig -e mailboxd_java_options="`zmlocalconfig -m nokey mailboxd_java_options` -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"

Note: You'll need to do a zmcontrol stop/start after making the above config change.

Get the output of these commands during high CPU usage:

2) Obtain some thread dumps in order to troubleshoot the problem (and before you restart):

$ /opt/zimbra/libexec/zmmailboxdmgr threaddump             # run this command 4 times (30 seconds apart)
$ cat /proc/<jetty_pid>/task/*/stat > jetty-task-stat.$(date +%Y%m%d-%H%M%S)
$ pstack <jetty_pid>                                       # where <jetty_pid> is the pid of the jetty process from "ps -ef | grep jetty".

3) Generate charts for the day that the problem occurred:

$ zmstat-chart -s /opt/zimbra/zmstat/<YYYY-MM-DD> -d <output-dir>

Then, scrutinize the following:

  a) generated charts
  b) /opt/zimbra/log/zmmailboxd.out,
  c) the data in /opt/zimbra/zmstat covering the period of the high CPU load,
  d) the jetty-task-stat files from above and
  e) the output from pstack.

4) Other important files to observe(or obtain) at this point:

  a) /opt/zimbra/conf/zmstat-chart.xml (if the file doesn't exist, you can obtain the output of zmstat-chart-config),
  b) /opt/zimbra/log/mailbox.log,
  c) /opt/zimbra/conf/my.cnf,
  d) /opt/zimbra/conf/slapd.conf,
  e) /opt/zimbra/openldap-data/DB_CONFIG,
  f) /opt/zimbra/zmstat/<mention the date here - when you observe high CPU load>/*.
(You can zip or tar them altogether, if you are transferring them somewhere)

5) For further information obtain the output of the following commands,

  a) /opt/zimbra/sleepycat/bin/db_stat -m -h /opt/zimbra/openldap-data | head -n 11
  b) /opt/zimbra/sleepycat/bin/db_stat -c -h /opt/zimbra/openldap-data | head -n 12
  c) grep dn: /opt/zimbra/backup/sessions/<recent_session>/ldap.bak | wc -l
  d) du -c -h /opt/zimbra/openldap-data/*.bdb
  e) cat /etc/fstab

Temporary Workaround:

Restart Zimbra:

  a) zmcontrol stop
  b) zmcontrol start

Verified Against: ZCS 5.0.x Date Created: 2/26/2008
Article ID: Date Modified: 2015-03-24

Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »

Jump to: navigation, search