Troubleshooting High CPU load average in ZCS 5.0.x as compared to 4.5.x

Revision as of 17:15, 31 March 2008 by Bburtin (talk | contribs) (Get the output of these commands during high CPU usage:)

In order to troubleshoot the high CPU usage observed in ZCS ver 5.0.x as compared to 4.5.x follow these steps: (Please refer to Bug# 23539 for more information: http://bugzilla.zimbra.com/show_bug.cgi?id=23539)

Collect the following information:

Make this change ASAP before you observe High CPU load average:

1) Please add the following java options (-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps) using this command:

$ zmlocalconfig -e mailboxd_java_options="`zmlocalconfig -m nokey mailboxd_java_options` -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"

Note: You'll need to do a zmcontrol stop/start after making the above config change.


Get the output of these commands during high CPU usage:

2) Obtain some thread dumps in order to troubleshoot the problem (and before you restart):

$ /opt/zimbra/libexec/zmmailboxdmgr threaddump             # run this command 4 times (30 seconds apart)
$ cat /proc/<jetty_pid>/task/*/stat > jetty-task-stat.$(date +%Y%m%d-%H%M%S)
$ pstack <jetty_pid>                                       # where <jetty_pid> is the pid of the jetty process from "ps -ef | grep jetty".

3) Generate charts for the day that the problem occurred:

$ zmstat-chart -s /opt/zimbra/zmstat/<YYYY-MM-DD> -d <output-dir>

Then, scrutinize the following:

  a) generated charts
  b) /opt/zimbra/log/zmmailboxd.out,
  c) the data in /opt/zimbra/zmstat covering the period of the high CPU load,
  d) the jetty-task-stat files from above and
  e) the output from pstack.

4) Other important files to observe(or obtain) at this point:

  a) /opt/zimbra/conf/zmstat-chart.xml (if the file doesn't exist, you can obtain the output of zmstat-chart-config),
  b) /opt/zimbra/log/mailbox.log,
  c) /opt/zimbra/conf/my.cnf,
  d) /opt/zimbra/conf/slapd.conf,
  e) /opt/zimbra/openldap-data/DB_CONFIG,
  f) /opt/zimbra/zmstat/<mention the date here - when you observe high CPU load>/*.
(You can zip or tar them altogether, if you are transferring them somewhere)

5) For further information obtain the output of the following commands,

  a) /opt/zimbra/sleepycat/bin/db_stat -m -h /opt/zimbra/openldap-data | head -n 11
  b) /opt/zimbra/sleepycat/bin/db_stat -c -h /opt/zimbra/openldap-data | head -n 12
  c) grep dn: /opt/zimbra/backup/sessions/<recent_session>/ldap.bak | wc -l
  d) du -c -h /opt/zimbra/openldap-data/*.bdb
  e) cat /etc/fstab

Temporary Workaround:

Restart Zimbra:

  a) zmcontrol stop
  b) zmcontrol start


Verified Against: ZCS 5.0.x Date Created: 2/26/2008
Article ID: https://wiki.zimbra.com/index.php?title=Troubleshooting_High_CPU_load_average_in_ZCS_5.0.x_as_compared_to_4.5.x Date Modified: 2008-03-31



Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »

Jump to: navigation, search