Troubleshooting High CPU load average in ZCS 5.0.x as compared to 4.5.x

Revision as of 10:43, 26 February 2008 by Skhanna (talk | contribs) (New page: In order to troubleshoot the high CPU usage observed in ZCS ver 5.0.x as compared to 4.5.x follow these steps: (Please refer to Bug# 23539 for more information: http://bugzilla.zimbra.com/...)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

In order to troubleshoot the high CPU usage observed in ZCS ver 5.0.x as compared to 4.5.x follow these steps: (Please refer to Bug# 23539 for more information: http://bugzilla.zimbra.com/show_bug.cgi?id=23539)

Collect the following information:

Make this change ASAP before you observe High CPU load average:

1) Please add the following java options (-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps) using this command:

$ zmlocalconfig -e mailboxd_java_options="`zmlocalconfig -m nokey mailboxd_java_options` -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"

Note: You'll need to do a zmcontrol stop/start after making the above config change.


Get the output of these commands during high CPU usage:

2) Obtain some thread dumps in order to troubleshoot the problem (and before you restart):

$ /opt/zimbra/libexec/zmmailboxdmgr threaddump             # run this command 4 times (30 seconds apart)
$ cat /proc/<jetty_pid>/task/*/stat > jetty-task-stat.$(date +%Y%m%d-%H%M%S)
$ pstack <jetty_pid>                                       # where <jetty_pid> is the pid of the jetty process from "ps -ef | grep jetty".

Then, scrutinize the following:

  a) /opt/zimbra/log/zmmailboxd.out,
  b) the data in /opt/zimbra/zmstat covering the period of the high CPU load,
  c) the jetty-task-stat files from above and
  d) the output from pstack.

3) Other important files to observe(or obtain) at this point:

  a) /opt/zimbra/conf/zmstat-chart.xml (if the file doesn't exist, you can obtain the output of zmstat-chart-config),
  b) /opt/zimbra/log/mailbox.log,
  c) /opt/zimbra/conf/my.cnf,
  d) /opt/zimbra/conf/slapd.conf,
  e) /opt/zimbra/openldap-data/DB_CONFIG,
  f) /opt/zimbra/zmstat/<mention the date here - when you observe high CPU load>/*.
(You can zip or rar them altogether, if you are transferring them somewhere)

4) For further information obtain the output of the following commands,

  a) /opt/zimbra/sleepycat/bin/db_stat -m -h /opt/zimbra/openldap-data | head -n 11
  b) /opt/zimbra/sleepycat/bin/db_stat -c -h /opt/zimbra/openldap-data | head -n 12
  c) grep dn: /opt/zimbra/backup/sessions/<recent_session>/ldap.bak | wc -l
  d) du -c -h /opt/zimbra/openldap-data/*.bdb
  e) cat /etc/fstab

Temporary Workaround:

Restart Zimbra:

  a) zmcontrol stop
  b) zmcontrol start
Jump to: navigation, search