Difference between revisions of "Troubleshooting High CPU load average in ZCS 5.0.x as compared to 4.5.x"

(Adding Article Footer & Category)
(Get the output of these commands during high CPU usage:)
Line 16: Line 16:
 
  $ cat /proc/<jetty_pid>/task/*/stat > jetty-task-stat.$(date +%Y%m%d-%H%M%S)
 
  $ cat /proc/<jetty_pid>/task/*/stat > jetty-task-stat.$(date +%Y%m%d-%H%M%S)
 
  $ pstack <jetty_pid>                                      # where <jetty_pid> is the pid of the jetty process from "ps -ef | grep jetty".
 
  $ pstack <jetty_pid>                                      # where <jetty_pid> is the pid of the jetty process from "ps -ef | grep jetty".
 +
 +
3) Generate charts for the day that the problem occurred:
 +
$ zmstat-chart -s /opt/zimbra/zmstat/<YYYY-MM-DD> -d <output-dir>
  
 
Then, scrutinize the following:
 
Then, scrutinize the following:
   a) /opt/zimbra/log/zmmailboxd.out,
+
   a) generated charts
   b) the data in /opt/zimbra/zmstat covering the period of the high CPU load,
+
  b) /opt/zimbra/log/zmmailboxd.out,
   c) the jetty-task-stat files from above and
+
   c) the data in /opt/zimbra/zmstat covering the period of the high CPU load,
   d) the output from pstack.
+
   d) the jetty-task-stat files from above and
 +
   e) the output from pstack.
  
3) Other important files to observe(or obtain) at this point:
+
4) Other important files to observe(or obtain) at this point:
 
   a) /opt/zimbra/conf/zmstat-chart.xml (if the file doesn't exist, you can obtain the output of zmstat-chart-config),
 
   a) /opt/zimbra/conf/zmstat-chart.xml (if the file doesn't exist, you can obtain the output of zmstat-chart-config),
 
   b) /opt/zimbra/log/mailbox.log,
 
   b) /opt/zimbra/log/mailbox.log,
Line 30: Line 34:
 
   e) /opt/zimbra/openldap-data/DB_CONFIG,
 
   e) /opt/zimbra/openldap-data/DB_CONFIG,
 
   f) /opt/zimbra/zmstat/<mention the date here - when you observe high CPU load>/*.
 
   f) /opt/zimbra/zmstat/<mention the date here - when you observe high CPU load>/*.
  (You can zip or rar them altogether, if you are transferring them somewhere)
+
  (You can zip or tar them altogether, if you are transferring them somewhere)
  
4) For further information obtain the output of the following commands,
+
5) For further information obtain the output of the following commands,
 
   a) /opt/zimbra/sleepycat/bin/db_stat -m -h /opt/zimbra/openldap-data | head -n 11
 
   a) /opt/zimbra/sleepycat/bin/db_stat -m -h /opt/zimbra/openldap-data | head -n 11
 
   b) /opt/zimbra/sleepycat/bin/db_stat -c -h /opt/zimbra/openldap-data | head -n 12
 
   b) /opt/zimbra/sleepycat/bin/db_stat -c -h /opt/zimbra/openldap-data | head -n 12

Revision as of 17:15, 31 March 2008

In order to troubleshoot the high CPU usage observed in ZCS ver 5.0.x as compared to 4.5.x follow these steps: (Please refer to Bug# 23539 for more information: http://bugzilla.zimbra.com/show_bug.cgi?id=23539)

Collect the following information:

Make this change ASAP before you observe High CPU load average:

1) Please add the following java options (-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps) using this command:

$ zmlocalconfig -e mailboxd_java_options="`zmlocalconfig -m nokey mailboxd_java_options` -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"

Note: You'll need to do a zmcontrol stop/start after making the above config change.


Get the output of these commands during high CPU usage:

2) Obtain some thread dumps in order to troubleshoot the problem (and before you restart):

$ /opt/zimbra/libexec/zmmailboxdmgr threaddump             # run this command 4 times (30 seconds apart)
$ cat /proc/<jetty_pid>/task/*/stat > jetty-task-stat.$(date +%Y%m%d-%H%M%S)
$ pstack <jetty_pid>                                       # where <jetty_pid> is the pid of the jetty process from "ps -ef | grep jetty".

3) Generate charts for the day that the problem occurred:

$ zmstat-chart -s /opt/zimbra/zmstat/<YYYY-MM-DD> -d <output-dir>

Then, scrutinize the following:

  a) generated charts
  b) /opt/zimbra/log/zmmailboxd.out,
  c) the data in /opt/zimbra/zmstat covering the period of the high CPU load,
  d) the jetty-task-stat files from above and
  e) the output from pstack.

4) Other important files to observe(or obtain) at this point:

  a) /opt/zimbra/conf/zmstat-chart.xml (if the file doesn't exist, you can obtain the output of zmstat-chart-config),
  b) /opt/zimbra/log/mailbox.log,
  c) /opt/zimbra/conf/my.cnf,
  d) /opt/zimbra/conf/slapd.conf,
  e) /opt/zimbra/openldap-data/DB_CONFIG,
  f) /opt/zimbra/zmstat/<mention the date here - when you observe high CPU load>/*.
(You can zip or tar them altogether, if you are transferring them somewhere)

5) For further information obtain the output of the following commands,

  a) /opt/zimbra/sleepycat/bin/db_stat -m -h /opt/zimbra/openldap-data | head -n 11
  b) /opt/zimbra/sleepycat/bin/db_stat -c -h /opt/zimbra/openldap-data | head -n 12
  c) grep dn: /opt/zimbra/backup/sessions/<recent_session>/ldap.bak | wc -l
  d) du -c -h /opt/zimbra/openldap-data/*.bdb
  e) cat /etc/fstab

Temporary Workaround:

Restart Zimbra:

  a) zmcontrol stop
  b) zmcontrol start


Verified Against: ZCS 5.0.x Date Created: 2/26/2008
Article ID: https://wiki.zimbra.com/index.php?title=Troubleshooting_High_CPU_load_average_in_ZCS_5.0.x_as_compared_to_4.5.x Date Modified: 2008-03-31



Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »

Jump to: navigation, search