Difference between revisions of "Troubleshooting High CPU load average in ZCS 5.0.x as compared to 4.5.x"

(New page: In order to troubleshoot the high CPU usage observed in ZCS ver 5.0.x as compared to 4.5.x follow these steps: (Please refer to Bug# 23539 for more information: http://bugzilla.zimbra.com/...)
 
 
(5 intermediate revisions by 2 users not shown)
Line 1: Line 1:
In order to troubleshoot the high CPU usage observed in ZCS ver 5.0.x as compared to 4.5.x follow these steps:
+
{{Archive}}{{Article Infobox|{{admin}}||{{ZCS 5.0}}|}}In order to troubleshoot the high CPU usage observed in ZCS ver 5.0.x as compared to 4.5.x follow these steps:
 
(Please refer to Bug# 23539 for more information: http://bugzilla.zimbra.com/show_bug.cgi?id=23539)
 
(Please refer to Bug# 23539 for more information: http://bugzilla.zimbra.com/show_bug.cgi?id=23539)
  
Line 16: Line 16:
 
  $ cat /proc/<jetty_pid>/task/*/stat > jetty-task-stat.$(date +%Y%m%d-%H%M%S)
 
  $ cat /proc/<jetty_pid>/task/*/stat > jetty-task-stat.$(date +%Y%m%d-%H%M%S)
 
  $ pstack <jetty_pid>                                      # where <jetty_pid> is the pid of the jetty process from "ps -ef | grep jetty".
 
  $ pstack <jetty_pid>                                      # where <jetty_pid> is the pid of the jetty process from "ps -ef | grep jetty".
 +
 +
3) Generate charts for the day that the problem occurred:
 +
$ zmstat-chart -s /opt/zimbra/zmstat/<YYYY-MM-DD> -d <output-dir>
  
 
Then, scrutinize the following:
 
Then, scrutinize the following:
   a) /opt/zimbra/log/zmmailboxd.out,
+
   a) generated charts
   b) the data in /opt/zimbra/zmstat covering the period of the high CPU load,
+
  b) /opt/zimbra/log/zmmailboxd.out,
   c) the jetty-task-stat files from above and
+
   c) the data in /opt/zimbra/zmstat covering the period of the high CPU load,
   d) the output from pstack.
+
   d) the jetty-task-stat files from above and
 +
   e) the output from pstack.
  
3) Other important files to observe(or obtain) at this point:
+
4) Other important files to observe(or obtain) at this point:
 
   a) /opt/zimbra/conf/zmstat-chart.xml (if the file doesn't exist, you can obtain the output of zmstat-chart-config),
 
   a) /opt/zimbra/conf/zmstat-chart.xml (if the file doesn't exist, you can obtain the output of zmstat-chart-config),
 
   b) /opt/zimbra/log/mailbox.log,
 
   b) /opt/zimbra/log/mailbox.log,
Line 30: Line 34:
 
   e) /opt/zimbra/openldap-data/DB_CONFIG,
 
   e) /opt/zimbra/openldap-data/DB_CONFIG,
 
   f) /opt/zimbra/zmstat/<mention the date here - when you observe high CPU load>/*.
 
   f) /opt/zimbra/zmstat/<mention the date here - when you observe high CPU load>/*.
  (You can zip or rar them altogether, if you are transferring them somewhere)
+
  (You can zip or tar them altogether, if you are transferring them somewhere)
  
4) For further information obtain the output of the following commands,
+
5) For further information obtain the output of the following commands,
 
   a) /opt/zimbra/sleepycat/bin/db_stat -m -h /opt/zimbra/openldap-data | head -n 11
 
   a) /opt/zimbra/sleepycat/bin/db_stat -m -h /opt/zimbra/openldap-data | head -n 11
 
   b) /opt/zimbra/sleepycat/bin/db_stat -c -h /opt/zimbra/openldap-data | head -n 12
 
   b) /opt/zimbra/sleepycat/bin/db_stat -c -h /opt/zimbra/openldap-data | head -n 12
Line 44: Line 48:
 
   a) zmcontrol stop
 
   a) zmcontrol stop
 
   b) zmcontrol start
 
   b) zmcontrol start
 +
 +
 +
{{Article Footer|ZCS 5.0.x|2/26/2008}}
 +
 +
[[Category:Performance and Tuning]]
 +
[[Category:ZCS 5.0]]

Latest revision as of 17:06, 24 March 2015

Admin Article

Article Information

This article applies to the following ZCS versions.

ZCS 5.0 Article ZCS 5.0

In order to troubleshoot the high CPU usage observed in ZCS ver 5.0.x as compared to 4.5.x follow these steps: (Please refer to Bug# 23539 for more information: http://bugzilla.zimbra.com/show_bug.cgi?id=23539)

Collect the following information:

Make this change ASAP before you observe High CPU load average:

1) Please add the following java options (-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps) using this command:

$ zmlocalconfig -e mailboxd_java_options="`zmlocalconfig -m nokey mailboxd_java_options` -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"

Note: You'll need to do a zmcontrol stop/start after making the above config change.


Get the output of these commands during high CPU usage:

2) Obtain some thread dumps in order to troubleshoot the problem (and before you restart):

$ /opt/zimbra/libexec/zmmailboxdmgr threaddump             # run this command 4 times (30 seconds apart)
$ cat /proc/<jetty_pid>/task/*/stat > jetty-task-stat.$(date +%Y%m%d-%H%M%S)
$ pstack <jetty_pid>                                       # where <jetty_pid> is the pid of the jetty process from "ps -ef | grep jetty".

3) Generate charts for the day that the problem occurred:

$ zmstat-chart -s /opt/zimbra/zmstat/<YYYY-MM-DD> -d <output-dir>

Then, scrutinize the following:

  a) generated charts
  b) /opt/zimbra/log/zmmailboxd.out,
  c) the data in /opt/zimbra/zmstat covering the period of the high CPU load,
  d) the jetty-task-stat files from above and
  e) the output from pstack.

4) Other important files to observe(or obtain) at this point:

  a) /opt/zimbra/conf/zmstat-chart.xml (if the file doesn't exist, you can obtain the output of zmstat-chart-config),
  b) /opt/zimbra/log/mailbox.log,
  c) /opt/zimbra/conf/my.cnf,
  d) /opt/zimbra/conf/slapd.conf,
  e) /opt/zimbra/openldap-data/DB_CONFIG,
  f) /opt/zimbra/zmstat/<mention the date here - when you observe high CPU load>/*.
(You can zip or tar them altogether, if you are transferring them somewhere)

5) For further information obtain the output of the following commands,

  a) /opt/zimbra/sleepycat/bin/db_stat -m -h /opt/zimbra/openldap-data | head -n 11
  b) /opt/zimbra/sleepycat/bin/db_stat -c -h /opt/zimbra/openldap-data | head -n 12
  c) grep dn: /opt/zimbra/backup/sessions/<recent_session>/ldap.bak | wc -l
  d) du -c -h /opt/zimbra/openldap-data/*.bdb
  e) cat /etc/fstab

Temporary Workaround:

Restart Zimbra:

  a) zmcontrol stop
  b) zmcontrol start


Verified Against: ZCS 5.0.x Date Created: 2/26/2008
Article ID: https://wiki.zimbra.com/index.php?title=Troubleshooting_High_CPU_load_average_in_ZCS_5.0.x_as_compared_to_4.5.x Date Modified: 2015-03-24



Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »

Jump to: navigation, search