Troubleshooting High CPU load average in ZCS 5.0.x as compared to 4.5.x: Difference between revisions
(Adding Article Footer & Category) |
|||
Line 16: | Line 16: | ||
$ cat /proc/<jetty_pid>/task/*/stat > jetty-task-stat.$(date +%Y%m%d-%H%M%S) | $ cat /proc/<jetty_pid>/task/*/stat > jetty-task-stat.$(date +%Y%m%d-%H%M%S) | ||
$ pstack <jetty_pid> # where <jetty_pid> is the pid of the jetty process from "ps -ef | grep jetty". | $ pstack <jetty_pid> # where <jetty_pid> is the pid of the jetty process from "ps -ef | grep jetty". | ||
3) Generate charts for the day that the problem occurred: | |||
$ zmstat-chart -s /opt/zimbra/zmstat/<YYYY-MM-DD> -d <output-dir> | |||
Then, scrutinize the following: | Then, scrutinize the following: | ||
a) /opt/zimbra/log/zmmailboxd.out, | a) generated charts | ||
b) /opt/zimbra/log/zmmailboxd.out, | |||
c) the data in /opt/zimbra/zmstat covering the period of the high CPU load, | |||
d) the jetty-task-stat files from above and | |||
e) the output from pstack. | |||
4) Other important files to observe(or obtain) at this point: | |||
a) /opt/zimbra/conf/zmstat-chart.xml (if the file doesn't exist, you can obtain the output of zmstat-chart-config), | a) /opt/zimbra/conf/zmstat-chart.xml (if the file doesn't exist, you can obtain the output of zmstat-chart-config), | ||
b) /opt/zimbra/log/mailbox.log, | b) /opt/zimbra/log/mailbox.log, | ||
Line 30: | Line 34: | ||
e) /opt/zimbra/openldap-data/DB_CONFIG, | e) /opt/zimbra/openldap-data/DB_CONFIG, | ||
f) /opt/zimbra/zmstat/<mention the date here - when you observe high CPU load>/*. | f) /opt/zimbra/zmstat/<mention the date here - when you observe high CPU load>/*. | ||
(You can zip or | (You can zip or tar them altogether, if you are transferring them somewhere) | ||
5) For further information obtain the output of the following commands, | |||
a) /opt/zimbra/sleepycat/bin/db_stat -m -h /opt/zimbra/openldap-data | head -n 11 | a) /opt/zimbra/sleepycat/bin/db_stat -m -h /opt/zimbra/openldap-data | head -n 11 | ||
b) /opt/zimbra/sleepycat/bin/db_stat -c -h /opt/zimbra/openldap-data | head -n 12 | b) /opt/zimbra/sleepycat/bin/db_stat -c -h /opt/zimbra/openldap-data | head -n 12 |
Revision as of 17:15, 31 March 2008
In order to troubleshoot the high CPU usage observed in ZCS ver 5.0.x as compared to 4.5.x follow these steps: (Please refer to Bug# 23539 for more information: http://bugzilla.zimbra.com/show_bug.cgi?id=23539)
Collect the following information:
Make this change ASAP before you observe High CPU load average:
1) Please add the following java options (-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps) using this command:
$ zmlocalconfig -e mailboxd_java_options="`zmlocalconfig -m nokey mailboxd_java_options` -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"
Note: You'll need to do a zmcontrol stop/start after making the above config change.
Get the output of these commands during high CPU usage:
2) Obtain some thread dumps in order to troubleshoot the problem (and before you restart):
$ /opt/zimbra/libexec/zmmailboxdmgr threaddump # run this command 4 times (30 seconds apart) $ cat /proc/<jetty_pid>/task/*/stat > jetty-task-stat.$(date +%Y%m%d-%H%M%S) $ pstack <jetty_pid> # where <jetty_pid> is the pid of the jetty process from "ps -ef | grep jetty".
3) Generate charts for the day that the problem occurred:
$ zmstat-chart -s /opt/zimbra/zmstat/<YYYY-MM-DD> -d <output-dir>
Then, scrutinize the following:
a) generated charts b) /opt/zimbra/log/zmmailboxd.out, c) the data in /opt/zimbra/zmstat covering the period of the high CPU load, d) the jetty-task-stat files from above and e) the output from pstack.
4) Other important files to observe(or obtain) at this point:
a) /opt/zimbra/conf/zmstat-chart.xml (if the file doesn't exist, you can obtain the output of zmstat-chart-config), b) /opt/zimbra/log/mailbox.log, c) /opt/zimbra/conf/my.cnf, d) /opt/zimbra/conf/slapd.conf, e) /opt/zimbra/openldap-data/DB_CONFIG, f) /opt/zimbra/zmstat/<mention the date here - when you observe high CPU load>/*. (You can zip or tar them altogether, if you are transferring them somewhere)
5) For further information obtain the output of the following commands,
a) /opt/zimbra/sleepycat/bin/db_stat -m -h /opt/zimbra/openldap-data | head -n 11 b) /opt/zimbra/sleepycat/bin/db_stat -c -h /opt/zimbra/openldap-data | head -n 12 c) grep dn: /opt/zimbra/backup/sessions/<recent_session>/ldap.bak | wc -l d) du -c -h /opt/zimbra/openldap-data/*.bdb e) cat /etc/fstab
Temporary Workaround:
Restart Zimbra:
a) zmcontrol stop b) zmcontrol start