Unresponsive Server Troubleshooting: Difference between revisions
No edit summary |
|||
Line 1: | Line 1: | ||
When you contact the Zimbra Support team regarding an unresponsive server, you may be requested to run several troubleshooting utilities and provide the output of those utilities to the support team. Additionally, you may be requested to submit stack traces and log files for the unresponsive server. | When you contact the Zimbra Support team regarding an unresponsive server, you may be requested to run several troubleshooting utilities and provide the output of those utilities to the support team. Additionally, you may be requested to submit stack traces and log files for the unresponsive server. | ||
This article provides instructions for running troubleshooting utilities, and also provides the locations of the stack trace and logging files | This article provides instructions for running troubleshooting utilities, and also provides the locations of the stack trace and logging files. | ||
=Overview of Log Files, Stack Trace Files, and Utilities= | =Overview of Log Files, Stack Trace Files, and Utilities= | ||
Line 23: | Line 21: | ||
==Utilities== | ==Utilities== | ||
Some C application debugging tools are useful for examining the state of current running processes. If you are experiencing issues with an unresponsive server, Zimbra Support may request that you run some or all of these utilities. | Some C application debugging tools are useful for examining the state of current running processes. If you are experiencing issues with an unresponsive server, Zimbra Support may request that you run some or all of these utilities. For each utility that you run, you will need to save the output in order to send it to Zimbra Support. | ||
'''''Note:''' Running these utilities may interfere with normal functioning of the ZCS applications involved.'' | '''''Note:''' Running these utilities may interfere with normal functioning of the ZCS applications involved.'' | ||
Line 37: | Line 35: | ||
To display statistics about currently running bdb databases (used by openldap, amavisd-new): | To display statistics about currently running bdb databases (used by openldap, amavisd-new): | ||
*db_stat | *db_stat | ||
=Steps for running utilities= | |||
The following steps will provide an order for running the utilities. Use the instructions for the server type with which you are having issues before restarting your server. For each utility that you run, you will need to save the output in order to send it to Zimbra Support. | |||
==OpenLDAP server== | |||
1. Log in as root. | |||
2. Determine the Zimbra openldap slapd process id [it’s 27567 in the attached example doc]. | |||
<nowiki>#</nowiki> ps u -C slapd | |||
3. Run lsof to generate a list of files opened by the unresponsive server. | |||
<nowiki>#</nowiki> lsof –p 27567 | |||
4. Use the db_stat utility to display statistics. | |||
$ /opt/zimbra/sleepycat/bin/db_stat -C A -h /opt/zimbra/openldap-data | |||
5. For ZCS 4.5.7 and later, if there are ldap replicas | |||
$ /opt/zimbra/sleepycat/bin/db_stat -C A -h /opt/zimbra/openldap-data/accesslog/db | |||
6. Use pstack to gather database statistics for the unresponsive server. | |||
<nowiki>#</nowiki> pstack 27567 | |||
7. Run gcore to create a core image. | |||
<nowiki>#</nowiki> gcore 27567 | |||
==Mail server== | |||
===ZCS 5.x: Jetty server=== | |||
1. Log in as root. | |||
2. Determine the jetty mailboxd process id [it’s 427 in the attached example doc]. | |||
3. Send some mailbox server thread dumps to /opt/zimbra/log/zmmailboxd.out; run this command two or three times 20 seconds apart. | |||
$ kill -3 427 | |||
4. Run lsof to generate a list of files opened by the unresponsive server. | |||
<nowiki>#</nowiki> lsof –p 427 | |||
5. Use pstack togenerate a thread dump for the unresponsive application. | |||
<nowiki>#</nowiki> pstack 427 | |||
6. Run gcore to create a core image. | |||
<nowiki>#</nowiki> gcore 427 | |||
===ZCS 4.x: Tomcat server=== | |||
1. Log in as root. | |||
2. Determine the tomcat mailbox jvm process id[it’s 23726 in the attached example doc]. | |||
3. Send some mailbox server thread dumps to /opt/zimbra/tomcat/logs/catalina.out; run this command two or three times 20 seconds apart. | |||
$ kill -3 23726 | |||
4. Run lsof to generate a list of files opened by the unresponsive server. | |||
<nowiki>#</nowiki> lsof –p 23726 | |||
5. Use pstack to generate a thread dump for the unresponsive application. | |||
<nowiki>#</nowiki> pstack 23726 | |||
6. Run gcore to create a core image. | |||
<nowiki>#</nowiki> gcore 23726 | |||
{{Article Footer|ZCS 4.5.x and 5.0.x|5/6/2008}} | |||
[[Category:LDAP]] | |||
[[Category:MTA]] | |||
[[Category:Troubleshooting]] |
Revision as of 07:53, 12 May 2008
When you contact the Zimbra Support team regarding an unresponsive server, you may be requested to run several troubleshooting utilities and provide the output of those utilities to the support team. Additionally, you may be requested to submit stack traces and log files for the unresponsive server.
This article provides instructions for running troubleshooting utilities, and also provides the locations of the stack trace and logging files.
Overview of Log Files, Stack Trace Files, and Utilities
The following files and utilities contain or collect information which helps analyze issues with unresponsive servers. The log files, stack trace files, and utilities should be collected and run before restarting your unresponsive server.
Log Files and Stack Trace Files
These are useful logs for beginning to troubleshoot an unresponsive mail server. Logs are generally rotated daily, so logs from previous days will have a time stamp appended to the filename, such as 5-6-2008. Any stacktrace files will have the process id of the JVM appended to the filename.
In all releases 5.0.x and 4.5.x, this is the main mailbox application log:
- /opt/zimbra/log/mailbox.log
In ZCS 5.0.x, the JVM output log and stacktrace files will be in /opt/zimbra/log/:
- /opt/zimbra/log/zmmailboxd.out (jvm output log)
- /opt/zimbra/log/stacktrace.<pid>
In ZCS 4.5.x, the JVM output log and stacktrace files will be in /opt/zimbra/tomcat/logs/:
- /opt/zimbra/tomcat/logs/catalina.out (tomcat jvm output log)
- /opt/zimbra/tomcat/logs/stacktrace.<pid>
Utilities
Some C application debugging tools are useful for examining the state of current running processes. If you are experiencing issues with an unresponsive server, Zimbra Support may request that you run some or all of these utilities. For each utility that you run, you will need to save the output in order to send it to Zimbra Support.
Note: Running these utilities may interfere with normal functioning of the ZCS applications involved.
- gdb
- gcore
- pstack
There are also utilities to examine the state of the operating system and specific applications.
To list open files on the system:
- lsof
To display statistics about currently running bdb databases (used by openldap, amavisd-new):
- db_stat
Steps for running utilities
The following steps will provide an order for running the utilities. Use the instructions for the server type with which you are having issues before restarting your server. For each utility that you run, you will need to save the output in order to send it to Zimbra Support.
OpenLDAP server
1. Log in as root.
2. Determine the Zimbra openldap slapd process id [it’s 27567 in the attached example doc].
# ps u -C slapd
3. Run lsof to generate a list of files opened by the unresponsive server.
# lsof –p 27567
4. Use the db_stat utility to display statistics.
$ /opt/zimbra/sleepycat/bin/db_stat -C A -h /opt/zimbra/openldap-data
5. For ZCS 4.5.7 and later, if there are ldap replicas
$ /opt/zimbra/sleepycat/bin/db_stat -C A -h /opt/zimbra/openldap-data/accesslog/db
6. Use pstack to gather database statistics for the unresponsive server.
# pstack 27567
7. Run gcore to create a core image.
# gcore 27567
Mail server
ZCS 5.x: Jetty server
1. Log in as root.
2. Determine the jetty mailboxd process id [it’s 427 in the attached example doc].
3. Send some mailbox server thread dumps to /opt/zimbra/log/zmmailboxd.out; run this command two or three times 20 seconds apart.
$ kill -3 427
4. Run lsof to generate a list of files opened by the unresponsive server.
# lsof –p 427
5. Use pstack togenerate a thread dump for the unresponsive application.
# pstack 427
6. Run gcore to create a core image.
# gcore 427
ZCS 4.x: Tomcat server
1. Log in as root.
2. Determine the tomcat mailbox jvm process id[it’s 23726 in the attached example doc].
3. Send some mailbox server thread dumps to /opt/zimbra/tomcat/logs/catalina.out; run this command two or three times 20 seconds apart.
$ kill -3 23726
4. Run lsof to generate a list of files opened by the unresponsive server.
# lsof –p 23726
5. Use pstack to generate a thread dump for the unresponsive application.
# pstack 23726
6. Run gcore to create a core image.
# gcore 23726