Unresponsive Server Troubleshooting

Revision as of 18:33, 8 October 2008 by Bobby (talk | contribs) (clarifications)

When you contact the Zimbra Support team regarding an unresponsive server, you may be requested to run several troubleshooting utilities and provide the output of those utilities to the support team. Additionally, you may be requested to submit stack traces and log files for the unresponsive server.

This article provides instructions for running troubleshooting utilities, and also provides the locations of the stack trace and logging files.

Overview of Log Files, Stack Trace Files, and Utilities

The following files and utilities contain or collect information which helps analyze issues with unresponsive servers. The log files, stack trace files, and utilities should be collected and run before restarting your unresponsive server.

Log Files and Stack Trace Files

These are useful logs for beginning to troubleshoot an unresponsive mail server. Logs are generally rotated daily, so logs from previous days will have a timestamp (such as ".2008-05-08") appended to the filename. Any stacktrace files will have the process id of the JVM appended to the filename.

In all releases 5.0.x and 4.5.x, this is the main mailbox application log:

  • /opt/zimbra/log/mailbox.log

In ZCS 5.0.x, the JVM output log and stacktrace files will be in /opt/zimbra/log/:

  • /opt/zimbra/log/zmmailboxd.out (jvm output log)
  • /opt/zimbra/log/stacktrace.<pid>

In ZCS 4.5.x, the JVM output log and stacktrace files will be in /opt/zimbra/tomcat/logs/:

  • /opt/zimbra/tomcat/logs/catalina.out (tomcat jvm output log)
  • /opt/zimbra/tomcat/logs/stacktrace.<pid>

Utilities

Some C application debugging tools are useful for examining the state of current running processes. If you are experiencing issues with an unresponsive server, Zimbra Support may request that you run some or all of these utilities. For each utility that you run, you will need to save the output in order to send it to Zimbra Support.

Note: Running these utilities may interfere with normal functioning of the ZCS applications involved.

  • gdb
  • gcore
  • pstack

There are also utilities to examine the state of the operating system and specific applications.

To list open files on the system:

  • lsof

To display statistics about currently running bdb databases (used by openldap, amavisd-new):

  • db_stat

Steps for running utilities

The following steps will provide an order for running the utilities. Use the instructions for the server type with which you are having issues before restarting your server. For each utility that you run, you will need to save the output in order to send it to Zimbra Support.

OpenLDAP server

1. Log in as root.

2. Determine the Zimbra openldap slapd process id [let’s assume 27567 for this example].

# ps u -C slapd

3. Run lsof to generate a list of files opened by the unresponsive server.

# lsof –p 27567

4. Use the db_stat utility to display statistics.

$ /opt/zimbra/sleepycat/bin/db_stat -C A -h /opt/zimbra/openldap-data

5. For ZCS 4.5.7 and later, if there are ldap replicas

$ /opt/zimbra/sleepycat/bin/db_stat -C A -h /opt/zimbra/openldap-data/accesslog/db

6. Use pstack to gather database statistics for the unresponsive server.

# pstack 27567

7. Run gcore to create a core image.

# gcore 27567

Mail server

ZCS 5.x: Jetty server

1. Log in as root.

2. Determine the jetty mailboxd process id [let’s assume 427 for this example].

# ps u -C java | grep jetty

3. Send some mailbox server thread dumps to /opt/zimbra/log/zmmailboxd.out; run this command two or three times 20 seconds apart.

$ kill -3 427

4. Run lsof to generate a list of files opened by the unresponsive server.

# lsof –p 427

5. Use pstack togenerate a thread dump for the unresponsive application.

# pstack 427

6. Run gcore to create a core image.

# gcore 427

ZCS 4.x: Tomcat server

1. Log in as root.

2. Determine the tomcat mailbox jvm process id [let’s assume 23726 for this example].

# ps u -C java | grep Tomcat

3. Send some mailbox server thread dumps to /opt/zimbra/tomcat/logs/catalina.out; run this command two or three times 20 seconds apart.

$ kill -3 23726

4. Run lsof to generate a list of files opened by the unresponsive server.

# lsof –p 23726

5. Use pstack to generate a thread dump for the unresponsive application.

# pstack 23726

6. Run gcore to create a core image.

# gcore 23726


Verified Against: ZCS 4.5.x and 5.0.x Date Created: 5/6/2008
Article ID: https://wiki.zimbra.com/index.php?title=Unresponsive_Server_Troubleshooting Date Modified: 2008-10-08



Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »

Jump to: navigation, search