Difference between revisions of "Troubleshooting Tomcat Issues With Java 1.6"

(Data to Collect)
Line 1: Line 1:
{|  width="100%" border="0"
 
|  bgcolor=#FF8C00 | [[Image:rev2.png]] - This article is '''currently being revised'''. This means that the information may not be up to date or accurate.
 
|}
 
 
 
 
== Introduction ==
 
== Introduction ==
 
To troubleshoot mailbox server tomcat OutOfMemory (OOME), or hang/unresponsive issues, Zimbra Support usually requests for stacktrace (catalina.out), mailbox.log and zimbrastats.csv files from customers.  When these files do *not* review any useful information, Zimbra will need java heap dumps to pinpoint the issue.  Until Zimbra ships Java 1.6 starting in ZCS 4.5.7, you'll be requested to follow the instructions in this article to install Java 1.6 manually, and collect all the necessary information.
 
To troubleshoot mailbox server tomcat OutOfMemory (OOME), or hang/unresponsive issues, Zimbra Support usually requests for stacktrace (catalina.out), mailbox.log and zimbrastats.csv files from customers.  When these files do *not* review any useful information, Zimbra will need java heap dumps to pinpoint the issue.  Until Zimbra ships Java 1.6 starting in ZCS 4.5.7, you'll be requested to follow the instructions in this article to install Java 1.6 manually, and collect all the necessary information.
Line 11: Line 6:
 
2. Java 1.6 is not going to fix the issue.  It'll only help us gather the heap dumps so that we can find out the root cause of the problem.<br>
 
2. Java 1.6 is not going to fix the issue.  It'll only help us gather the heap dumps so that we can find out the root cause of the problem.<br>
 
3. Before upgrading to the next version, you should revert back to Java 1.5.<br>
 
3. Before upgrading to the next version, you should revert back to Java 1.5.<br>
 
  
 
== Procedures to Install and Configure Java 1.6==
 
== Procedures to Install and Configure Java 1.6==
Line 35: Line 29:
  
 
7. To verify that everything starts OK, as the [[zimbra user]], run:
 
7. To verify that everything starts OK, as the [[zimbra user]], run:
  <tt>zmcontrol stats</tt>
+
  <tt>zmcontrol status</tt>
 
 
 
As root, run:
 
As root, run:
 
  <tt>/opt/zimbra/java/bin/jps</tt>
 
  <tt>/opt/zimbra/java/bin/jps</tt>
Line 46: Line 39:
 
The above means that two JVM processes are alive.
 
The above means that two JVM processes are alive.
  
 +
== Procedures to Install the zmstat Scripts ==
 +
1. Download the tarball [http://wiki.zimbra.com/images/5/54/Zmstat.tar.zip Zmstat.tar.zip] and transfer it to your Zimbra server.<br>
 +
2. As the [[zimbra user]], run the following commands:<br>
 +
<tt>mkdir -p /opt/zimbra/log/zmstat/bin
 +
cd /opt/zimbra/log/zmstat/bin
 +
tar xvfz Zmstat.tar.zip</tt>
 +
3. As the zimbra user, run zmstat:
 +
<tt>/opt/zimbra/log/zmstat/bin/zmstat start</tt>
 +
This will run several processes that capture stats every 10 seconds.  They are saved in various /opt/zimbra/log/zmstat/*.csv files, and they rotate at midnight to 2007-mm-dd subdirectory.
  
 
== Data to Collect ==
 
== Data to Collect ==
 
 
1. Set up a cron job to run the following two commands, as the [[zimbra user]], every 30 (or 60) minutes, and redirect the output to a file.  This will give us some trend data.
 
1. Set up a cron job to run the following two commands, as the [[zimbra user]], every 30 (or 60) minutes, and redirect the output to a file.  This will give us some trend data.
 
  <tt>jmap -histo:live <tomcat_pid>
 
  <tt>jmap -histo:live <tomcat_pid>
Line 58: Line 59:
 
  <tt>/opt/zimbra/java/bin/jmap -heap <tomcat_pid>
 
  <tt>/opt/zimbra/java/bin/jmap -heap <tomcat_pid>
 
  /opt/zimbra/java/bin/jmap -permstat <tomcat_pid>
 
  /opt/zimbra/java/bin/jmap -permstat <tomcat_pid>
  /opt/zimbra/java/bin/jmap -finalizerinfo <tomcat_pid>  
+
  /opt/zimbra/java/bin/jmap -finalizerinfo <tomcat_pid></tt>  
  
 
Also, run these commands as the [[zimbra user]]:
 
Also, run these commands as the [[zimbra user]]:
  jmap -histo:live <tomcat_pid>  
+
  <tt>jmap -histo:live <tomcat_pid>  
 
  jmap -histo <tomcat_pid>  
 
  jmap -histo <tomcat_pid>  
 
  jmap -dump:live,format=b,file=/opt/zimbra/log/heap_dump.dat
 
  jmap -dump:live,format=b,file=/opt/zimbra/log/heap_dump.dat
Line 70: Line 71:
 
3.  Also, send the followings:
 
3.  Also, send the followings:
  
  <tt>/opt/zimbra/tomcat/logs/cataina.out (need previous day’s files as well)
+
  <tt>/opt/zimbra/log/zmstat/*.csv (including previous day's subdir)
  /opt/zimbra/log/mailbox.log (need previous day’s as well)
+
/opt/zimbra/tomcat/logs/cataina.out (need previous day’s file as well)
  /opt/Zimbra/log/zimbrastats.csv
+
  /opt/zimbra/log/mailbox.log (need previous day’s file as well)
 +
  /opt/zimbra/log/zimbrastats.csv
 
  output of "df –h"</tt>
 
  output of "df –h"</tt>
----
+
 
<br> Keywords: ''revisiond''
+
{{Article Footer|Zimbra Collaboration Suite 4.5.4 on RHEL4 32-bit|7/12/2007}}
{| class="toc" border="0" cellpadding="0" cellspacing="0" width=100%
+
[[Category:Troubleshooting]]
|  align="left" | <b>Verified Against:</b> Zimbra Collaboration Suite 4.5.4 on RHEL4 32-bit
 
|}
 
{|  class="toc" border="0" cellpadding="0" cellspacing="0" width=100%
 
|  align="left" | <b>Article ID:</b> http://wiki.zimbra.com/index.php?title=Troubleshooting_Tomcat_Issues_With_Java_1.6
 
|  align="right" | <strong>Date Created:</strong> July 12, 2007  
 
|  align="right" | <strong>Date Modified:</strong> July 12, 2007
 
|}
 

Revision as of 17:47, 1 August 2007

Introduction

To troubleshoot mailbox server tomcat OutOfMemory (OOME), or hang/unresponsive issues, Zimbra Support usually requests for stacktrace (catalina.out), mailbox.log and zimbrastats.csv files from customers. When these files do *not* review any useful information, Zimbra will need java heap dumps to pinpoint the issue. Until Zimbra ships Java 1.6 starting in ZCS 4.5.7, you'll be requested to follow the instructions in this article to install Java 1.6 manually, and collect all the necessary information.

Note:
1. Zimbra currently ships with Java 1.5 on non-Mac platforms.
2. Java 1.6 is not going to fix the issue. It'll only help us gather the heap dumps so that we can find out the root cause of the problem.
3. Before upgrading to the next version, you should revert back to Java 1.5.

Procedures to Install and Configure Java 1.6

1. Download JDK 6u1 from http://java.sun.com/javase/downloads/index.jsp.
(Note: The procedures here were verified against 6u1, but should be applicable to 6u2 as well, except for the path name).

2. Follow the instructions included with the download to install Java 1.6.

3. As the zimbra user, run:

zmcontrol stop

4. As root, run:

mv /opt/zimbra/jdk1.5.0_08 /opt/zimbra/jdk1.5.0_08_orig
cp -a /usr/java/jdk1.6.0_01 /opt/zimbra/jdk1.5.0_08
cd /opt/zimbra/libexec
mv zmtomcatmgr zmtomcatmgr.orig
ln -s zmtomcatmgr.unrestricted zmtomcatmgr

5. As the zimbra user, run:

zmlocalconfig -e tomcat_java_options="-client -XX:NewRatio=2 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/zimbra/log -XX:ErrorFile=/opt/zimbra/log/java_error%p.log"
zmcontrol start

7. To verify that everything starts OK, as the zimbra user, run:

zmcontrol status  

As root, run:

/opt/zimbra/java/bin/jps

which should show an output similar to the following:

12221 TomcatLauncher 
12313 TransformationSomething

The above means that two JVM processes are alive.

Procedures to Install the zmstat Scripts

1. Download the tarball Zmstat.tar.zip and transfer it to your Zimbra server.
2. As the zimbra user, run the following commands:

mkdir -p /opt/zimbra/log/zmstat/bin
cd /opt/zimbra/log/zmstat/bin
tar xvfz Zmstat.tar.zip

3. As the zimbra user, run zmstat:

/opt/zimbra/log/zmstat/bin/zmstat start

This will run several processes that capture stats every 10 seconds. They are saved in various /opt/zimbra/log/zmstat/*.csv files, and they rotate at midnight to 2007-mm-dd subdirectory.

Data to Collect

1. Set up a cron job to run the following two commands, as the zimbra user, every 30 (or 60) minutes, and redirect the output to a file. This will give us some trend data.

jmap -histo:live <tomcat_pid>
jmap -histo <tomcat_pid>

where <tomcat_pid> is the PID of TomcatLauncher from the jps command.

2. When the problem occurs again, run these commands as root:

/opt/zimbra/java/bin/jmap -heap <tomcat_pid>
/opt/zimbra/java/bin/jmap -permstat <tomcat_pid>
/opt/zimbra/java/bin/jmap -finalizerinfo <tomcat_pid> 

Also, run these commands as the zimbra user:

jmap -histo:live <tomcat_pid> 
jmap -histo <tomcat_pid> 
jmap -dump:live,format=b,file=/opt/zimbra/log/heap_dump.dat
kill -3 <tomcat_pid>

Note: All the jmap commands (except for the dump) write to stdout. The jmap -permstat command may take several minutes to run.

3. Also, send the followings:

/opt/zimbra/log/zmstat/*.csv (including previous day's subdir)
/opt/zimbra/tomcat/logs/cataina.out (need previous day’s file as well)
/opt/zimbra/log/mailbox.log (need previous day’s file as well)
/opt/zimbra/log/zimbrastats.csv
output of "df –h"
Verified Against: Zimbra Collaboration Suite 4.5.4 on RHEL4 32-bit Date Created: 7/12/2007
Article ID: https://wiki.zimbra.com/index.php?title=Troubleshooting_Tomcat_Issues_With_Java_1.6 Date Modified: 2007-08-01



Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »

Jump to: navigation, search