Troubleshooting Tomcat Issues With Java 1.6: Difference between revisions
No edit summary |
|||
(6 intermediate revisions by 3 users not shown) | |||
Line 1: | Line 1: | ||
== Introduction == | {{Archive}}{{Article Infobox|{{admin}}|||{{ZCS 4.5}}}}== Introduction == | ||
To troubleshoot mailbox server tomcat OutOfMemory (OOME), or hang/unresponsive issues, Zimbra Support usually requests for stacktrace (catalina.out), mailbox.log and zimbrastats.csv files from customers. When these files do *not* review any useful information, Zimbra will need java heap dumps to pinpoint the issue. Until Zimbra ships Java 1.6 starting in ZCS 4.5.7, you'll be requested to follow the instructions in this article to install Java 1.6 manually, and collect all the necessary information. | To troubleshoot mailbox server tomcat OutOfMemory (OOME), or hang/unresponsive issues, Zimbra Support usually requests for stacktrace (catalina.out), mailbox.log and zimbrastats.csv files from customers. When these files do *not* review any useful information, Zimbra will need java heap dumps to pinpoint the issue. Until Zimbra ships Java 1.6 starting in ZCS 4.5.7, you'll be requested to follow the instructions in this article to install Java 1.6 manually, and collect all the necessary information. | ||
Line 40: | Line 40: | ||
The above means that two JVM processes are alive. | The above means that two JVM processes are alive. | ||
===Issues after Upgrade=== | |||
1. If you see something like: | |||
<pre>Error occurred: java.security.cert.CertificateException: Untrusted Server Certificate Chain</pre> | |||
please look at [[SSL Certificate Problems]]. | |||
2. If all the shared folders disappear and you get the following error while trying to view them: | |||
<pre>msg - system failure: IOException | |||
code - service.FAILURE | |||
method - ZmCsfeCommand.prototype.invoke | |||
detail - soap:Receiver</pre> | |||
please restart Zimbra and try again. | |||
== Procedures to Install the zmstat Scripts == | == Procedures to Install the zmstat Scripts == | ||
Line 66: | Line 79: | ||
<tt>jmap -histo:live <tomcat_pid> | <tt>jmap -histo:live <tomcat_pid> | ||
jmap -histo <tomcat_pid> | jmap -histo <tomcat_pid> | ||
jmap -dump:live,format=b,file=/opt/zimbra/log/heap_dump.dat | jmap -dump:live,format=b,file=/opt/zimbra/log/heap_dump.dat <tomcat_pid> | ||
kill -3 <tomcat_pid></tt> | kill -3 <tomcat_pid></tt> | ||
Line 74: | Line 87: | ||
<tt>/opt/zimbra/log/zmstat/*.csv (including previous day's subdir) | <tt>/opt/zimbra/log/zmstat/*.csv (including previous day's subdir) | ||
/opt/zimbra/tomcat/logs/ | /opt/zimbra/tomcat/logs/catalina.out (need previous day’s file as well) | ||
/opt/zimbra/log/mailbox.log (need previous day’s file as well) | /opt/zimbra/log/mailbox.log (need previous day’s file as well) | ||
/opt/zimbra/log/zimbrastats.csv | /opt/zimbra/log/zimbrastats.csv | ||
Line 80: | Line 93: | ||
{{Article Footer|Zimbra Collaboration Suite 4.5.4 on RHEL4 32-bit|7/12/2007}} | {{Article Footer|Zimbra Collaboration Suite 4.5.4 on RHEL4 32-bit|7/12/2007}} | ||
[[Category:Troubleshooting]] | [[Category:Troubleshooting MTA]] | ||
[[Category:Troubleshooting Server]] |
Latest revision as of 17:25, 24 March 2015
Article Information |
---|
This article applies to the following ZCS versions. |
Introduction
To troubleshoot mailbox server tomcat OutOfMemory (OOME), or hang/unresponsive issues, Zimbra Support usually requests for stacktrace (catalina.out), mailbox.log and zimbrastats.csv files from customers. When these files do *not* review any useful information, Zimbra will need java heap dumps to pinpoint the issue. Until Zimbra ships Java 1.6 starting in ZCS 4.5.7, you'll be requested to follow the instructions in this article to install Java 1.6 manually, and collect all the necessary information.
Note:
1. Zimbra currently ships with Java 1.5 on non-Mac platforms.
2. Java 1.6 is not going to fix the issue. It'll only help us gather the heap dumps so that we can find out the root cause of the problem.
3. Before upgrading to the next version, you should revert back to Java 1.5.
Procedures to Install and Configure Java 1.6
1. Download JDK 6u1 from http://java.sun.com/javase/downloads/index.jsp.
(Note: The procedures here were verified against 6u1, but should be applicable to 6u2 as well, except for the path name).
2. Follow the instructions included with the download to install Java 1.6.
3. As the zimbra user, run:
zmcontrol stop
4. As root, run:
mv /opt/zimbra/jdk1.5.0_08 /opt/zimbra/jdk1.5.0_08_orig cp -a /usr/java/jdk1.6.0_01 /opt/zimbra/jdk1.5.0_08 cd /opt/zimbra/libexec mv zmtomcatmgr zmtomcatmgr.orig ln -s zmtomcatmgr.unrestricted zmtomcatmgr
5. As the zimbra user, run:
zmlocalconfig -e tomcat_java_options="-client -XX:NewRatio=2 -Djava.awt.headless=true -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/zimbra/log -XX:ErrorFile=/opt/zimbra/log/java_error%p.log"
zmcontrol start
7. To verify that everything starts OK, as the zimbra user, run:
zmcontrol status
As root, run:
/opt/zimbra/java/bin/jps
which should show an output similar to the following:
12221 TomcatLauncher 12313 TransformationSomething
The above means that two JVM processes are alive.
Issues after Upgrade
1. If you see something like:
Error occurred: java.security.cert.CertificateException: Untrusted Server Certificate Chain
please look at SSL Certificate Problems.
2. If all the shared folders disappear and you get the following error while trying to view them:
msg - system failure: IOException code - service.FAILURE method - ZmCsfeCommand.prototype.invoke detail - soap:Receiver
please restart Zimbra and try again.
Procedures to Install the zmstat Scripts
1. Download the tarball Zmstat.tar.zip and transfer it to your Zimbra server.
2. As the zimbra user, run the following commands:
mkdir -p /opt/zimbra/log/zmstat/bin cd /opt/zimbra/log/zmstat/bin tar xvfz Zmstat.tar.zip
3. As the zimbra user, run zmstat:
/opt/zimbra/log/zmstat/bin/zmstat start
This will run several processes that capture stats every 10 seconds. They are saved in various /opt/zimbra/log/zmstat/*.csv files, and they rotate at midnight to 2007-mm-dd subdirectory.
Data to Collect
1. Set up a cron job to run the following two commands, as the zimbra user, every 30 (or 60) minutes, and redirect the output to a file. This will give us some trend data.
jmap -histo:live <tomcat_pid> jmap -histo <tomcat_pid>
where <tomcat_pid> is the PID of TomcatLauncher from the jps command.
2. When the problem occurs again, run these commands as root:
/opt/zimbra/java/bin/jmap -heap <tomcat_pid> /opt/zimbra/java/bin/jmap -permstat <tomcat_pid> /opt/zimbra/java/bin/jmap -finalizerinfo <tomcat_pid>
Also, run these commands as the zimbra user:
jmap -histo:live <tomcat_pid> jmap -histo <tomcat_pid> jmap -dump:live,format=b,file=/opt/zimbra/log/heap_dump.dat <tomcat_pid> kill -3 <tomcat_pid>
Note: All the jmap commands (except for the dump) write to stdout. The jmap -permstat command may take several minutes to run.
3. Also, send the followings:
/opt/zimbra/log/zmstat/*.csv (including previous day's subdir) /opt/zimbra/tomcat/logs/catalina.out (need previous day’s file as well) /opt/zimbra/log/mailbox.log (need previous day’s file as well) /opt/zimbra/log/zimbrastats.csv output of "df –h"