|This article applies to the following ZCS versions.|
# /opt/zimbra/libexec/zmdiaglog -h Usage: zmdiaglog [-dthazj] -a - Do everything: also gather JVM heapdump/coredump -d - Log destination (Default /opt/zimbra/data/tmp/zmdiaglog.PID) -t - Timeout in seconds for hanging commands (Default 120) -j - Also include the output of /opt/zimbra/libexec/zmjavawatch -z - Zip output -h - This help message
zmdiaglog is the standard tool that Zimbra provides for collecting performance or troubleshooting data. Zimbra has standardized this tool in order to capture the full set of data during an event - data that Zimbra Support and Engineering regularly requires and determines essential in investigating all sides of the event. It is imperative that zmdiaglog is run during the trouble event, and prior to the restart of any processes.
Included in the zmdiaglog is the following:
- zmthrdump threaddumps at 10 second intervals for 100 seconds
- dmesg & /var/log/messages
- all logs
- all config files
- heap dump and heap histogram (when -a flag is used)
zmdiaglog is executed as root as:
- Standard execution, without heap dump:
- Standard execution, without heap dump, output zipped:
# /opt/zimbra/libexec/zmdiaglog -z
- Execution with heap dump and output zipped:
# /opt/zimbra/libexec/zmdiaglog -a -z
- Execution with heap dump, zmjavawatch output, and output zipped:
# /opt/zimbra/libexec/zmdiaglog -a -j -z
- Execution with zmjavawatch output, and output zipped. Heap dumps can take longer (see below), so this is probably the most common usage:
# /opt/zimbra/libexec/zmdiaglog -j -z
As a general rule, plan to run this last command /opt/zimbra/libexec/zmdiaglog -a -j -z anytime the server is behaving poorly - this will be sure to take a snapshot of the server profile at the time of the event.
A "zmdiaglog -a" also provides a heap dump, but should only be run in this manner on ZCS 6.0 or later. Running "zmdiaglog -a" on a 5.0.x server is not valuable, as JVM 1.5 does not provide consistent or worthwhile heap dumps, and also this process is invasive on JVM 1.5 and will cause the mailboxd server to pause for the duration of the heap dump. Once on ZCS 6.0.x or later, which uses Java JVM6, the heap dump can be taken non-invasively and can be valuable under conditions of memory events or OOMEs.
The entire standard zmdiaglog should take about 100-200 seconds, and builds a .gz file at /opt/zimbra/data/tmp (/tmp on ZCS 5.0.x) that can be uploaded to Zimbra Support, and without the -a flag is non-invasive (other than the additional gzip load on the server). The "zmdiaglog -a" will take longer to write a 4-6GB heap file to disk, but again will not affect the running process once on JVM6.
Given the length of a common event, it is very rare that one cannot be taken successfully, and the value of doing so is great when compared to allowing an event to happen and not capturing the data. The time invested in running the zmdiaglog once will hopefully allow identification of the root cause, and prevent future outages.
As soon as the zmdiaglog completes its JVM functions (thread dumps, procstats, and optionally heap dump), a "zmmailboxdctl restart" can be run to restart a non-functioning JVM process if necessary.