Log Files

Log Files

   KB 1302        Last updated on 2023-07-17  




0.00
(0 votes)

Depending on what's installed on the server, and how your system is configured, some logs may not be on the server that is experiencing the error.

Installation Logs

These are named with a pid file extension (3-5 digit number) - generally, the most recent one is the one you're looking for
/tmp/install.log - installation of zimbra software packages
/tmp/zmsetup.log - site-specific configuration

Runtime Logs

Most logs are found in /opt/zimbra/log/:

Note: Before ZCS 4.5, mailbox.log was called zimbra.log

mailbox.log - formerly tomcat, now jetty mail services
audit.log - authentication
clamd.log - antivirus db
convertd.log - attachment conversion 
freshclam.log - clam antivirus updates
logger_myslow.log - slow logger db queries
myslow.log - slow db queries
spamtrain.log - spam/ham training
sync.log - zimbra mobile
zimbrastats.csv - server performance statistics
zmconvertd.log - conversion server monitor
Some additional logs are in /opt/zimbra/tomcat/logs/:
access_log.<date> - traditional http access log for each http "hit"
catalina.out - tomcat application log
stacktrace.<pid> - full thread dump from "kill -3 <tomcat pid>"; generated each time tomcat is stopped
Other log locations:
/var/log/zimbra.log - MTA and system status log; postfix, amavisd
Output of various scripts:
/tmp/gengraphs.out
/tmp/logprocess.out
/tmp/logswatch.out
/tmp/swatch.out
/tmp/zmlogger.out

mailbox.log categories

The mailboxd logging is described extensively on a dedicated page.

Notes

For some log facilities, you may need to modify the existing lines in the log4j.properties.in file. For example, the default zimbra.sync entry (for Zimbra Mobile Sync) is the following:

log4j.logger.zimbra.sync=INFO,SYNC

For this entry, you would want to modify INFO to TRACE:

log4j.logger.zimbra.sync=TRACE,SYNC

You can also change log settings for a single user with zmprov:

$ zmprov addAccountLogger user@domain.com zimbra.soap debug

Run zmprov help log or zmprov help misc for more details. Note that account-level log settings are reset when the server [mailboxd service] is restarted. bug 26642 was filed for this issue.

Once the logging collection is complete, disable the account logger with removeAccountLogger:

$ zmprov removeAccountLogger user@domain.com zimbra.soap

Performance Statistics

Many useful Zimbra server performance statistics are logged to /opt/zimbra/zmstat/mailboxd.csv. This file can be opened in Excel or another utility that reads files in CSV format. Here's a summary of the statistics logged. All pairs of timing statistics named xxx_count and xxx_avg specify the number of times an operation was performed and the average execution time. Additional logs after 4.5.7 include cpu.csv and proc.csv.

  • timestamp - timestamp of a given statistic collection period
  • lmtp_rcvd_msgs,lmtp_rcvd_bytes,lmtp_rcvd_rcpt,lmtp_dlvd_msgs,lmtp_dlvd_bytes - number and size of messages received and delivered
  • db_pool_size - database connection pool size
  • db_conn_count,db_conn_ms_avg - timing data for getting a database connection
  • ldap_dc_count,ldap_dc_ms_avg - timing data for getting an LDAP connection
  • mbox_add_msg_count,mbox_add_msg_ms_avg - timing data for adding a message
  • mbox_msg_cache,mbox_item_cache - hit rate for the message body (blob) cache and mail item (database metadata) cache. 0 = 0%, 1 = 100%.
  • soap_count,soap_ms_avg,imap_count,imap_ms_avg,pop_count,pop_ms_avg - timing data for SOAP, IMAP and POP operations
  • idx_wrt_avg - Lucene index writer count
  • java_heap_MB - Amount of memory used by the Java VM
  • mysql_opened_tables,mysql_slow_queries,mysql_threads_connected,innodb_pages_read,innodb_pages_written,innodb_bp_hit_rate - MySQL statistics. See MySQL documentation for more info.
  • pop_conn,pop_ssl_conn,imap_conn,imap_ssl_conn - Number of active POP and IMAP connections

MTA Logs

The MTA (postfix) logs via syslog to the mail facility. By default, this goes to /var/log/maillog, but Zimbra sets it to /var/log/zimbra.log

Tomcat (web server) Logs

Tomcat logs via syslog to the local0 facility. By default, this goes to /var/log/messages AND /opt/zimbra/log/mailbox.log
Some tomcat information (including thread dumps) will also be in /opt/zimbra/tomcat/logs/catalina.out; stopping (or restarting) tomcat outputs the current thread dump to /opt/zimbra/tomcat/logs/stacktrace.<tomcat pid>

Spell Server Logs

Aspell logs can be found at httpd_access.log.<date> and httpd_error.log.<date> in /opt/zimbra/log

Swatch Logs

Swatch logs to /tmp/swatch.out

iSync Connector Logs

The iSync Connector logs are in different locations based on the version you are using.

In 4.5.x ~/Library/Application Support/Zimbra/com.zimbra.XXXXXXX.sync. There will be at most 5 log files there, sync.log, sync.log.2, ....

In 5.0.x /Users/<user>/Library/Logs/Zimbra/com.zimbra.XXXXXXXXXX/sync.X.log. You can specify how many log files to keep in the Zimbra preference page.

Logging the Originating IP

In a multi-server environment, or any environment running a proxy, the mailboxd server may only log the IP of the connecting proxy. In order to fix this, Zimbra added a feature in Bug 31633 in 6.0 and later to track the originating IP of the user's client machine:

  • X-Originating-IP support for nginx or other fronting proxy [1] - Fixed: 6.0

The actual log entry used in messaging headers is defined in this variable, and by default is "X-Forwarded-For" (see https://en.wikipedia.org/wiki/X-Forwarded-For for further background info):

$ zmlocalconfig zimbra_http_originating_ip_header
zimbra_http_originating_ip_header = X-Forwarded-For

In order to use this feature, you must define the Trusted IPs of your internal nginx proxies, so that the mailstores will instead capture the data on the originating IP from the HTTP traffic. By default, the zimbraMailTrustedIP is empty:

$ zmprov gcf zimbraMailTrustedIP

By adding your nginx proxy addresses (as seen by the mailstores) to this attribute, it should then log appropriately. For example, if there are two nginx proxies:

$ zmprov mcf +zimbraMailTrustedIP {IP of nginx-1} +zimbraMailTrustedIP {IP of nginx-2}

Example:

$ zmprov mcf +zimbraMailTrustedIP 10.11.12.1 +zimbraMailTrustedIP 10.11.12.2

Please note here too, if you are running nginx on the same node as the mailstore, you will need to add both 127.0.0.1 and the real address of that node:

$ zmprov mcf +zimbraMailTrustedIP 127.0.0.1 +zimbraMailTrustedIP 10.11.12.13

Check your result:

$ zmprov gcf zimbraMailTrustedIP
zimbraMailTrustedIP: 10.11.12.1
zimbraMailTrustedIP: 10.11.12.2
zimbraMailTrustedIP: 10.11.12.13
zimbraMailTrustedIP: 127.0.0.1

Then restart mailboxd:

$ zmmailboxdctl restart

In mailbox.log, you will then see the "oip" (Originating IP) for the real client IP:

2013-08-30 11:19:41,043 INFO  [qtp2050551931-94:http://127.0.0.1:8080/service/soap/AuthRequest] [name=user1@example.com;oip=5.6.7.8;ua=zclient/8.0.4_GA_5737;] mbxmgr - Mailbox 3 account abcdef8f-1234-5678-9012-8abcdefe2658 LOADED

For IPv6 configuration of originating IP see: https://wiki.zimbra.com/wiki/Configuring_for_IP_V6

Using DEBUG log level with Zimbra Extensions

To enable the storing of debug level messages for extensions append to /opt/zimbra/conf/log4j.properties.in and /opt/zimbra/conf/log4j.properties the following:

logger.extensions.name = zimbra.extensions
logger.extensions.level = debug
logger.extensions.additivity = false
logger.extensions.appenderRef.LOGFILE.ref = mailboxFile

You will have to restart Zimbra using:

sudo su zimbra -
zmmailboxdctl restart

You can now use the following:

ZimbraLog.extensions.debug("this is a debug message that will show up in /opt/zimbra/log/mailbox.log if debug logging level is set for extensions");

This will show up in the /opt/zimbra/log/mailbox.log like this:

2022-10-26 11:12:25,174 DEBUG [qtp48914743-18:https://zimbra10.barrydegraaff.nl/service/extension/mytest] [] extensions - this is a debug message that will show up in /opt/zimbra/log/mailbox.log if debug logging level is set for extensions

audit.log and nginx access and error log to syslog

If you want the messages from /opt/zimbra/log/audit.log and /opt/zimbra/log/nginx.access.log to appear in syslog for centralized logging with rsyslog. You can use the following syntax in [b]/etc/rsyslog.conf[/b] on Ubuntu 20

$ModLoad imfile

# error log
$InputFileName /opt/zimbra/log/nginx.error.log
$InputFileTag nginx:
$InputFileStateFile stat-nginx-error
$InputFileSeverity info
$InputFileFaility local6
$InputFilePollInterval 1
$InputRunFileMonitor

# access log
$InputFileName /opt/zimbra/log/nginx.access.log
$InputFileTag nginx:
$InputFileStateFile stat-nginx-access
$InputFileSeverity info
$InputFileFaility local6
$InputFilePollInterval 1
$InputRunFileMonitor

$InputFileName /opt/zimbra/log/audit.log
$InputFileTag zimbra-audit:
$InputFileStateFile zimbra-audit
$InputFileSeverity info
$InputFileFaility local0
$InputFilePollInterval 1
$InputRunFileMonitor

For full example please check: https://github.com/Zimbra/elastic-stack

Verified Against: Zimbra Collaboration 8.0, 7.0 Date Created: 04/16/2014
Article ID: https://wiki.zimbra.com/index.php?title=Log_Files Date Modified: 2023-07-17



Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »

Jump to: navigation, search