Troubleshooting Course Content Rough Drafts-Zimbra Logs: Difference between revisions
Line 193: | Line 193: | ||
* See also http://www.zimbra.com/forums/zimbra-connector-outlook/1083-tips-how-report-errors-zcs-outlook-connector.html | * See also http://www.zimbra.com/forums/zimbra-connector-outlook/1083-tips-how-report-errors-zcs-outlook-connector.html | ||
=== Quiz Questions === | |||
* What |
Revision as of 22:21, 4 March 2015
Introduction
Log Files
Brief list of the major log files in Zimbra and their significance.
- /opt/zimbra/log/mailbox.log - Zimbra Mailboxd (jetty) writes all mailbox activity to this log file. This activity can be anything from moving a message to deletion of an account.
- /opt/zimbra/log/zmmailboxd.out – Mailboxd(jetty) logs to this file during startup.
- /opt/zimbra/log/audit.log – Logs authentication activity.
- /opt/zimbra/log/sync.log – Mobile sync operations get logged to this log file.
- /opt/zimbra/log/clamd.log – Clamd (Antivirus daemon) logs all antivirus activity to this file.
- /opt/zimbra/log/freshclam.log – Antivirus definition update.
- /opt/zimbra/log/mysql_error.log – Mysql errors and startup messages. This log should be looked up to determine if innodb corruption has occurred.
- /opt/zimbra/log/myslow.log – Consists sql statements which took more than long_query_time seconds (zimbra default – 1second) to execute and required atleast min_examined_row_limit (zimbra default - 0) rows to be examined.
- /opt/zimbra/log/nginx.access.log - NGINX writes information about client requests after the request is processed.
- /opt/zimbra/log/nginx.log – Nginx error log which defaults to “info” in Zimbra.
- /opt/zimbra/log/spamtrain.log – Spam training log.
- /opt/zimbra/log/ews.log – EWS server side activity.
- /var/log/zimbra.log – All MTA (Postfix, amavisd, antispam, antivirus,cyrus-sasl)/Ldap logs go to this file.
- /var/log/zimbra-stats.log – Logger data gets logged to this file.
Log Rotation
Two different log rotation daemons handle log rotation of the zimbra log files.
- Log4j - Log4j is responsible for logging to mailbox.log, audit.log, sync.log, synctrace.log,webxml.log and ews.log. For config : /opt/zimbra/conf/log4j.properties
Example, below the are the config options for the rotation of /opt/zimbra/log/mailbox.log
log4j.appender.LOGFILE=org.apache.log4j.rolling.RollingFileAppender log4j.appender.LOGFILE.RollingPolicy=org.apache.log4j.rolling.TimeBasedRollingPolicy log4j.appender.LOGFILE.RollingPolicy.FileNamePattern=/opt/zimbra/log/mailbox.log.%d{yyyy-MM-dd} log4j.appender.LOGFILE.File=/opt/zimbra/log/mailbox.log log4j.appender.LOGFILE.layout=com.zimbra.common.util.ZimbraPatternLayout log4j.appender.LOGFILE.layout.ConversionPattern=%d %-5p [%t] [%z] %c{1} - %m%n
- Logrotate.d - The OS daemon logrotate used to rotate other log files.
Config file : /etc/logrotate.d/zimbra
Files rotated by logrotate : zimbra.log, zimbra-stats.log, myslow.log, zmconfigd.log, zmconfigd.log, clamd.log, freshclam.log, nginx.log, nginx_access.log, cbpolicyd.log and spamtrain.log.
Debug Logging
Mailbox
The ZCS server uses log4j, a Java logging package as the log manager
- By default, the ZC server has log4j configured to log to the local file system
- Global debug logging : Doing this will enable debug logging for any log4j property.
Edit - /opt/zimbra/conf/log4j.properties.in log4j.logger.zimbra.imap=DEBUG Restart mailboxd – zmmailboxdctl restart
- Per user debug logging :
addAccountLogger : zmprov aal user@domain.com zimbra.imap debug removeAccountLogger : zmprov ral user@domain.com zimbra.imap
To list all possible loggers -
zmprov help log
- Debug logs for mailbox get written to /opt/zimbra/log/mailbox.log, ews.log for ews, sync.log for sync and webxml.log for webxml.
Logging Levels
Logging levels determine what types of events are logged. The following are logging levels in log4j:
- FATAL
- ERROR
- WARN
- INFO
- DEBUG
The FATAL logging level will only log very severe error events. The DEBUG logging level will log detailed informational events useful for debugging purposes. Each additional logging level will log more events, including the events belonging to the previous logging levels. For example, enabling the WARN logging level will record log events belonging to the FATAL, ERROR, and WARN logging levels, but will not record events belonging to the INFO or DEBUG levels. By default, most predefined log4j categories in ZCS are set to the INFO logging level.
Protocol Trace
Protocol trace is available in the following logging categories with TRACE logging level:
- zimbra.smtp
- zimbra.lmtp
- zimbra.soap
- zimbra.imap
- zimbra.imap-client
- zimbra.pop
- zimbra.pop-client
For more information on logging levels, see the ZCS Admin Guide > Monitoring Zimbra Servers > Log Files > Logging Levels.
Ldap
- Ldap debug logging – Controlled by “ldap_common_loglevel”, this writes the config ldap attribute "olcLogLevel".
Examples -
zmlocalconfig –e ldap_common_loglevel=256 or zmlocalconfig –e ldap_common_loglevel=“stats”
Restart - ldap stop; ldap start.
For various log levels please see -
http://www.openldap.org/software/man.cgi?query=slapd.conf&apropos=0&sektion=0&manpath=OpenLDAP+2.4-Release&format=html
- Debug logs written to /var/log/zimbra.log
Postfix
- Edit /opt/zimbra/postfix/conf/master.cf.in
Change - smtp inet n - n - - smtpd To - smtp inet n - n - - smtpd –vv
Restart of postfix required
zmmtactl restart
- This will write debug postfix logs to /var/log/zimbra.log
SpamAssassin
- Debug logging for SpamAssassin controlled by “zimbraAmavisSALogLevel”, this writes "sa_debug" to /opt/zimbra/conf/amavisd.conf.
- Value can be 0 or 1(info or all)
zmprov mcf zimbraAmavisSALogLevel 1 zmprov ms `zmhostname` zimbraAmavisSALogLevel 1
- Debug logs get written to /var/log/zimbra.log
Amavis
- Amavisd debug logging - Controlled by zimbra attribute "zimbraAmavisLogLevel", which writes value of "$log_level" in file /opt/zimbra/conf/amavisd.conf
- Valid range is 0-5, defaults to 1.
zmprov mcf zimbraAmavisLogLevel 5 zmprov ms `zmhostname` "zimbraAmavisLogLevel" 5
- This will write debug log to - /var/log/zimbra.log
Proxy
- Controlled by zimbra attribute zimbraReverseProxyLogLevel.
zmprov ms `zmhostname` zimbraReverseProxyLogLevel debug zmnginxctl restart
It will log a lot, so you can set it back to info at any time:
zmprov ms `zmhostname` zimbraReverseProxyLogLevel info
- This will write debug logs to /opt/zimbra/log/nginx.log
ZCO Logs
- For all but the simplest queries, ZCO Dev needs ZCO logs, dumps and event viewer logs. Here's how to gather them:
Disable Windows Search start->run->services->Windows Search->Properties->Stop + “Disabled” Start Outlook Disable Foreign Addins i.e. any addins that dont start with "Zimbra" OL2003: Tools->Options->Other->Advanced Options->COM Add-Ins->Remove any addins that don't start with "Zimbra" OL2007: Tools->Trust Center->Add-ins->Manage COM Addins->Go->Untick any addins that don't start with "Zimbra" OL2010: File->Options->Add-ins->Manage COM Addins->Go->Untick any addins that don't start with "Zimbra" OL2013: as OL2010 Close Outlook
- Empty ZCOLogs Directory (to avoid sending ZCO Dev stale/irrelevant logs)
- Set Log Settings (using ZCOLogCtl.exe) to the following -
Click the Default button Max size of single log file : 50Mb Preserve Log File Backups ticked Max Size of ZCOLogs directory UNticked
ZCO Troubleshooting
- Start Outlook + Repro the Issue
- Take detailed notes of the steps
- If possible, make a note of the time the problem ocurred
- If the problem relates to a particular item, include its ZimbraID and/or Subject
- Include Screen shots; Screen video if possible
- Close Outlook. If Outlook hung, grab a Hang Dump:
- Task Manager->Processes Tab->Right click OUTLOOK.EXE->Create Dump File
- Export Outlook Events from the Windows Event Viewer:
- Start Event Viewer
- Select "Windows Logs" | Application from the left pane.
- In Actions (right pane) select "Filter Current Log..."
- In the Filter dialog select
- select "Last 7 days" from the "Logged:" dropdown
- check Outlook in the "Event sources:"
- they will probably also need to limit output to the current "User:"
- Click OK
- Save with "Save Filtered Log File...", and pass us the resulting .evtx file
- Check logs before sending to ZCO Dev
- Must cover the repro period (note that each log file name includes the date and time at which a client session started)
- Check for obvious issues (e.g. server unreachable)
## ERROR ## Usually indicates an unexpected problem ## WARNING ## Usually benign but worth knowing
Quiz Questions
- What