Difference between revisions of "Troubleshooting Course Content Rough Drafts-Zimbra Logs"

m (Log Rotation And Management)
m (Mailbox)
Line 92: Line 92:
 
   resetAllLoggers(rlog) [-s/--server hostname]
 
   resetAllLoggers(rlog) [-s/--server hostname]
  
* Debug logs for mailbox required when troubleshooting a specific problem with the mailbox component, for example  
+
* Debug logs for mailbox required when troubleshooting a specific problem with the mailbox component, for example :
 
+
** Trying to add an appointment but not happening
Trying to add an appointment but not happening
+
** Specific message not getting delivered
Specific message not getting delivered
+
** Specific user unable to log into his acocunt using web/imap/pop etc.
Specific user unable to log into his acocunt using web/imap/pop etc.
+
** Unable to download specific message using imap/pop or activesync
Unable to download specific message using imap/pop or activesync
 
 
   
 
   
 
* Each component of mailboxd has a control to increase logging for that specific component. These are known as loggers, To list all possible loggers -
 
* Each component of mailboxd has a control to increase logging for that specific component. These are known as loggers, To list all possible loggers -

Revision as of 21:03, 12 March 2015

Verified Against: Zimbra Collaboration Suite 8.6 Date Created: 01/22/2015
Article ID: https://wiki.zimbra.com/index.php?title=Troubleshooting_Course_Content_Rough_Drafts-Zimbra_Logs Date Modified: 2015-03-12



Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »

Introduction

Ajcody - Need a general introduction of the course for the trainer to start with.


Log Files

A brief list of the major log files in Zimbra and their significance.

  • Mailstore Service Related Logs
    • /opt/zimbra/log/mailbox.log - Zimbra Mailboxd (jetty) writes all mailbox activity to this log file. This activity can be anything from moving a message to deletion of an account.
    • /opt/zimbra/log/zmmailboxd.out – Mailboxd (jetty) logs to this file during startup. Should be the log file to refer if you have trouble getting your mailboxd to start.
    • /opt/zimbra/log/mysql_error.log – Mysql errors and startup messages. This log should be looked up to determine if innodb corruption has occurred.
    • /opt/zimbra/log/myslow.log – Consists of sql statements which took more than long_query_time seconds (zimbra default – 1second) to execute and required at least min_examined_row_limit (zimbra default - 0) rows to be examined.
    • /opt/zimbra/log/audit.log – Logs authentication activity.
    • /opt/zimbra/log/sync.log – Mobile sync operations get logged to this log file. Should be used for reference when troubleshooting activesync issues.
    • /opt/zimbra/log/ews.log – All EWS server side activity gets logged here. Should be used when trying to identify issues with EWS clients.
  • LDAP Service Related Logs
    • /var/log/zimbra.log – Ldap log events go to this file.
  • MTA Service Related Logs
    • /var/log/zimbra.log – MTA (Postfix, amavisd, antispam, antivirus, cyrus-sasl) log events go to this file.
    • /opt/zimbra/log/clamd.log – Clamd (Antivirus daemon) logs all antivirus activity to this file.
    • /opt/zimbra/log/freshclam.log – Antivirus definition update log. Review this file if your server is not updating its virus definitions.
    • /opt/zimbra/log/spamtrain.log – The spam training log. Useful when trying to debug the action of "marking" a mail as spam and ham.
  • Proxy Service Related Logs
    • /opt/zimbra/log/nginx.access.log - Nginx writes information about client requests after the request is processed here. Can be used to track incoming requests and their response codes being sent by the server.
    • /opt/zimbra/log/nginx.log – Nginx error log which defaults to “info” level for logging. Log file to begin with for troubleshooting proxy related issues, gives a good overview of what the proxy is doing.
  • Stats Services Related Logs
    • /var/log/zimbra-stats.log – Logger data gets logged to this file via the OS logger, syslogd.
  • For All ZCS Servers
    • /var/log/zimbra.log – Tracking of the various ZC service status [starting/stopping] via the OS logger, syslogd, are logged in this file as well.
    • /var/log/syslog – Check here for possible server issues that might be causing ZC issues also.
    • /var/log/messages – Check here for possible server issues that might be causing ZC issues also.

Log Rotation And Management

Two different log rotation daemons handle log rotation of the zimbra log files - Log4j and Logrotate.d .

  • Log4j - Log4j is responsible for logging to :

mailbox.log, audit.log, sync.log, synctrace.log,webxml.log and ews.log.

  • Configuration file for Log4j is : /opt/zimbra/conf/log4j.properties

Example, below the are the config options for the rotation of /opt/zimbra/log/mailbox.log

log4j.appender.LOGFILE=org.apache.log4j.rolling.RollingFileAppender
log4j.appender.LOGFILE.RollingPolicy=org.apache.log4j.rolling.TimeBasedRollingPolicy
log4j.appender.LOGFILE.RollingPolicy.FileNamePattern=/opt/zimbra/log/mailbox.log.%d{yyyy-MM-dd}
log4j.appender.LOGFILE.File=/opt/zimbra/log/mailbox.log
log4j.appender.LOGFILE.layout=com.zimbra.common.util.ZimbraPatternLayout
log4j.appender.LOGFILE.layout.ConversionPattern=%d %-5p [%t] [%z] %c{1} - %m%n
  • Logrotate.d - The OS daemon logrotate used to rotate other log files. Files rotated by logrotate are :

zimbra.log, zimbra-stats.log, myslow.log, zmconfigd.log, zmconfigd.log, clamd.log, freshclam.log, nginx.log, nginx_access.log, cbpolicyd.log and spamtrain.log.

  • Configuration file for Logrotate is : /etc/logrotate.d/zimbra

There is also log management commands in the zimbra crontab. You can review the crontab lines by doing :

su - zimbra
crontab -l

Debug Logging

Mailbox

The ZCS server uses log4j, a Java logging package as the log manager

  • By default, the ZC server has log4j configured to log to the local file system
  • Global debug logging : Doing this will enable debug logging for any log4j property.
Edit - /opt/zimbra/conf/log4j.properties.in
log4j.logger.zimbra.imap=DEBUG
Restart mailboxd – zmmailboxdctl restart
  • Per user debug logging (does not requires mailboxd restart) :

[Examples]

addAccountLogger : zmprov aal user@domain.com zimbra.imap debug
removeAccountLogger : zmprov ral user@domain.com zimbra.imap

[zmprov options for logger]

 addAccountLogger(aal) [-s/--server hostname] {name@domain|id} {logging-category} {trace|debug|info|warn|error}
 getAccountLoggers(gal) [-s/--server hostname] {name@domain|id}
 getAllAccountLoggers(gaal) [-s/--server hostname]
 removeAccountLogger(ral) [-s/--server hostname] [{name@domain|id}] [{logging-category}]
 resetAllLoggers(rlog) [-s/--server hostname]
  • Debug logs for mailbox required when troubleshooting a specific problem with the mailbox component, for example :
    • Trying to add an appointment but not happening
    • Specific message not getting delivered
    • Specific user unable to log into his acocunt using web/imap/pop etc.
    • Unable to download specific message using imap/pop or activesync
  • Each component of mailboxd has a control to increase logging for that specific component. These are known as loggers, To list all possible loggers -
zmprov help log
  • Debug logs for mailbox get written to /opt/zimbra/log/mailbox.log, /opt/zimbra/log/ews.log for ews, /opt/zimbra/log/sync.log for sync and /opt/zimbra/log/webxml.log for webxml.


Logging Levels Logging levels determine what types of events are logged. The following are logging levels in log4j:

  • FATAL
  • TRACE
  • ERROR
  • WARN
  • INFO
  • DEBUG

The FATAL logging level will only log very severe error events. The DEBUG logging level will log detailed informational events useful for debugging purposes. Each additional logging level will log more events, including the events belonging to the previous logging levels. For example, enabling the WARN logging level will record log events belonging to the FATAL, ERROR, and WARN logging levels, but will not record events belonging to the INFO or DEBUG levels. By default, most predefined log4j categories in ZCS are set to the INFO logging level.

Protocol Trace Protocol trace is available in the following logging categories with TRACE logging level:

  • zimbra.smtp
  • zimbra.lmtp
  • zimbra.soap
  • zimbra.imap
  • zimbra.imap-client
  • zimbra.pop
  • zimbra.pop-client


  • For all logging levels and other reference material please see
https://www.zimbra.com/docs/ne/latest/administration_guide/wwhelp/wwhimpl/js/html/wwhelp.htm#href=860_admin_ne.Logging_Levels.html

Ldap

  • Ldap debug logging – Controlled by “ldap_common_loglevel”, this writes the config ldap attribute "olcLogLevel".
  • Typically required when trying to troubleshoot a specific request which is taking time, or when there some weird scenario which is causing a specific replica sync to fail.

Examples -

 zmlocalconfig –e ldap_common_loglevel=256
         or
 zmlocalconfig –e ldap_common_loglevel=“stats”
 Restart -  ldap stop; ldap start.  
  • For various log levels please see -
http://www.openldap.org/software/man.cgi?query=slapd.conf&apropos=0&sektion=0&manpath=OpenLDAP+2.4-Release&format=html
  • Debug logs written to /var/log/zimbra.log

Postfix

  • Typically required when trying to troubleshoot email flow issues. For example specific mail getting dropped or getting stuck in the queue. Or if troubleshooting issues with the postfix process itself, for example postfix crashing. The debug log will help understand what exactly postfix was doing when the crash occurred.
  • Edit /opt/zimbra/postfix/conf/master.cf.in
Change - smtp      inet  n       -       n       -       -       smtpd
To - smtp      inet  n       -       n       -       -       smtpd –vv

Restart of postfix required

zmmtactl restart
  • This will write debug postfix logs to /var/log/zimbra.log

SpamAssassin

  • Required when troubleshooting issues specific to spamassassin, for example a message being incorrectly dropped or spamassassin taking too long to scan emails.
  • Debug logging for SpamAssassin controlled by “zimbraAmavisSALogLevel”, this writes "sa_debug" to /opt/zimbra/conf/amavisd.conf.
  • Value can be 0 or 1, where 1 means debug log enabled and 0 means minimalistic info logging.
zmprov mcf zimbraAmavisSALogLevel 1 
zmprov ms `zmhostname` zimbraAmavisSALogLevel 1 
  • Debug logs get written to /var/log/zimbra.log

Amavis

  • Amavisd debug logging - Controlled by zimbra attribute "zimbraAmavisLogLevel", which writes value of "$log_level" in file /opt/zimbra/conf/amavisd.conf
  • Typically required when trying to identify issues -
    • Amavis causing delay in scanning a message
    • Domain disclaimer not working as expected
    • Amavis causing disruption in mail flow
  • Logging level - 0-5, where 5 represents maximum debug logging. Value defaults to 1.
zmprov mcf zimbraAmavisLogLevel 5
zmprov ms `zmhostname` "zimbraAmavisLogLevel" 5
  • This will write debug log to - /var/log/zimbra.log


Proxy

  • Controlled by zimbra attribute zimbraReverseProxyLogLevel.
zmprov ms `zmhostname` zimbraReverseProxyLogLevel debug
zmnginxctl restart

It will log a lot, so you can set it back to info at any time:

 zmprov ms `zmhostname` zimbraReverseProxyLogLevel info
  • This will write debug logs to /opt/zimbra/log/nginx.log
  • Typically required to trouble shoot proxy issues like -
    • Proxy unable to fetch user data
    • Unable to look up correct user's mailbox
    • Nginx process crash


ZCO Logs

  • For all but the simplest queries, ZCO Dev needs ZCO logs, dumps and event viewer logs. Here's how to gather them:
Disable Windows Search 
start->run->services->Windows Search->Properties->Stop + “Disabled” 
Start Outlook 
Disable Foreign Addins i.e. any addins that dont start with "Zimbra" 
OL2003: Tools->Options->Other->Advanced Options->COM Add-Ins->Remove any addins that don't start  with "Zimbra" 
OL2007: Tools->Trust Center->Add-ins->Manage COM Addins->Go->Untick any addins that don't start with "Zimbra" 
OL2010: File->Options->Add-ins->Manage COM Addins->Go->Untick any addins that don't start with "Zimbra" 
OL2013: as OL2010 
Close Outlook 
  • Empty ZCOLogs Directory (to avoid sending ZCO Dev stale/irrelevant logs)
  • Set Log Settings (using ZCOLogCtl.exe) to the following -
 Click the Default button 
 Max size of single log file : 50Mb 
 Preserve Log File Backups ticked 
 Max Size of ZCOLogs directory UNticked

ZCO Troubleshooting

  • Start Outlook + Repro the Issue
  • Take detailed notes of the steps
  • If possible, make a note of the time the problem ocurred
  • If the problem relates to a particular item, include its ZimbraID and/or Subject
  • Include Screen shots; Screen video if possible
  • Close Outlook. If Outlook hung, grab a Hang Dump:
  • Task Manager->Processes Tab->Right click OUTLOOK.EXE->Create Dump File
  • Export Outlook Events from the Windows Event Viewer:
  • Start Event Viewer
  • Select "Windows Logs" | Application from the left pane.
  • In Actions (right pane) select "Filter Current Log..."
  • In the Filter dialog select
  • select "Last 7 days" from the "Logged:" dropdown
  • check Outlook in the "Event sources:"
  • they will probably also need to limit output to the current "User:"
  • Click OK
  • Save with "Save Filtered Log File...", and pass us the resulting .evtx file
  • Check logs before sending to ZCO Dev
  • Must cover the repro period (note that each log file name includes the date and time at which a client session started)
  • Check for obvious issues (e.g. server unreachable)
## ERROR ## Usually indicates an unexpected problem 
## WARNING ## Usually benign but worth knowing 

Quiz Questions

  • Which log file(s) an ZC Administrator should review if there are reports that incoming mail has stopped.
  • You need to enable SOAP TRACE logging for user joe@xyz.com, what is the command needed to enable without the need of restarting service?
Jump to: navigation, search