Troubleshooting Course Content Rough Drafts-Zimbra Logs: Difference between revisions

 
(60 intermediate revisions by 2 users not shown)
Line 4: Line 4:
=Introduction=
=Introduction=


Ajcody - Need a general introduction of the course for the trainer to start with.
'''''Ajcody - Need a general introduction of the course for the trainer to start with.'''''


== OS Commands To Be Familiar With ==
=== VI ===
* arrow keys - moves the "cursor" location  up/down , left/right
* i for insert
* r for replace character
* R replace and continue replacing to right
* u - undo change
* /term  search for "term" and goto in file
** n - when in search mode, will goto next match
** shift+n - go back to last match
* q! - forced quit without saving
* wq! - writes , quits , forces save
=== Grep  ===
grep FATAL mailbox.log
Ignore case [-i] :
grep -i fatal mailbox.log
OR or extended regexp operator [-E] , any line that has fatal or error [might not be available in all versions of grep] :
grep -iE 'fatal|error' mailbox.log
NOT, Negative or invert matching [-v] , any line that DOES NOT have info or warn:
grep -iEv 'info|warn' mailbox.log
Print additional lines around context [-C#] :
grep -iEv -C3 'info|warn' mailbox.log
Print additional lines after context [-A#] :
grep -iEv -A3 'info|warn' mailbox.log
Print with line numbers [-n] :
grep -iEvn -A3 'info|warn' mailbox.log
If your terminal shows colors, this is useful in seeing where the match is in the output [--color] :
grep -iEvn -A3 --color 'info|warn' mailbox.log
AND match. Match happens left to right with first variable and then second :
grep -iE 'user1.*user2' /var/log/zimbra.log
AND match going either way :
grep -iE 'user1.*user2|user2.*user1' /var/log/zimbra.log
AND match by piping to another grep :
  grep -iE user1 /var/log/zimbra.log | grep user2
Print only the matched (non-empty) parts of a matching line, with each part on a separate output line. Very useful with you combine with the -E option and are looking for uniform log events and want to discard unnecessary info without doing/knowing sed or awk.
grep -i smtp /var/log/zimbra.log | grep -ioE 'to=.*status=.*[ \t] ' | head -5
  to=<admin@mail-172.example.com>, relay=127.0.0.1[127.0.0.1]:10026, delay=4.4, delays=0.22/0.03/0.4/3.7, dsn=2.0.0, status=sent
  to=<admin@mail-172.example.com>, relay=127.0.0.1[127.0.0.1]:10032, delay=2.7, delays=0.3/0.04/0.01/2.3, dsn=2.0.0, status=sent
  to=<admin@mail-172.example.com>, relay=127.0.0.1[127.0.0.1]:10026, delay=2.3, delays=0.07/0.1/0.27/1.9, dsn=2.0.0, status=sent 
  to=<admin@mail-172.example.com>, relay=127.0.0.1[127.0.0.1]:10032, delay=1.5, delays=0.24/0.02/0.02/1.2, dsn=2.0.0, status=sent
  to=<admin@mail-172.example.com>, relay=127.0.0.1[127.0.0.1]:10026, delay=7.9, delays=0.3/0.03/2.3/5.3, dsn=2.0.0, status=sent
zgrep -i "triggers filter" /var/log/zimbra.* | grep -ioE 'from=.*to=.*>[ \t]' | grep example | uniq
  from=<admin@mail-172.example.com> to=<admin@mail-172.example.com>
  from=<user1@mail-172.example.com> to=<admin@mail-172.example.com>
  from=<admin@mail-172.example.com> to=<admin@mail-172.example.com>
=== Zgrep  ===
You can use all the options above in grep with zgrep .
zgrep -i fatal mailbox.log.*.gz
zgrep -iEv -C3 'info|warn' mailbox.log.*.gz
=== Head ===
Useful, for example, when you want to see how a long file formats date/time to use later for your grep:
head mailbox.log
=== Tail  ===
If you wanted to monitor multiple files at once in 'real time' . Note, you could also pipe the output to grep also :
tail -f mailbox.log -f trace_log.2015_03_11 -f access_log.2015-03-11 -f sync.log -f ews.log
tail -f mailbox.log -f trace_log.2015_03_11 -f access_log.2015-03-11 -f sync.log -f ews.log | grep -i user1
=== Less  ===
less +F /opt/zimbra/log/mailbox.log
=== wc - Word Count ===
=== uniq - Unique ===
=== sort ===


== Log Files ==
== Log Files ==


A brief list of the major log files in Zimbra and their significance.
A brief list of the major log files in Zimbra and their significance -
   
   
* Mailstore Service Related Logs
=== Mailstore Service Related Logs ===
** /opt/zimbra/log/mailbox.log - Zimbra Mailboxd (jetty) writes all mailbox activity to this log file. This activity can be anything from moving a message to deletion of an account.  
* /opt/zimbra/log/mailbox.log - Zimbra Mailboxd (jetty) writes all mailbox activity to this log file. This activity can be anything from moving a message to deletion of an account.  
** /opt/zimbra/log/zmmailboxd.out – Mailboxd (jetty) logs to this file during startup. Should be the log file to refer if you have trouble getting your mailboxd to start.
* /opt/zimbra/log/zmmailboxd.out – Mailboxd (jetty) logs to this file during startup. Should be the log file to refer if you have trouble getting your mailboxd to start.
** /opt/zimbra/log/mysql_error.log – Mysql errors and startup messages. This log should be looked up to determine if innodb corruption has occurred.  
* /opt/zimbra/log/mysql_error.log – Mysql errors and startup messages. This log should be looked up to determine if innodb corruption has occurred.  
** /opt/zimbra/log/myslow.log – Consists of sql statements which took more than long_query_time seconds (zimbra default – 1second) to execute and required at least min_examined_row_limit (zimbra default - 0) rows to be examined.  
* /opt/zimbra/log/myslow.log – Consists of sql statements which took more than long_query_time seconds (zimbra default – 1second) to execute and required at least min_examined_row_limit (zimbra default - 0) rows to be examined.  
** /opt/zimbra/log/audit.log – Logs authentication activity.
* /opt/zimbra/log/audit.log – Logs authentication activity.
** /opt/zimbra/log/sync.log – Mobile sync operations get logged to this log file. Should be used for reference when troubleshooting activesync issues.
* /opt/zimbra/log/sync.log – Mobile sync operations get logged to this log file. Should be used for reference when troubleshooting activesync issues.
** /opt/zimbra/log/ews.log – All EWS server side activity gets logged here. Should be used when trying to identify issues with EWS clients.
* /opt/zimbra/log/syncstate.log - Sync state information logging.
* LDAP Service Related Logs
* /opt/zimbra/log/wbxml.log - WAP binary xml log.
** /var/log/zimbra.log – Ldap log events go to this file.
* /opt/zimbra/log/ews.log – All EWS server side activity gets logged here. Should be used when trying to identify issues with EWS clients.
* MTA Service Related Logs
 
** /var/log/zimbra.log – MTA (Postfix, amavisd, antispam,  antivirus, cyrus-sasl) log events go to this file.
=== LDAP Service Related Logs ===
** /opt/zimbra/log/clamd.log – Clamd (Antivirus daemon) logs all antivirus activity to this file.
* /var/log/zimbra.log – Ldap log events go to this file.
** /opt/zimbra/log/freshclam.log – Antivirus definition update log. Review this file if your server is not updating its virus definitions.
=== MTA Service Related Logs ===
** /opt/zimbra/log/spamtrain.log – The spam training log. Useful when trying to debug the action of "marking" a mail as spam and ham.  
* /var/log/zimbra.log – MTA (Postfix, amavisd, antispam,  antivirus, cyrus-sasl) log events go to this file.
* Proxy Service Related Logs
* /opt/zimbra/log/clamd.log – Clamd (Antivirus daemon) logs all antivirus activity to this file.
** /opt/zimbra/log/nginx.access.log - Nginx writes information about client requests after the request is processed here. Can be used to track incoming requests and their response codes being sent by the server.  
* /opt/zimbra/log/freshclam.log – Antivirus definition update log. Review this file if your server is not updating its virus definitions.
** /opt/zimbra/log/nginx.log – Nginx error log which defaults to “info” level for logging. Log file to begin with for troubleshooting proxy related issues, gives a good overview of what the proxy is doing.  
* /opt/zimbra/log/spamtrain.log – The spam training log. Useful when trying to debug the action of "marking" a mail as spam and ham.  
* Stats Services Related Logs
=== Proxy Service Related Logs ===
** /var/log/zimbra-stats.log – Logger data gets logged to this file via the OS logger, syslogd.
* /opt/zimbra/log/nginx.access.log - Nginx writes information about client requests after the request is processed here. Can be used to track incoming requests and their response codes being sent by the server.  
* For All ZCS Servers  
* /opt/zimbra/log/nginx.log – Nginx error log which defaults to “info” level for logging. Log file to begin with for troubleshooting proxy related issues, gives a good overview of what the proxy is doing.  
** /var/log/zimbra.log – Tracking of the various ZC service status [starting/stopping] via the OS logger, syslogd, are logged in this file as well.
=== Stats Services Related Logs ===
** /var/log/syslog
* /var/log/zimbra-stats.log – Stats data gets logged to this file via the OS logger, syslogd.
** /var/log/messages
 
=== For All ZCS Servers ===
* /var/log/zimbra.log – Tracking of the various ZC service status [starting/stopping] via the OS logger, syslogd, are logged in this file as well.
* /var/log/syslog – Check here for possible server issues that might be causing ZC issues also.
* /var/log/messages – Check here for possible server issues that might be causing ZC issues also.


== Log Rotation And Management ==
== Log Rotation And Management ==
Line 40: Line 148:
Two different log rotation daemons handle log rotation of the zimbra log files - Log4j and Logrotate.d .  
Two different log rotation daemons handle log rotation of the zimbra log files - Log4j and Logrotate.d .  


* Log4j - Log4j is responsible for logging to mailbox.log, audit.log, sync.log, synctrace.log,webxml.log and ews.log. For config : /opt/zimbra/conf/log4j.properties
* Log4j - Log4j is responsible for logging to :
 
mailbox.log, audit.log, sync.log,wbxml.log and ews.log.  
 
* Configuration file for Log4j is : /opt/zimbra/conf/log4j.properties


Example, below the are the config options for the rotation of /opt/zimbra/log/mailbox.log  
Example, below the are the config options for the rotation of /opt/zimbra/log/mailbox.log  
Line 51: Line 163:
  log4j.appender.LOGFILE.layout.ConversionPattern=%d %-5p [%t] [%z] %c{1} - %m%n
  log4j.appender.LOGFILE.layout.ConversionPattern=%d %-5p [%t] [%z] %c{1} - %m%n


* Logrotate.d - The OS daemon logrotate used to rotate other log files.  
* Logrotate.d - The OS daemon logrotate used to rotate other log files. Files rotated by logrotate are :


Config file : /etc/logrotate.d/zimbra  
zimbra.log, zimbra-stats.log, myslow.log, zmconfigd.log, zmconfigd.log, clamd.log, freshclam.log, nginx.log, nginx_access.log, cbpolicyd.log and spamtrain.log.


Files rotated by logrotate : zimbra.log, zimbra-stats.log, myslow.log, zmconfigd.log, zmconfigd.log, clamd.log, freshclam.log, nginx.log, nginx_access.log, cbpolicyd.log and spamtrain.log.
* Configuration file for Logrotate is : /etc/logrotate.d/zimbra  


There is also log management commands in the zimbra crontab. You can review the crontab lines by doing :
There is also log management commands in the zimbra crontab. You can review the crontab lines by doing :
Line 64: Line 176:
== Debug Logging ==
== Debug Logging ==
=== Mailbox ===
=== Mailbox ===
The ZCS server uses log4j, a Java logging package as the log manager
 
The ZCS server uses log4j, a Java logging package as the log manager.
 
* By default, the ZC server has log4j configured to log to the local file system
* By default, the ZC server has log4j configured to log to the local file system
* To modify the server debug logging level, you would edit /opt/zimbra/conf/log4j.properties.in . Doing this will adjust the logging level for the log4j property you adjusted. 
==== Example using config file  log4j.properties ====       
[Example adjusts all imap activity to be set to DEBUG for all users]


* Global debug logging : Doing this will enable debug logging for any log4j property. 
* As zimbra, edit - /opt/zimbra/conf/log4j.properties.in  
         
* Add the following line to the bottom once you confirm one doesn't already exist for zimbra.imap :
Edit - /opt/zimbra/conf/log4j.properties.in
** log4j.logger.zimbra.imap=DEBUG
log4j.logger.zimbra.imap=DEBUG
Restart mailboxd – zmmailboxdctl restart


* Per user debug logging (does not requires mailboxd restart) :
* Restart the mailboxd service with :  zmmailboxdctl restart


[Examples]
Note: The above is config file change.


addAccountLogger : zmprov aal user@domain.com zimbra.imap debug
==== Example using zmprov ====
removeAccountLogger : zmprov ral user@domain.com zimbra.imap
* To adjust logging levels on a per user basis, you can do the following below and it does NOT require a mailboxd restart. Also note that this is a command as opposed to the above approach of changing a config file -


[zmprov options for logger]
[Example adjusts zimbra.imap to be set to DEBUG for a user]


* addAccountLogger :
zmprov aal user@domain.com zimbra.imap debug
* removeAccountLogger :
zmprov ral user@domain.com zimbra.imap
* zmprov options for logger
   addAccountLogger(aal) [-s/--server hostname] {name@domain|id} {logging-category} {trace|debug|info|warn|error}
   addAccountLogger(aal) [-s/--server hostname] {name@domain|id} {logging-category} {trace|debug|info|warn|error}
   getAccountLoggers(gal) [-s/--server hostname] {name@domain|id}
   getAccountLoggers(gal) [-s/--server hostname] {name@domain|id}
Line 88: Line 210:
   resetAllLoggers(rlog) [-s/--server hostname]
   resetAllLoggers(rlog) [-s/--server hostname]


* Debug logs for mailbox required when troubleshooting a specific problem with the mailbox component, for example


Trying to add an appointment but not happening
==== When to use ====
Specific message not getting delivered
* Debug logs for mailbox required when troubleshooting a specific problem with the mailbox component, for example :
Specific user unable to log into his acocunt using web/imap/pop etc.
** Trying to add an appointment but not happening
Unable to download specific message using imap/pop or activesync
** Specific message not getting delivered
** Specific user unable to log into his acocunt using web/imap/pop etc.
* Each component of mailboxd has a control to increase logging for that specific component. These are known as loggers, To list all possible loggers -
** Unable to download specific message using imap/pop or activesync
 
==== List of Log Categories ====
Each component of mailboxd has a variable to increase logging for that specific component. These are known as logging-categories. In the above listing for zmprov addAccountLogger, you'll see reference to the option {logging-category}. To get a list all possible logging categories {logging-category} -  
  zmprov help log
  zmprov help log


* Debug logs for mailbox get written to /opt/zimbra/log/mailbox.log, /opt/zimbra/log/ews.log for ews, /opt/zimbra/log/sync.log for sync and /opt/zimbra/log/webxml.log for webxml.
==== List of Log Levels ====
Logging levels determine what types of events are logged. The following explains log levels and what level of logging each will product. Log levels in log4j listed by highest level of verbosity to the least :
 
TRACE - Designates finer-grained informational events than the DEBUG.
DEBUG - The DEBUG logging level will log detailed informational events useful for debugging purposes.
INFO - Designates informational messages that highlight the progress of the application at coarse-grained level. By default, most predefined log4j categories in ZCS are set to the INFO logging level.
WARN - Designates potentially harmful situations.
ERROR - Designates error events that might still allow the application to continue running.
FATAL - The FATAL logging level will only log very severe error events.
 
Each additional logging level will log more events, including the events belonging to the previous logging levels. For example, enabling the WARN logging level will record log events belonging to the WARN, ERROR, and FATAL logging levels, but will not record events belonging to the INFO, DEBUG, or TRACE levels.


* Protocol Trace
Protocol trace is available in the following logging categories with TRACE logging level:


Logging Levels
zimbra.smtp
Logging levels determine what types of events are logged. The following are logging levels in log4j:
zimbra.lmtp
zimbra.soap
zimbra.imap
zimbra.imap-client
zimbra.pop
zimbra.pop-client


*FATAL
*TRACE
*ERROR
*WARN
*INFO
*DEBUG


The FATAL logging level will only log very severe error events. The DEBUG logging level will log detailed informational events useful for debugging purposes. Each additional logging level will log more events, including the events belonging to the previous logging levels. For example, enabling the WARN logging level will record log events belonging to the FATAL, ERROR, and WARN logging levels, but will not record events belonging to the INFO or DEBUG levels. By default, most predefined log4j categories in ZCS are set to the INFO logging level.
==== Debug log files ====
* Debug logs for mailbox get written to :
** /opt/zimbra/log/mailbox.log for most categories,
** /opt/zimbra/log/ews.log for ews
** /opt/zimbra/log/sync.log for sync
** /opt/zimbra/log/webxml.log for webxml


Protocol Trace
==== Reference ====
Protocol trace is available in the following logging categories with TRACE logging level:
* For all logging levels and other reference material please see  -  
*zimbra.smtp
*zimbra.lmtp
*zimbra.soap
*zimbra.imap
*zimbra.imap-client
*zimbra.pop
*zimbra.pop-client


  https://www.zimbra.com/docs/ne/latest/administration_guide/wwhelp/wwhimpl/js/html/wwhelp.htm#href=860_admin_ne.Logging_Levels.html


* For all logging levels and other reference material please see
  http://community.zimbra.com/projects/collaboration_kb/w/log-reference
  https://www.zimbra.com/docs/ne/latest/administration_guide/wwhelp/wwhimpl/js/html/wwhelp.htm#href=860_admin_ne.Logging_Levels.html


=== Ldap ===
=== Ldap ===
*Ldap debug logging – Controlled by “ldap_common_loglevel”, this writes the config ldap attribute "olcLogLevel".


*Typically required when trying to troubleshoot a specific request which is taking time, or when there some weird scenario which is causing a specific replica sync to fail.  
* Ldap debug logging – Controlled by “ldap_common_loglevel” variable, this writes the config ldap attribute "olcLogLevel".
* Typically required when trying to troubleshoot a specific request which is taking time or when there some weird scenario which is causing a specific replica sync to fail.  
* Debug logs are written to /var/log/zimbra.log
 
Examples [as the zimbra user]:


Examples  -
   zmlocalconfig –e ldap_common_loglevel=256
   zmlocalconfig –e ldap_common_loglevel=256
          or
 
   zmlocalconfig –e ldap_common_loglevel=“stats”
   zmlocalconfig –e ldap_common_loglevel=“stats”


  Restart -  ldap stop; ldap start. 
Restarting of ldap is required for the change to take effect:


* For various log levels please see -
  ldap stop; ldap start
  http://www.openldap.org/software/man.cgi?query=slapd.conf&apropos=0&sektion=0&manpath=OpenLDAP+2.4-Release&format=html


* Debug logs written to /var/log/zimbra.log
For various log levels please see :
* http://www.openldap.org/software/man.cgi?query=slapd.conf&apropos=0&sektion=0&manpath=OpenLDAP+2.4-Release&format=html


=== Postfix ===
=== Postfix ===
* Typically required when trying to troubleshoot email flow issues. For example specific mail getting dropped or getting stuck in the queue. Or if troubleshooting issues with the postfix process itself, for example postfix crashing. The debug log will help understand what exactly postfix was doing when the crash occurred.  
* Typically required when trying to troubleshoot email flow issues. For example specific mail getting dropped or getting stuck in the queue. Or if troubleshooting issues with the postfix process itself, for example postfix crashing. The debug log will help understand what exactly postfix was doing when the crash occurred.  
* Postfix log events goto /var/log/zimbra.log
* To modify default logging level you edit /opt/zimbra/postfix/conf/master.cf.in
Example, change the following line [default] in /opt/zimbra/postfix/conf/master.cf.in :


* Edit /opt/zimbra/postfix/conf/master.cf.in
  smtp      inet  n      -      n      -      -      smtpd
 
 
  Change - smtp      inet  n      -      n      -      -      smtpd
To be the following [as the zimbra user] :
  To - smtp      inet  n      -      n      -      -      smtpd –vv
 
  smtp      inet  n      -      n      -      -      smtpd –vv
 
A restart of postfix required


Restart of postfix required
  zmmtactl restart
  zmmtactl restart


* This will write debug postfix logs to /var/log/zimbra.log
=== SpamAssassin ===


=== SpamAssassin ===
* Required when troubleshooting issues specific to spamassassin, for example a message being incorrectly dropped or spamassassin taking too long to scan emails.  
* Required when troubleshooting issues specific to spamassassin, for example a message being incorrectly dropped or spamassassin taking too long to scan emails.  
* Debug logs get written to /var/log/zimbra.log
* Debug logging for SpamAssassin is controlled by “zimbraAmavisSALogLevel”, this writes "sa_debug" to /opt/zimbra/conf/amavisd.conf.
* The value can be 0 or 1, where 1 means debug log enabled and 0 means minimalistic info logging.


* Debug logging for SpamAssassin controlled by “zimbraAmavisSALogLevel”, this writes "sa_debug" to /opt/zimbra/conf/amavisd.conf.
Example [as the zimbra user]:
 
* Value can be 0 or 1, where 1 means debug log enabled and 0 means minimalistic info logging.


  zmprov mcf zimbraAmavisSALogLevel 1  
  zmprov mcf zimbraAmavisSALogLevel 1  
  zmprov ms `zmhostname` zimbraAmavisSALogLevel 1  
  zmprov ms `zmhostname` zimbraAmavisSALogLevel 1


* Debug logs get written to /var/log/zimbra.log
=== Amavis ===


=== Amavis ===
* Amavisd debug logging - Controlled by zimbra attribute "zimbraAmavisLogLevel", which writes value of "$log_level" in file /opt/zimbra/conf/amavisd.conf
* Amavisd debug logging - Controlled by zimbra attribute "zimbraAmavisLogLevel", which writes value of "$log_level" in file /opt/zimbra/conf/amavisd.conf
 
* Amavis debug logging goes to  - /var/log/zimbra.log
* Typically required when trying to identify issues -
* Typically required when trying to identify issues such as:
** Amavis causing delay in scanning a message
** Amavis causing delay in scanning a message
** Domain disclaimer not working as expected
** Domain disclaimer not working as expected
** Amavis causing disruption in mail flow
** Amavis causing disruption in mail flow
* Logging level can be from 0 to 5, where 5 represents maximum debug logging. Default value is 1.


* Logging level - 0-5, where 5 represents maximum debug logging. Value defaults to 1.
Example [as zimbra user]:


  zmprov mcf zimbraAmavisLogLevel 5
  zmprov mcf zimbraAmavisLogLevel 5
  zmprov ms `zmhostname` "zimbraAmavisLogLevel" 5
  zmprov ms `zmhostname` "zimbraAmavisLogLevel" 5


* This will write debug log to  - /var/log/zimbra.log
=== Proxy ===


* Controlled by zimbra attribute zimbraReverseProxyLogLevel .
* Logging events tied to zimbraReverseProxyLogLevel are written to /opt/zimbra/log/nginx.log
* Typically required to trouble shoot proxy issues like the following situations :
** Proxy unable to fetch user data
** Unable to look up correct user's mailbox
** Nginx process crash
* The default for zimbraReverseProxyLogLevel is INFO . Possible values are [ per zimbra-attrs.xml ]:
** debug, info, notice, warn, error, crit, debug_http, debug_mail, debug_core, debug_zimbra


=== Proxy ===
Example to change zimbraReverseProxyLogLevel to DEBUG [as the zimbra user]:
* Controlled by zimbra attribute zimbraReverseProxyLogLevel.


  zmprov ms `zmhostname` zimbraReverseProxyLogLevel debug
  zmprov ms `zmhostname` zimbraReverseProxyLogLevel debug
  zmnginxctl restart
  zmnginxctl restart


It will log a lot, so you can set it back to info at any time:
It will log a lot of information; you can set it back to info at any time with :
 
   zmprov ms `zmhostname` zimbraReverseProxyLogLevel info
   zmprov ms `zmhostname` zimbraReverseProxyLogLevel info
  zmnginxctl restart
== ZCO Logs ==
For all but the simplest queries, ZCO Dev needs ZCO logs, dumps and event viewer logs.


* This will write debug logs to /opt/zimbra/log/nginx.log
Here's how to gather them:


* Typically required to trouble shoot proxy issues like -
===Preparing ZCO And Outlooking For Debug Log Gathering===
** Proxy unable to fetch user data
** Unable to look up correct user's mailbox
** Nginx process crash


# Disable Windows Search
# start->run->services->Windows Search->Properties->Stop + “Disabled”
# Start Outlook
# Disable Foreign Addins i.e. any addins that dont start with "Zimbra"
#* Then, depending on your version of Outlook:
#** OL2003: Tools->Options->Other->Advanced Options->COM Add-Ins->Remove any addins that don't start  with "Zimbra"
#** OL2007: Tools->Trust Center->Add-ins->Manage COM Addins->Go->Untick any addins that don't start with "Zimbra"
#** OL2010: File->Options->Add-ins->Manage COM Addins->Go->Untick any addins that don't start with "Zimbra"
#** OL2013: as OL2010
# Close Outlook
# Empty ZCOLogs Directory (to avoid sending ZCO Dev stale/irrelevant logs)
# Set Log Settings (using ZCOLogCtl.exe) to the following :
#* Click the Default button
#* Max size of single log file : 50Mb
#* Preserve Log File Backups : Ticked
#* Max Size of ZCOLogs directory : Un-ticked


== ZCO Logs ==
=== ZCO Troubleshooting And Gathering Of Information ===
* For all but the simplest queries, ZCO Dev needs ZCO logs, dumps and event viewer logs. Here's how to gather them:


Disable Windows Search
''''' Ajcody - There is no listed step on gathering the actual ZCO logs '''''
start->run->services->Windows Search->Properties->Stop + “Disabled”
Start Outlook
Disable Foreign Addins i.e. any addins that dont start with "Zimbra"
OL2003: Tools->Options->Other->Advanced Options->COM Add-Ins->Remove any addins that don't start  with "Zimbra"
OL2007: Tools->Trust Center->Add-ins->Manage COM Addins->Go->Untick any addins that don't start with "Zimbra"
OL2010: File->Options->Add-ins->Manage COM Addins->Go->Untick any addins that don't start with "Zimbra"
OL2013: as OL2010
Close Outlook


* Empty ZCOLogs Directory (to avoid sending ZCO Dev stale/irrelevant logs)  
# Start Outlook and reproduce the issue
* Set Log Settings (using ZCOLogCtl.exe) to the following -
#* Take detailed notes of the steps
#* If possible, make a note of the time the problem occurred
#* If the problem relates to a particular item, include its ZimbraID and/or Subject
#* Include Screen shots; Screen video if possible
# Close Outlook. If Outlook hung, grab a Hang Dump:
#* Task Manager->Processes Tab->Right click OUTLOOK.EXE->Create Dump File
# Export Outlook Events from the Windows Event Viewer:
#* Start Event Viewer
#* Select "Windows Logs" | Application from the left pane.
#* In Actions (right pane) select "Filter Current Log..."
#* In the Filter dialog select
#* select "Last 7 days" from the "Logged:" dropdown
#* check Outlook in the "Event sources:"
#** It's recommended to limit output to the current "User:"
#* Click OK
#* Save with "Save Filtered Log File...", and pass us the resulting .evtx file
# Check logs before sending to ZCO Dev or Zimbra Support
#* Must cover the repro period (note that each log file name includes the date and time at which a client session started)  
#* Check for obvious issues (e.g. server unreachable)  
#* ## ERROR ## Usually indicates an unexpected problem
#* ## WARNING ## Usually benign but worth knowing


  Click the Default button
See also :
  Max size of single log file : 50Mb
* http://www.zimbra.com/forums/zimbra-connector-outlook/1083-tips-how-report-errors-zcs-outlook-connector.html
  Preserve Log File Backups ticked
  Max Size of ZCOLogs directory UNticked


=== ZCO Troubleshooting ===
== Zimbra Web Client Debug Logging ==
* Start Outlook + Repro the Issue
* Take detailed notes of the steps
* If possible, make a note of the time the problem ocurred
* If the problem relates to a particular item, include its ZimbraID and/or Subject
* Include Screen shots; Screen video if possible


* Close Outlook. If Outlook hung, grab a Hang Dump:
* Client side issues like a JS error at times needs client side logs to understand what is breaking.  
* Task Manager->Processes Tab->Right click OUTLOOK.EXE->Create Dump File
* Export Outlook Events from the Windows Event Viewer:
* Start Event Viewer
* Select "Windows Logs" | Application from the left pane.
* In Actions (right pane) select "Filter Current Log..."
* In the Filter dialog select
* select "Last 7 days" from the "Logged:" dropdown
* check Outlook in the "Event sources:"
* they will probably also need to limit output to the current "User:"
* Click OK
* Save with "Save Filtered Log File...", and pass us the resulting .evtx file
* Check logs before sending to ZCO Dev
* Must cover the repro period (note that each log file name includes the date and time at which a client session started)
* Check for obvious issues (e.g. server unreachable)


## ERROR ## Usually indicates an unexpected problem
* Chrome and Firefox both allow "debug mode" which help.
## WARNING ## Usually benign but worth knowing


* See also http://www.zimbra.com/forums/zimbra-connector-outlook/1083-tips-how-report-errors-zcs-outlook-connector.html
* Firebug is a very commonly used utility used to monitor calls to the server and the response, the below explains how to set it up collect diagnostics -
  https://wiki.zimbra.com/wiki/Enable_firebug


=Quiz Questions=
=Quiz Questions=
Line 257: Line 415:
* You need to enable SOAP TRACE logging for user joe@xyz.com, what is the command needed to enable without the need of restarting service?
* You need to enable SOAP TRACE logging for user joe@xyz.com, what is the command needed to enable without the need of restarting service?


*
* Mailboxd will not start, having run "zmmailboxdctl start", the service status still shows up as stopped, which file should be reviewed to understand what the problem is.
 
* Users randomly get a 502 bad gateway when trying to login into the web client, need to troubleshoot why proxy is reporting an error instead of fetching the data from the user account.
 
* Junk messages are not getting marked as Spam, need to review why Spamassassin is not doing a scan of messages.
 
* Collect ZCO diagnostic log

Latest revision as of 16:09, 8 April 2015

Verified Against: Zimbra Collaboration Suite 8.6 Date Created: 01/22/2015
Article ID: https://wiki.zimbra.com/index.php?title=Troubleshooting_Course_Content_Rough_Drafts-Zimbra_Logs Date Modified: 2015-04-08



Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »

Introduction

Ajcody - Need a general introduction of the course for the trainer to start with.

OS Commands To Be Familiar With

VI

  • arrow keys - moves the "cursor" location up/down , left/right
  • i for insert
  • r for replace character
  • R replace and continue replacing to right
  • u - undo change
  • /term search for "term" and goto in file
    • n - when in search mode, will goto next match
    • shift+n - go back to last match
  • q! - forced quit without saving
  • wq! - writes , quits , forces save

Grep

grep FATAL mailbox.log

Ignore case [-i] :

grep -i fatal mailbox.log

OR or extended regexp operator [-E] , any line that has fatal or error [might not be available in all versions of grep] :

grep -iE 'fatal|error' mailbox.log

NOT, Negative or invert matching [-v] , any line that DOES NOT have info or warn:

grep -iEv 'info|warn' mailbox.log

Print additional lines around context [-C#] :

grep -iEv -C3 'info|warn' mailbox.log

Print additional lines after context [-A#] :

grep -iEv -A3 'info|warn' mailbox.log

Print with line numbers [-n] :

grep -iEvn -A3 'info|warn' mailbox.log

If your terminal shows colors, this is useful in seeing where the match is in the output [--color] :

grep -iEvn -A3 --color 'info|warn' mailbox.log

AND match. Match happens left to right with first variable and then second :

grep -iE 'user1.*user2' /var/log/zimbra.log

AND match going either way :

grep -iE 'user1.*user2|user2.*user1' /var/log/zimbra.log 

AND match by piping to another grep :

 grep -iE user1 /var/log/zimbra.log | grep user2

Print only the matched (non-empty) parts of a matching line, with each part on a separate output line. Very useful with you combine with the -E option and are looking for uniform log events and want to discard unnecessary info without doing/knowing sed or awk.

grep -i smtp /var/log/zimbra.log | grep -ioE 'to=.*status=.*[ \t] ' | head -5
 to=<admin@mail-172.example.com>, relay=127.0.0.1[127.0.0.1]:10026, delay=4.4, delays=0.22/0.03/0.4/3.7, dsn=2.0.0, status=sent 
 to=<admin@mail-172.example.com>, relay=127.0.0.1[127.0.0.1]:10032, delay=2.7, delays=0.3/0.04/0.01/2.3, dsn=2.0.0, status=sent 
 to=<admin@mail-172.example.com>, relay=127.0.0.1[127.0.0.1]:10026, delay=2.3, delays=0.07/0.1/0.27/1.9, dsn=2.0.0, status=sent  
 to=<admin@mail-172.example.com>, relay=127.0.0.1[127.0.0.1]:10032, delay=1.5, delays=0.24/0.02/0.02/1.2, dsn=2.0.0, status=sent 
 to=<admin@mail-172.example.com>, relay=127.0.0.1[127.0.0.1]:10026, delay=7.9, delays=0.3/0.03/2.3/5.3, dsn=2.0.0, status=sent
zgrep -i "triggers filter" /var/log/zimbra.* | grep -ioE 'from=.*to=.*>[ \t]' | grep example | uniq
 from=<admin@mail-172.example.com> to=<admin@mail-172.example.com> 
 from=<user1@mail-172.example.com> to=<admin@mail-172.example.com> 
 from=<admin@mail-172.example.com> to=<admin@mail-172.example.com>

Zgrep

You can use all the options above in grep with zgrep .

zgrep -i fatal mailbox.log.*.gz
zgrep -iEv -C3 'info|warn' mailbox.log.*.gz

Head

Useful, for example, when you want to see how a long file formats date/time to use later for your grep:

head mailbox.log

Tail

If you wanted to monitor multiple files at once in 'real time' . Note, you could also pipe the output to grep also :

tail -f mailbox.log -f trace_log.2015_03_11 -f access_log.2015-03-11 -f sync.log -f ews.log
tail -f mailbox.log -f trace_log.2015_03_11 -f access_log.2015-03-11 -f sync.log -f ews.log | grep -i user1

Less

less +F /opt/zimbra/log/mailbox.log

wc - Word Count

uniq - Unique

sort

Log Files

A brief list of the major log files in Zimbra and their significance -

Mailstore Service Related Logs

  • /opt/zimbra/log/mailbox.log - Zimbra Mailboxd (jetty) writes all mailbox activity to this log file. This activity can be anything from moving a message to deletion of an account.
  • /opt/zimbra/log/zmmailboxd.out – Mailboxd (jetty) logs to this file during startup. Should be the log file to refer if you have trouble getting your mailboxd to start.
  • /opt/zimbra/log/mysql_error.log – Mysql errors and startup messages. This log should be looked up to determine if innodb corruption has occurred.
  • /opt/zimbra/log/myslow.log – Consists of sql statements which took more than long_query_time seconds (zimbra default – 1second) to execute and required at least min_examined_row_limit (zimbra default - 0) rows to be examined.
  • /opt/zimbra/log/audit.log – Logs authentication activity.
  • /opt/zimbra/log/sync.log – Mobile sync operations get logged to this log file. Should be used for reference when troubleshooting activesync issues.
  • /opt/zimbra/log/syncstate.log - Sync state information logging.
  • /opt/zimbra/log/wbxml.log - WAP binary xml log.
  • /opt/zimbra/log/ews.log – All EWS server side activity gets logged here. Should be used when trying to identify issues with EWS clients.

LDAP Service Related Logs

  • /var/log/zimbra.log – Ldap log events go to this file.

MTA Service Related Logs

  • /var/log/zimbra.log – MTA (Postfix, amavisd, antispam, antivirus, cyrus-sasl) log events go to this file.
  • /opt/zimbra/log/clamd.log – Clamd (Antivirus daemon) logs all antivirus activity to this file.
  • /opt/zimbra/log/freshclam.log – Antivirus definition update log. Review this file if your server is not updating its virus definitions.
  • /opt/zimbra/log/spamtrain.log – The spam training log. Useful when trying to debug the action of "marking" a mail as spam and ham.

Proxy Service Related Logs

  • /opt/zimbra/log/nginx.access.log - Nginx writes information about client requests after the request is processed here. Can be used to track incoming requests and their response codes being sent by the server.
  • /opt/zimbra/log/nginx.log – Nginx error log which defaults to “info” level for logging. Log file to begin with for troubleshooting proxy related issues, gives a good overview of what the proxy is doing.

Stats Services Related Logs

  • /var/log/zimbra-stats.log – Stats data gets logged to this file via the OS logger, syslogd.

For All ZCS Servers

  • /var/log/zimbra.log – Tracking of the various ZC service status [starting/stopping] via the OS logger, syslogd, are logged in this file as well.
  • /var/log/syslog – Check here for possible server issues that might be causing ZC issues also.
  • /var/log/messages – Check here for possible server issues that might be causing ZC issues also.

Log Rotation And Management

Two different log rotation daemons handle log rotation of the zimbra log files - Log4j and Logrotate.d .

  • Log4j - Log4j is responsible for logging to :

mailbox.log, audit.log, sync.log,wbxml.log and ews.log.

  • Configuration file for Log4j is : /opt/zimbra/conf/log4j.properties

Example, below the are the config options for the rotation of /opt/zimbra/log/mailbox.log

log4j.appender.LOGFILE=org.apache.log4j.rolling.RollingFileAppender
log4j.appender.LOGFILE.RollingPolicy=org.apache.log4j.rolling.TimeBasedRollingPolicy
log4j.appender.LOGFILE.RollingPolicy.FileNamePattern=/opt/zimbra/log/mailbox.log.%d{yyyy-MM-dd}
log4j.appender.LOGFILE.File=/opt/zimbra/log/mailbox.log
log4j.appender.LOGFILE.layout=com.zimbra.common.util.ZimbraPatternLayout
log4j.appender.LOGFILE.layout.ConversionPattern=%d %-5p [%t] [%z] %c{1} - %m%n
  • Logrotate.d - The OS daemon logrotate used to rotate other log files. Files rotated by logrotate are :

zimbra.log, zimbra-stats.log, myslow.log, zmconfigd.log, zmconfigd.log, clamd.log, freshclam.log, nginx.log, nginx_access.log, cbpolicyd.log and spamtrain.log.

  • Configuration file for Logrotate is : /etc/logrotate.d/zimbra

There is also log management commands in the zimbra crontab. You can review the crontab lines by doing :

su - zimbra
crontab -l

Debug Logging

Mailbox

The ZCS server uses log4j, a Java logging package as the log manager.

  • By default, the ZC server has log4j configured to log to the local file system
  • To modify the server debug logging level, you would edit /opt/zimbra/conf/log4j.properties.in . Doing this will adjust the logging level for the log4j property you adjusted.

Example using config file log4j.properties

[Example adjusts all imap activity to be set to DEBUG for all users]

  • As zimbra, edit - /opt/zimbra/conf/log4j.properties.in
  • Add the following line to the bottom once you confirm one doesn't already exist for zimbra.imap :
    • log4j.logger.zimbra.imap=DEBUG
  • Restart the mailboxd service with : zmmailboxdctl restart

Note: The above is config file change.

Example using zmprov

  • To adjust logging levels on a per user basis, you can do the following below and it does NOT require a mailboxd restart. Also note that this is a command as opposed to the above approach of changing a config file -

[Example adjusts zimbra.imap to be set to DEBUG for a user]

  • addAccountLogger :
zmprov aal user@domain.com zimbra.imap debug
  • removeAccountLogger :
zmprov ral user@domain.com zimbra.imap
  • zmprov options for logger
 addAccountLogger(aal) [-s/--server hostname] {name@domain|id} {logging-category} {trace|debug|info|warn|error}
 getAccountLoggers(gal) [-s/--server hostname] {name@domain|id}
 getAllAccountLoggers(gaal) [-s/--server hostname]
 removeAccountLogger(ral) [-s/--server hostname] [{name@domain|id}] [{logging-category}]
 resetAllLoggers(rlog) [-s/--server hostname]


When to use

  • Debug logs for mailbox required when troubleshooting a specific problem with the mailbox component, for example :
    • Trying to add an appointment but not happening
    • Specific message not getting delivered
    • Specific user unable to log into his acocunt using web/imap/pop etc.
    • Unable to download specific message using imap/pop or activesync

List of Log Categories

Each component of mailboxd has a variable to increase logging for that specific component. These are known as logging-categories. In the above listing for zmprov addAccountLogger, you'll see reference to the option {logging-category}. To get a list all possible logging categories {logging-category} -

zmprov help log

List of Log Levels

Logging levels determine what types of events are logged. The following explains log levels and what level of logging each will product. Log levels in log4j listed by highest level of verbosity to the least :

TRACE - Designates finer-grained informational events than the DEBUG.
DEBUG - The DEBUG logging level will log detailed informational events useful for debugging purposes.
INFO - Designates informational messages that highlight the progress of the application at coarse-grained level. By default, most predefined log4j categories in ZCS are set to the INFO logging level.
WARN - Designates potentially harmful situations.
ERROR - Designates error events that might still allow the application to continue running.
FATAL - The FATAL logging level will only log very severe error events.

Each additional logging level will log more events, including the events belonging to the previous logging levels. For example, enabling the WARN logging level will record log events belonging to the WARN, ERROR, and FATAL logging levels, but will not record events belonging to the INFO, DEBUG, or TRACE levels.

  • Protocol Trace

Protocol trace is available in the following logging categories with TRACE logging level:

zimbra.smtp
zimbra.lmtp
zimbra.soap
zimbra.imap
zimbra.imap-client
zimbra.pop
zimbra.pop-client


Debug log files

  • Debug logs for mailbox get written to :
    • /opt/zimbra/log/mailbox.log for most categories,
    • /opt/zimbra/log/ews.log for ews
    • /opt/zimbra/log/sync.log for sync
    • /opt/zimbra/log/webxml.log for webxml

Reference

  • For all logging levels and other reference material please see -
 https://www.zimbra.com/docs/ne/latest/administration_guide/wwhelp/wwhimpl/js/html/wwhelp.htm#href=860_admin_ne.Logging_Levels.html
http://community.zimbra.com/projects/collaboration_kb/w/log-reference

Ldap

  • Ldap debug logging – Controlled by “ldap_common_loglevel” variable, this writes the config ldap attribute "olcLogLevel".
  • Typically required when trying to troubleshoot a specific request which is taking time or when there some weird scenario which is causing a specific replica sync to fail.
  • Debug logs are written to /var/log/zimbra.log

Examples [as the zimbra user]:

 zmlocalconfig –e ldap_common_loglevel=256
 zmlocalconfig –e ldap_common_loglevel=“stats”

Restarting of ldap is required for the change to take effect:

ldap stop; ldap start

For various log levels please see :

Postfix

  • Typically required when trying to troubleshoot email flow issues. For example specific mail getting dropped or getting stuck in the queue. Or if troubleshooting issues with the postfix process itself, for example postfix crashing. The debug log will help understand what exactly postfix was doing when the crash occurred.
  • Postfix log events goto /var/log/zimbra.log
  • To modify default logging level you edit /opt/zimbra/postfix/conf/master.cf.in

Example, change the following line [default] in /opt/zimbra/postfix/conf/master.cf.in :

smtp      inet  n       -       n       -       -       smtpd

To be the following [as the zimbra user] :

smtp      inet  n       -       n       -       -       smtpd –vv

A restart of postfix required

zmmtactl restart

SpamAssassin

  • Required when troubleshooting issues specific to spamassassin, for example a message being incorrectly dropped or spamassassin taking too long to scan emails.
  • Debug logs get written to /var/log/zimbra.log
  • Debug logging for SpamAssassin is controlled by “zimbraAmavisSALogLevel”, this writes "sa_debug" to /opt/zimbra/conf/amavisd.conf.
  • The value can be 0 or 1, where 1 means debug log enabled and 0 means minimalistic info logging.

Example [as the zimbra user]:

zmprov mcf zimbraAmavisSALogLevel 1 
zmprov ms `zmhostname` zimbraAmavisSALogLevel 1

Amavis

  • Amavisd debug logging - Controlled by zimbra attribute "zimbraAmavisLogLevel", which writes value of "$log_level" in file /opt/zimbra/conf/amavisd.conf
  • Amavis debug logging goes to - /var/log/zimbra.log
  • Typically required when trying to identify issues such as:
    • Amavis causing delay in scanning a message
    • Domain disclaimer not working as expected
    • Amavis causing disruption in mail flow
  • Logging level can be from 0 to 5, where 5 represents maximum debug logging. Default value is 1.

Example [as zimbra user]:

zmprov mcf zimbraAmavisLogLevel 5
zmprov ms `zmhostname` "zimbraAmavisLogLevel" 5

Proxy

  • Controlled by zimbra attribute zimbraReverseProxyLogLevel .
  • Logging events tied to zimbraReverseProxyLogLevel are written to /opt/zimbra/log/nginx.log
  • Typically required to trouble shoot proxy issues like the following situations :
    • Proxy unable to fetch user data
    • Unable to look up correct user's mailbox
    • Nginx process crash
  • The default for zimbraReverseProxyLogLevel is INFO . Possible values are [ per zimbra-attrs.xml ]:
    • debug, info, notice, warn, error, crit, debug_http, debug_mail, debug_core, debug_zimbra

Example to change zimbraReverseProxyLogLevel to DEBUG [as the zimbra user]:

zmprov ms `zmhostname` zimbraReverseProxyLogLevel debug
zmnginxctl restart

It will log a lot of information; you can set it back to info at any time with :

 zmprov ms `zmhostname` zimbraReverseProxyLogLevel info
 zmnginxctl restart

ZCO Logs

For all but the simplest queries, ZCO Dev needs ZCO logs, dumps and event viewer logs.

Here's how to gather them:

Preparing ZCO And Outlooking For Debug Log Gathering

  1. Disable Windows Search
  2. start->run->services->Windows Search->Properties->Stop + “Disabled”
  3. Start Outlook
  4. Disable Foreign Addins i.e. any addins that dont start with "Zimbra"
    • Then, depending on your version of Outlook:
      • OL2003: Tools->Options->Other->Advanced Options->COM Add-Ins->Remove any addins that don't start with "Zimbra"
      • OL2007: Tools->Trust Center->Add-ins->Manage COM Addins->Go->Untick any addins that don't start with "Zimbra"
      • OL2010: File->Options->Add-ins->Manage COM Addins->Go->Untick any addins that don't start with "Zimbra"
      • OL2013: as OL2010
  5. Close Outlook
  6. Empty ZCOLogs Directory (to avoid sending ZCO Dev stale/irrelevant logs)
  7. Set Log Settings (using ZCOLogCtl.exe) to the following :
    • Click the Default button
    • Max size of single log file : 50Mb
    • Preserve Log File Backups : Ticked
    • Max Size of ZCOLogs directory : Un-ticked

ZCO Troubleshooting And Gathering Of Information

Ajcody - There is no listed step on gathering the actual ZCO logs

  1. Start Outlook and reproduce the issue
    • Take detailed notes of the steps
    • If possible, make a note of the time the problem occurred
    • If the problem relates to a particular item, include its ZimbraID and/or Subject
    • Include Screen shots; Screen video if possible
  2. Close Outlook. If Outlook hung, grab a Hang Dump:
    • Task Manager->Processes Tab->Right click OUTLOOK.EXE->Create Dump File
  3. Export Outlook Events from the Windows Event Viewer:
    • Start Event Viewer
    • Select "Windows Logs" | Application from the left pane.
    • In Actions (right pane) select "Filter Current Log..."
    • In the Filter dialog select
    • select "Last 7 days" from the "Logged:" dropdown
    • check Outlook in the "Event sources:"
      • It's recommended to limit output to the current "User:"
    • Click OK
    • Save with "Save Filtered Log File...", and pass us the resulting .evtx file
  4. Check logs before sending to ZCO Dev or Zimbra Support
    • Must cover the repro period (note that each log file name includes the date and time at which a client session started)
    • Check for obvious issues (e.g. server unreachable)
    • ## ERROR ## Usually indicates an unexpected problem
    • ## WARNING ## Usually benign but worth knowing

See also :

Zimbra Web Client Debug Logging

  • Client side issues like a JS error at times needs client side logs to understand what is breaking.
  • Chrome and Firefox both allow "debug mode" which help.
  • Firebug is a very commonly used utility used to monitor calls to the server and the response, the below explains how to set it up collect diagnostics -
 https://wiki.zimbra.com/wiki/Enable_firebug

Quiz Questions

  • Which log file(s) an ZC Administrator should review if there are reports that incoming mail has stopped.
  • You need to enable SOAP TRACE logging for user joe@xyz.com, what is the command needed to enable without the need of restarting service?
  • Mailboxd will not start, having run "zmmailboxdctl start", the service status still shows up as stopped, which file should be reviewed to understand what the problem is.
  • Users randomly get a 502 bad gateway when trying to login into the web client, need to troubleshoot why proxy is reporting an error instead of fetching the data from the user account.
  • Junk messages are not getting marked as Spam, need to review why Spamassassin is not doing a scan of messages.
  • Collect ZCO diagnostic log
Jump to: navigation, search