Ajcody-Logging: Difference between revisions

mNo edit summary
 
(132 intermediate revisions by 5 users not shown)
Line 1: Line 1:
{|  width="100%" border="0"
{{BC|Zeta Alliance}}                        <!-- Note, this will also add [[Category: Zeta Alliance]] to bottom of wiki page. -->
| bgcolor="orange" | [[Image:Attention.png]] - This article is NOT official Zimbra documentation. It is a user contribution and may include unsupported customizations, references, suggestions, or information.
__FORCETOC__                              <!-- Will force a TOC regards of size of article. __NOTOC__ if no TOC is wanted. -->
|}
<div class="col-md-12 ibox-content">
=Logging=             <!-- Normally will reflect page title. Is listed at very top of page. -->
{{KB|{{ZETA}}|{{ZCS 8.5}}|{{ZCS 8.0}}|{{ZCS 7.0}}|}}            <!-- Can only handle 3 ZCS versions. -->


=Logging=


==Actual Logging Homepage==
==Actual Logging Homepage==
Line 11: Line 12:
==Server==
==Server==


===zmdiaglog===
===RFE's Related To Better Logging And Historical Data Of Systems===


As root:
See : [[Ajcody-Testing-Debugging#RFE.27s_Related_To_Better_Logging_And_Historical_Data_Of_Systems]]
/opt/zimbra/libexec/zmdiaglog
 
===Debugging===
 
See : [[Ajcody-Testing-Debugging]] for more complete debugging information.
 
====When Was A ZCS Service Enabled Or Disabled====


This will create a zip file with allot of relevant data for troubleshooting issues.
See : [[Ajcody-Notes-Archive-Discovery#When_Was_A_ZCS_Service_Enabled_Or_Disabled]]


===Syslog Items===
===Syslog Items===


tail /etc/syslog.conf
====Single Server Setup====
 
/etc/syslog.conf should have lines similiar to:
 
[towards bottom of conf file]
  local0.*                -/var/log/zimbra.log
  local0.*                -/var/log/zimbra.log
  auth.*                  -/var/log/zimbra.log
  auth.*                  -/var/log/zimbra.log
  mail.*               -/var/log/zimbra.log
  mail.*                 -/var/log/zimbra.log
 
Make sure syslog allows messages from log4j to be written, log4j doesn't do unix pipes. Specifically, it uses internet domain sockets (514/upd) instead of unix domain sockets (/dev/log).  On a single server setup, this means log4j talking to localhost via 514/udp. Again, /dev/log is never used by log4j (AFAIK, this could change in the future). Adding the "-r" applies for the centralized syslog server as usual, but it also applies for the case where you want log4j data to be logged via syslog locally [single server setup]:


Restarting syslog
<pre>
[root@zimbra sysconfig]# diff -u /etc/sysconfig/syslog.ORIG /etc/sysconfig/syslog
--- /etc/sysconfig/syslog.ORIG  2008-03-25 09:14:28.000000000 -0400
+++ /etc/sysconfig/syslog      2009-06-09 16:01:13.000000000 -0400
@@ -3,7 +3,7 @@
# -r enables logging from remote machines
# -x disables DNS lookups on messages recieved with -r
# See syslogd(8) for more details
-SYSLOGD_OPTIONS="-m 0"
+SYSLOGD_OPTIONS="-r -m 0"
# Options to klogd
# -2 prints all kernel oops messages twice; once for klogd to decode, and
#    once for processing with 'ksymoops'
</pre>
 
Restart syslog:


  /etc/init.d/syslog restart
  /etc/init.d/syslog restart


====Syslog Server====
Setup ZCS to be aware of syslog [we'll restart Zimbra later once all the changes are done]:
 
zmprov mcf zimbraLogToSyslog TRUE
 
This will set the appropriate ldap values (after a zimbra restart) to make log4j.properties be written correctly:
 
/opt/zimbra/conf/log4j.properties will now show some modifications, for example:
 
[snips - DON'T cut cut/paste this info below for your server]
#log4j.rootLogger=INFO,LOGFILE
log4j.rootLogger=INFO,LOGFILE,SYSLOG
...
# Syslog appender
log4j.appender.SYSLOG=org.apache.log4j.net.SyslogAppender
log4j.appender.SYSLOG.SyslogHost=localhost
log4j.appender.SYSLOG.Facility=LOCAL0
log4j.appender.SYSLOG.layout=com.zimbra.common.util.ZimbraPatternLayout
log4j.appender.SYSLOG.layout.ConversionPattern=mailboxd: %-5p [%t] [%z] %c{1} - %m
 
This will actually cause very little to goto syslog though, it's a very minimalist setup. Now adjust /opt/zimbra/conf/log4j.properties.in to log more items to syslog.
 
'''''Note: You may or may not want all these things going to syslog and you may be able to send more/other things to syslog as well.  I've not spend lots of time with log4j and I'm not familiar enough with all the code to know which pieces are using which logger/settings.'''''
 
'''''WARNING: the AUDIT logs may have sensitive data so be careful to protect the logs so that sensitive data is not leaked accidentally.'''''
 
<pre>
[zimbra@zimbra conf]$ diff -u log4j.properties.in.ORIG log4j.properties.in
--- log4j.properties.in.ORIG    2009-06-05 15:31:20.000000000 -0400
+++ log4j.properties.in 2009-06-09 15:54:07.000000000 -0400
@@ -35,7 +35,7 @@
 
# Save zimbra.security to AUDIT appender
log4j.additivity.zimbra.security=false
-log4j.logger.zimbra.security=INFO,AUDIT
+log4j.logger.zimbra.security=INFO,AUDIT,SYSLOG
 
# Syslog appender
log4j.appender.SYSLOG=org.apache.log4j.net.SyslogAppender
@@ -88,7 +88,7 @@
log4j.additivity.zimbra.wbxml=false
log4j.logger.zimbra.wbxml=DEBUG,WBXML
 
-log4j.logger.zimbra=INFO
+log4j.logger.zimbra=INFO,SYSLOG
log4j.logger.zimbra.op=WARN
-log4j.logger.com.zimbra=INFO
+log4j.logger.com.zimbra=INFO,SYSLOG
</pre>
 
At this point, you can restart zimbra for all the changes to be in effect:
 
zmcontrol stop ; zmcontrol start
 
====Centralized Syslog Server====


Please see:
Please see:
Line 36: Line 116:
This will show you how to setup a syslog server and then to configure the "clients" to log to the local filesystem as well as the syslog server.
This will show you how to setup a syslog server and then to configure the "clients" to log to the local filesystem as well as the syslog server.


The syslog events are very minimal with Zimbra. Most of the logging is control by Log4j (tomcat/jetty). There's some conf files in /opt/zimbra/conf/log4j.properties* . You can adjust this file to also output into the syslog environment but syslog has limited sections for "facility". You'll end up getting huge files and g-d only knows how many log events per second.
You'll need to then take into account the information in [[Ajcody-Logging#Single_Server_Setup]] to handle the "Zimbra" part as well as dealing with log4j.


====Centralized Log Server Project Ideas====
=====Centralized Log Server Project Ideas=====


'''Problems:'''
'''Problems:'''
*Logs are to large to manage on each server.
*Logs are to large to manage on each server.
*Want to build a central logging server for all logging events.
*Want to build a central logging server for all logging events.
*The syslog events are very minimal with Zimbra (/var/log/zimbra.log). Most of the logging is control by Log4j (tomcat/jetty). Output from log4j generally goes into the /opt/zimbra/log/ directory. There's some conf files in /opt/zimbra/conf/log4j.properties* for log4j. You can adjust this file to also output into the syslog environment but syslog has very limited choices for "facility". See [http://logging.apache.org/log4net/release/sdk/log4net.Appender.LocalSyslogAppender.SyslogFacility.html Log4j & Syslog facility] for details. You'll end up getting huge files and god knows how many log events per second if you adjusted log4j to dump all zimbra logging into syslog control.
*The syslog events are very minimal with Zimbra (/var/log/zimbra.log). Most of the logging is control by Log4j (tomcat/jetty). Output from log4j generally goes into the /opt/zimbra/log/ directory. There's some conf files in /opt/zimbra/conf/log4j.properties* for log4j. You can adjust this file to also output into the syslog environment but syslog has very limited choices for "facility". See [[Ajcody-Logging#Single_Server_Setup]] and [http://logging.apache.org/log4net/release/sdk/log4net.Appender.LocalSyslogAppender.SyslogFacility.html Log4j & Syslog facility] for details. You'll end up getting huge files and god knows how many log events per second if you adjusted log4j to dump all zimbra logging into syslog control.


General Thought On This:
General Thought On This:
Line 49: Line 129:
Use syslog-ng with mysql for the central log server. Add on php-syslog-ng for comfort.
Use syslog-ng with mysql for the central log server. Add on php-syslog-ng for comfort.


Leave the log4j stuff along and configure syslog-ng to monitor the actual log files as they are generated by log4j in /opt/zimbra/log/* .
Leave the log4j stuff alone and configure syslog-ng to monitor the actual log files as they are generated by log4j in /opt/zimbra/log/* .


Then decrease the log sizes and archiving done on the local servers, i.e. zimbraLogRawLifetime  &  zimbraLogSummaryLifetime .
Then decrease the log sizes and archiving done on the local servers, i.e. zimbraLogRawLifetime  &  zimbraLogSummaryLifetime .
Line 65: Line 145:
*** http://nms.gdd.net/index.php/Installation_Guide
*** http://nms.gdd.net/index.php/Installation_Guide


===Where's It All Go?===


Any stacktrace files will have the process id of the JVM appended to the filename.


In all releases 5.0.x and 4.5.x, this is the main mailbox application log:
===Log Rotation and Removal===
 
====Logs In /opt/zimbra/log Directory====
 
Please look at the output of your crontab - AS ZIMBRA, look for the "Log pruning" line.
 
crontab -l
 
And also the config files for logrotate:
 
/etc/logrotate.conf
 
/etc/logrotate.d/zimbra
 
=====Example From 5.0.11 - Complete ZCS Install Single Server=====
 
crontab for zimbra:
 
<pre>
# Log pruning
#
30 2 * * * find /opt/zimbra/log/ -type f -name \*.log\* -mtime +8 -exec rm {} \; > /dev/null 2>&1
35 2 * * * find /opt/zimbra/log/ -type f -name \*.out.???????????? -mtime +8 -exec rm {} \; > /dev/null 2>&1
 
# Log pruning
#
30 2 * * * find /opt/zimbra/mailboxd/logs/ -type f -name \*log\* -mtime +8 -exec rm {} \; > /dev/null 2>&1
</pre>
 
And the logrotate files:
 
<pre>
$ cat /etc/logrotate.conf
# see "man logrotate" for details
# rotate log files weekly
weekly
 
# keep 4 weeks worth of backlogs
rotate 4
 
# create new (empty) log files after rotating old ones
create
 
# uncomment this if you want your log files compressed
#compress
 
# RPM packages drop log rotation information into this directory
include /etc/logrotate.d


* /opt/zimbra/log/mailbox.log
# no packages own wtmp -- we'll rotate them here
/var/log/wtmp {
    monthly
    minsize 1M
    create 0664 root utmp
    rotate 1
}


In ZCS 5.0.x, the JVM output log and stacktrace files will be in /opt/zimbra/log/:
# system-specific logs may be also be configured here.
</pre>


* /opt/zimbra/log/zmmailboxd.out (jvm output log)
And also:
* /opt/zimbra/log/stacktrace.<pid>


Meaning, changes to /opt/zimbra/conf/log4j.properties.in
<pre>
$ cat /etc/logrotate.d/zimbra
/var/log/zimbra.log {
    daily
    missingok
    notifempty
    create 0644 zimbra zimbra
    postrotate
      killall -HUP syslogd 2> /dev/null || true
      kill -HUP `cat /opt/zimbra/log/swatch.pid 2> /dev/null` 2> /dev/null || true
      kill -HUP `cat /opt/zimbra/log/logswatch.pid 2> /dev/null` 2> /dev/null || true
    endscript
    compress
}


This file, /opt/zimbra/conf/log4j.properties.in , also controls where some events are logged to.
/opt/zimbra/log/myslow.log {
    daily
    missingok
    copytruncate
    rotate 30
    notifempty
    create 0644 zimbra zimbra
    compress
}


Example:
/opt/zimbra/log/logger_myslow.log {
    daily
    missingok
    copytruncate
    notifempty
    create 0660 zimbra zimbra
    compress
    size 5000k
    rotate 7
}


# Appender AUDIT writes to the file "audit.log".
/opt/zimbra/log/clamd.log {
log4j.appender.AUDIT=org.apache.log4j.DailyRollingFileAppender
    daily
'''log4j.appender.AUDIT.File=/opt/zimbra/log/audit.log'''
    missingok
log4j.appender.AUDIT.DatePattern='.'yyyy-MM-dd
    copytruncate
log4j.appender.AUDIT.Append=true
    notifempty
log4j.appender.AUDIT.layout=com.zimbra.common.util.ZimbraPatternLayout
    create 0660 zimbra zimbra
log4j.appender.AUDIT.layout.ConversionPattern=%d %-5p [%t] [%z] %c{1} - %m%n
    postrotate
    kill -HUP `cat /opt/zimbra/log/clamd.pid 2> /dev/null` 2> /dev/null || true
    endscript
    compress
    size 5000k
    rotate 7
}
 
/opt/zimbra/log/zmlogswatch.out {
    daily
    missingok
    copytruncate
    notifempty
    create 0740 zimbra zimbra
    postrotate
      su - zimbra -c "/opt/zimbra/bin/zmlogswatchctl stop"
      su - zimbra -c "/opt/zimbra/bin/zmlogswatchctl start"
    endscript
    rotate 5
    compress
}


===Log Rotation and Removal===
/opt/zimbra/log/zmswatch.out {
    daily
    missingok
    copytruncate
    notifempty
    create 0740 zimbra zimbra
    postrotate
      su - zimbra -c "/opt/zimbra/bin/zmswatchctl stop"
      su - zimbra -c "/opt/zimbra/bin/zmswatchctl start"
    endscript
    rotate 5
    compress
}
 
/opt/zimbra/log/zmmtaconfig.log {
    daily
    missingok
    copytruncate
    notifempty
    create 0740 zimbra zimbra
    postrotate
      su - zimbra -c "/opt/zimbra/bin/zmmtaconfigctl restart"
    endscript
    rotate 5
    compress
}
 
/opt/zimbra/log/nginx.log {
    daily
    missingok
    notifempty
    create 0644 zimbra zimbra
    postrotate
      kill -USR1 `cat /opt/zimbra/log/nginx.pid 2> /dev/null` 2> /dev/null || true
    endscript
    rotate 7
    compress
}
 
/opt/zimbra/log/zmconvertd.log {
    daily
    missingok
    copytruncate
    notifempty
    create 0644 zimbra zimbra
    rotate 7
}
 
/opt/zimbra/zmstat/zmstat.out {
    daily
    missingok
    copytruncate
    rotate 7
    notifempty
    compress
}
</pre>
 
====Openldap Logs - In /opt/zimbra/openldap-data/====
 
Ldap logs files in /opt/zimbra/openldap-data/ and in /opt/zimbra/data/ldap/* subdirectories [ZCS 6+] that are like log.0000000001 , log.0000000002 , etc.
 
This logpurge is controlled by:
 
<pre>
$ cd /opt/zimbra/openldap/etc/openldap/
[zimbra@mail3 openldap]$ grep logpurge *
master-accesslog-overlay.conf:logpurge 07+00:00 01+00:00
</pre>
 
=====Logpurge Directive Description=====
 
From [http://www.zytrax.com/books/ldap/ch6/accesslog.html Accesslog Chapter]


Please look at the output of your crontab - AS ZIMBRA, look for the "Log pruning" line.
logpurge Directive


crontab -l
logpurge age interval


And also the config files for logrotate:
Defines both the maximum age for log entries to be retained in the database and how often to scan the database for old entries. Bothage and interval are specified as a time span in days, hours, minutes, and seconds. The time format is [ddd+]hh:mm[:ss], for example, the days and seconds components are optional but hours and minutes are required. Except for days, which can be up to 5 digits, each numeric field must be exactly two digits. Example:


/etc/logrotate.conf
# the log database will be scanned every day
# entries older than two days will be deleted.
logpurge 2+00:00 1+00:00


/etc/logrotate.d/zimbra
When using a log database that supports ordered indexing on generalizedTime attributes, specifying an eq index on the reqStart attribute will increase the performance of purge operations.


===What's up with all the logs?===
===What's up with all the logs?===


Logging page resources:
====Other Logging Page Resources====


http://www.zimbra.com/docs/ne/latest/administration_guide/9_Monitoring.14.1.html
http://www.zimbra.com/docs/ne/latest/administration_guide/9_Monitoring.14.1.html


http://wiki.zimbra.com/index.php?title=Server_Monitoring
http://wiki.zimbra.com/index.php?title=Server_Monitoring
http://wiki.zimbra.com/index.php?title=Monitoring_Zimbra_Servers


http://wiki.zimbra.com/index.php?title=Log_Files
http://wiki.zimbra.com/index.php?title=Log_Files
Line 120: Line 373:
====The Bread And Butter Logs====
====The Bread And Butter Logs====


* '''/opt/zimbra/log/mailbox.log'''
=====/opt/zimbra/log/mailbox.log - where most of your mailbox store activity is logged=====
** This log is a mailboxd log4j server log containing the logs from the mailbox server. This includes the mailbox store, LMTP server, IMAP and POP servers, and Index server.
 
* '''/opt/zimbra/log/zmmailbox.out'''
:* This log is the mailboxd log4j server log containing the logs from the mailbox server. This includes activity from the mailbox store, LMTP server, IMAP and POP servers, and Index server.
** Mailstore not coming up and nothing is being logged in mailbox.log, check here for errors.
:** Location:  '''/opt/zimbra/log/mailbox.log'''
* '''/var/log/zimbra.log'''
 
** The Zimbra syslog details the activities of the Zimbra MTA (Postfix, amavisd, antispam, antivirus), Logger, Authentication (cyrus-sasl), and Directory (OpenLDAP). By default LDAP activity is logged to Zimbra.log.
=====/opt/zimbra/log/zmmailboxd.out - mailboxd/jvm output log=====
* '''/opt/zimbra/db/data/YOURHOSTNAME.err'''
** This is the message store database error log.
* '''/var/log/messages'''


===Increase Logging===
:* Mailstore not coming up and nothing is being logged in mailbox.log, check here for errors.
:** Location:  '''/opt/zimbra/log/zmmailboxd.out'''
 
=====/opt/zimbra/log/stacktrace.<pid> - stacktrace logs=====
 
:* stacktraces related to mailboxd
:** Location:  '''/opt/zimbra/log/stacktrace.<pid> '''
 
=====/opt/zimbra/db/data/YOURHOSTNAME.err - errors for MySQL - ZCS 4.x & ZCS 5.x=====
 
:* This is the message store database error log.
:** Location:  '''/opt/zimbra/db/data/YOURHOSTNAME.err'''
 
=====/var/log/zimbra.log - mta and system status log, postfix, amavisd=====
 
:* The Zimbra syslog details the activities of the Zimbra MTA (Postfix, amavisd, antispam, antivirus), Logger, Authentication (cyrus-sasl), and Directory (OpenLDAP). By default LDAP activity is logged to Zimbra.log.
:** Location:  '''/var/log/zimbra.log'''
 
=====/var/log/messages - mta and OS related log events=====
 
:* Probably has mta logging events [which also show in /var/log/zimbra.log] and also log events related to your OS
:** Location:  '''/var/log/messages'''
 
=====/opt/zimbra/log/mysql_error.log - problems with MySQL=====
 
:* If there is data corruption or another problem causing direct mysql errors, events will be logged here.
:** '''/opt/zimbra/log/mysql_error.log'''
 
=====/opt/zimbra/log/myslow.log - slow db/MySQL queries=====
 
:* If certain search requests are taking longer to complete than others, they will be logged here.
:** '''/opt/zimbra/log/myslow.log'''
 
====Other Logs====
 
=====/opt/zimbra/log/audit.log - authentication events=====


====Increase Logging Per User====
:* A log of all admin actions taken as well as logins to the server.
:** '''/opt/zimbra/log/audit.log'''


Depending on the ZCS version the below will clear all the per account loggers on zimbra reboot. IMAP is used for these examples.
:Want to see all uses of the "View Mail" button from the admin console?  Will have the IP address, User Agent of the browser, user that the admin is logged in as, and the account that they are viewing.


* addAccoutLogger
:<pre> grep DelegateAuth /opt/zimbra/log/audit.log </pre>
zmprov aal user@domain.com zimbra.imap debug
* removeAccountLogger
zmprov ral user@domain.com zimbra.imap


Example usuage:
=====/opt/zimbra/log/clamd.log - antivrius db=====


zmprov aal user@domain.com zimbra.soap debug
:* Status and checks for clamav
:** '''/opt/zimbra/log/clamd.log'''


And tail the log file, while attempting to login to the AJAX client.:
:If your clamav db is out of date, you'll see a log event here like this:


  tail -f /opt/zimbra/log/mailbox.log
: <pre>
: LibClamAV Warning: **************************************************                               
: LibClamAV Warning: *** The virus database is older than 7 days!  ***                               
: LibClamAV Warning: ***  Please update it as soon as possible.    ***                               
: LibClamAV Warning: **************************************************
</pre>


See [[Ajcody-Logging#Log4J_Variables]] for variables that can be used.
=====/opt/zimbra/log/convertd.log - attachment conversion=====


====Increase Logging Per Server====
:* If your having problems with your attachments and the ability to view them in ZWC, check here for errors.
:** '''/opt/zimbra/log/convertd.log'''


If you want to do it globally edit this file:
=====/opt/zimbra/log/freshclam.log - clam antivirus updates=====


vi /opt/zimbra/conf/log4j.properties
:* Shows the actual attempt to update the clamav updates.
:** '''/opt/zimbra/log/freshclam.log'''


Add a line at the end that's similar to something like this: log4j.logger.zimbra.imap=DEBUG.
=====/opt/zimbra/log/spamtrain.log - spam/ham training details=====


No restart of any service is needed, '''BUT''' if something happens that causes a regeneration of the log4j.properties file then your changes will be overwritten. A regeneration of this file takes the contents of /opt/zimbra/conf/log4.properties.in .
:* Spam/Ham training details are logged here. Errors are also logged here related to this service.
:* From a default single ZCS configurations crontab [zimbra]:


To permanently make a change, modify the /opt/zimbra/conf/log4.properties.in file, then a restart would be necessary.
:<pre>
: # Spam training
: #:
: 0 :23 * * * /opt/zimbra/bin/zmtrainsa >> /opt/zimbra/log/spamtrain.log 2>&1
: #
: # Spam training cleanup
: #
: 45 23 * * * /opt/zimbra/bin/zmtrainsa --cleanup >> /opt/zimbra/log/spamtrain.log 2>&1
</pre>


$ zmcontrol stop
=====/opt/zimbra/log/sync.log - zimbra mobile and activesync=====
$ zmcontrol start


or you could run this on the server:
:* Your mobile devices will basically log here.
:** '''/opt/zimbra/log/sync.log'''


$ zmmtaconfig mailboxd
=====/opt/zimbra/log/synctrace.log - zimbra mobile and activesync=====


Wait for a minute for the server to pick up your change.
:* Still looking for description that describes difference with sync.log
:** '''/opt/zimbra/log/synctrace.log'''


See [[Ajcody-Logging#Log4J_Variables]] for variables that can be used.
=====/opt/zimbra/log/syncstate.log - no description found=====


====Log4J Variables====
:* No description found.
 
:** '''/opt/zimbra/log/syncstate.log'''
"zimbra.misc" logger. For all events that don't have a specific-category.
   
"zimbra.index" logger. For indexing-related events.
   
"zimbra.index.lucene" logger. For logging of low-level lucene operations (debug-level only)
   
"zimbra.searchstat" logger.  For logging statistics about what kinds of searches are run
   
"zimbra.redolog" logger. For redolog-releated events.
   
"zimbra.lmtp" logger. For LMTP-related events.
   
"zimbra.smtp" logger. For SMTP-related events.
   
"zimbra.nio" logger. For NIO-related events.
   
"zimbra.imap" logger. For IMAP-related events.
   
"zimbra.pop" logger. For POP-related events.
   
"zimbra.mailbox" logger. For mailbox-related events.
   
"zimbra.calendar" logger. For calendar-related events.
   
"zimbra.im" logger. For instant messaging-related events.
   
"zimbra.account" logger. For account-related events.
   
"zimbra.gal" logger. For account-related events.
   
"zimbra.ldap" logger. For ldap-related events.
   
"zimbra.security" logger. For security-related events


"zimbra.soap" logger. For soap-related events
=====/opt/zimbra/log/wbxml.log - no description found=====
   
"zimbra.test" logger. For testing-related events
   
"zimbra.sqltrace" logger. For tracing SQL statements sent to the database
   
"zimbra.dbconn" logger. For tracing database connections
   
"zimbra.perf" logger. For logging performance statistics
   
"zimbra.cache" logger. For tracing object cache activity
   
"zimbra.filter" logger. For filter-related logs.
   
"zimbra.session" logger. For session- and notification-related logs.
   
"zimbra.backup" logger. For backup/restore-related logs.
   
"zimbra.system" logger. For startup/shutdown and other related logs.
   
"zimbra.sync" logger. For sync client interface logs.
   
"zimbra.synctrace" logger. For sync client interface logs.
   
"zimbra.syncstate" logger. For sync client interface logs.
   
"zimbra.wbxml" logger. For wbxml client interface logs.
   
"zimbra.extensions" logger. For logging extension loading related info.
   
"zimbra.zimlet" logger. For logging zimlet related info.
   
"zimbra.wiki" logger. For wiki and document sharing.
   
"zimbra.op" logger. Logs server operations
   
"zimbra.dav" logger. Logs dav operations
   
"zimbra.io" logger.  Logs file IO operations.
   
"zimbra.datasource" logger.  Logs data source operations.
   
remote management.
   
public static final Log rmgmt = LogFactory.getLog("zimbra.rmgmt");
   
"zimbra.webclient" logger. Logs ZimbraWebClient servlet and jsp operations.
   
"zimbra.scheduler" logger.  Logs scheduled task operations.
   
"zimbra.store" logger.  Logs filesystem storage operations.
   
"zimbra.fb" logger.  Logs free/busy operations.
   
"zimbra.purge" logger.  Logs mailbox purge operations.
   
"zimbra.mailop" logger.  Logs changes to items in the mailbox.


===Thread Dump To Mailbox.log===
:* No description found.
:** '''/opt/zimbra/log/wbxml.log'''


Run this command as root
=====/opt/zimbra/log/zmlogswatch.out - =====


for i in 1 2 3; do /opt/zimbra/libexec/zmmailboxdmgr threaddump; sleep 30; done


Hopefully the zmmailboxd.out will reflect the thread dumps.
=====/opt/zimbra/log/zmmtaconfig.log - =====


=====A Better Way To Thread Dump?=====


Usage of the below script:
=====/opt/zimbra/log/zmmyinit.log - =====


::$0 FILENAME -dl [-frames #_ stack_frames] [-id REGEXP] [-all]
::$0 FILENAME -dt [-frames #_ stack_frames] [-sort id|state] [-stack REGEXP] [-id REGEXP] [-state REGEXP] [-waiting NUMBER]


::frames:  controls the # lines of stack trace included in the output
=====/opt/zimbra/log/zmsetup.`date`-'pid'.log - =====
::id:      only include where the id matches REGEXP


::all:    include ALL locks (not just ones that other threads are blocked-on)


::sort:    controls the order threads are printed out (locks always printed in lock ID order)
=====/tmp/install.log - =====
::stack:  only include threads where the thread's stack output matches REGEXP
::state:  only include threads where the thread state (e.g. RUNNABLE) matches REGEXP
::waiting: only inclure threads are blocking other threads


Examples:
After install, gets moved to : /opt/zimbra/.install/


::$0 threads.txt -dt -stack MailboxIndex.java -state RUNNABLE -f 20
=====/tmp/zmsetup.log - =====
::::-- dumps all RUNNABLE threads with MailboxIndex.java on the stack (1st 20 lines of the stack)


::$0 threads.txt -dl -f 0
After install, gets moved to : /opt/zimbra/log/zmsetup.log.[some numbers]
::::-- dumps the list of locks in the system that are blocking at least one thread


::$0 threads.txt -dt -f 0 -sort state
=====/opt/zimbra/httpd_access.log - =====
::::-- dumps a list of all the threads in the system and tells you their run state


::$0 threads.txt -dt -f 20 -w 1
::::-- dumps a list of all the threads in the system that are blocking other threads (quick way to hunt for contention)


Working Example:
=====/opt/zimbra/httpd_error.log - =====


:: threadDumpAnalyzer.pl zmmailboxd.out -dl -f 100 ~/Desktop
====New Data Directory In ZCS6+====


======threadDumpAnalyzer.pl Script======
In ZCS 6+ , there's a directory to organize 'data' from our various components.


<pre>
<pre>
#!/usr/bin/perl -w
[root@mail3 data]# pwd
#
/opt/zimbra/data
# ***** BEGIN LICENSE BLOCK *****
 
#
[root@mail3 data]# ls -F
# Zimbra Collaboration Suite Server
altermime/  amavisd/  clamav/  dspam/ ldap/ postfix/ tmp/
# Copyright (C) 2004, 2005, 2006, 2007 Zimbra, Inc.
#
# The contents of this file are subject to the Yahoo! Public License
# Version 1.0 ("License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
# http://www.zimbra.com/license.
#
# Software distributed under the License is distributed on an "AS IS"
# basis, WITHOUT WARRANTY OF ANY KIND, either express or implied.
#
# ***** END LICENSE BLOCK *****
#
use strict;
use warnings;
use Getopt::Long;


my %locks;                      # by LockID - hash w/ 'owner','type'
</pre>
my %threads;          # by threadId - hash w/ 'state' 'waitingOnLock'


my $filename = $ARGV[0];
This information could prove useful as well with trouble-shooting and debugging problems.
my ($dumpLocks, $dumpThreads, $searchThreadStack, $searchThreadId, $stackFrames, $sort, $filterByState, $allLocks);
my ($waiting);


$stackFrames = 10;
===Increase Logging===
$sort = "state";


GetOptions(
====Increase Logging Per User====
          "dl" => \$dumpLocks,
          "dt" => \$dumpThreads,
          "frames=s" => \$stackFrames,
          "sort=s" => \$sort,
          "id=s" => \$searchThreadId,
          "state=s" => \$filterByState,
          "stack=s" => \$searchThreadStack,
          "waiting=s" => \$waiting,
          "all" => \$allLocks,
          );


sub dumpLocks();
Depending on the ZCS version the below will clear all the per account loggers on zimbra reboot. IMAP is used for these examples.
sub handleThread();
sub dumpThreads();
sub usage();
sub readFile($);
sub getBlockedThreads($);


if (!defined $filename) {
* addAccoutLogger
  usage();
zmprov aal user@domain.com zimbra.imap debug
}
* removeAccountLogger
zmprov ral user@domain.com zimbra.imap


readFile($filename);
Example usuage:


if (defined $dumpLocks) {
zmprov aal user@domain.com zimbra.soap debug
  dumpLocks();
} elsif (defined $dumpThreads) {
  dumpThreads();
} else {
  usage();
}


exit(0);
And tail the log file, while attempting to login to the AJAX client.:


sub usage() {
tail -f /opt/zimbra/log/mailbox.log
  my $usage = <<END_OF_USAGE;
   
USAGE:
    $0 FILENAME -dl [-frames #_ stack_frames] [-id REGEXP] [-all]
    $0 FILENAME -dt [-frames #_ stack_frames] [-sort id|state] [-stack REGEXP] [-id REGEXP] [-state REGEXP] [-waiting NUMBER]


    frames:  controls the # lines of stack trace included in the output
See [[Ajcody-Logging#Log4J_Variables]] for variables that can be used.
    id:      only include where the id matches REGEXP


    all:    include ALL locks (not just ones that other threads are blocked-on)
====To See What Accounts Have Extended Logging Enabled====


    sort:    controls the order threads are printed out (locks always printed in lock ID order)
There's the gaal option for zmprov . It list the accounts on a per mailstore basis :
    stack:  only include threads where the thread's stack output matches REGEXP
    state:  only include threads where the thread state (e.g. RUNNABLE) matches REGEXP
    waiting: only inclure threads are blocking other threads


  Examples:
<pre> getAllAccountLoggers(gaal) [-s/--server hostname]</pre>


    $0 threads.txt -dt -stack MailboxIndex.java -state RUNNABLE -f 20
<pre>
        -- dumps all RUNNABLE threads with MailboxIndex.java on the stack (1st 20 lines of the stack)
[zimbra@zcs806 ~]$ zmprov gaal `zmhostname`
# name test01@zcs806.domain.com
zimbra.imap=debug
</pre>


    $0 threads.txt -dl -f 0
=====Bugs And RFE's=====
        -- dumps the list of locks in the system that are blocking at least one thread


    $0 threads.txt -dt -f 0 -sort state
I filed the following RFE:
        -- dumps a list of all the threads in the system and tells you their run state


    $0 threads.txt -dt -f 20 -w 1
* "Allow "secondary" logging file for zmprov addAccountLogger"
        -- dumps a list of all the threads in the system that are blocking other threads (quick way to hunt for contention)
** http://bugzilla.zimbra.com/show_bug.cgi?id=37780
   


END_OF_USAGE
====Increase Logging Per Server====
  die $usage;
}


sub mySort($$) {
If you want to do it globally edit this file:
  my ($a, $b) = @_;
  if ($sort eq "state") {
    my $state1 = $threads{$a}{state};
    my $state2 = $threads{$b}{state};
    if (!defined $state1) {
      return 1;
    } elsif (!defined $state2) {
      return -1;
    }
    return $state1 cmp $state2;
  } else {
    return $a cmp $b;
  }
}


sub padToWidth($$) {
vi /opt/zimbra/conf/log4j.properties
  (my $str, my $width) = @_;
  if (!defined($str)) {
    $str = "";
  }
  return sprintf "%-".$width.".".$width."s", $str;
}


sub formatStackTrace($$) {
Add a line at the end that's similar to something like this: log4j.logger.zimbra.imap=DEBUG.  
  my ($stack, $indent) = @_;
  my $ret;
  if ($stackFrames == 0) { return ""; }
  my $num = $stackFrames;
 
  foreach my $line (split /\n/, $stack) {
    if ($num <= 0) {
      return $ret;
    }
    $ret .= $indent.$line."\n";
    $num--;
  }
  return $ret;
}


sub formatLock($) {
No restart of any service is needed, '''BUT''' if something happens that causes a regeneration of the log4j.properties file then your changes will be overwritten. A regeneration of this file takes the contents of /opt/zimbra/conf/log4.properties.in .  
  my $lockId = shift;
  my $output = "$lockId - ";
  my $numData = 0;
  for my $data (keys %{ $locks{$lockId}}) {
    $output .= " $data=";
    $output .= $locks{$lockId}{$data};
    $numData++;
  }
  if ($numData == 0) { $output .= " UNKNOWN"; }
  return $output;
}


sub formatThread($) {
To permanently make a change, modify the /opt/zimbra/conf/log4.properties.in file, then a restart would be necessary.
  my $threadId = shift;
  if (!defined $threadId) { return ""; }
  my $foo = padToWidth($threadId, 50);
  if (!defined($foo) || $foo eq "") {
    $foo = "ASDF";
  }
  my $bar = $threads{$threadId}{state};
  if (!defined($bar) || $bar eq "") {
    $bar = "HJKL";
  }
  my $ret = $foo.$bar."\n";
  if (defined $threads{$threadId}{waitingOnLock}) {
    $ret .= "\tWaiting for: ".formatLock($threads{$threadId}{waitingOnLock})."\n";
  }


  my @blockedThreads = getBlockedThreads($threadId);
$ zmcontrol stop
  for my $blockedThread (sort @blockedThreads) {
  $ zmcontrol start
    $ret .= "\t$blockedThread is waiting on this thread\n";
  }
 
  $ret .= formatStackTrace($threads{$threadId}{stack}, "\t ");
  if ($stackFrames > 0) {
    $ret .= "\n";
  }
  return $ret;
}


# given a lockId, return a list of the threads that are blocked on it
or you could run this on the server:
sub getLockWaiters($) {
  my $lockId = shift;
  my @ret;


  foreach my $threadId ( keys %threads ) {
$ zmmtaconfig mailboxd
    if (defined $threads{$threadId}{waitingOnLock}) {
      if ($threads{$threadId}{waitingOnLock} eq $lockId) {
        push @ret, $threadId;
      }
    }
  }
  return @ret;
}


sub getLockOwner($) {
Wait for a minute for the server to pick up your change.
  my $lockId = shift();
  if (defined $locks{$lockId}) {
    if (defined $locks{$lockId}{owner}) {
      return $locks{$lockId}{owner};
    } else {
      return "";
    }
  } else {
    return "";
  }
}


# given a threadId, get a list of all other threads that are blocked
See [[Ajcody-Logging#Log4J_Variables]] for variables that can be used.
# on locks it is holding
sub getBlockedThreads($) {
  my $threadId = shift;
  my @ret;
  foreach my $lockId ( sort keys %locks ) {
    if (getLockOwner($lockId) eq $threadId) { # a lock we own
      my @blockedThreads = getLockWaiters($lockId);
      foreach my $blockedThread (@blockedThreads) {
        push @ret, $blockedThread;
        push @ret, getBlockedThreads($blockedThread); #recurse!
      }
    }
  }
  return @ret;
}


sub dumpLocks() {
====Log4J Variables====
  foreach my $lockId ( sort keys %locks ) {
    my $ret = "";
    my $numWaiters = 0;
    $ret .= "LOCK: $lockId  ";
    for my $data (keys %{ $locks{$lockId}}) {
      $ret .= "$data=";
      $ret .= $locks{$lockId}{$data};
      $ret .= ", ";
    }
    $ret .= "\n";
    foreach my $threadId ( keys %threads ) {
      if (defined $threads{$threadId}{waitingOnLock}) {
        if ($threads{$threadId}{waitingOnLock} eq $lockId) {
          $ret .= "\tThread $threadId is waiting for this lock\n";
          $numWaiters++;
        }
      }
    }
    $ret .= formatStackTrace($threads{getLockOwner($lockId)}{stack}, "\t");


    if ((!defined $searchThreadId) || ($lockId =~ /$searchThreadId/)) {
An Official page has been made for this topic now per my RFE request, [http://bugzilla.zimbra.com/show_bug.cgi?id=41894 bug 41894]
      if ($numWaiters > 0 || defined $allLocks) {
        if ($stackFrames > 0) {
          $ret .= "\n";
        }
        print $ret;
      }
    }
  }
}


sub dumpThreads() {
Please see:
  foreach my $threadId ( sort { mySort($a, $b) } keys %threads ) {
    if (!defined $threadId) {
      # continue
    } elsif (defined $searchThreadStack && !($threads{$threadId}{stack} =~ /$searchThreadStack/)) {
      # continue
    } elsif (defined $searchThreadId && !($threadId =~ /$searchThreadId/)) {
      # continue
    } elsif (defined $filterByState && !($threads{$threadId}{state} =~ /$filterByState/)) {
      # continue
    } elsif (defined $waiting && (getBlockedThreads($threadId) < $waiting)) {
      # continue
    } else {
      print formatThread($threadId);
    }
  }
}


sub readFile($) {
* [[Using_log4j_to_Configure_mailboxd_Logging]]
  my $filename = shift;
 
  open IN, "<$filename" or die "couldn't open $filename";
 
  my @curThread;


  while (<>) {
====OpenLDAP Logging Levels====
    chomp;
    s/\r//g;
    if ($_ eq "") {
      if (@curThread) {
        my $threadId;
        my @locksHeld;
        my $waitingOnLock;
        my $threadState;
        my $output;
       
        my $firstLineState;
       
        # 1stline
        my $line = shift @curThread;
        $output .= $line."\n";
        if ($line =~ /"(.*)"/) {
          $threadId = $1;
          if ($line =~/nid=0x[0-9a-f]+\s([a-zA-Z\s\.()]+)/) {
            $threads{$threadId}{state} = $1;
          }


        } else {
Please see [[King0770-Notes-Change-LDAP-Log-Levels]]
          $threadId = $line;
        }
        if ($threadId eq "") { $threadId = "none"; }
       
        # 2nd line
        $line = shift @curThread;
        if (defined $line) {
          $output .= $line."\n";
          if ($line =~ /State: ([A-Z_]+)/) {
            $threadState = $1;
            $threads{$threadId}{state} = $1;
          }
         
          foreach $line (@curThread) {
            $output .= $line."\n";
            if ($line =~ /locked <(0x[0-9a-f]+)>\s?(.*)?/) {
              push @locksHeld, $1;
             
              $locks{$1}{owner} = $threadId;
              $locks{$1}{type} = $2;
             
            } elsif ($line =~ /- waiting to lock <(0x[0-9a-f]+)>/) {
              $waitingOnLock = $1;
              $threads{$threadId}{waitingOnLock} = $1;
            } elsif ($line =~ /- waiting to lock <(0x[0-9a-f]+)>/) {
              $waitingOnLock = $1;
              $threads{$threadId}{waitingOnLock} = $1;             
            }
          }
        } else {
          $threads{$threadId}{state} = "unknown";
        }
       
        $threads{$threadId}{stack} = $output;
        undef @curThread;
      }
    } else {
      push @curThread, $_;
    }
  }
 
  close IN;
}
</pre>


===Want To See What's In Redolog Files===
===Want To See What's In Redolog Files===
Line 675: Line 606:
Pick the right redolog file, either redo.log or one of the files under archive/, based on timestamp.
Pick the right redolog file, either redo.log or one of the files under archive/, based on timestamp.


See [[Ajcody-Notes#Redolog_Files]] for more details.
See [[Ajcody-Backup-Restore-Issues#Redolog_Files]] for more details.


===Internal Zimbra Charting - zmstat-chart===
===Internal Zimbra Charting - zmstat-chart===
Line 684: Line 615:
** http://bugzilla.zimbra.com/show_bug.cgi?id=12196
** http://bugzilla.zimbra.com/show_bug.cgi?id=12196
* "zmstat - expand documentation"
* "zmstat - expand documentation"
*** See:
**** http://wiki.zimbra.com/index.php?title=Zmstats
**** http://wiki.zimbra.com/index.php?title=Server_Monitoring
** http://bugzilla.zimbra.com/show_bug.cgi?id=32135
** http://bugzilla.zimbra.com/show_bug.cgi?id=32135
* "zmstat-chart - graphing besides just one day"
* "zmstat-chart - graphing besides just one day"
Line 704: Line 638:
Please see the following:
Please see the following:


http://www.zimbra.com/docs/ne/latest/administration_guide/A_app-command-line.17.24.html
[http://www.zimbra.com/docs/os/latest/administration_guide/A_app-command-line.13.1.html zmstat-chart CLI Administration Guide]


http://wiki.zimbra.com/index.php?title=Server_Monitoring
[http://www.zimbra.com/docs/os/latest/administration_guide/A_app-command-line.13.20.html#1171137 zmstat-chart-config CLI Administration Guide]
 
[http://www.zimbra.com/docs/os/latest/administration_guide/A_app-command-line.13.21.html#1232962 zmstatctl CLI Administration Guide]
 
[[Zmstats|Zmstats Wiki]]
 
[[Server_Monitoring|Server Monitoring Wiki]]


http://www.zimbra.com/docs/os/latest/administration_guide/9_Monitoring.12.1.html
[http://www.zimbra.com/docs/os/latest/administration_guide/9_Monitoring.12.1.html Monitoring Administration Guide]


[http://www.jfree.org/jfreechart/ JFreeChart] is the project/binary behind the charting tool. You'll find the jar files here [ZCS 5.0.9 example]:
[http://www.jfree.org/jfreechart/ JFreeChart] is the project/binary behind the charting tool. You'll find the jar files here [ZCS 5.0.9 example]:
Line 763: Line 703:
====Like To Have zmstat-chart Data Integrated With Zimbra====
====Like To Have zmstat-chart Data Integrated With Zimbra====


Ok, this is just down and dirty. You'll need to handle security issues yourself.
I've made an RFE for this integration:
 
* "zmstat - integrate webpage setup and allow configuration of chart run"
** http://bugzilla.zimbra.com/show_bug.cgi?id=32139
 
=====With Zimbra Jetty=====
 
:'''Note:'''
::'''The default Jetty behavior will NOT display directory listing. You'll need a valid html type file in the directory target.'''
::'''I would recommend the apache solution over this one as you can avoid hitting performance issues this might cause.'''
 
Dump some graphing directories in there:
 
zmstat-chart -s /opt/zimbra/zmstat -d /opt/zimbra/jetty/webapps/zimbra/downloads/zmstat-chart/`date +%F-%H-%M`
 
You'll see there's the directory and when you go into it - there's your charts.
 
You can now place this command in your crontab to run on a schedule basis.
 
Things you'll want to check before deploying:
 
#Place charting directory in a location where space is available and will not create a risk to the mail services if it's full.
#Look at the zmstat-chart options to see what other parameters you want to use - especially if it's going in your crontab.
#Make sure your following and security guidelines for your environment.
 
You'll be able to view the data with the following url:
 
http://hostname/zimbra/downloads/zmstat-chart/
 
=====With Zimbra Apache Or Non-Zimbra Server With Apache=====
 
======Before You Proceed======
 
Things to check or confirm first before deploying:
 
* Place your directory root [/opt/zimbra/support in this example] in a location where space is available and will not create a risk to the mail services if it's full.
* If you'll be using NFS to have a centralized storage point:
** Your NFS mount point on the Zimbra servers would be /opt/zimbra/support in this example.
** You'll be exporting /opt/zimbra/support in this example from the NFS server.
** If this is a non-zimbra server, copy the zimbra entry from a zimbra server's /etc/passwd and /etc/group into the non-zimbra servers passwd and group file. This way, if NFS is also used the UID/GID matches.
* Apache Directory variable will be /opt/zimbra/support/data in this example.
* Apache .htpasswd location will be /opt/zimbra/support/.htpasswd in this example, putting it above of the apache Directory variable.
** Make sure your following the security guidelines for your environment.
* Look at the zmstat-chart options to see what other parameters you want to use - especially if it's going in your crontab.
 
======Directory Layout======
 
As root:
mdir -p /opt/zimbra/support/data
cd /opt/zimbra/support/data
for i in `/opt/zimbra/bin/zmprov gas`
  do
  mkdir $i
  done
 
If your not on a ZCS server, you'll be manually creating directories named after the output of zmhostname from each of your ZCS servers.


  mkdir /opt/zimbra/zmstat-chart
Now, create subdirectory paths under each zmhsotname directory.
  chown zimbra:zimbra /opt/zimbra/zmstat-chart
 
  cd /opt/zimbra/support/data/
for i in `ls`
  do
  mkdir $i/debug $i/logs $i/zmstat $i/tops
  done
 
Set permissions on the directories we made:
 
  chown -R zimbra:zimbra /opt/zimbra/support
chmod -R 755 /opt/zimbra/support
 
With this structure and with NFS setup, you'll have a common repository that will have a standard path to use on a per server basis. Example :
 
script-command -PathOption /opt/zimbra/support/data/`zmhostname`/TYPE_OF_DATA/`date +%F-%H-%M`
 
======Apache Configuration======


Let's configure apache to show this directory.
Let's configure apache to show this directory.


* For ZCS Apache Setup:
  vi /opt/zimbra/conf/httpd.conf
  vi /opt/zimbra/conf/httpd.conf
  ** Towards the end, add the following**
  ** Towards the end, add the following**
  # Include zmstat-chart directory
  # Include zmstat-chart directory
  Include /opt/zimbra/httpd/conf/extra/zmstat-chart.conf
  Include /opt/zimbra/httpd/conf/extra/zimbra-support.conf
 
* Non-ZCS Server With Apache. Check that there's an existing Include directive that will see our zimbra-support.conf:
vi /etc/httpd/conf/httpd.conf
** Look for something that will allow the following**
** /etc/httpd/conf.d/zimbra-support.conf to be used**
** Your distro might differ on apache paths as well**
** as Include statement for other conf files**
Include conf.d/*.conf


Let's now make the apache conf file for zmstat-chart.conf
Let's now make the apache conf file for zimbra-support.conf


  vi /opt/zimbra/httpd/conf/extra/zmstat-chart.conf
* For ZCS Apache Setup:
  Alias /zmstat-chart "/opt/zimbra/zmstat-chart"
  vi /opt/zimbra/httpd/conf/extra/zimbra-support.conf
  <Directory "/opt/zimbra/zmstat-chart">
  Alias /support "/opt/zimbra/support/data"
  <Directory "/opt/zimbra/support/data">
  AuthName "Secure Area For Zimbra Support"
  AuthType Basic
  AuthUserFile /opt/zimbra/support/.htpasswd
  require valid-user
   Options Indexes
   Options Indexes
   IndexOptions FancyIndexing VersionSort
   IndexOptions FancyIndexing VersionSort
Line 786: Line 811:
   Allow from all
   Allow from all
  </Directory>
  </Directory>
* Non-ZCS Server With Apache:
vi /etc/httpd/conf.d/zimbra-support.conf
Alias /support "/opt/zimbra/support/data"
<Directory "/opt/zimbra/support/data">
  AuthName "Secure Area For Zimbra Support"
  AuthType Basic
  AuthUserFile /opt/zimbra/support/.htpasswd
  require valid-user
  Options Indexes
  IndexOptions FancyIndexing VersionSort
  AllowOverride None
  Order allow,deny
  Allow from all
</Directory>
Make sure permissions are set correctly on zimbra-support.conf so apache will use it.
Now let's setup a username and password to use with the .htpasswd file.
cd /opt/zimbra/support/
htpasswd -c .htpasswd zimbrasupport
chmod 644 .htpasswd


Restart apache so the changes take effect:
Restart apache so the changes take effect:


* For ZCS Apache Setup:
  zmapachectl stop
  zmapachectl stop
  zmapachectl start
  zmapachectl start
* Non-ZCS Server With Apache:
httpd restart


Now, let's dump some graphing directories in there:
Now, let's dump some graphing directories in there:


  zmstat-chart -s /opt/zimbra/zmstat -d /opt/zimbra/zmstat-chart/`date +%F-%H-%M`
  zmstat-chart -s /opt/zimbra/zmstat -d /opt/zimbra/support/data/`zmhostname`/zmstat/`date +%F-%H-%M`


To now view the charts, adjust for your servername.
To now view the charts. Adjust, SERVERHOSTNAME, for your servername.


http://servername:7780/zmstat-chart/
* For ZCS Apache Setup:
http://servername:7780/support/data/SERVERHOSTNAME/zmstat
 
* Non-ZCS Server With Apache:
http://servername/support/data/SERVERHOSTNAME/zmstat


You'll see there's the directory and when you go into it - there's your charts.
You'll see there's the directory and when you go into it - there's your charts.


You can now place this command in your crontab to run on a schedule basis.
You can now place this command in your crontab to run on a schedule basis.
Things you'll want to check before deploying:
#Place charting directory in a location where space is available and will not create a risk to the mail services if it's full.
#Look at the zmstat-chart options to see what other parameters you want to use - especially if it's going in your crontab.
#Make sure your following and security guidelines for your environment.


====Tweaking zmstat-chart====
====Tweaking zmstat-chart====
Line 879: Line 929:
*Mailboxd  JVM Permanent Generation and Code Cache
*Mailboxd  JVM Permanent Generation and Code Cache


====Getting All User Quota Data (not zmstat related really)====
====Getting All User Quota Data With Zmsoap (not zmstat related really)====


To get quota stats, run. The output is a little bit difficult to read since it's in bytes:
Moved to [[Getting_All_Users_Quota_Data]]


zmprov gqu [server-name]
====Getting All User Quota Data With Zmprov (not zmstat related really)====


Here's a script to put into user readable format below [by Tony P.]. You'll need to modify the $hostname variable near the top:
Moved to [[Getting_All_Users_Quota_Data]]


<pre>
===Third Party Tools (Monitoring & Logging)===
#!/usr/bin/perl
 
====FYI - Support For Real Time Counters (snmp,etc.)====
 
This is for those that need to go beyond what zmstat-chart is doing or need to intergrate within a third party monitoring systems.
 
Please see RFE:
* "Support for real time counters (snmp, jmx, etc)"
** http://bugzilla.zimbra.com/show_bug.cgi?id=12196


$hostname = localhost;
Currently, this RFE has only resolved support for JMX, not SNMP. Customers who need SNMP can use a  product like [http://www.jmanage.org/ jManage] to do the translation from JMX to SNMP.


@quotas = `zmprov gqu $hostname`;
====SNMP And Zimbra====


foreach $quotaline(@quotas)
=====SNMP Related Bugs And RFEs=====


{
Please read the following below as they'll have little bits of information that you might need for your customizations / external setup for snmp use against Zimbra. :


chop($quotaline);
* "support for real time counters (snmp, jmx, etc)"
** http://bugzilla.zimbra.com/show_bug.cgi?id=12196
* "zimbra snmp is broken"
** http://bugzilla.zimbra.com/show_bug.cgi?id=14760
* "snmp disk checks report incorrect usage."
** http://bugzilla.zimbra.com/show_bug.cgi?id=26060
* "replace zmdisklog / zimbra-snmp integration"
** http://bugzilla.zimbra.com/show_bug.cgi?id=38528


($email, $quota, $usage) = split(/\s/,$quotaline);
=====SNMP Setup On Zimbra To Notify A Remote Host=====


$quota=utils_convert_bytes_to_optimal_unit($quota);
Our SNMP support is pretty basic; currently, we only send traps when a service (mta, mailbox, ldap)
changes state (stop/start).


$usage=utils_convert_bytes_to_optimal_unit($usage);
'''First, you'll need net-snmp package install on the zimbra host for the notifications to be sent to a remote host.'''
:* When you install the net-snmp package, it creates /etc/snmp/snmpd.conf. This file needs to be edited with the correct community string to allow for snmp mibs to be read from remote machines The default value is "public" with the net-snmp package.
:** /opt/zimbra/conf/swatchrc.in is set to use [ perlcode 0 my $snmpargs="-v 2c -c zimbra localhost ''"; ] for the community string. Adjust this if needed for your corporate snmp environment.
:** Your security policy for your company might require you to review the other options there as well - i.e. limit to read-only.
:** Also, if there is a firewall between the zimbra server and the snmp server host you'll need to open up port 161/UDP.


print "$email - $usage\n";
Then your zimbra host with the zimbra snmp service installed, do the following for a basic default setup:


}
zmlocalconfig -e snmp_notify=1
zmlocalconfig -e snmp_trap_host=your.host.name
 
/opt/zimbra/libexec/zmsnmpinit
 
zmswatchctl stop
zmswatchctl start


sub utils_convert_bytes_to_optimal_unit{
We watch for something matching /err: Service status change/ and send the trap with:


my($bytes) = @_;
/opt/zimbra/snmp/bin/snmptrap


return '' if ($bytes eq '');
See other topics below for customizations that might be needed for your snmp environment.


my($size);
Other reference for Zimbra and SNMP:
* [http://www.zimbra.com/docs/ne/latest/administration_guide/9_Monitoring.16.10.html#1092315 SNMP]


$size = $bytes . ' Bytes' if ($bytes < 1024);
======Files To Review For SNMP======


$size = sprintf("%.2f", ($bytes/1024)) . ' KB' if ($bytes >= 1024 && $bytes < 1048576);
Look at the contents of the following files:


$size = sprintf("%.2f", ($bytes/1048576)) . ' MB' if ($bytes >= 1048576 && $bytes < 1073741824);
* /opt/zimbra/libexec/zmsnmpinit
** /opt/zimbra/conf/swatchrc.in
*** zmsnmpinit reads swatchrc.in and writes out the file swatchrc for the running configuration
*** /opt/zimbra/conf/swatchrc
** /opt/zimbra/snmp/share/snmp/snmpd.conf.in which is the SOURCE file to
*** /opt/zimbra/conf/snmpd.conf [see /opt/zimbra/libexec/zmsnmpinit ]
**** The two above are used by zmsnmpinit to generate the /opt/zimbra/conf/swatchrc
* /opt/zimbra/net-snmp/share/snmp/mibs/zimbra.mib
* /opt/zimbra/net-snmp/share/snmp/mibs/zimbra_traps.mib
* /opt/zimbra/log/zmswatch.out
** Monitor this to see the services when they go up and down successfully send out the email notification for your snmp configuration


$size = sprintf("%.2f", ($bytes/1073741824)) . ' GB' if ($bytes >= 1073741824 && $bytes < 1099511627776);
=====Zimbra MIBS=====


$size = sprintf("%.2f", ($bytes/1099511627776)) . ' TB' if ($bytes >= 1099511627776);
You'll find zimbra.mib and zimbra_traps.mib in the following directory. This listing is mine under 5.0.19 :


return $size;
<pre>
[root@mail3 ~]# cd /opt/zimbra/net-snmp/share/snmp/mibs/


}
[root@mail3 mibs]# ls
AGENTX-MIB.txt                      IPV6-TC.txt                SNMP-USER-BASED-SM-MIB.txt
DISMAN-EVENT-MIB.txt                IPV6-UDP-MIB.txt          SNMP-USM-AES-MIB.txt
DISMAN-SCHEDULE-MIB.txt              NET-SNMP-AGENT-MIB.txt    SNMP-USM-DH-OBJECTS-MIB.txt
DISMAN-SCRIPT-MIB.txt                NET-SNMP-EXAMPLES-MIB.txt  SNMPv2-CONF.txt
EtherLike-MIB.txt                    NET-SNMP-EXTEND-MIB.txt    SNMPv2-MIB.txt
HCNUM-TC.txt                        NET-SNMP-MIB.txt          SNMPv2-SMI.txt
HOST-RESOURCES-MIB.txt              NET-SNMP-TC.txt            SNMPv2-TC.txt
HOST-RESOURCES-TYPES.txt            NET-SNMP-VACM-MIB.txt      SNMPv2-TM.txt
IANA-ADDRESS-FAMILY-NUMBERS-MIB.txt  NOTIFICATION-LOG-MIB.txt  SNMP-VIEW-BASED-ACM-MIB.txt
IANAifType-MIB.txt                  RFC1155-SMI.txt            TCP-MIB.txt
IANA-LANGUAGE-MIB.txt                RFC1213-MIB.txt            TRANSPORT-ADDRESS-MIB.txt
IANA-RTPROTO-MIB.txt                RFC-1215.txt              UCD-DEMO-MIB.txt
IF-INVERTED-STACK-MIB.txt            RMON-MIB.txt              UCD-DISKIO-MIB.txt
IF-MIB.txt                          SMUX-MIB.txt              UCD-DLMOD-MIB.txt
INET-ADDRESS-MIB.txt                SNMP-COMMUNITY-MIB.txt    UCD-IPFWACC-MIB.txt
IP-FORWARD-MIB.txt                  SNMP-FRAMEWORK-MIB.txt    UCD-SNMP-MIB.txt
IP-MIB.txt                          SNMP-MPD-MIB.txt          UDP-MIB.txt
IPV6-ICMP-MIB.txt                    SNMP-NOTIFICATION-MIB.txt  zimbra.mib
IPV6-MIB.txt                        SNMP-PROXY-MIB.txt        zimbra_traps.mib
IPV6-TCP-MIB.txt                    SNMP-TARGET-MIB.txt
</pre>
</pre>


===Third Party Tools (Monitoring & Logging)===
=====What Is Looked For=====
 
Take a look at your /opt/zimbra/conf/swatchrc - this is mine under 5.0.19
 
<pre>
perlcode 0 my %notifications=();
perlcode 0 $notifications{smtp}="yes";
perlcode 0 $notifications{snmp}="yes";
 
perlcode 0 my $fr='admin@mail3.zimbra.REMOVED.com';
perlcode 0 my $pwc='admin@mail3.zimbra.REMOVED.com';
 
perlcode 0 my $snmpargs="-v 2c -c zimbra localhost ''";
perlcode 0 my $snmptrap="/opt/zimbra/snmp/bin/snmptrap $snmpargs";
perlcode 0 my $snmpsvctrap="ZIMBRA-TRAP-MIB::zmServiceStatusTrap";
perlcode 0 my $snmpsvcname="ZIMBRA-MIB::zmServiceName";
perlcode 0 my $snmpsvcstatus="ZIMBRA-MIB::zmServiceStatus";
 
perlcode 0 my %statuses=('started'=>1,'stopped'=>0);
 
perlcode 0 my $hostname="mail3.zimbra.homeunix.com";
 
perlcode 0 sub donotify {  my %args = (@_); if ($args{HOST} eq "localhost") {$args{HOST}=$hostname;};
if ($notifications{smtp}) { dosmtp(%args) if $args{SERVICE}; dodisksmtp(%args) if $args{DISK};};
if ($notifications{snmp}) {dosnmp(%args);};  }
 
perlcode 0 sub dosmtp {  my %args = (@_);  print "SMTP notification: $args{MESSAGE}\n";
open (FOO, "|/opt/zimbra/postfix/sbin/sendmail -Am -t");
print FOO "To: $pwc\nFrom: $fr\nSubject: Service $args{SERVICE} $args{STATUS} on $args{HOST}\n\n$args{MESSAGE}\n";
close FOO; }
 
perlcode 0 sub dodisksmtp {  my %args = (@_);  print "SMTP notification: $args{MESSAGE}\n";
open (FOO, "|/opt/zimbra/postfix/sbin/sendmail -Am -t");
print FOO "To: $pwc\nFrom: $fr\nSubject: Disk $args{DISK} at $args{UTIL}\% on $args{HOST}\n\n$args{MESSAGE}\n";
close FOO; }
 
perlcode 0 sub dosnmp {  my %args = (@_); print "SNMP notification: $args{MESSAGE}\n";
`$snmptrap $snmpsvctrap $snmpsvcname s $args{SERVICE} $snmpsvcstatus i $statuses{$args{STATUS}}`; }
 
ignore /DEBUG/
 
watchfor /err: Service status change: (\S+) (.*) changed from stopped to running/
donotify SERVICE=$2,STATUS=started,HOST=$1
watchfor /err: Service status change: (\S+) (.*) changed from running to stopped/
donotify SERVICE=$2,STATUS=stopped,HOST=$1
 
watchfor /err: Disk warning: (\S+) (\S+) at (\d+)/
        donotify DISK=$2,UTIL=$3,HOST=$1
watchfor /crit: Disk warning: (\S+) (\S+) at (\d+)/
        donotify DISK=$2,UTIL=$3,HOST=$1
</pre>


====FYI - Support For Real Time Counters (snmp,etc.)====
=====Enhanced MIB Files For HP OpenView=====


This is for those that need to go beyond what zmstat-chart is doing or need to intergrate within a third party monitoring systems.
I've created an RFE for this:


Please see RFE:
* "enchanced MIB and OID information to work with HP Openview"
* "Support for real time counters (snmp, jmx, etc)"
** http://bugzilla.zimbra.com/show_bug.cgi?id=44354
** http://bugzilla.zimbra.com/show_bug.cgi?id=12196


Currently, this RFE has only resolved support for JMX, not SNMP. Customers who need SNMP can use a  product like [http://www.jmanage.org/ jManage] to do the translation from JMX to SNMP.
Zimbra does not provide "enhanced" mib files" at this time. Thresholds can be set by the customer within their individual monitoring system. Zimbra is alerting on is service up or service down, see the other information above in [[Ajcody-Logging#SNMP_And_Zimbra]] .


====Some Choices====
====Some Choices====
Line 1,116: Line 1,272:
**If you get warning about pop-up, accept it.  
**If you get warning about pop-up, accept it.  
**If the debug window doesn't show, just mouse in the url field and hit the return key again. It should now pop up.
**If the debug window doesn't show, just mouse in the url field and hit the return key again. It should now pop up.
----


[[Category: Community Sandbox]]
[[Category: Community Sandbox]]
[[Category:Logger]]
[[Category:Logging]]
[[Category: Author:Ajcody]]
[[Category: Zeta Alliance]]

Latest revision as of 19:23, 20 June 2016

Logging

   KB 2491        Last updated on 2016-06-20  




0.00
(0 votes)
24px ‎  - This is Zeta Alliance Certified Documentation. The content has been tested by the Community.


Actual Logging Homepage

Please see Ajcody-Logging

Server

RFE's Related To Better Logging And Historical Data Of Systems

See : Ajcody-Testing-Debugging#RFE.27s_Related_To_Better_Logging_And_Historical_Data_Of_Systems

Debugging

See : Ajcody-Testing-Debugging for more complete debugging information.

When Was A ZCS Service Enabled Or Disabled

See : Ajcody-Notes-Archive-Discovery#When_Was_A_ZCS_Service_Enabled_Or_Disabled

Syslog Items

Single Server Setup

/etc/syslog.conf should have lines similiar to:

[towards bottom of conf file]
local0.*                -/var/log/zimbra.log
auth.*                  -/var/log/zimbra.log
mail.*                  -/var/log/zimbra.log

Make sure syslog allows messages from log4j to be written, log4j doesn't do unix pipes. Specifically, it uses internet domain sockets (514/upd) instead of unix domain sockets (/dev/log). On a single server setup, this means log4j talking to localhost via 514/udp. Again, /dev/log is never used by log4j (AFAIK, this could change in the future). Adding the "-r" applies for the centralized syslog server as usual, but it also applies for the case where you want log4j data to be logged via syslog locally [single server setup]:

[root@zimbra sysconfig]# diff -u /etc/sysconfig/syslog.ORIG /etc/sysconfig/syslog
--- /etc/sysconfig/syslog.ORIG  2008-03-25 09:14:28.000000000 -0400
+++ /etc/sysconfig/syslog       2009-06-09 16:01:13.000000000 -0400
@@ -3,7 +3,7 @@
 # -r enables logging from remote machines
 # -x disables DNS lookups on messages recieved with -r
 # See syslogd(8) for more details
-SYSLOGD_OPTIONS="-m 0"
+SYSLOGD_OPTIONS="-r -m 0"
 # Options to klogd
 # -2 prints all kernel oops messages twice; once for klogd to decode, and
 #    once for processing with 'ksymoops'

Restart syslog:

/etc/init.d/syslog restart

Setup ZCS to be aware of syslog [we'll restart Zimbra later once all the changes are done]:

zmprov mcf zimbraLogToSyslog TRUE

This will set the appropriate ldap values (after a zimbra restart) to make log4j.properties be written correctly:

/opt/zimbra/conf/log4j.properties will now show some modifications, for example:

[snips - DON'T cut cut/paste this info below for your server]
#log4j.rootLogger=INFO,LOGFILE
log4j.rootLogger=INFO,LOGFILE,SYSLOG
...
# Syslog appender
log4j.appender.SYSLOG=org.apache.log4j.net.SyslogAppender
log4j.appender.SYSLOG.SyslogHost=localhost
log4j.appender.SYSLOG.Facility=LOCAL0
log4j.appender.SYSLOG.layout=com.zimbra.common.util.ZimbraPatternLayout
log4j.appender.SYSLOG.layout.ConversionPattern=mailboxd: %-5p [%t] [%z] %c{1} - %m

This will actually cause very little to goto syslog though, it's a very minimalist setup. Now adjust /opt/zimbra/conf/log4j.properties.in to log more items to syslog.

Note: You may or may not want all these things going to syslog and you may be able to send more/other things to syslog as well. I've not spend lots of time with log4j and I'm not familiar enough with all the code to know which pieces are using which logger/settings.

WARNING: the AUDIT logs may have sensitive data so be careful to protect the logs so that sensitive data is not leaked accidentally.

[zimbra@zimbra conf]$ diff -u log4j.properties.in.ORIG log4j.properties.in
--- log4j.properties.in.ORIG    2009-06-05 15:31:20.000000000 -0400
+++ log4j.properties.in 2009-06-09 15:54:07.000000000 -0400
@@ -35,7 +35,7 @@

 # Save zimbra.security to AUDIT appender
 log4j.additivity.zimbra.security=false
-log4j.logger.zimbra.security=INFO,AUDIT
+log4j.logger.zimbra.security=INFO,AUDIT,SYSLOG

 # Syslog appender
 log4j.appender.SYSLOG=org.apache.log4j.net.SyslogAppender
@@ -88,7 +88,7 @@
 log4j.additivity.zimbra.wbxml=false
 log4j.logger.zimbra.wbxml=DEBUG,WBXML

-log4j.logger.zimbra=INFO
+log4j.logger.zimbra=INFO,SYSLOG
 log4j.logger.zimbra.op=WARN
-log4j.logger.com.zimbra=INFO
+log4j.logger.com.zimbra=INFO,SYSLOG

At this point, you can restart zimbra for all the changes to be in effect:

zmcontrol stop ; zmcontrol start

Centralized Syslog Server

Please see:

This will show you how to setup a syslog server and then to configure the "clients" to log to the local filesystem as well as the syslog server.

You'll need to then take into account the information in Ajcody-Logging#Single_Server_Setup to handle the "Zimbra" part as well as dealing with log4j.

Centralized Log Server Project Ideas

Problems:

  • Logs are to large to manage on each server.
  • Want to build a central logging server for all logging events.
  • The syslog events are very minimal with Zimbra (/var/log/zimbra.log). Most of the logging is control by Log4j (tomcat/jetty). Output from log4j generally goes into the /opt/zimbra/log/ directory. There's some conf files in /opt/zimbra/conf/log4j.properties* for log4j. You can adjust this file to also output into the syslog environment but syslog has very limited choices for "facility". See Ajcody-Logging#Single_Server_Setup and Log4j & Syslog facility for details. You'll end up getting huge files and god knows how many log events per second if you adjusted log4j to dump all zimbra logging into syslog control.

General Thought On This:

Use syslog-ng with mysql for the central log server. Add on php-syslog-ng for comfort.

Leave the log4j stuff alone and configure syslog-ng to monitor the actual log files as they are generated by log4j in /opt/zimbra/log/* .

Then decrease the log sizes and archiving done on the local servers, i.e. zimbraLogRawLifetime & zimbraLogSummaryLifetime .


References:


Log Rotation and Removal

Logs In /opt/zimbra/log Directory

Please look at the output of your crontab - AS ZIMBRA, look for the "Log pruning" line.

crontab -l

And also the config files for logrotate:

/etc/logrotate.conf

/etc/logrotate.d/zimbra

Example From 5.0.11 - Complete ZCS Install Single Server

crontab for zimbra:

# Log pruning
#
30 2 * * * find /opt/zimbra/log/ -type f -name \*.log\* -mtime +8 -exec rm {} \; > /dev/null 2>&1
35 2 * * * find /opt/zimbra/log/ -type f -name \*.out.???????????? -mtime +8 -exec rm {} \; > /dev/null 2>&1

# Log pruning
#
30 2 * * * find /opt/zimbra/mailboxd/logs/ -type f -name \*log\* -mtime +8 -exec rm {} \; > /dev/null 2>&1

And the logrotate files:

$ cat /etc/logrotate.conf 
# see "man logrotate" for details
# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# uncomment this if you want your log files compressed
#compress

# RPM packages drop log rotation information into this directory
include /etc/logrotate.d

# no packages own wtmp -- we'll rotate them here
/var/log/wtmp {
    monthly
    minsize 1M
    create 0664 root utmp
    rotate 1
}

# system-specific logs may be also be configured here.

And also:

$ cat /etc/logrotate.d/zimbra 
/var/log/zimbra.log {
    daily
    missingok
    notifempty
    create 0644 zimbra zimbra
    postrotate
      killall -HUP syslogd 2> /dev/null || true
      kill -HUP `cat /opt/zimbra/log/swatch.pid 2> /dev/null` 2> /dev/null || true
      kill -HUP `cat /opt/zimbra/log/logswatch.pid 2> /dev/null` 2> /dev/null || true
    endscript
    compress
}

/opt/zimbra/log/myslow.log {
    daily
    missingok
    copytruncate
    rotate 30
    notifempty
    create 0644 zimbra zimbra
    compress
}

/opt/zimbra/log/logger_myslow.log {
    daily
    missingok
    copytruncate
    notifempty
    create 0660 zimbra zimbra
    compress
    size 5000k
    rotate 7
} 

/opt/zimbra/log/clamd.log {
    daily
    missingok
    copytruncate
    notifempty
    create 0660 zimbra zimbra
    postrotate
     kill -HUP `cat /opt/zimbra/log/clamd.pid 2> /dev/null` 2> /dev/null || true
    endscript
    compress
    size 5000k
    rotate 7
}

/opt/zimbra/log/zmlogswatch.out {
    daily
    missingok
    copytruncate
    notifempty
    create 0740 zimbra zimbra
    postrotate
      su - zimbra -c "/opt/zimbra/bin/zmlogswatchctl stop"
      su - zimbra -c "/opt/zimbra/bin/zmlogswatchctl start"
    endscript
    rotate 5
    compress
}

/opt/zimbra/log/zmswatch.out {
    daily
    missingok
    copytruncate
    notifempty
    create 0740 zimbra zimbra
    postrotate
      su - zimbra -c "/opt/zimbra/bin/zmswatchctl stop"
      su - zimbra -c "/opt/zimbra/bin/zmswatchctl start"
    endscript
    rotate 5
    compress
}

/opt/zimbra/log/zmmtaconfig.log {
    daily
    missingok
    copytruncate
    notifempty
    create 0740 zimbra zimbra
    postrotate
      su - zimbra -c "/opt/zimbra/bin/zmmtaconfigctl restart"
    endscript
    rotate 5
    compress
}

/opt/zimbra/log/nginx.log {
    daily
    missingok
    notifempty
    create 0644 zimbra zimbra
    postrotate
      kill -USR1 `cat /opt/zimbra/log/nginx.pid 2> /dev/null` 2> /dev/null || true
    endscript
    rotate 7
    compress
}

/opt/zimbra/log/zmconvertd.log {
    daily
    missingok
    copytruncate
    notifempty
    create 0644 zimbra zimbra
    rotate 7
}

/opt/zimbra/zmstat/zmstat.out {
    daily
    missingok
    copytruncate
    rotate 7
    notifempty
    compress
}

Openldap Logs - In /opt/zimbra/openldap-data/

Ldap logs files in /opt/zimbra/openldap-data/ and in /opt/zimbra/data/ldap/* subdirectories [ZCS 6+] that are like log.0000000001 , log.0000000002 , etc.

This logpurge is controlled by:

$ cd /opt/zimbra/openldap/etc/openldap/
[zimbra@mail3 openldap]$ grep logpurge *
master-accesslog-overlay.conf:logpurge	07+00:00	01+00:00
Logpurge Directive Description

From Accesslog Chapter

logpurge Directive

logpurge age interval

Defines both the maximum age for log entries to be retained in the database and how often to scan the database for old entries. Bothage and interval are specified as a time span in days, hours, minutes, and seconds. The time format is [ddd+]hh:mm[:ss], for example, the days and seconds components are optional but hours and minutes are required. Except for days, which can be up to 5 digits, each numeric field must be exactly two digits. Example:

  1. the log database will be scanned every day
  2. entries older than two days will be deleted.

logpurge 2+00:00 1+00:00

When using a log database that supports ordered indexing on generalizedTime attributes, specifying an eq index on the reqStart attribute will increase the performance of purge operations.

What's up with all the logs?

Other Logging Page Resources

http://www.zimbra.com/docs/ne/latest/administration_guide/9_Monitoring.14.1.html

http://wiki.zimbra.com/index.php?title=Server_Monitoring

http://wiki.zimbra.com/index.php?title=Log_Files

http://wiki.zimbra.com/index.php?title=Unresponsive_Server_Troubleshooting

The Bread And Butter Logs

/opt/zimbra/log/mailbox.log - where most of your mailbox store activity is logged
  • This log is the mailboxd log4j server log containing the logs from the mailbox server. This includes activity from the mailbox store, LMTP server, IMAP and POP servers, and Index server.
    • Location: /opt/zimbra/log/mailbox.log
/opt/zimbra/log/zmmailboxd.out - mailboxd/jvm output log
  • Mailstore not coming up and nothing is being logged in mailbox.log, check here for errors.
    • Location: /opt/zimbra/log/zmmailboxd.out
/opt/zimbra/log/stacktrace.<pid> - stacktrace logs
  • stacktraces related to mailboxd
    • Location: /opt/zimbra/log/stacktrace.<pid>
/opt/zimbra/db/data/YOURHOSTNAME.err - errors for MySQL - ZCS 4.x & ZCS 5.x
  • This is the message store database error log.
    • Location: /opt/zimbra/db/data/YOURHOSTNAME.err
/var/log/zimbra.log - mta and system status log, postfix, amavisd
  • The Zimbra syslog details the activities of the Zimbra MTA (Postfix, amavisd, antispam, antivirus), Logger, Authentication (cyrus-sasl), and Directory (OpenLDAP). By default LDAP activity is logged to Zimbra.log.
    • Location: /var/log/zimbra.log
/var/log/messages - mta and OS related log events
  • Probably has mta logging events [which also show in /var/log/zimbra.log] and also log events related to your OS
    • Location: /var/log/messages
/opt/zimbra/log/mysql_error.log - problems with MySQL
  • If there is data corruption or another problem causing direct mysql errors, events will be logged here.
    • /opt/zimbra/log/mysql_error.log
/opt/zimbra/log/myslow.log - slow db/MySQL queries
  • If certain search requests are taking longer to complete than others, they will be logged here.
    • /opt/zimbra/log/myslow.log

Other Logs

/opt/zimbra/log/audit.log - authentication events
  • A log of all admin actions taken as well as logins to the server.
    • /opt/zimbra/log/audit.log
Want to see all uses of the "View Mail" button from the admin console? Will have the IP address, User Agent of the browser, user that the admin is logged in as, and the account that they are viewing.
 grep DelegateAuth /opt/zimbra/log/audit.log 
/opt/zimbra/log/clamd.log - antivrius db
  • Status and checks for clamav
    • /opt/zimbra/log/clamd.log
If your clamav db is out of date, you'll see a log event here like this:
LibClamAV Warning: **************************************************
LibClamAV Warning: *** The virus database is older than 7 days! ***
LibClamAV Warning: *** Please update it as soon as possible. ***
LibClamAV Warning: **************************************************
/opt/zimbra/log/convertd.log - attachment conversion
  • If your having problems with your attachments and the ability to view them in ZWC, check here for errors.
    • /opt/zimbra/log/convertd.log
/opt/zimbra/log/freshclam.log - clam antivirus updates
  • Shows the actual attempt to update the clamav updates.
    • /opt/zimbra/log/freshclam.log
/opt/zimbra/log/spamtrain.log - spam/ham training details
  • Spam/Ham training details are logged here. Errors are also logged here related to this service.
  • From a default single ZCS configurations crontab [zimbra]:
# Spam training
#:
0 :23 * * * /opt/zimbra/bin/zmtrainsa >> /opt/zimbra/log/spamtrain.log 2>&1
#
# Spam training cleanup
#
45 23 * * * /opt/zimbra/bin/zmtrainsa --cleanup >> /opt/zimbra/log/spamtrain.log 2>&1
/opt/zimbra/log/sync.log - zimbra mobile and activesync
  • Your mobile devices will basically log here.
    • /opt/zimbra/log/sync.log
/opt/zimbra/log/synctrace.log - zimbra mobile and activesync
  • Still looking for description that describes difference with sync.log
    • /opt/zimbra/log/synctrace.log
/opt/zimbra/log/syncstate.log - no description found
  • No description found.
    • /opt/zimbra/log/syncstate.log
/opt/zimbra/log/wbxml.log - no description found
  • No description found.
    • /opt/zimbra/log/wbxml.log
/opt/zimbra/log/zmlogswatch.out -
/opt/zimbra/log/zmmtaconfig.log -
/opt/zimbra/log/zmmyinit.log -
/opt/zimbra/log/zmsetup.`date`-'pid'.log -
/tmp/install.log -

After install, gets moved to : /opt/zimbra/.install/

/tmp/zmsetup.log -

After install, gets moved to : /opt/zimbra/log/zmsetup.log.[some numbers]

/opt/zimbra/httpd_access.log -
/opt/zimbra/httpd_error.log -

New Data Directory In ZCS6+

In ZCS 6+ , there's a directory to organize 'data' from our various components.

[root@mail3 data]# pwd
/opt/zimbra/data

[root@mail3 data]# ls -F
altermime/  amavisd/  clamav/  dspam/  ldap/  postfix/  tmp/

This information could prove useful as well with trouble-shooting and debugging problems.

Increase Logging

Increase Logging Per User

Depending on the ZCS version the below will clear all the per account loggers on zimbra reboot. IMAP is used for these examples.

  • addAccoutLogger
zmprov aal user@domain.com zimbra.imap debug
  • removeAccountLogger
zmprov ral user@domain.com zimbra.imap

Example usuage:

zmprov aal user@domain.com zimbra.soap debug 

And tail the log file, while attempting to login to the AJAX client.:

tail -f /opt/zimbra/log/mailbox.log

See Ajcody-Logging#Log4J_Variables for variables that can be used.

To See What Accounts Have Extended Logging Enabled

There's the gaal option for zmprov . It list the accounts on a per mailstore basis :

  getAllAccountLoggers(gaal) [-s/--server hostname]
[zimbra@zcs806 ~]$ zmprov gaal `zmhostname`
# name test01@zcs806.domain.com
zimbra.imap=debug
Bugs And RFE's

I filed the following RFE:

Increase Logging Per Server

If you want to do it globally edit this file:

vi /opt/zimbra/conf/log4j.properties

Add a line at the end that's similar to something like this: log4j.logger.zimbra.imap=DEBUG.

No restart of any service is needed, BUT if something happens that causes a regeneration of the log4j.properties file then your changes will be overwritten. A regeneration of this file takes the contents of /opt/zimbra/conf/log4.properties.in .

To permanently make a change, modify the /opt/zimbra/conf/log4.properties.in file, then a restart would be necessary.

$ zmcontrol stop
$ zmcontrol start

or you could run this on the server:

$ zmmtaconfig mailboxd

Wait for a minute for the server to pick up your change.

See Ajcody-Logging#Log4J_Variables for variables that can be used.

Log4J Variables

An Official page has been made for this topic now per my RFE request, bug 41894

Please see:

OpenLDAP Logging Levels

Please see King0770-Notes-Change-LDAP-Log-Levels

Want To See What's In Redolog Files

If you suspect there's too much redolog activity during a time window or have another need to inspect the contents of the redolog, dump it and examine it:

$ zmjava com.zimbra.cs.redolog.util.RedoLogVerify /opt/zimbra/redolog/redo.log > out.file

Pick the right redolog file, either redo.log or one of the files under archive/, based on timestamp.

See Ajcody-Backup-Restore-Issues#Redolog_Files for more details.

Internal Zimbra Charting - zmstat-chart

RFE/Bugs You Might Be Interested In

zmstat IS NOT Logger (graphs in admin console)

Please note, zmstat is different than the processes and graphs involved with Logger. Logger is behind the graphs that show up in the admin web console.

Please see Logger for details and trouble-shooting steps with Logger.

Rick has some other steps for King0770-Notes#Reinitialize_the_Logger_DB

Main References For zmstat-chart Command

Please see the following:

zmstat-chart CLI Administration Guide

zmstat-chart-config CLI Administration Guide

zmstatctl CLI Administration Guide

Zmstats Wiki

Server Monitoring Wiki

Monitoring Administration Guide

JFreeChart is the project/binary behind the charting tool. You'll find the jar files here [ZCS 5.0.9 example]:

  • /opt/zimbra/lib/jars/jfreechart-1.0.1.jar
  • /opt/zimbra/jetty-6.1.5/common/lib/jfreechart-1.0.1.jar

Issues Being Investigated With zmstat And Other "Charting" Items

I currently have an internal thread going with the dev's about some issues with zmstat and also with attempts of getting the active sessions data from the admin web console to be seen in trends.

I'll just be dumping my notes here and then cleaning them up once I'm done with the internal conversation.

  • Question: (mailboxd: active connections by client protocol) What exactly is this tracking? I'm on a test box with no connections and it has the different threads in the 100's/1000's. Customer was hoping (as was I) that this would be giving data much like is shown in the admin console about user connections. Also, there doesn't seem to be a comparable "active connections" for the web client (soap)?
    • Answer: For the connections issue, it's the number of clients that are currently connected. There's no analog for SOAP because SOAP doesn't have persistent connections. We may be able to log stats for number of SOAP sessions, but that would include noise for cases where the user closes the browser window and the session hasn't timed out yet. If you think this is useful, please file an RFE.
  • Question: (graph time plot) If I run zmstat-chart with the --aggregate-start-at option with something like "08/07/2008 01:00:00" I notice that the graphs don't actually reference "days" at the bottom but still use what looks to be "hours". Is this a bug where the bottom reference of the graph doesn't get adjusted or is the syntax to run zmstat-chart to do this more tricky that I think.
    • Answer: zmstat-chart doesn't support multi-day ranges, it really only supports a single directory with a single day of charts, the aggregate options are for specifying ranges at which to calculate the max(col)/min(col)/avg(col) values. Our usual practice is to generate one set of charts for each day we want to look at; I think it gives us a better look at when peaks occur and how they compare day-to-day (it's much easier to look at them in an overlay fashion).
    • Answer: Seems like a reasonable request to be able to generate charts for multiple days. Please file an RFE if you think this would be useful.
  • Question: How reliable are these numbers are in admin console....
    • Answer: I imagine they should be quite reliable; but HTTP (Web) is stateless, so you can't really determine whether a session is really active or not. I believe multiple sessions appearing for a single user would be the result of reloading or closing and then reopening the browser to log in again.
      • Perhaps it is best to ask the users in question what their usage is like? Is there any concern over the high number of sessions? I don't think it should affect server performance.
  • Question: Could you please explain the following:
    • Mailboxd Mailbox Add Rate (Delivery Rate)
      • Answer: The rate at which messages are being added to the mailbox server, e.g. 1 message being delivered per second is... 1 m/s
    • Mailboxd Mailbox Add Latency (Delivery Speed)
      • Answer: How long it takes on average to add a message to a mailbox (averaged over the period of a minute)
    • Mailboxd Active Connections by Client Protocol- This one does NOT give us counts for web client. Host zcs2.mail.XXXX.xxx: Mailboxd: Active Connections by Client Protocol avg(IMAP) = 169.86 avg(IMAP SSL) = 0.00 avg(POP) = 0.21 avg(POP SSL) = 0.00
      • Answer: It doesn't give counts for web client because the web client does not have persistent connections (being HTTP-based). SOAP requests (used by the Web client) use transient connections that often close after they are used; i.e. if you took the number of "SOAP" connections and compared it to the number of "active" sessions, it'd likely be something like 0-1% of the latter figure.
    • Mailboxd Mailbox Get Count
      • Answer: The number of mailboxes that got opened in the last minute (I don't recall if this counting cache hits or not?)
    • Mailboxd Mailbox Get Latency
      • Answer: How long it takes to open a mailbox, on average (over a minute)
    • Also, we don't get stats for any of the soap parameters:
      • SOAP Invocation Count Summary (Top 10 max)
      • SOAP Average Call Duration AuthRequest
        • Answer: I don't know why this could be; perhaps zmstat-chart-config.xml needs to be regenerated, is /opt/zimbra/zmstat/soap.csv available and does it contain data?

Running zmstat-chart

Confirm it's running:

zmprov gs [mailserver hostname] | grep -i stats
  zimbraServiceEnabled:  stats
  zimbraServiceinstalled: stats

Default location of stat files is:

/opt/zimbra/zmstat/

Example command to run:

mkdir /tmp/stats
zmstat-chart -s /opt/zimbra/zmstat -d /tmp/stats/

Like To Have zmstat-chart Data Integrated With Zimbra

I've made an RFE for this integration:

With Zimbra Jetty
Note:
The default Jetty behavior will NOT display directory listing. You'll need a valid html type file in the directory target.
I would recommend the apache solution over this one as you can avoid hitting performance issues this might cause.

Dump some graphing directories in there:

zmstat-chart -s /opt/zimbra/zmstat -d /opt/zimbra/jetty/webapps/zimbra/downloads/zmstat-chart/`date +%F-%H-%M`

You'll see there's the directory and when you go into it - there's your charts.

You can now place this command in your crontab to run on a schedule basis.

Things you'll want to check before deploying:

  1. Place charting directory in a location where space is available and will not create a risk to the mail services if it's full.
  2. Look at the zmstat-chart options to see what other parameters you want to use - especially if it's going in your crontab.
  3. Make sure your following and security guidelines for your environment.

You'll be able to view the data with the following url:

http://hostname/zimbra/downloads/zmstat-chart/
With Zimbra Apache Or Non-Zimbra Server With Apache
Before You Proceed

Things to check or confirm first before deploying:

  • Place your directory root [/opt/zimbra/support in this example] in a location where space is available and will not create a risk to the mail services if it's full.
  • If you'll be using NFS to have a centralized storage point:
    • Your NFS mount point on the Zimbra servers would be /opt/zimbra/support in this example.
    • You'll be exporting /opt/zimbra/support in this example from the NFS server.
    • If this is a non-zimbra server, copy the zimbra entry from a zimbra server's /etc/passwd and /etc/group into the non-zimbra servers passwd and group file. This way, if NFS is also used the UID/GID matches.
  • Apache Directory variable will be /opt/zimbra/support/data in this example.
  • Apache .htpasswd location will be /opt/zimbra/support/.htpasswd in this example, putting it above of the apache Directory variable.
    • Make sure your following the security guidelines for your environment.
  • Look at the zmstat-chart options to see what other parameters you want to use - especially if it's going in your crontab.
Directory Layout

As root:

mdir -p /opt/zimbra/support/data
cd /opt/zimbra/support/data
for i in `/opt/zimbra/bin/zmprov gas`
 do
 mkdir $i
 done

If your not on a ZCS server, you'll be manually creating directories named after the output of zmhostname from each of your ZCS servers.

Now, create subdirectory paths under each zmhsotname directory.

cd /opt/zimbra/support/data/
for i in `ls`
  do
  mkdir $i/debug $i/logs $i/zmstat $i/tops
  done

Set permissions on the directories we made:

chown -R zimbra:zimbra /opt/zimbra/support
chmod -R 755 /opt/zimbra/support

With this structure and with NFS setup, you'll have a common repository that will have a standard path to use on a per server basis. Example :

script-command -PathOption /opt/zimbra/support/data/`zmhostname`/TYPE_OF_DATA/`date +%F-%H-%M`
Apache Configuration

Let's configure apache to show this directory.

  • For ZCS Apache Setup:
vi /opt/zimbra/conf/httpd.conf
** Towards the end, add the following**
# Include zmstat-chart directory
Include /opt/zimbra/httpd/conf/extra/zimbra-support.conf
  • Non-ZCS Server With Apache. Check that there's an existing Include directive that will see our zimbra-support.conf:
vi /etc/httpd/conf/httpd.conf
** Look for something that will allow the following**
** /etc/httpd/conf.d/zimbra-support.conf to be used**
** Your distro might differ on apache paths as well**
** as Include statement for other conf files**
Include conf.d/*.conf

Let's now make the apache conf file for zimbra-support.conf

  • For ZCS Apache Setup:
vi /opt/zimbra/httpd/conf/extra/zimbra-support.conf
Alias /support "/opt/zimbra/support/data"
<Directory "/opt/zimbra/support/data">
  AuthName "Secure Area For Zimbra Support"
  AuthType Basic
  AuthUserFile /opt/zimbra/support/.htpasswd
  require valid-user
  Options Indexes
  IndexOptions FancyIndexing VersionSort
  AllowOverride None
  Order allow,deny
  Allow from all
</Directory>
  • Non-ZCS Server With Apache:
vi /etc/httpd/conf.d/zimbra-support.conf
Alias /support "/opt/zimbra/support/data"
<Directory "/opt/zimbra/support/data">
  AuthName "Secure Area For Zimbra Support"
  AuthType Basic
  AuthUserFile /opt/zimbra/support/.htpasswd
  require valid-user
  Options Indexes
  IndexOptions FancyIndexing VersionSort
  AllowOverride None
  Order allow,deny
  Allow from all
</Directory>

Make sure permissions are set correctly on zimbra-support.conf so apache will use it.

Now let's setup a username and password to use with the .htpasswd file.

cd /opt/zimbra/support/
htpasswd -c .htpasswd zimbrasupport
chmod 644 .htpasswd

Restart apache so the changes take effect:

  • For ZCS Apache Setup:
zmapachectl stop
zmapachectl start
  • Non-ZCS Server With Apache:
httpd restart

Now, let's dump some graphing directories in there:

zmstat-chart -s /opt/zimbra/zmstat -d /opt/zimbra/support/data/`zmhostname`/zmstat/`date +%F-%H-%M`

To now view the charts. Adjust, SERVERHOSTNAME, for your servername.

  • For ZCS Apache Setup:

http://servername:7780/support/data/SERVERHOSTNAME/zmstat

  • Non-ZCS Server With Apache:

http://servername/support/data/SERVERHOSTNAME/zmstat

You'll see there's the directory and when you go into it - there's your charts.

You can now place this command in your crontab to run on a schedule basis.

Tweaking zmstat-chart

zmstat-chart also has the option to use a configuration file. The default configuration file is:

/opt/zimbra/conf/zmstat-chart.xml

Copy this to another location and edit away. Then add the -c /location/to/config/file to use your new configuration.

Default Items Charted

Here's a list of the default items that will be charted with zmstat-chart:

  • Postfix Queue Size
  • convertd CPU time used
  • convertd Resident Memory
  • convertd Processes and Threads
  • Total CPU
  • Process CPU
  • Process Total Memory
  • Process Resident Memory
  • Virtual Memory
  • Context Switches
  • Run/Blocked Process Queue Size
  • Disk Partition Throughput
  • Disk Utilization
  • Disk Throughput
  • Disk IOPs
  • Swap Activity
  • Total file descriptors open
  • SOAP Invocation Count Summary (Top 10 max)
  • SOAP Average Call Duration Summary (Top 10 avg)
  • SOAP Invocation Count GetServerRequest
  • SOAP Invocation Count BackupRequest
  • SOAP Invocation Count GetAllConfigRequest
  • SOAP Invocation Count GetDomainRequest
  • SOAP Invocation Count GetAllDomainsRequest
  • SOAP Invocation Count AuthRequest
  • SOAP Average Call Duration GetServerRequest
  • SOAP Average Call Duration BackupRequest
  • SOAP Average Call Duration GetAllConfigRequest
  • SOAP Average Call Duration GetDomainRequest
  • SOAP Average Call Duration GetAllDomainsRequest
  • SOAP Average Call Duration AuthRequest
  • MySQL Database Connections
  • MySQL InnoDB Buffer Pool Pages
  • MySQL InnoDB Buffer Pool Hit Rate
  • MySQL Tables Open/Opened
  • MySQL Total Slow Queries Count
  • Mailboxd Connection Pool Get Latency
  • Mailboxd Dirty Lucene Index Writers
  • Mailboxd Lucene IndexWriterCache Hitrate
  • Mailboxd Lucene IO
  • Mailboxd LMTP Delivery Throughput
  • Mailboxd LMTP Delivery Rate
  • Mailboxd Mailbox Add Rate (Delivery Rate)
  • Mailboxd Mailbox Add Latency (Delivery Speed)
  • Mailboxd Request Rate by Client Protocol
  • Mailboxd Response Time by Client Protocol
  • Mailboxd Active Connections by Client Protocol
  • Mailboxd Mailbox Get Count
  • Mailboxd Mailbox Get Latency
  • Mailboxd Mailbox Cache Hit Rate
  • Mailboxd Mailbox Item/Blob Cache Hit Rate
  • Mailboxd Garbage Collection Time
  • Mailboxd Garbage Collection Count
  • Mailboxd JVM Heap Used
  • Mailboxd JVM Heap Free
  • Mailboxd JVM Permanent Generation and Code Cache

Getting All User Quota Data With Zmsoap (not zmstat related really)

Moved to Getting_All_Users_Quota_Data

Getting All User Quota Data With Zmprov (not zmstat related really)

Moved to Getting_All_Users_Quota_Data

Third Party Tools (Monitoring & Logging)

FYI - Support For Real Time Counters (snmp,etc.)

This is for those that need to go beyond what zmstat-chart is doing or need to intergrate within a third party monitoring systems.

Please see RFE:

Currently, this RFE has only resolved support for JMX, not SNMP. Customers who need SNMP can use a product like jManage to do the translation from JMX to SNMP.

SNMP And Zimbra

SNMP Related Bugs And RFEs

Please read the following below as they'll have little bits of information that you might need for your customizations / external setup for snmp use against Zimbra. :

SNMP Setup On Zimbra To Notify A Remote Host

Our SNMP support is pretty basic; currently, we only send traps when a service (mta, mailbox, ldap) changes state (stop/start).

First, you'll need net-snmp package install on the zimbra host for the notifications to be sent to a remote host.

  • When you install the net-snmp package, it creates /etc/snmp/snmpd.conf. This file needs to be edited with the correct community string to allow for snmp mibs to be read from remote machines The default value is "public" with the net-snmp package.
    • /opt/zimbra/conf/swatchrc.in is set to use [ perlcode 0 my $snmpargs="-v 2c -c zimbra localhost "; ] for the community string. Adjust this if needed for your corporate snmp environment.
    • Your security policy for your company might require you to review the other options there as well - i.e. limit to read-only.
    • Also, if there is a firewall between the zimbra server and the snmp server host you'll need to open up port 161/UDP.

Then your zimbra host with the zimbra snmp service installed, do the following for a basic default setup:

zmlocalconfig -e snmp_notify=1
zmlocalconfig -e snmp_trap_host=your.host.name
/opt/zimbra/libexec/zmsnmpinit
zmswatchctl stop
zmswatchctl start

We watch for something matching /err: Service status change/ and send the trap with:

/opt/zimbra/snmp/bin/snmptrap

See other topics below for customizations that might be needed for your snmp environment.

Other reference for Zimbra and SNMP:

Files To Review For SNMP

Look at the contents of the following files:

  • /opt/zimbra/libexec/zmsnmpinit
    • /opt/zimbra/conf/swatchrc.in
      • zmsnmpinit reads swatchrc.in and writes out the file swatchrc for the running configuration
      • /opt/zimbra/conf/swatchrc
    • /opt/zimbra/snmp/share/snmp/snmpd.conf.in which is the SOURCE file to
      • /opt/zimbra/conf/snmpd.conf [see /opt/zimbra/libexec/zmsnmpinit ]
        • The two above are used by zmsnmpinit to generate the /opt/zimbra/conf/swatchrc
  • /opt/zimbra/net-snmp/share/snmp/mibs/zimbra.mib
  • /opt/zimbra/net-snmp/share/snmp/mibs/zimbra_traps.mib
  • /opt/zimbra/log/zmswatch.out
    • Monitor this to see the services when they go up and down successfully send out the email notification for your snmp configuration
Zimbra MIBS

You'll find zimbra.mib and zimbra_traps.mib in the following directory. This listing is mine under 5.0.19 :

[root@mail3 ~]# cd /opt/zimbra/net-snmp/share/snmp/mibs/

[root@mail3 mibs]# ls
AGENTX-MIB.txt                       IPV6-TC.txt                SNMP-USER-BASED-SM-MIB.txt
DISMAN-EVENT-MIB.txt                 IPV6-UDP-MIB.txt           SNMP-USM-AES-MIB.txt
DISMAN-SCHEDULE-MIB.txt              NET-SNMP-AGENT-MIB.txt     SNMP-USM-DH-OBJECTS-MIB.txt
DISMAN-SCRIPT-MIB.txt                NET-SNMP-EXAMPLES-MIB.txt  SNMPv2-CONF.txt
EtherLike-MIB.txt                    NET-SNMP-EXTEND-MIB.txt    SNMPv2-MIB.txt
HCNUM-TC.txt                         NET-SNMP-MIB.txt           SNMPv2-SMI.txt
HOST-RESOURCES-MIB.txt               NET-SNMP-TC.txt            SNMPv2-TC.txt
HOST-RESOURCES-TYPES.txt             NET-SNMP-VACM-MIB.txt      SNMPv2-TM.txt
IANA-ADDRESS-FAMILY-NUMBERS-MIB.txt  NOTIFICATION-LOG-MIB.txt   SNMP-VIEW-BASED-ACM-MIB.txt
IANAifType-MIB.txt                   RFC1155-SMI.txt            TCP-MIB.txt
IANA-LANGUAGE-MIB.txt                RFC1213-MIB.txt            TRANSPORT-ADDRESS-MIB.txt
IANA-RTPROTO-MIB.txt                 RFC-1215.txt               UCD-DEMO-MIB.txt
IF-INVERTED-STACK-MIB.txt            RMON-MIB.txt               UCD-DISKIO-MIB.txt
IF-MIB.txt                           SMUX-MIB.txt               UCD-DLMOD-MIB.txt
INET-ADDRESS-MIB.txt                 SNMP-COMMUNITY-MIB.txt     UCD-IPFWACC-MIB.txt
IP-FORWARD-MIB.txt                   SNMP-FRAMEWORK-MIB.txt     UCD-SNMP-MIB.txt
IP-MIB.txt                           SNMP-MPD-MIB.txt           UDP-MIB.txt
IPV6-ICMP-MIB.txt                    SNMP-NOTIFICATION-MIB.txt  zimbra.mib
IPV6-MIB.txt                         SNMP-PROXY-MIB.txt         zimbra_traps.mib
IPV6-TCP-MIB.txt                     SNMP-TARGET-MIB.txt
What Is Looked For

Take a look at your /opt/zimbra/conf/swatchrc - this is mine under 5.0.19

perlcode 0 my %notifications=();
perlcode 0 $notifications{smtp}="yes";
perlcode 0 $notifications{snmp}="yes";

perlcode 0 my $fr='admin@mail3.zimbra.REMOVED.com';
perlcode 0 my $pwc='admin@mail3.zimbra.REMOVED.com';

perlcode 0 my $snmpargs="-v 2c -c zimbra localhost ''";
perlcode 0 my $snmptrap="/opt/zimbra/snmp/bin/snmptrap $snmpargs";
perlcode 0 my $snmpsvctrap="ZIMBRA-TRAP-MIB::zmServiceStatusTrap";
perlcode 0 my $snmpsvcname="ZIMBRA-MIB::zmServiceName";
perlcode 0 my $snmpsvcstatus="ZIMBRA-MIB::zmServiceStatus";

perlcode 0 my %statuses=('started'=>1,'stopped'=>0);

perlcode 0 my $hostname="mail3.zimbra.homeunix.com";

perlcode 0 sub donotify {   my %args = (@_); if ($args{HOST} eq "localhost") {$args{HOST}=$hostname;}; 
 if ($notifications{smtp}) { dosmtp(%args) if $args{SERVICE}; dodisksmtp(%args) if $args{DISK};}; 
 if ($notifications{snmp}) {dosnmp(%args);};  }

perlcode 0 sub dosmtp {   my %args = (@_);  print "SMTP notification: $args{MESSAGE}\n"; 
 open (FOO, "|/opt/zimbra/postfix/sbin/sendmail -Am -t"); 
 print FOO "To: $pwc\nFrom: $fr\nSubject: Service $args{SERVICE} $args{STATUS} on $args{HOST}\n\n$args{MESSAGE}\n";
 close FOO; }

perlcode 0 sub dodisksmtp {   my %args = (@_);  print "SMTP notification: $args{MESSAGE}\n"; 
 open (FOO, "|/opt/zimbra/postfix/sbin/sendmail -Am -t"); 
 print FOO "To: $pwc\nFrom: $fr\nSubject: Disk $args{DISK} at $args{UTIL}\% on $args{HOST}\n\n$args{MESSAGE}\n"; 
 close FOO; } 

perlcode 0 sub dosnmp {   my %args = (@_); print "SNMP notification: $args{MESSAGE}\n"; 
 `$snmptrap $snmpsvctrap $snmpsvcname s $args{SERVICE} $snmpsvcstatus i $statuses{$args{STATUS}}`; }

ignore /DEBUG/

watchfor /err: Service status change: (\S+) (.*) changed from stopped to running/ 
	donotify SERVICE=$2,STATUS=started,HOST=$1
watchfor /err: Service status change: (\S+) (.*) changed from running to stopped/ 
	donotify SERVICE=$2,STATUS=stopped,HOST=$1

watchfor /err: Disk warning: (\S+) (\S+) at (\d+)/ 
        donotify DISK=$2,UTIL=$3,HOST=$1
watchfor /crit: Disk warning: (\S+) (\S+) at (\d+)/ 
        donotify DISK=$2,UTIL=$3,HOST=$1
Enhanced MIB Files For HP OpenView

I've created an RFE for this:

Zimbra does not provide "enhanced" mib files" at this time. Thresholds can be set by the customer within their individual monitoring system. Zimbra is alerting on is service up or service down, see the other information above in Ajcody-Logging#SNMP_And_Zimbra .

Some Choices

Charting & Graphing The Data
Montoring Software

Nagios On Zimbra

This is a really rough draft for ideas I have in background. Shouldn't be used by anyone.

Configure Nagios to run on single server Zimbra box - Centos 5.x

Configure yum with repo and install nagios

vi /etc/yum.repos.d/Dag.repo
yum update
yum install nagios nagios-plugins nagios-devel nagios-plugins-nrpe

Move nagios.conf http file into main zimbra directory.

cp /etc/httpd/conf.d/nagios.conf /opt/zimbra/httpd/conf/extra/

Setup nagios to run as zimbra

vi /etc/nagios/nagios.cfg
**Change nagios user to zimbra**
nagios_user=zimbra
nagios_group=zimbra

Change ownership of directories from nagios to zimbra.

chown -R 500:500 /var/log/nagios/
chown -R 500:500 /etc/nagios/
chown -R 500:500 /usr/share/nagios/

Configure authentication within Nagios

vi /etc/nagios/nagios.cfg
# AUTHENTICATION USAGE
use_authentication=1
# SYSTEM/PROCESS INFORMATION ACCESS
authorized_for_system_information=nagiosadmin
# CONFIGURATION INFORMATION ACCESS
authorized_for_configuration_information=nagiosadmin
# SYSTEM/PROCESS COMMAND ACCESS
authorized_for_system_commands=nagiosadmin
# GLOBAL HOST/SERVICE VIEW ACCESS
authorized_for_all_services=nagiosadmin
authorized_for_all_hosts=nagiosadmin
# GLOBAL HOST/SERVICE COMMAND ACCESS
authorized_for_all_service_commands=nagiosadmin
authorized_for_all_host_commands=nagiosadmin

Set up httpasswd's for the accounts for Nagios

htpasswd -c /etc/nagios/htpasswd.users nagiosadmin
htpasswd /etc/nagios/htpasswd.users guest

Configure Zimbra's http/apache to use nagios http config file

vi /opt/zimbra/conf/httpd.conf
**Add the following towards bottom**
# Include Nagios
Include /opt/zimbra/httpd/conf/extra/nagios.conf

Starting nagios is done as root

/etc/init.d/nagios start

Restarting apache for nagios issues would be done with (as zimbra)

zmapachectl stop
zmapachectl start

The webpage address to view Nagios will be like this:

http://IP_OF_SERVER:7780/nagios/

Use the rest of this how-to to configure it now: http://wiki.centos.org/HowTos/Nagios

MRTG - SNMP On Zimbra

This is a really rough draft for ideas I have in background. Shouldn't be used by anyone.

Configure yum with repo and install mrtg, net-snmp, net-snmp-utils

vi /etc/yum.repos.d/Dag.repo
yum update
yum instal mrtg net-snmp net-snmp-utils

Follow some how-to on setting up the basics.

Create a http config:

vi /opt/zimbra/httpd/conf/extra/mrtg.conf
Alias /mrtg "/opt/zimbra/mrtg"
<Directory "/opt/zimbra/mrtg">
#  SSLRequireSSL
  Options None
  AllowOverride None
  Order allow,deny
  Allow from all
#  Order deny,allow
#  Deny from all
#  Allow from 127.0.0.1
</Directory>

Add mrtg to http configuration within zimbra:

vi /opt/zimbra/conf/httpd.conf
# Include Mrtg
Include /opt/zimbra/httpd/conf/extra/mrtg.conf

Restart apache:

zmapachectl stop
zmapachectl start

Create directory to hold mrtg data:

mkdir /opt/zimbra/mrtg

Address will be something like:

http://IP_OF_SERVER:7780/mrtg/index.html

Mailq Pointing To Right Binary

# ls -la /usr/bin/mailq
 lrwxrwxrwx 1 root root 27 Sep  3 17:00 /usr/bin/mailq -> /etc/alternatives/mta-mailq
# ls -la /etc/alternatives/mta-mailq
 lrwxrwxrwx 1 root root 23 Apr  1 10:17 /etc/alternatives/mta-mailq -> /usr/bin/mailq.sendmail
# rm /usr/bin/mailq
# ln -s /opt/zimbra/postfix/sbin/mailq /usr/bin/mailq
# mailq
 Mail queue is empty

Web Client Logging

Active Sessions

Please see Zmsoap#Active_Server_Sessions_With_DumpSessionsRequest

Debug (SOAP) via Browser

See http://wiki.zimbra.com/index.php?title=Web_Client_URL_Tricks&redirect=no

Admins To View Client Issues

Within the admin console, you can view users mail.

  • Goto accounts and highlight the user having the problem.
  • Click on the View Mail button above that frame.
  • Then goto the url field of that new window and modify it to look like this [replace mailserver with yours]:
  • Hit your return key to cause the browser to reload.
    • If you get warning about pop-up, accept it.
    • If the debug window doesn't show, just mouse in the url field and hit the return key again. It should now pop up.

Jump to: navigation, search