Ajcody-Logging
- This article is NOT official Zimbra documentation. It is a user contribution and may include unsupported customizations, references, suggestions, or information. |
Logging
Actual Logging Homepage
Please see Ajcody-Logging
Server
Debugging
See Ajcody-Testing-Debugging for more complete debugging information.
Syslog Items
Single Server Setup
/etc/syslog.conf should have lines similiar to:
local0.* -/var/log/zimbra.log auth.* -/var/log/zimbra.log mail.* -/var/log/zimbra.log
Restart syslog:
/etc/init.d/syslog restart
Setup ZCS to be aware of Syslog:
zmprov mcf zimbraLogTOSysLog TRUE zmcontrol stop ; zmcontrol start
This will set the appropriate ldap values (after a zimbra restart) to make log4j.properties be written correctly:
/opt/zimbra/conf/log4j.properties will now show some modifications, for example:
[snips - DON'T cut cut/paste this info below for your server] #log4j.rootLogger=INFO,LOGFILE log4j.rootLogger=INFO,LOGFILE,SYSLOG ... # Syslog appender log4j.appender.SYSLOG=org.apache.log4j.net.SyslogAppender log4j.appender.SYSLOG.SyslogHost=localhost log4j.appender.SYSLOG.Facility=LOCAL0 log4j.appender.SYSLOG.layout=com.zimbra.common.util.ZimbraPatternLayout log4j.appender.SYSLOG.layout.ConversionPattern=mailboxd: %-5p [%t] [%z] %c{1} - %m
This will actually cause very little to goto syslog though, it's a very minimalist setup.
The following is what a customer shared with me:
Adjust log4j.conf to log more items to syslog. Note: You may or may not want all these things going to syslog and you may be able to send more/other things to syslog as well. I've not spend lots of time with log4j and I'm not familiar enough with all the code to know which pieces are using which logger/settings. WARNING: the AUDIT logs may have sensitive data so be careful to protect the logs so that sensitive data is not leaked accidentally... [zimbra@zimbra conf]$ diff -u log4j.properties.in.ORIG log4j.properties.in --- log4j.properties.in.ORIG 2009-06-05 15:31:20.000000000 -0400 +++ log4j.properties.in 2009-06-09 15:54:07.000000000 -0400 @@ -35,7 +35,7 @@ # Save zimbra.security to AUDIT appender log4j.additivity.zimbra.security=false -log4j.logger.zimbra.security=INFO,AUDIT +log4j.logger.zimbra.security=INFO,AUDIT,SYSLOG # Syslog appender log4j.appender.SYSLOG=org.apache.log4j.net.SyslogAppender @@ -88,7 +88,7 @@ log4j.additivity.zimbra.wbxml=false log4j.logger.zimbra.wbxml=DEBUG,WBXML -log4j.logger.zimbra=INFO +log4j.logger.zimbra=INFO,SYSLOG log4j.logger.zimbra.op=WARN -log4j.logger.com.zimbra=INFO +log4j.logger.com.zimbra=INFO,SYSLOG
Centralized Syslog Server
Please see:
This will show you how to setup a syslog server and then to configure the "clients" to log to the local filesystem as well as the syslog server.
The syslog events are very minimal with Zimbra. Most of the logging is control by Log4j (tomcat/jetty). There's some conf files in /opt/zimbra/conf/log4j.properties* . You can adjust this file to also output into the syslog environment but syslog has limited sections for "facility". You'll end up getting huge files and g-d only knows how many log events per second.
Centralized Log Server Project Ideas
Problems:
- Logs are to large to manage on each server.
- Want to build a central logging server for all logging events.
- The syslog events are very minimal with Zimbra (/var/log/zimbra.log). Most of the logging is control by Log4j (tomcat/jetty). Output from log4j generally goes into the /opt/zimbra/log/ directory. There's some conf files in /opt/zimbra/conf/log4j.properties* for log4j. You can adjust this file to also output into the syslog environment but syslog has very limited choices for "facility". See Log4j & Syslog facility for details. You'll end up getting huge files and god knows how many log events per second if you adjusted log4j to dump all zimbra logging into syslog control.
General Thought On This:
Use syslog-ng with mysql for the central log server. Add on php-syslog-ng for comfort.
Leave the log4j stuff along and configure syslog-ng to monitor the actual log files as they are generated by log4j in /opt/zimbra/log/* .
Then decrease the log sizes and archiving done on the local servers, i.e. zimbraLogRawLifetime & zimbraLogSummaryLifetime .
References:
- Syslog-ng Homepage
- "O'Reilly - Building Secure Servers With Linux - Chapter On Syslog-ng"
- http://eduunix.ccut.edu.cn/index2/html/linux/O'Reilly%20-%20Building.Secure%20Servers%20with%20Linux/bssrvrlnx-CHP-10-SECT-2.html
- Most notably, the section on "sources"
- Php-syslog-ng
- http://code.google.com/p/php-syslog-ng/
- How-To Setup and Install Syslog-ng, mysql, and php-syslog-ng
Where's It All Go?
Any stacktrace files will have the process id of the JVM appended to the filename.
In all releases 5.0.x and 4.5.x, this is the main mailbox application log:
- /opt/zimbra/log/mailbox.log
In ZCS 5.0.x, the JVM output log and stacktrace files will be in /opt/zimbra/log/:
- /opt/zimbra/log/zmmailboxd.out (jvm output log)
- /opt/zimbra/log/stacktrace.<pid>
Meaning, changes to /opt/zimbra/conf/log4j.properties.in
This file, /opt/zimbra/conf/log4j.properties.in , also controls where some events are logged to.
Example:
# Appender AUDIT writes to the file "audit.log". log4j.appender.AUDIT=org.apache.log4j.DailyRollingFileAppender log4j.appender.AUDIT.File=/opt/zimbra/log/audit.log log4j.appender.AUDIT.DatePattern='.'yyyy-MM-dd log4j.appender.AUDIT.Append=true log4j.appender.AUDIT.layout=com.zimbra.common.util.ZimbraPatternLayout log4j.appender.AUDIT.layout.ConversionPattern=%d %-5p [%t] [%z] %c{1} - %m%n
Log Rotation and Removal
Logs In /opt/zimbra/log Directory
Please look at the output of your crontab - AS ZIMBRA, look for the "Log pruning" line.
crontab -l
And also the config files for logrotate:
/etc/logrotate.conf
/etc/logrotate.d/zimbra
Example From 5.0.11 - Complete ZCS Install Single Server
crontab for zimbra:
# Log pruning # 30 2 * * * find /opt/zimbra/log/ -type f -name \*.log\* -mtime +8 -exec rm {} \; > /dev/null 2>&1 35 2 * * * find /opt/zimbra/log/ -type f -name \*.out.???????????? -mtime +8 -exec rm {} \; > /dev/null 2>&1 # Log pruning # 30 2 * * * find /opt/zimbra/mailboxd/logs/ -type f -name \*log\* -mtime +8 -exec rm {} \; > /dev/null 2>&1
And the logrotate files:
$ cat /etc/logrotate.conf # see "man logrotate" for details # rotate log files weekly weekly # keep 4 weeks worth of backlogs rotate 4 # create new (empty) log files after rotating old ones create # uncomment this if you want your log files compressed #compress # RPM packages drop log rotation information into this directory include /etc/logrotate.d # no packages own wtmp -- we'll rotate them here /var/log/wtmp { monthly minsize 1M create 0664 root utmp rotate 1 } # system-specific logs may be also be configured here.
And also:
$ cat /etc/logrotate.d/zimbra /var/log/zimbra.log { daily missingok notifempty create 0644 zimbra zimbra postrotate killall -HUP syslogd 2> /dev/null || true kill -HUP `cat /opt/zimbra/log/swatch.pid 2> /dev/null` 2> /dev/null || true kill -HUP `cat /opt/zimbra/log/logswatch.pid 2> /dev/null` 2> /dev/null || true endscript compress } /opt/zimbra/log/myslow.log { daily missingok copytruncate rotate 30 notifempty create 0644 zimbra zimbra compress } /opt/zimbra/log/logger_myslow.log { daily missingok copytruncate notifempty create 0660 zimbra zimbra compress size 5000k rotate 7 } /opt/zimbra/log/clamd.log { daily missingok copytruncate notifempty create 0660 zimbra zimbra postrotate kill -HUP `cat /opt/zimbra/log/clamd.pid 2> /dev/null` 2> /dev/null || true endscript compress size 5000k rotate 7 } /opt/zimbra/log/zmlogswatch.out { daily missingok copytruncate notifempty create 0740 zimbra zimbra postrotate su - zimbra -c "/opt/zimbra/bin/zmlogswatchctl stop" su - zimbra -c "/opt/zimbra/bin/zmlogswatchctl start" endscript rotate 5 compress } /opt/zimbra/log/zmswatch.out { daily missingok copytruncate notifempty create 0740 zimbra zimbra postrotate su - zimbra -c "/opt/zimbra/bin/zmswatchctl stop" su - zimbra -c "/opt/zimbra/bin/zmswatchctl start" endscript rotate 5 compress } /opt/zimbra/log/zmmtaconfig.log { daily missingok copytruncate notifempty create 0740 zimbra zimbra postrotate su - zimbra -c "/opt/zimbra/bin/zmmtaconfigctl restart" endscript rotate 5 compress } /opt/zimbra/log/nginx.log { daily missingok notifempty create 0644 zimbra zimbra postrotate kill -USR1 `cat /opt/zimbra/log/nginx.pid 2> /dev/null` 2> /dev/null || true endscript rotate 7 compress } /opt/zimbra/log/zmconvertd.log { daily missingok copytruncate notifempty create 0644 zimbra zimbra rotate 7 } /opt/zimbra/zmstat/zmstat.out { daily missingok copytruncate rotate 7 notifempty compress }
Openldap Logs - In /opt/zimbra/openldap-data/
Ldap logs files in /opt/zimbra/openldap-data/ that are like log.0000000001 , log.0000000002 , etc.
This logpurge is controlled by:
$ cd /opt/zimbra/openldap/etc/openldap/ [zimbra@mail3 openldap]$ grep logpurge * master-accesslog-overlay.conf:logpurge 07+00:00 01+00:00
Logpurge Directive Description
From Accesslog Chapter
logpurge Directive
logpurge age interval
Defines both the maximum age for log entries to be retained in the database and how often to scan the database for old entries. Bothage and interval are specified as a time span in days, hours, minutes, and seconds. The time format is [ddd+]hh:mm[:ss], for example, the days and seconds components are optional but hours and minutes are required. Except for days, which can be up to 5 digits, each numeric field must be exactly two digits. Example:
- the log database will be scanned every day
- entries older than two days will be deleted.
logpurge 2+00:00 1+00:00
When using a log database that supports ordered indexing on generalizedTime attributes, specifying an eq index on the reqStart attribute will increase the performance of purge operations.
What's up with all the logs?
Logging page resources:
http://www.zimbra.com/docs/ne/latest/administration_guide/9_Monitoring.14.1.html
http://wiki.zimbra.com/index.php?title=Server_Monitoring
http://wiki.zimbra.com/index.php?title=Monitoring_Zimbra_Servers
http://wiki.zimbra.com/index.php?title=Log_Files
http://wiki.zimbra.com/index.php?title=Unresponsive_Server_Troubleshooting
The Bread And Butter Logs
- /opt/zimbra/log/mailbox.log
- This log is a mailboxd log4j server log containing the logs from the mailbox server. This includes the mailbox store, LMTP server, IMAP and POP servers, and Index server.
- /opt/zimbra/log/zmmailbox.out
- Mailstore not coming up and nothing is being logged in mailbox.log, check here for errors.
- /var/log/zimbra.log
- The Zimbra syslog details the activities of the Zimbra MTA (Postfix, amavisd, antispam, antivirus), Logger, Authentication (cyrus-sasl), and Directory (OpenLDAP). By default LDAP activity is logged to Zimbra.log.
- /opt/zimbra/db/data/YOURHOSTNAME.err
- This is the message store database error log.
- /var/log/messages
Increase Logging
Increase Logging Per User
Depending on the ZCS version the below will clear all the per account loggers on zimbra reboot. IMAP is used for these examples.
- addAccoutLogger
zmprov aal user@domain.com zimbra.imap debug
- removeAccountLogger
zmprov ral user@domain.com zimbra.imap
Example usuage:
zmprov aal user@domain.com zimbra.soap debug
And tail the log file, while attempting to login to the AJAX client.:
tail -f /opt/zimbra/log/mailbox.log
See Ajcody-Logging#Log4J_Variables for variables that can be used.
Bugs And RFE's
I filed the following RFE:
- "Allow "secondary" logging file for zmprov addAccountLogger"
Increase Logging Per Server
If you want to do it globally edit this file:
vi /opt/zimbra/conf/log4j.properties
Add a line at the end that's similar to something like this: log4j.logger.zimbra.imap=DEBUG.
No restart of any service is needed, BUT if something happens that causes a regeneration of the log4j.properties file then your changes will be overwritten. A regeneration of this file takes the contents of /opt/zimbra/conf/log4.properties.in .
To permanently make a change, modify the /opt/zimbra/conf/log4.properties.in file, then a restart would be necessary.
$ zmcontrol stop $ zmcontrol start
or you could run this on the server:
$ zmmtaconfig mailboxd
Wait for a minute for the server to pick up your change.
See Ajcody-Logging#Log4J_Variables for variables that can be used.
Log4J Variables
"log4j.logger.zimbra.misc" logger. For all events that don't have a specific-category.
"log4j.logger.zimbra.index" logger. For indexing-related events.
"log4j.logger.zimbra.index.lucene" logger. For logging of low-level lucene operations (debug-level only)
"log4j.logger.zimbra.searchstat" logger. For logging statistics about what kinds of searches are run
"log4j.logger.zimbra.redolog" logger. For redolog-releated events.
"log4j.logger.zimbra.lmtp" logger. For LMTP-related events.
"log4j.logger.zimbra.smtp" logger. For SMTP-related events.
"log4j.logger.zimbra.nio" logger. For NIO-related events.
"log4j.logger.zimbra.imap" logger. For IMAP-related events.
"log4j.logger.zimbra.pop" logger. For POP-related events.
"log4j.logger.zimbra.mailbox" logger. For mailbox-related events.
"log4j.logger.zimbra.calendar" logger. For calendar-related events.
"log4j.logger.zimbra.im" logger. For instant messaging-related events.
"log4j.logger.zimbra.account" logger. For account-related events.
"log4j.logger.zimbra.gal" logger. For account-related events.
"log4j.logger.zimbra.ldap" logger. For ldap-related events.
"log4j.logger.zimbra.security" logger. For security-related events
"log4j.logger.zimbra.soap" logger. For soap-related events
"log4j.logger.zimbra.test" logger. For testing-related events
"log4j.logger.zimbra.sqltrace" logger. For tracing SQL statements sent to the database
"log4j.logger.zimbra.dbconn" logger. For tracing database connections
"log4j.logger.zimbra.perf" logger. For logging performance statistics
"log4j.logger.zimbra.cache" logger. For tracing object cache activity
"log4j.logger.zimbra.filter" logger. For filter-related logs.
"log4j.logger.zimbra.session" logger. For session- and notification-related logs.
"log4j.logger.zimbra.backup" logger. For backup/restore-related logs.
"log4j.logger.zimbra.system" logger. For startup/shutdown and other related logs.
"log4j.logger.zimbra.sync" logger. For sync client interface logs.
"log4j.logger.zimbra.synctrace" logger. For sync client interface logs.
"log4j.logger.zimbra.syncstate" logger. For sync client interface logs.
"log4j.logger.zimbra.wbxml" logger. For wbxml client interface logs.
"log4j.logger.zimbra.extensions" logger. For logging extension loading related info.
"log4j.logger.zimbra.zimlet" logger. For logging zimlet related info.
"log4j.logger.zimbra.wiki" logger. For wiki and document sharing.
"log4j.logger.zimbra.op" logger. Logs server operations
"log4j.logger.zimbra.dav" logger. Logs dav operations
"log4j.logger.zimbra.io" logger. Logs file IO operations.
"log4j.logger.zimbra.datasource" logger. Logs data source operations.
"log4j.logger.zimbra.webclient" logger. Logs ZimbraWebClient servlet and jsp operations.
"log4j.logger.zimbra.scheduler" logger. Logs scheduled task operations.
"log4j.logger.zimbra.store" logger. Logs filesystem storage operations.
"log4j.logger.zimbra.fb" logger. Logs free/busy operations.
"log4j.logger.zimbra.purge" logger. Logs mailbox purge operations.
"log4j.logger.zimbra.mailop" logger. Logs changes to items in the mailbox.
OpenLDAP Logging Levels
Please see King0770-Notes-Change-LDAP-Log-Levels
Want To See What's In Redolog Files
If you suspect there's too much redolog activity during a time window or have another need to inspect the contents of the redolog, dump it and examine it:
$ zmjava com.zimbra.cs.redolog.util.RedoLogVerify /opt/zimbra/redolog/redo.log > out.file
Pick the right redolog file, either redo.log or one of the files under archive/, based on timestamp.
See Ajcody-Notes#Redolog_Files for more details.
Internal Zimbra Charting - zmstat-chart
RFE/Bugs You Might Be Interested In
- "support for real time counters (snmp, jmx, etc)"
- "zmstat - expand documentation"
- "zmstat-chart - graphing besides just one day"
- "zmstat - integrate webpage setup and allow configuration of chart run"
- "zmstat - should have some trending data on ALL client connections (web/soap)"
zmstat IS NOT Logger (graphs in admin console)
Please note, zmstat is different than the processes and graphs involved with Logger. Logger is behind the graphs that show up in the admin web console.
Please see Logger for details and trouble-shooting steps with Logger.
Rick has some other steps for King0770-Notes#Reinitialize_the_Logger_DB
Main References For zmstat-chart Command
Please see the following:
zmstat-chart CLI Administration Guide
zmstat-chart-config CLI Administration Guide
zmstatctl CLI Administration Guide
Monitoring Administration Guide
JFreeChart is the project/binary behind the charting tool. You'll find the jar files here [ZCS 5.0.9 example]:
- /opt/zimbra/lib/jars/jfreechart-1.0.1.jar
- /opt/zimbra/jetty-6.1.5/common/lib/jfreechart-1.0.1.jar
Issues Being Investigated With zmstat And Other "Charting" Items
I currently have an internal thread going with the dev's about some issues with zmstat and also with attempts of getting the active sessions data from the admin web console to be seen in trends.
I'll just be dumping my notes here and then cleaning them up once I'm done with the internal conversation.
- Question: (mailboxd: active connections by client protocol) What exactly is this tracking? I'm on a test box with no connections and it has the different threads in the 100's/1000's. Customer was hoping (as was I) that this would be giving data much like is shown in the admin console about user connections. Also, there doesn't seem to be a comparable "active connections" for the web client (soap)?
- Answer: For the connections issue, it's the number of clients that are currently connected. There's no analog for SOAP because SOAP doesn't have persistent connections. We may be able to log stats for number of SOAP sessions, but that would include noise for cases where the user closes the browser window and the session hasn't timed out yet. If you think this is useful, please file an RFE.
- Question: (graph time plot) If I run zmstat-chart with the --aggregate-start-at option with something like "08/07/2008 01:00:00" I notice that the graphs don't actually reference "days" at the bottom but still use what looks to be "hours". Is this a bug where the bottom reference of the graph doesn't get adjusted or is the syntax to run zmstat-chart to do this more tricky that I think.
- Answer: zmstat-chart doesn't support multi-day ranges, it really only supports a single directory with a single day of charts, the aggregate options are for specifying ranges at which to calculate the max(col)/min(col)/avg(col) values. Our usual practice is to generate one set of charts for each day we want to look at; I think it gives us a better look at when peaks occur and how they compare day-to-day (it's much easier to look at them in an overlay fashion).
- Answer: Seems like a reasonable request to be able to generate charts for multiple days. Please file an RFE if you think this would be useful.
- Question: How reliable are these numbers are in admin console....
- Answer: I imagine they should be quite reliable; but HTTP (Web) is stateless, so you can't really determine whether a session is really active or not. I believe multiple sessions appearing for a single user would be the result of reloading or closing and then reopening the browser to log in again.
- Perhaps it is best to ask the users in question what their usage is like? Is there any concern over the high number of sessions? I don't think it should affect server performance.
- Answer: I imagine they should be quite reliable; but HTTP (Web) is stateless, so you can't really determine whether a session is really active or not. I believe multiple sessions appearing for a single user would be the result of reloading or closing and then reopening the browser to log in again.
- Question: Could you please explain the following:
- Mailboxd Mailbox Add Rate (Delivery Rate)
- Answer: The rate at which messages are being added to the mailbox server, e.g. 1 message being delivered per second is... 1 m/s
- Mailboxd Mailbox Add Latency (Delivery Speed)
- Answer: How long it takes on average to add a message to a mailbox (averaged over the period of a minute)
- Mailboxd Active Connections by Client Protocol- This one does NOT give us counts for web client. Host zcs2.mail.XXXX.xxx: Mailboxd: Active Connections by Client Protocol avg(IMAP) = 169.86 avg(IMAP SSL) = 0.00 avg(POP) = 0.21 avg(POP SSL) = 0.00
- Answer: It doesn't give counts for web client because the web client does not have persistent connections (being HTTP-based). SOAP requests (used by the Web client) use transient connections that often close after they are used; i.e. if you took the number of "SOAP" connections and compared it to the number of "active" sessions, it'd likely be something like 0-1% of the latter figure.
- Mailboxd Mailbox Get Count
- Answer: The number of mailboxes that got opened in the last minute (I don't recall if this counting cache hits or not?)
- Mailboxd Mailbox Get Latency
- Answer: How long it takes to open a mailbox, on average (over a minute)
- Also, we don't get stats for any of the soap parameters:
- SOAP Invocation Count Summary (Top 10 max)
- SOAP Average Call Duration AuthRequest
- Answer: I don't know why this could be; perhaps zmstat-chart-config.xml needs to be regenerated, is /opt/zimbra/zmstat/soap.csv available and does it contain data?
- Mailboxd Mailbox Add Rate (Delivery Rate)
Running zmstat-chart
Confirm it's running:
zmprov gs [mailserver hostname] | grep -i stats zimbraServiceEnabled: stats zimbraServiceinstalled: stats
Default location of stat files is:
/opt/zimbra/zmstat/
Example command to run:
mkdir /tmp/stats zmstat-chart -s /opt/zimbra/zmstat -d /tmp/stats/
Like To Have zmstat-chart Data Integrated With Zimbra
I've made an RFE for this integration:
- "zmstat - integrate webpage setup and allow configuration of chart run"
With Zimbra Jetty
- Note:
- The default Jetty behavior will NOT display directory listing. You'll need a valid html type file in the directory target.
- I would recommend the apache solution over this one as you can avoid hitting performance issues this might cause.
Dump some graphing directories in there:
zmstat-chart -s /opt/zimbra/zmstat -d /opt/zimbra/jetty/webapps/zimbra/downloads/zmstat-chart/`date +%F-%H-%M`
You'll see there's the directory and when you go into it - there's your charts.
You can now place this command in your crontab to run on a schedule basis.
Things you'll want to check before deploying:
- Place charting directory in a location where space is available and will not create a risk to the mail services if it's full.
- Look at the zmstat-chart options to see what other parameters you want to use - especially if it's going in your crontab.
- Make sure your following and security guidelines for your environment.
You'll be able to view the data with the following url:
http://hostname/zimbra/downloads/zmstat-chart/
With Zimbra Apache Or Non-Zimbra Server With Apache
Before You Proceed
Things to check or confirm first before deploying:
- Place your directory root [/opt/zimbra/support in this example] in a location where space is available and will not create a risk to the mail services if it's full.
- If you'll be using NFS to have a centralized storage point:
- Your NFS mount point on the Zimbra servers would be /opt/zimbra/support in this example.
- You'll be exporting /opt/zimbra/support in this example from the NFS server.
- If this is a non-zimbra server, copy the zimbra entry from a zimbra server's /etc/passwd and /etc/group into the non-zimbra servers passwd and group file. This way, if NFS is also used the UID/GID matches.
- Apache Directory variable will be /opt/zimbra/support/data in this example.
- Apache .htpasswd location will be /opt/zimbra/support/.htpasswd in this example, putting it above of the apache Directory variable.
- Make sure your following the security guidelines for your environment.
- Look at the zmstat-chart options to see what other parameters you want to use - especially if it's going in your crontab.
Directory Layout
As root:
mdir -p /opt/zimbra/support/data cd /opt/zimbra/support/data for i in `/opt/zimbra/bin/zmprov gas` do mkdir $i done
If your not on a ZCS server, you'll be manually creating directories named after the output of zmhostname from each of your ZCS servers.
Now, create subdirectory paths under each zmhsotname directory.
cd /opt/zimbra/support/data/ for i in `ls` do mkdir $i/debug $i/logs $i/zmstat done
Set permissions on the directories we made:
chown -R zimbra:zimbra /opt/zimbra/support chmod -R 755 /opt/zimbra/support
With this structure and with NFS setup, you'll have a common repository that will have a standard path to use on a per server basis. Example :
script-command -PathOption /opt/zimbra/support/data/`zmhostname`/TYPE_OF_DATA/`date +%F-%H-%M`
Apache Configuration
Let's configure apache to show this directory.
- For ZCS Apache Setup:
vi /opt/zimbra/conf/httpd.conf ** Towards the end, add the following** # Include zmstat-chart directory Include /opt/zimbra/httpd/conf/extra/zimbra-support.conf
- Non-ZCS Server With Apache. Check that there's an existing Include directive that will see our zimbra-support.conf:
vi /etc/httpd/conf/httpd.conf ** Look for something that will allow the following** ** /etc/httpd/conf.d/zimbra-support.conf to be used** ** Your distro might differ on apache paths as well** ** as Include statement for other conf files** Include conf.d/*.conf
Let's now make the apache conf file for zimbra-support.conf
- For ZCS Apache Setup:
vi /opt/zimbra/httpd/conf/extra/zimbra-support.conf Alias /support "/opt/zimbra/support/data" <Directory "/opt/zimbra/support/data"> AuthName "Secure Area For Zimbra Support" AuthType Basic AuthUserFile /opt/zimbra/support/.htpasswd require valid-user Options Indexes IndexOptions FancyIndexing VersionSort AllowOverride None Order allow,deny Allow from all </Directory>
- Non-ZCS Server With Apache:
vi /etc/httpd/conf.d/zimbra-support.conf Alias /support "/opt/zimbra/support/data" <Directory "/opt/zimbra/support/data"> AuthName "Secure Area For Zimbra Support" AuthType Basic AuthUserFile /opt/zimbra/support/.htpasswd require valid-user Options Indexes IndexOptions FancyIndexing VersionSort AllowOverride None Order allow,deny Allow from all </Directory>
Make sure permissions are set correctly on zimbra-support.conf so apache will use it.
Now let's setup a username and password to use with the .htpasswd file.
cd /opt/zimbra/support/ htpasswd -c .htpasswd zimbrasupport chmod 644 .htpasswd
Restart apache so the changes take effect:
- For ZCS Apache Setup:
zmapachectl stop zmapachectl start
- Non-ZCS Server With Apache:
httpd restart
Now, let's dump some graphing directories in there:
zmstat-chart -s /opt/zimbra/zmstat -d /opt/zimbra/support/data/`zmhostname`/zmstat/`date +%F-%H-%M`
To now view the charts. Adjust, SERVERHOSTNAME, for your servername.
- For ZCS Apache Setup:
http://servername:7780/support/data/SERVERHOSTNAME/zmstat
- Non-ZCS Server With Apache:
http://servername/support/data/SERVERHOSTNAME/zmstat
You'll see there's the directory and when you go into it - there's your charts.
You can now place this command in your crontab to run on a schedule basis.
Tweaking zmstat-chart
zmstat-chart also has the option to use a configuration file. The default configuration file is:
/opt/zimbra/conf/zmstat-chart.xml
Copy this to another location and edit away. Then add the -c /location/to/config/file to use your new configuration.
Default Items Charted
Here's a list of the default items that will be charted with zmstat-chart:
- Postfix Queue Size
- convertd CPU time used
- convertd Resident Memory
- convertd Processes and Threads
- Total CPU
- Process CPU
- Process Total Memory
- Process Resident Memory
- Virtual Memory
- Context Switches
- Run/Blocked Process Queue Size
- Disk Partition Throughput
- Disk Utilization
- Disk Throughput
- Disk IOPs
- Swap Activity
- Total file descriptors open
- SOAP Invocation Count Summary (Top 10 max)
- SOAP Average Call Duration Summary (Top 10 avg)
- SOAP Invocation Count GetServerRequest
- SOAP Invocation Count BackupRequest
- SOAP Invocation Count GetAllConfigRequest
- SOAP Invocation Count GetDomainRequest
- SOAP Invocation Count GetAllDomainsRequest
- SOAP Invocation Count AuthRequest
- SOAP Average Call Duration GetServerRequest
- SOAP Average Call Duration BackupRequest
- SOAP Average Call Duration GetAllConfigRequest
- SOAP Average Call Duration GetDomainRequest
- SOAP Average Call Duration GetAllDomainsRequest
- SOAP Average Call Duration AuthRequest
- MySQL Database Connections
- MySQL InnoDB Buffer Pool Pages
- MySQL InnoDB Buffer Pool Hit Rate
- MySQL Tables Open/Opened
- MySQL Total Slow Queries Count
- Mailboxd Connection Pool Get Latency
- Mailboxd Dirty Lucene Index Writers
- Mailboxd Lucene IndexWriterCache Hitrate
- Mailboxd Lucene IO
- Mailboxd LMTP Delivery Throughput
- Mailboxd LMTP Delivery Rate
- Mailboxd Mailbox Add Rate (Delivery Rate)
- Mailboxd Mailbox Add Latency (Delivery Speed)
- Mailboxd Request Rate by Client Protocol
- Mailboxd Response Time by Client Protocol
- Mailboxd Active Connections by Client Protocol
- Mailboxd Mailbox Get Count
- Mailboxd Mailbox Get Latency
- Mailboxd Mailbox Cache Hit Rate
- Mailboxd Mailbox Item/Blob Cache Hit Rate
- Mailboxd Garbage Collection Time
- Mailboxd Garbage Collection Count
- Mailboxd JVM Heap Used
- Mailboxd JVM Heap Free
- Mailboxd JVM Permanent Generation and Code Cache
To get quota stats, run. The output is a little bit difficult to read since it's in bytes:
zmprov gqu [server-name]
Here's a script to put into user readable format below [by Tony P.]. You'll need to modify the $hostname variable near the top:
#!/usr/bin/perl $hostname = localhost; @quotas = `zmprov gqu $hostname`; foreach $quotaline(@quotas) { chop($quotaline); ($email, $quota, $usage) = split(/\s/,$quotaline); $quota=utils_convert_bytes_to_optimal_unit($quota); $usage=utils_convert_bytes_to_optimal_unit($usage); print "$email - $usage\n"; } sub utils_convert_bytes_to_optimal_unit{ my($bytes) = @_; return '' if ($bytes eq ''); my($size); $size = $bytes . ' Bytes' if ($bytes < 1024); $size = sprintf("%.2f", ($bytes/1024)) . ' KB' if ($bytes >= 1024 && $bytes < 1048576); $size = sprintf("%.2f", ($bytes/1048576)) . ' MB' if ($bytes >= 1048576 && $bytes < 1073741824); $size = sprintf("%.2f", ($bytes/1073741824)) . ' GB' if ($bytes >= 1073741824 && $bytes < 1099511627776); $size = sprintf("%.2f", ($bytes/1099511627776)) . ' TB' if ($bytes >= 1099511627776); return $size; }
Third Party Tools (Monitoring & Logging)
FYI - Support For Real Time Counters (snmp,etc.)
This is for those that need to go beyond what zmstat-chart is doing or need to intergrate within a third party monitoring systems.
Please see RFE:
- "Support for real time counters (snmp, jmx, etc)"
Currently, this RFE has only resolved support for JMX, not SNMP. Customers who need SNMP can use a product like jManage to do the translation from JMX to SNMP.
Some Choices
Charting & Graphing The Data
Montoring Software
- Nagios
- Centreon - based upon Nagios
- Splunk
- You could also look at using Splunk and Nagios together.
- BigSister - take off of BigBrother
- Zenoss
- Cacti
- OpenNMS
- Munin
Nagios On Zimbra
This is a really rough draft for ideas I have in background. Shouldn't be used by anyone.
Configure Nagios to run on single server Zimbra box - Centos 5.x
Configure yum with repo and install nagios
vi /etc/yum.repos.d/Dag.repo yum update yum install nagios nagios-plugins nagios-devel nagios-plugins-nrpe
Move nagios.conf http file into main zimbra directory.
cp /etc/httpd/conf.d/nagios.conf /opt/zimbra/httpd/conf/extra/
Setup nagios to run as zimbra
vi /etc/nagios/nagios.cfg **Change nagios user to zimbra** nagios_user=zimbra nagios_group=zimbra
Change ownership of directories from nagios to zimbra.
chown -R 500:500 /var/log/nagios/ chown -R 500:500 /etc/nagios/ chown -R 500:500 /usr/share/nagios/
Configure authentication within Nagios
vi /etc/nagios/nagios.cfg # AUTHENTICATION USAGE use_authentication=1 # SYSTEM/PROCESS INFORMATION ACCESS authorized_for_system_information=nagiosadmin # CONFIGURATION INFORMATION ACCESS authorized_for_configuration_information=nagiosadmin # SYSTEM/PROCESS COMMAND ACCESS authorized_for_system_commands=nagiosadmin # GLOBAL HOST/SERVICE VIEW ACCESS authorized_for_all_services=nagiosadmin authorized_for_all_hosts=nagiosadmin # GLOBAL HOST/SERVICE COMMAND ACCESS authorized_for_all_service_commands=nagiosadmin authorized_for_all_host_commands=nagiosadmin
Set up httpasswd's for the accounts for Nagios
htpasswd -c /etc/nagios/htpasswd.users nagiosadmin htpasswd /etc/nagios/htpasswd.users guest
Configure Zimbra's http/apache to use nagios http config file
vi /opt/zimbra/conf/httpd.conf **Add the following towards bottom** # Include Nagios Include /opt/zimbra/httpd/conf/extra/nagios.conf
Starting nagios is done as root
/etc/init.d/nagios start
Restarting apache for nagios issues would be done with (as zimbra)
zmapachectl stop zmapachectl start
The webpage address to view Nagios will be like this:
http://IP_OF_SERVER:7780/nagios/
Use the rest of this how-to to configure it now: http://wiki.centos.org/HowTos/Nagios
MRTG - SNMP On Zimbra
This is a really rough draft for ideas I have in background. Shouldn't be used by anyone.
Configure yum with repo and install mrtg, net-snmp, net-snmp-utils
vi /etc/yum.repos.d/Dag.repo yum update yum instal mrtg net-snmp net-snmp-utils
Follow some how-to on setting up the basics.
Create a http config:
vi /opt/zimbra/httpd/conf/extra/mrtg.conf Alias /mrtg "/opt/zimbra/mrtg" <Directory "/opt/zimbra/mrtg"> # SSLRequireSSL Options None AllowOverride None Order allow,deny Allow from all # Order deny,allow # Deny from all # Allow from 127.0.0.1 </Directory>
Add mrtg to http configuration within zimbra:
vi /opt/zimbra/conf/httpd.conf # Include Mrtg Include /opt/zimbra/httpd/conf/extra/mrtg.conf
Restart apache:
zmapachectl stop zmapachectl start
Create directory to hold mrtg data:
mkdir /opt/zimbra/mrtg
Address will be something like:
http://IP_OF_SERVER:7780/mrtg/index.html
Mailq Pointing To Right Binary
# ls -la /usr/bin/mailq lrwxrwxrwx 1 root root 27 Sep 3 17:00 /usr/bin/mailq -> /etc/alternatives/mta-mailq # ls -la /etc/alternatives/mta-mailq lrwxrwxrwx 1 root root 23 Apr 1 10:17 /etc/alternatives/mta-mailq -> /usr/bin/mailq.sendmail # rm /usr/bin/mailq # ln -s /opt/zimbra/postfix/sbin/mailq /usr/bin/mailq # mailq Mail queue is empty
Web Client Logging
Active Sessions
Please see Zmsoap#Active_Server_Sessions_With_DumpSessionsRequest
Debug (SOAP) via Browser
See http://wiki.zimbra.com/index.php?title=Web_Client_URL_Tricks&redirect=no
Admins To View Client Issues
Within the admin console, you can view users mail.
- Goto accounts and highlight the user having the problem.
- Click on the View Mail button above that frame.
- Then goto the url field of that new window and modify it to look like this [replace mailserver with yours]:
- Hit your return key to cause the browser to reload.
- If you get warning about pop-up, accept it.
- If the debug window doesn't show, just mouse in the url field and hit the return key again. It should now pop up.