Brian-Notes

From Zimbra :: Wiki

Jump to: navigation, search

Overview

There are 3 primary data gathering services, logger, stats and mailboxd

The logger service is a separate package that can be installed on a mailbox host and is responsible for aggregating data on mail, spam, service state and disk capacity. This data is used to Admin Console to display data in the statistics tabs as well as the service status s tab. The data is also used by cli tools like zmdailyreport and zmmsgtrace.

The stats services is installed on all ZCS instances with the zimbra-core service and is responsible for gathering background stats on things such as io, memory, cpu, process, mysql and mta queues. The data is stored in csv files under /opt/zimbra/zmstat and currently is not incorporated into a visual display in the Admin Console.

A third source of data is located in /opt/zimbra/log/zimbrastats.csv. This file is generated by the mailboxd server and contains data related to the state and performance of the server. This data is also not currently incorporated into the Admin Console graphs.

Components and data flow within the logger service.

Logger is comprised of several tools and processes, including zmlogswatch, zmlogger, zmlogprocess, zmgengraph, zmdailyreport, zmmsgtrace and a mysql instance for data storage. Data sources include postfix, amavis, zmstatuslog, zmdisklog, and zmqueuelog.

The data sources are required to log their data via syslog to either the mail or local0 facility. This data is aggregrated into /var/log/zimbra.log by the syslog daemon via configuration changes done by the ZCS installation tools. zmstatuslog, zmdisklog and zmqueuelog are executed via the zimbra crontab and log the current state of the zcs components to syslog.

The logger service uses an application called zmlogswatch to watch /var/log/zimbra.log for changes. New additions to the log file are passed on to zmlogger which is responsible for parsing the data and inserting into the raw_logs table of zimbra_logger mysql database. This process is done in real time as long as the logger service is running.

Backend processing of the logger data is handled primarily by zmlogprocess. zmlogprocess is run from the zimbra crontab every two minutes. This interval can be reduced for performance considerations by adjusting the localconfig key logger_logprocess_interval. zmlogprocess is responsible for processing data from the raw_logs table into the appropriate sections, aggregating data for amavis, mta and disk statistics and purging old data. Data lifetime is determined by global config keys zimbraLogRawLifetime and zimbraLogSummaryLifetime.

Conversion of the data into graphs displayed by the Admin Console is handled by zmgengraphs. Like zmlogprocess this is executed from the zimbra crontab every 10 minutes. This interval can be reduced by adjusting the localconfig key logger_gengraphs_interval. zmgengraphs extracts the data from mysql and creates rrd databases from which the graphs are generated.

Troubleshooting logger.

General problems.

 1. Ensure logger service is installed.

% zimbra gs `zmhostname` | grep zimbraServiceInstalled | grep logger zimbraServiceInstalled: logger

 2. Ensure logger service is enabled.

% zimbra gs `zmhostname` | grep zimbraServiceEnabled | grep logger zimbraServiceEnabled: logger

 3. Ensure logger service is running.

% zmcontrol status

Admin Console Service Status reports Service status data is notavailable. To see the server status, logger service must be installed.

 1. Verify all items under general problems.
 2. Verify crontab entry for zmstatuslog
   % crontab -l | grep zmstatuslog
     2 * * * * /opt/zimbra/libexec/zmstatuslog
 3. Verify status log entries in /var/log/zimbra.log
   % /opt/zimbra/libexec/zmstatuslog
   % tail -20 /var/log/zimbra.log | grep STATUS
     Oct 11 11:19:14 mail zimbramon[1389]: 1389:info: 2007-10-11 11:19:07, STATUS: mail.example.com: mta: Running
     Oct 11 11:19:14 mail zimbramon[1389]: 1389:info: 2007-10-11 11:19:07, STATUS: mail.example.com: mailbox: Running
     Oct 11 11:19:14 mail zimbramon[1389]: 1389:info: 2007-10-11 11:19:07, STATUS: mail.example.com: logger: Running
 4. Verify the log entries are making it into the database.  
    It's important to note that you must have entries in the service_status
    table for all other logger functions/graphs to work.  If this 
   %  logmysql -Dzimbra_logger -e "select * from service_status"
     +-------------------+-----------+---------------------+--------+-------------+
     | server                  | service    | time                    | status | loghostname |
     +-------------------+-----------+---------------------+--------+-------------+
     | mail.example.com | mta         | 2007-10-11 11:19:07 |      1 | mail        | 
     | mail.example.com | mailbox   | 2007-10-11 11:19:07 |      1 | mail        | 
     | mail.example.com | logger     | 2007-10-11 11:19:07 |      1 | mail        | 
     +-------------------+-----------+---------------------+--------+-------------+
 5. Check the log files for errors.
     /opt/zimbra/log/zmlogger.log
     /opt/zimbra/log/zmlogswatch.out
 6. Submit the output of these files to support/engineering if you
    still are having issues.


Admin Console Server Statistics are not being updated.

 1. Verify all items under general problems.
 2. Verify all server status items.
 3. Verify crontab entry for zmlogprocess
   % crontab -l | grep zmlogprocess
     10,20,30,40,50 * * * * /opt/zimbra/libexec/zmlogprocess > /tmp/logprocess.out 2>&1
 4. Check the log files for errors.
   /tmp/logprocess.out
 5. Submit the log files to support/engineering if you are still having
    issues.


Additional debugging output.

 Making these modifications should only be necessary when support or
 engineering asks for additional information.  Making this modification
 will increase the volume of data in the logfiles substantially and may
 create disk space problems if left unmonitored.
 1. Enabled debug logging in zmlogger
    Modify /opt/zimbra/libexec/zmlogger
    Change $debug = 0 to $debug = 1;
 2. Start logswatch with additional debug
    /opt/zimbra/libexec/logswatch \
     --config-file=/opt/zimbra/conf/logswatchrc --use-cpan-file-tail \
     --debug-level=10 --script-dir=/tmp -t /var/log/zimbra.log > \
     /opt/zimbra/log/zmlogswatch.out &
Personal tools