Logger (ZCS 6.0.x and later): Difference between revisions

m (minor corrections.)
m (- added note about zmmsgtrace being reintroduced in 7.1.1)
 
(12 intermediate revisions by 7 users not shown)
Line 1: Line 1:
{{BC|Community Sandbox}}
__FORCETOC__
<div class="col-md-12 ibox-content">
=Logger (ZCS 6.0 and later)=
{{KB|{{Unsupported}}|{{ZCS 6.0}}||}}
{{Archive}}{{WIP}}
= Logger in version 6.0 (GnR) =
= Logger in version 6.0 (GnR) =


Line 15: Line 21:
to the old service are no longer available.
to the old service are no longer available.


zmmsgtrace has been removed with no plan for a replacement.
[[CLI_zmmsgtrace|zmmsgtrace]] was removed in ZCS 6, however it was reintroduced in ZCS 7.1.1.


== Architecture and Implementation ==
== Architecture and Implementation ==
Line 31: Line 37:
=== zmlogprocess ===
=== zmlogprocess ===


Zmlogprocess is launched out of cron on the logger host every 5 minutes and
Zmlogprocess is launched out of cron on the logger host every 10 minutes.
reads the 'unprocessed.log' created by zmlogger.  Upon launch, zmlogprocess
renames 'unprocessed.log' so that zmlogger can start spooling to a new file.


This job reads MTA data from unprocessed.log: MTA traffic and volume, SPAM
This job reads MTA data from zimbra.log: MTA traffic and volume, SPAM
and Virus infections.  The data is then sent back to zmlogger as stat counters
and Virus infections.  The data is then sent back to zmlogger as stat counters
via syslog.
via syslog.
Line 54: Line 58:
   message-type: headers(csv):: data-columns(csv)
   message-type: headers(csv):: data-columns(csv)


Messages over 900 characters in length are broken up using a uuid eliding
Messages over 800 characters in length are broken up using a uuid eliding
scheme.  This scheme is required due to many syslogd implementations
scheme.  This scheme is required due to many syslogd implementations
restricting message length to be 1022 bytes or shorter.  The scheme works
restricting message length to be 1022 bytes or shorter.  The scheme works
Line 106: Line 110:
the RRD files.  The information will often be used to form graphs and charts
the RRD files.  The information will often be used to form graphs and charts
in the admin console.
in the admin console.


== Troubleshooting ==
== Troubleshooting ==
Line 116: Line 119:
=== Statistics show no data available ===
=== Statistics show no data available ===


# zmloggerctl status; ps ax | grep zmlogger
zmloggerctl status; echo $?; ps ax | grep zmlogger
# /opt/zimbra/libexec/zmrrdfetch -f zmmtastats
/opt/zimbra/libexec/zmrrdfetch -f zmmtastats
# grep -w MTA /var/log/zimbra-stats.log
/opt/zimbra/libexec/zmrrdfetch -c -f zmmtastats
# tail /var/log/zimbra-stats.log
grep -w MTA /var/log/zimbra-stats.log
# zmsoap -z GetLoggerStatsRequest stats/@name=zmmtastats | head -20
tail /var/log/zimbra-stats.log
zmsoap -z GetLoggerStatsRequest stats/@name=zmmtastats | head -20
zmsoap -z GetLoggerStatsRequest stats/@name=zmmtastats @limit=1 | head -50


Take the above information, and post/attach the information to support or the Zimbra Forums
Take the above information, and post/attach the information to support or the Zimbra Forums
Line 126: Line 131:
{{Article Footer|Zimbra Collaboration Suite 6.0.x|4/9/2009}}
{{Article Footer|Zimbra Collaboration Suite 6.0.x|4/9/2009}}
[[Category:Logger]]
[[Category:Logger]]
[[Category:ZCS 6.0]]

Latest revision as of 20:19, 7 October 2015

Logger (ZCS 6.0 and later)

   KB 2827        Last updated on 2015-10-7  




0.00
(0 votes)

Logger in version 6.0 (GnR)

Description

Logger is a non-essential ZCS service that processes logs produced by Zimbra, and associated services. Logs are processed into a format that is readily displayed in the Zimbra Admin Console indicating real-time service status, MTA/spam/virus traffic and performance statistics.

What's changed since 5.0 (and earlier)

The logger service has been completely redesigned and redeveloped for ZCS 6.0

The service no longer relies on mysql and most command-line utilities related to the old service are no longer available.

zmmsgtrace was removed in ZCS 6, however it was reintroduced in ZCS 7.1.1.

Architecture and Implementation

zmlogger

This process is launched through zmloggerctl and runs on a pipe from zmlogswatch, it tails /var/log/zimbra-stats.log and captures information from the 'zimbramon' and 'mailboxd' services and parses out statistics information.

Zmlogger reads an sqlite database to determine the type of each stat counter and mappings to RRD file:data-columns and writes the stat counters to the appropriate RRD file(s).

zmlogprocess

Zmlogprocess is launched out of cron on the logger host every 10 minutes.

This job reads MTA data from zimbra.log: MTA traffic and volume, SPAM and Virus infections. The data is then sent back to zmlogger as stat counters via syslog.

zmstat

Zmstats is also integrated into the logger service by broadcasting its stat counters over syslog in addition to using the old CSV files.

Stats will still be readable using the zmstat-chart tool in addition to live, real-time data displayed in the admin console.

syslog

The logger service will use syslog as a mechanism for aggregating data to a centralized server. Messages are generally sent in the format of

 message-type: headers(csv):: data-columns(csv)

Messages over 800 characters in length are broken up using a uuid eliding scheme. This scheme is required due to many syslogd implementations restricting message length to be 1022 bytes or shorter. The scheme works by appending :::uuid to prepending :::uuid to messages that are split. The UUID allows easily matching long messages together in the event multiple long messages occur at once (common).

sqlite

SQLite is used as an integration database. It maps dns names to zmhostname when there are mismatches, stores information about stat counter types and a mapping of a stat counter to an RRD file's datasource column.

Schema

Our integration database schema is as follows:

  • hosts: columns (id, dns_hostname, zm_hostname); generated dynamically as logger works

Self-explanatory

  • rrds: columns (id, host_id, col_name, col_name_19, col_num, csv_file, rrd_file); also generated dynamically as logger works
    • host_id is the host for which this column is mapping
    • col_name_19 is the name of the column in the rrd file
    • col_num is the index of the column in the rrd_file
    • csv_file is the grouping of the stat counter
    • rrd_file is a number that, combined with the host_id, determines the rrd filename
  • datatypes: columns (csv_file, col_name, col_type); static mappings table (coming in beta2)
    • csv_file is the grouping of the stat counter
    • col_name is the column for which this is determining the datatype
    • col_type is the type of the column GAUGE, DERIVED, COUNTER or ABSOLUTE (rrd types)
  • config: columns (schema_version); single row table used for schema versioning (coming in beta2)

zmloggerhostmap

This is a command used to manipulate the DNS hostname to zmhostname mappings when there is a mismatch in the information reported to Logger

rrd

RRDs are used to store stat counter information, it manages aggregation and averaging. Generally, one RRD file is generated per group of statistics and information is retained for 2 years in varying levels of granularity (from 30 second to 1 hour resolution).

zmrrdfetch

This is an internal command that ZCS will use to fetch and combine data from the RRD files. The information will often be used to form graphs and charts in the admin console.

Troubleshooting

No chart shows

Is flash installed? The new charts now require flash to generate images on the client-side.

Statistics show no data available

zmloggerctl status; echo $?; ps ax | grep zmlogger
/opt/zimbra/libexec/zmrrdfetch -f zmmtastats
/opt/zimbra/libexec/zmrrdfetch -c -f zmmtastats
grep -w MTA /var/log/zimbra-stats.log
tail /var/log/zimbra-stats.log
zmsoap -z GetLoggerStatsRequest stats/@name=zmmtastats | head -20
zmsoap -z GetLoggerStatsRequest stats/@name=zmmtastats @limit=1 | head -50

Take the above information, and post/attach the information to support or the Zimbra Forums

Verified Against: Zimbra Collaboration Suite 6.0.x Date Created: 4/9/2009
Article ID: https://wiki.zimbra.com/index.php?title=Logger_(ZCS_6.0.x_and_later) Date Modified: 2015-10-07



Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »

Jump to: navigation, search