View source for Logger (ZCS 6.0.x and later)

{{BC|Community Sandbox}}
__FORCETOC__
<div class="col-md-12 ibox-content">
=Logger (ZCS 6.0 and later)=
{{KB|{{Unsupported}}|{{ZCS 6.0}}||}}
{{Archive}}{{WIP}}
= Logger in version 6.0 (GnR) =

== Description ==

Logger is a non-essential ZCS service that processes logs produced by Zimbra,
and associated services.  Logs are processed into a format that is readily
displayed in the Zimbra Admin Console indicating real-time service status,
MTA/spam/virus traffic and performance statistics.

=== What's changed since 5.0 (and earlier) ===

The logger service has been completely redesigned and redeveloped for ZCS 6.0

The service no longer relies on mysql and most command-line utilities related
to the old service are no longer available.

[[CLI_zmmsgtrace|zmmsgtrace]] was removed in ZCS 6, however it was reintroduced in ZCS 7.1.1.

== Architecture and Implementation ==

=== zmlogger ===

This process is launched through zmloggerctl and runs on a pipe from
zmlogswatch, it tails /var/log/zimbra-stats.log and captures information from the
'zimbramon' and 'mailboxd' services and parses out statistics information.

Zmlogger reads an sqlite database to determine the type of each stat counter
and mappings to RRD file:data-columns and writes the stat counters to
the appropriate RRD file(s).

=== zmlogprocess ===

Zmlogprocess is launched out of cron on the logger host every 10 minutes.

This job reads MTA data from zimbra.log: MTA traffic and volume, SPAM
and Virus infections.  The data is then sent back to zmlogger as stat counters
via syslog.

=== zmstat ===

Zmstats is also integrated into the logger service by broadcasting its stat
counters over syslog in addition to using the old CSV files.

Stats will still be readable using the zmstat-chart tool in addition to live,
real-time data displayed in the admin console.

=== syslog ===

The logger service will use syslog as a mechanism for aggregating data to a
centralized server.  Messages are generally sent in the format of

  message-type: headers(csv):: data-columns(csv)

Messages over 800 characters in length are broken up using a uuid eliding
scheme.  This scheme is required due to many syslogd implementations
restricting message length to be 1022 bytes or shorter.  The scheme works
by appending :::uuid to prepending :::uuid to messages that are split.  The
UUID allows easily matching long messages together in the event multiple
long messages occur at once (common).

=== sqlite ===

SQLite is used as an integration database.  It maps dns names to zmhostname
when there are mismatches, stores information about stat counter types and
a mapping of a stat counter to an RRD file's datasource column.

==== Schema ====

Our integration database schema is as follows:

* hosts: columns (id, dns_hostname, zm_hostname); generated dynamically as logger works

Self-explanatory

* rrds: columns (id, host_id, col_name, col_name_19, col_num, csv_file, rrd_file); also generated dynamically as logger works
** host_id is the host for which this column is mapping
** col_name_19 is the name of the column in the rrd file
** col_num is the index of the column in the rrd_file
** csv_file is the grouping of the stat counter
** rrd_file is a number that, combined with the host_id, determines the rrd filename

* datatypes: columns (csv_file, col_name, col_type); static mappings table (coming in beta2)
** csv_file is the grouping of the stat counter
** col_name is the column for which this is determining the datatype
** col_type is the type of the column GAUGE, DERIVED, COUNTER or ABSOLUTE (rrd types)

* config: columns (schema_version); single row table used for schema versioning (coming in beta2)

==== zmloggerhostmap ====

This is a command used to manipulate the DNS hostname to zmhostname mappings
when there is a mismatch in the information reported to Logger

=== rrd ===

RRDs are used to store stat counter information, it manages aggregation and
averaging.  Generally, one RRD file is generated per group of statistics and
information is retained for 2 years in varying levels of granularity (from 30
second to 1 hour resolution).

==== zmrrdfetch ====

This is an internal command that ZCS will use to fetch and combine data from
the RRD files.  The information will often be used to form graphs and charts
in the admin console.

== Troubleshooting ==

=== No chart shows ===

Is flash installed?  The new charts now require flash to generate images on the client-side.

=== Statistics show no data available ===

 zmloggerctl status; echo $?; ps ax | grep zmlogger
 /opt/zimbra/libexec/zmrrdfetch -f zmmtastats
 /opt/zimbra/libexec/zmrrdfetch -c -f zmmtastats
 grep -w MTA /var/log/zimbra-stats.log
 tail /var/log/zimbra-stats.log
 zmsoap -z GetLoggerStatsRequest stats/@name=zmmtastats | head -20
 zmsoap -z GetLoggerStatsRequest stats/@name=zmmtastats @limit=1 | head -50

Take the above information, and post/attach the information to support or the Zimbra Forums

{{Article Footer|Zimbra Collaboration Suite 6.0.x|4/9/2009}}
[[Category:Logger]]
[[Category:ZCS 6.0]]