Logger (ZCS 5.0.x and earlier): Difference between revisions

mNo edit summary
No edit summary
 
(37 intermediate revisions by 9 users not shown)
Line 1: Line 1:
Although the logger is not essential for Zimbra to operate, this article will describe how to troubleshoot Logger configuration, setup, and troubleshooting.
{{BC|Community Sandbox}}
__FORCETOC__
<div class="col-md-12 ibox-content">
=Logger (ZCS 5.0 and earlier)=
{{KB|{{Unsupported}}|{{ZCS 5.0}}||}}
{{Archive}}{{WIP}}
Although the logger is not essential for Zimbra to operate, this article will describe setup, configuration, and troubleshooting of the logger service for ZCS 5.0.x and earlier.
 
=Setup and Configuration=
==Installation==
If the logger service is installed during ZCS installation, zmsetup.pl runs zmloggerinit, which A) creates the '''zimbra_logger''' database and the tables shown in the output below, and B) generates the zimbra and root mysql user passwords, and stores them in '''[[CLI_zmlocalconfig_%28Local_Configuration)|zmlocalconfig]]'''.
 
==Multi-node Installations==
''Note: See also the [http://www.zimbra.com/docs/ne/latest/administration_guide/9_Monitoring.16.01.html Monitoring Zimbra Servers].
 
1. Set the log hostname to the name of the node running the logger service. For this example, it's called '''mail1.domain.com'''.
 
$ zmprov getConfig zimbraLogHostname
$ zmprov modifyConfig zimbraLogHostname mail1.domain.com
 
2. Configure syslog on each node. This must be done as root on each node.
 
# /opt/zimbra/bin/zmsyslogsetup
 
3. In order to allow the log host to accept log messages from the other nodes, add the "-r" option to the SYSLOGD_OPTIONS line in /etc/sysconfig/syslog (this must also be done as root); for example:
 
SYSLOGD_OPTIONS="-r -m 0"
 
=Troubleshooting=
 
==Accessing the logger MySQL database manually==


====Accessing Logger MySQL database====
  $ logmysql zimbra_logger
  $ logmysql zimbra_logger
 
  mysql> show tables;
  mysql> show tables;
  +-------------------------+
  +-------------------------+
Line 20: Line 49:
  +-------------------------+
  +-------------------------+


==Checking and Repairing the tables in the logger database==
Mysql has built-in tools for checking and repairing the database.  You should check all of the tables in the logger database, and repair each one that indicates it needs repair. 
Here is an example, using the "raw_logs" table:
$ logmysql zimbra_logger
mysql> check table raw_logs;
+------------------------+-------+----------+----------+
| Table                  | Op    | Msg_type | Msg_text |
+------------------------+-------+----------+----------+
| zimbra_logger.raw_logs | check | status  | OK      |
+------------------------+-------+----------+----------+
1 row in set (1.06 sec)
If a table does not show OK status, try repairing:
mysql> repair table raw_logs;
+------------------------+--------+----------+----------+
| Table                  | Op    | Msg_type | Msg_text |
+------------------------+--------+----------+----------+
| zimbra_logger.raw_logs | repair | status  | OK      |
+------------------------+--------+----------+----------+
1 row in set (2.32 sec)
See [http://dev.mysql.com/doc/refman/5.0/en/repair.html MySQL documentation] for more information.
==Overview of logger pipeline==
1. The mta components (postfix, amavis) and system status scripts (zmstatuslog, zmdisklog, zmqueuelog) log to /var/log/zimbra.log.
2. The logswatch script monitors /var/log/zimbra.log and sends new lines to the zmlogger script.
3. The zmlogger script inserts the log lines into the raw_logs table, and updates the service_status table, in the zimbra_logger database.
4. The zmlogprocess script breaks down the lines from the raw_logs table and inserts the data into the mta, mta_aggregate, amavis, amavis_aggregate, disk_status, disk_aggregate, and processing_history tables of the zimbra_logger db.
==How to determine why logger isn't working==
Why does the logger service stop? The two main causes are log rotation and a bug in zmlogswatchctl before ZCS 4.5.
===Check the logger mysql error log===
The logger mysql error log will be /opt/zimbra/logger/db/data/<hostname>.err. For example, if your server's hostname is "zimbra.domain.com", the file will be /opt/zimbra/logger/db/data/zimbra.domain.com.err.
Try [[Repair_Logger_Data_Corruption|repairing the corrupt tables]] if you see any lines like this:
070927 14:30:01 [ERROR] /opt/zimbra/logger/mysql/libexec/mysqld: Table './zimbra_logger/raw_logs' is marked as crashed and last (automatic?) repair failed
===Check "zmcontrol status"===
1. If "'''logmysql.server''' is not running", verify whether A) the file /opt/zimbra/logger/db/mysql.pid exists, and B) there is an /opt/zimbra/logger/mysql/libexec/mysqld process.
cat /opt/zimbra/logger/db/mysql.pid
ps aux | grep logger/mysql
Under normal operation, the id of this mysqld process will be in the mysql.pid file.
2. If "'''zmlogswatchctl''' is not running", verify whether A) the file /opt/zimbra/log/logswatch.pid exists, and B) there is a single /opt/zimbra/libexec/logswatch process.
cat /opt/zimbra/log/logswatch.pid
ps aux | grep logswatch
3. If "'''logger''' Running", verify that the logger database is accessible and that there are not multiple logswatch scripts running (see #2).
logmysqladmin status
ps aux | grep logswatch
===Check the MTA log===
1. Is there MTA activity?
grep postfix /var/log/zimbra.log | tail
grep amavis /var/log/zimbra.log | tail
2. Is there ZCS system information?
grep STATUS /var/log/zimbra.log | tail
grep DISK /var/log/zimbra.log | tail
grep QUEUE /var/log/zimbra.log | tail
===Check the logger database===
1. Is data making it to the '''service_status''' and '''raw_logs''' tables (choose the current date)?
logmysql zimbra_logger
select * from service_status;
select * from raw_logs where log_date >= curdate();    # Today's entries
# or
select * from raw_logs where log_date > '2007-03-15';  # Entries newer than 2007-03-15
==Symptoms==
[[Logger#Check_.22zmcontrol_status.22|zmlogswatchctl is not running]]
[[Logger#Multi-node_Installations|Admin console only showing data for the log host in a multi-node installation]]
==What queries the logger database==
1. The '''zmdailyreport''' script processes data from the mta and amavis tables, and emails the results to root, which is normally an alias to the original zimbra admin account.
[zimbra@mail ~]$ crontab -l | grep zmdailyreport
10 1 * * * /opt/zimbra/libexec/zmdailyreport | /opt/zimbra/postfix/sbin/sendmail root
[zimbra@mail ~]$ zmprov getAccount admin | grep Alias
zimbraMailAlias: root@mail.domain.com
zimbraMailAlias: postmaster@mail.domain.com
2. The '''zmmsgtrace''' tool queries the mta and amavis tables and outputs basic info about matching messages.
3. The '''zmgengraphs''' script queries the disk_aggregate, mta_aggregate, and amavis_aggregate tables and creates images that will be displayed in the admin console.
4. Loading the '''[[Administration_Console|admin console]]''' queries the service_status table, and viewing server statistics will display the graphs created by zmgengraphs.
[screenshot]
== How to shrink logger database ==


For first time is good to clean db manually if the database is very big.
The commands bellow will delete all data in three tables (mta, amavis, raw_logs). If you need this data don't execute them!
  $zmlogswatchctl stop (don't execute "zmloggerctl stop" this also stops logger mysqld)
  $logmysql -D zimbra_logger
  mysql> delete from amavis;
  mysql> optimize table amavis;
  mysql> delete from mta;
  mysql> optimize table mta;
  mysql> delete from raw_logs;
  mysql> optimize table raw_logs;
  mysql> quit
  $zmlogswatchctl start
Be patient, each deleting query may lasts for a long time!


[http://www.3wcasinos.com/roulette-tips/index.html roulette tips]
Here is a script for database cleaning. Run it under zimbra account. If the database is big and the execution time is long enough do zmlogswatchctl stop command to prevent zimbra accessing database. And don't forget make zmlogswatchctl start after the script.
[http://www.casino-games-wiki.com/index.php/online_casino_games online casino games]
  #!/bin/bash
[http://www.slots-wiki.com/index.php/slots_hints slots hints]
 
[http://www.casinos-go.com/online-casino-tips/free-online-casino-slot.html free online casino slot]
  AMAVIS=10 #keep last 10 days
[http://www.online-casino-wiki.com/index.php/online_casino_tips online casino tips]
  MTA=30 #keep last 30 days
[http://www.casino-games-wiki.com/index.php/casino_games_rules casino games rules]
  RAW=2 #keep last 2 days
[http://www.casinos-new.com/blackjack.html Online casino - Blackjack online]
 
[http://www.gambling-online-theory.com/casinos/strategy-casinos.html strategy casinos]
  /opt/zimbra/bin/logmysql -D zimbra_logger << EOF
[http://www.casinos-go.com/online-casino-tips/online-casino-games.html online casino games]
  delete from amavis where arrive_time < adddate(curdate(),interval -$AMAVIS day);
[http://www.magical-casino.com/casino_games.html Casino games.]
  optimize table amavis;
  select count(*) AS amavis_rec_left from amavis;
  delete from mta where (arrive_time > "2000-01-01" and arrive_time < adddate(curdate(),interval -$MTA day)) or
    (leave_time > "2000-01-01" and leave_time < adddate(curdate(),interval -$MTA day));
  optimize table mta;
  select count(*) AS MTA_rec_left from mta;
  delete from raw_logs where log_date < adddate(curdate(),interval -$RAW day);
  optimize table raw_logs;
  select count(*) AS raw_rec_left from raw_logs;
  quit
  EOF
{{Article Footer|Zimbra Collaboration 5.0|04/16/2014}}
[[Category:Logger]]
[[Category:Troubleshooting Logger]]
[[Category:ZCS 5.0]]

Latest revision as of 10:17, 12 July 2015

Logger (ZCS 5.0 and earlier)

   KB 1377        Last updated on 2015-07-12  




0.00
(0 votes)

Although the logger is not essential for Zimbra to operate, this article will describe setup, configuration, and troubleshooting of the logger service for ZCS 5.0.x and earlier.

Setup and Configuration

Installation

If the logger service is installed during ZCS installation, zmsetup.pl runs zmloggerinit, which A) creates the zimbra_logger database and the tables shown in the output below, and B) generates the zimbra and root mysql user passwords, and stores them in zmlocalconfig.

Multi-node Installations

Note: See also the Monitoring Zimbra Servers.

1. Set the log hostname to the name of the node running the logger service. For this example, it's called mail1.domain.com.

$ zmprov getConfig zimbraLogHostname
$ zmprov modifyConfig zimbraLogHostname mail1.domain.com

2. Configure syslog on each node. This must be done as root on each node.

# /opt/zimbra/bin/zmsyslogsetup

3. In order to allow the log host to accept log messages from the other nodes, add the "-r" option to the SYSLOGD_OPTIONS line in /etc/sysconfig/syslog (this must also be done as root); for example:

SYSLOGD_OPTIONS="-r -m 0"

Troubleshooting

Accessing the logger MySQL database manually

$ logmysql zimbra_logger

mysql> show tables;
+-------------------------+
| Tables_in_zimbra_logger |
+-------------------------+
| amavis                  |
| amavis_aggregate        |
| config                  |
| disk_aggregate          |
| disk_status             |
| mta                     |
| mta_aggregate           |
| processing_history      |
| raw_logs                |
| service_status          |
+-------------------------+

Checking and Repairing the tables in the logger database

Mysql has built-in tools for checking and repairing the database. You should check all of the tables in the logger database, and repair each one that indicates it needs repair.

Here is an example, using the "raw_logs" table:

$ logmysql zimbra_logger

mysql> check table raw_logs;
+------------------------+-------+----------+----------+
| Table                  | Op    | Msg_type | Msg_text |
+------------------------+-------+----------+----------+
| zimbra_logger.raw_logs | check | status   | OK       | 
+------------------------+-------+----------+----------+
1 row in set (1.06 sec)

If a table does not show OK status, try repairing:

mysql> repair table raw_logs;
+------------------------+--------+----------+----------+
| Table                  | Op     | Msg_type | Msg_text |
+------------------------+--------+----------+----------+
| zimbra_logger.raw_logs | repair | status   | OK       | 
+------------------------+--------+----------+----------+
1 row in set (2.32 sec)

See MySQL documentation for more information.

Overview of logger pipeline

1. The mta components (postfix, amavis) and system status scripts (zmstatuslog, zmdisklog, zmqueuelog) log to /var/log/zimbra.log.

2. The logswatch script monitors /var/log/zimbra.log and sends new lines to the zmlogger script.

3. The zmlogger script inserts the log lines into the raw_logs table, and updates the service_status table, in the zimbra_logger database.

4. The zmlogprocess script breaks down the lines from the raw_logs table and inserts the data into the mta, mta_aggregate, amavis, amavis_aggregate, disk_status, disk_aggregate, and processing_history tables of the zimbra_logger db.

How to determine why logger isn't working

Why does the logger service stop? The two main causes are log rotation and a bug in zmlogswatchctl before ZCS 4.5.

Check the logger mysql error log

The logger mysql error log will be /opt/zimbra/logger/db/data/<hostname>.err. For example, if your server's hostname is "zimbra.domain.com", the file will be /opt/zimbra/logger/db/data/zimbra.domain.com.err.

Try repairing the corrupt tables if you see any lines like this:

070927 14:30:01 [ERROR] /opt/zimbra/logger/mysql/libexec/mysqld: Table './zimbra_logger/raw_logs' is marked as crashed and last (automatic?) repair failed

Check "zmcontrol status"

1. If "logmysql.server is not running", verify whether A) the file /opt/zimbra/logger/db/mysql.pid exists, and B) there is an /opt/zimbra/logger/mysql/libexec/mysqld process.

cat /opt/zimbra/logger/db/mysql.pid
ps aux | grep logger/mysql

Under normal operation, the id of this mysqld process will be in the mysql.pid file.

2. If "zmlogswatchctl is not running", verify whether A) the file /opt/zimbra/log/logswatch.pid exists, and B) there is a single /opt/zimbra/libexec/logswatch process.

cat /opt/zimbra/log/logswatch.pid
ps aux | grep logswatch

3. If "logger Running", verify that the logger database is accessible and that there are not multiple logswatch scripts running (see #2).

logmysqladmin status
ps aux | grep logswatch

Check the MTA log

1. Is there MTA activity?

grep postfix /var/log/zimbra.log | tail
grep amavis /var/log/zimbra.log | tail

2. Is there ZCS system information?

grep STATUS /var/log/zimbra.log | tail
grep DISK /var/log/zimbra.log | tail
grep QUEUE /var/log/zimbra.log | tail

Check the logger database

1. Is data making it to the service_status and raw_logs tables (choose the current date)?

logmysql zimbra_logger
select * from service_status;
select * from raw_logs where log_date >= curdate();     # Today's entries
# or
select * from raw_logs where log_date > '2007-03-15';  # Entries newer than 2007-03-15

Symptoms

zmlogswatchctl is not running

Admin console only showing data for the log host in a multi-node installation


What queries the logger database

1. The zmdailyreport script processes data from the mta and amavis tables, and emails the results to root, which is normally an alias to the original zimbra admin account.

[zimbra@mail ~]$ crontab -l | grep zmdailyreport
10 1 * * * /opt/zimbra/libexec/zmdailyreport | /opt/zimbra/postfix/sbin/sendmail root
[zimbra@mail ~]$ zmprov getAccount admin | grep Alias
zimbraMailAlias: root@mail.domain.com
zimbraMailAlias: postmaster@mail.domain.com

2. The zmmsgtrace tool queries the mta and amavis tables and outputs basic info about matching messages.

3. The zmgengraphs script queries the disk_aggregate, mta_aggregate, and amavis_aggregate tables and creates images that will be displayed in the admin console.

4. Loading the admin console queries the service_status table, and viewing server statistics will display the graphs created by zmgengraphs.

[screenshot]

How to shrink logger database

For first time is good to clean db manually if the database is very big. The commands bellow will delete all data in three tables (mta, amavis, raw_logs). If you need this data don't execute them!

 $zmlogswatchctl stop (don't execute "zmloggerctl stop" this also stops logger mysqld)
 $logmysql -D zimbra_logger
 mysql> delete from amavis;
 mysql> optimize table amavis;
 mysql> delete from mta;
 mysql> optimize table mta;
 mysql> delete from raw_logs;
 mysql> optimize table raw_logs;
 mysql> quit
 $zmlogswatchctl start

Be patient, each deleting query may lasts for a long time!

Here is a script for database cleaning. Run it under zimbra account. If the database is big and the execution time is long enough do zmlogswatchctl stop command to prevent zimbra accessing database. And don't forget make zmlogswatchctl start after the script.

 #!/bin/bash
 
 AMAVIS=10 #keep last 10 days
 MTA=30 #keep last 30 days
 RAW=2 #keep last 2 days
 
 /opt/zimbra/bin/logmysql -D zimbra_logger << EOF
 delete from amavis where arrive_time < adddate(curdate(),interval -$AMAVIS day);
 optimize table amavis;
 select count(*) AS amavis_rec_left from amavis;
 delete from mta where (arrive_time > "2000-01-01" and arrive_time < adddate(curdate(),interval -$MTA day)) or 
   (leave_time > "2000-01-01" and leave_time < adddate(curdate(),interval -$MTA day));
 optimize table mta;
 select count(*) AS MTA_rec_left from mta;
 delete from raw_logs where log_date < adddate(curdate(),interval -$RAW day);
 optimize table raw_logs;
 select count(*) AS raw_rec_left from raw_logs;
 quit
 EOF
Verified Against: Zimbra Collaboration 5.0 Date Created: 04/16/2014
Article ID: https://wiki.zimbra.com/index.php?title=Logger_(ZCS_5.0.x_and_earlier) Date Modified: 2015-07-12



Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »

Jump to: navigation, search