Ajcody-Logger-Issues: Difference between revisions

Revision as of 19:42, 17 July 2014

- This article is NOT official Zimbra documentation. It is a user contribution and may include unsupported customizations, references, suggestions, or information.

Logger Issues

You might want to cross review these other sections as well:

Actual Logger Issues Homepage

Please see Ajcody-Logger-Issues

Logger and ZCS 6.x+

Please see Logger(GnR) , as logger has changed dramatically in 6.x+ as compared to 5.x.

Missing Stats

Are You Using rsyslog, syslog-ng, or syslog

See this forum thread: upgrade to 6.0.2 stats and status no longer working

See the bugs listed below as well.

Confirm All hosts Are Still Present

Follow the instructions at Ajcody-Logger-Issues#What_Tables_And_To_View_Them and confirm all the hosts displayed are still present in your environment. You'll probably also see an error about this in the output from the instructions given at Logger(GnR)#Statistics_show_no_data_available

Logger Not Running

You might want to check /opt/zimbra/log/zmlogswatch.out and see if it mentions a missing file or directory. For example:

tail -f log/zmlogswatch.out
   Error opening /var/log/zimbra-stats.log: No such file or directory at
   /opt/zimbra/data/tmp/.swatch_script.21119 line 92

Here's the ownership as that file should be:

ls -la /var/log/zimbra-stats.log
-rw-r--r-- 1 zimbra zimbra 2500220 Feb 17 05:37 /var/log/zimbra-stats.log

Logger 6.x Bugs To Review

Some bugs to review:

"Server statistics fail after logrotate with rsyslogd"
- http://bugzilla.zimbra.com/show_bug.cgi?id=43081
"Services status: all servers reported as not running/error in web console"
- http://bugzilla.zimbra.com/show_bug.cgi?id=42053
""no data available" message after upgrading from 5.0.18 to 6.0.2"
- http://bugzilla.zimbra.com/show_bug.cgi?id=42293
"48 hour server specific no stats - loading data for chart..."
- https://bugzilla.zimbra.com/show_bug.cgi?id=42332
"Stats service failing due to folder permissions getting set wrong"
- http://bugzilla.zimbra.com/show_bug.cgi?id=45010
"zmloggerinit [in zcs5+] for zcs6+"
- http://bugzilla.zimbra.com/show_bug.cgi?id=45471
"rework logger"
- http://bugzilla.zimbra.com/show_bug.cgi?id=45705
  - Dev comments on this bug:
    - "Lots of issues have been filed against logger problems. For affected deployments, the current recommendation is to disable the logger until the problems are solved."

sqlite3 and Logger

External Reference: Command Line Shell For SQLite

Login To DB and Quit

How to log into the logger db and to quit:

[zimbra@mail3 data]$ pwd
/opt/zimbra/logger/db/data
[zimbra@mail3 data]$ sqlite3 logger.sqlitedb
SQLite version 3.3.6
Enter ".help" for instructions
sqlite> .quit
[zimbra@mail3 data]$

What Tables And To View Them

$ pwd
 /opt/zimbra/logger/db/data

$ sqlite3 logger.sqlitedb
    SQLite version 3.3.6
    Enter ".help" for instructions
    sqlite> .mode list
    sqlite> .tables
      config           hosts            rrd_column_type  rrds
    sqlite> select * from hosts;
      1|mail3|mail3.zimbra.REMOVED.com
      2|mail3.zimbra.REMOVED.com|mail3.zimbra.REMOVED.com
    sqlite>.quit

$

Delete A Host

This is the command to delete a host, id is the number shown on the left of the line where the hostname is display from "select * from hosts;"

 sqlite>  DELETE  from hosts WHERE id=1;

That would delete the first hostname [id 1] .

Query Of ZCS Service Status History Via zmrrdfetch

Example were done on ZCS 8.0.6 and a single ZCS server setup. First, if you need a converter for unix time see : http://www.epochconverter.com/ . Secondy, note the columns are described below. Some of my examples will not include this first row. :

timestamp,opendkim,zmconfigd,mailbox,proxy,spell,logger,antivirus,archiving,snmp,ldap,memcached,convertd,stats,antispam,mta

The below example checks to see when the archiving service was reported as up. Archiving is the 9th column.

/opt/zimbra/libexec/zmrrdfetch -f zmstatuslog -h zcs806.us.DOMAIN.com -s 1388534400 -e 1405618603 | awk -F, '$9 == 1'
[cut]
1405454400,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
1405458000,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
1405461600,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
1405465200,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
1405468800,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
1405472400,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
1405476000,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
1405479600,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
[cut]

The below example checks to see when the archiving service was reported as unavailable 100%. Archiving is the 9th column. [zimbra@zcs806 data]$ /opt/zimbra/libexec/zmrrdfetch -f zmstatuslog -h zcs806.us.DOMAIN.com -s 1388534400 -e 1405618603 | awk -F, '$9 == ""'

1405069200,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
1405072800,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
1405076400,1,0.964354991388889,1,1,1,1,1,,1,1,1,1,1,1,1
1405080000,1,0.946143796944445,1,1,1,1,1,,1,1,1,1,1,1,1
1405083600,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
1405087200,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
1405090800,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
1405094400,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
1405098000,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
1405101600,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
1405105200,1,0.987032061666667,1,1,1,1,1,,1,1,1,1,1,1,1
1405108800,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
1405112400,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
1405116000,1,0.982143184166667,1,1,1,1,1,,1,1,1,1,1,1,1
1405119600,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
1405123200,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
1405126800,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
1405130400,1,0.967078515555556,1,1,1,1,1,,1,1,1,1,1,1,1
1405134000,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
1405137600,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
1405141200,1,0.920520648333333,1,1,1,1,1,,1,1,1,1,1,1,1
1405144800,1,0.9539186725,1,1,1,1,1,,1,1,1,1,1,1,1
1405148400,1,0.981826185277778,1,1,1,1,1,,1,1,1,1,1,1,1
1405152000,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
1405155600,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
1405159200,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
1405162800,1,0.996330343333333,1,1,1,1,1,,1,1,1,1,1,1,1
1405166400,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
1405170000,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
1405173600,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
1405177200,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
[cut]

The below example checks to see when the mailbox service was reported as NOT available 100% or the time but was NOT 100% unavailable either. Mailbox is the 3rd column.

[zimbra@zcs806 data]$ /opt/zimbra/libexec/zmrrdfetch -f zmstatuslog -h zcs806.us.DOMAIN.com -s 1388534400 -e 1405618603 | awk -F, '$3 ~ 0'
[cut]
1405076400,1,0.964354991388889,1,1,1,1,1,,1,1,1,1,1,1,1
1405080000,1,0.946143796944445,1,1,1,1,1,,1,1,1,1,1,1,1
1405105200,1,0.987032061666667,1,1,1,1,1,,1,1,1,1,1,1,1
1405116000,1,0.982143184166667,1,1,1,1,1,,1,1,1,1,1,1,1
1405130400,1,0.967078515555556,1,1,1,1,1,,1,1,1,1,1,1,1
1405141200,1,0.920520648333333,1,1,1,1,1,,1,1,1,1,1,1,1
1405144800,1,0.9539186725,1,1,1,1,1,,1,1,1,1,1,1,1
1405148400,1,0.981826185277778,1,1,1,1,1,,1,1,1,1,1,1,1
1405162800,1,0.996330343333333,1,1,1,1,1,,1,1,1,1,1,1,1
[cut]

An example of a zmcontrol restart and what zmrrdfetch will show. Note I adjusted the ending flag to be : -e `date +%s --date="1 minute ago" : as compared to the above examples :

[zimbra@zcs806 data]$ /opt/zimbra/libexec/zmrrdfetch -f zmstatuslog -h zcs806.us.DOMAIN.com -s `date +%s --date="15 minute ago"` -e `date +%s --date="1 minute ago"`
timestamp,opendkim,zmconfigd,mailbox,proxy,spell,logger,antivirus,archiving,snmp,ldap,memcached,convertd,stats,antispam,mta
1405621920,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
1405621950,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
1405621980,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
1405622010,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
1405622040,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
1405622070,0.105026416666667,0.1050323,1,1,0.105029991666667,1,0.105016,0.105017466666667,0.105028733333333,1,1,1,0.10503115,0.105014483333333,0.105025008333333
1405622100,0.105026416666667,0.1050323,1,1,0.105029991666667,1,0.105016,0.105017466666667,0.105028733333333,1,1,1,0.10503115,0.105014483333333,0.105025008333333
1405622130,0.105026416666667,0.1050323,1,1,0.105029991666667,1,0.105016,0.105017466666667,0.105028733333333,1,1,1,0.10503115,0.105014483333333,0.105025008333333
1405622160,0.105026416666667,0.1050323,1,1,0.105029991666667,1,0.105016,0.105017466666667,0.105028733333333,1,1,1,0.10503115,0.105014483333333,0.105025008333333
1405622190,,0,,,,,,,,,,,,,
1405622220,,0,,,,,,,,,,,,,
1405622250,,0,,,,,,,,,,,,,
1405622280,,0,,,,,,,,,,,,,
1405622310,,0,,,,,,,,,,,,,
1405622340,,0.893621444444444,,,,,,,,,,,,,
1405622370,,0.893621444444444,,,,,,,,,,,,,
1405622400,,0.893621444444444,,,,,,,,,,,,,
1405622430,,1,,,,,,,,,,,,,
1405622460,,1,,,,,,,,,,,,,
1405622490,,1,,,,,,,,,,,,,
1405622520,,1,,,,,,,,,,,,,
1405622550,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
1405622580,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1

Logger Related Bugs

5.0.11 - Fixed in 5.0.12 But See Below Before Upgrading

UPDATE: From release notes: kill all zmlogger processes before upgrading to 5.0.12 from 5.0.10+

"zmlogger fails to terminate at zmcontrol stop"
- Which generally is causing a whole host of problems. Shows up in 5.0.11 release
- http://bugzilla.zimbra.com/show_bug.cgi?id=33730 dup of below
  - Please get the script (attachment) within this bug that updates the control script.
- http://bugzilla.zimbra.com/show_bug.cgi?id=33604

Steps To Resolve

Edit /opt/zimbra/conf/my.logger.cnf, and modify this line for 20 connections.
- max_connections = 20
Review the following bug and download the zmlogswatchctl script listed in the attachments section.
- http://bugzilla.zimbra.com/show_bug.cgi?id=33604

Shutdown logger.

```
zmloggerctl stop
```
find any orphaned logger or swatch processes, and kill them
- ```
 ps -ef | grep logg
```
- ```
 ps -ef | grep watch
```

Replace the logswatch control file

mv /opt/zimbra/bin/zmlogswatchctl /opt/zimbra/bin/original.zmlogswatchctl

Add the new zmlogswatchctl file from the above bug to /opt/zimbra/bin

as root

chown root:root /opt/zimbra/bin/zmlogswatchctl

as root

chmod 755 /opt/zimbra/bin/zmlogswatchctl

Start up the logger process.
- ```
zmloggerctl start
```

You can be checking the output in the following as well to see if any errors are still being reported:

/tmp/logprocess.out
/tmp/gengraphs.out
/opt/zimbra/logger/db/data/<hostname>.err

Logger Table Check And Repair

Other items to check:

Also, check the logger tables to see if any of the tables need to be repaired

logmysql -e "check table zimbra_logger.amavis"
logmysql -e "check table zimbra_logger.amavis_aggregate"
logmysql -e "check table zimbra_logger.config"
logmysql -e "check table zimbra_logger.disk_aggregate"
logmysql -e "check table zimbra_logger.disk_status"
logmysql -e "check table zimbra_logger.mta"
logmysql -e "check table zimbra_logger.mta_aggregate"
logmysql -e "check table zimbra_logger.processing_history"
logmysql -e "check table zimbra_logger.raw_logs"
logmysql -e "check table zimbra_logger.service_status"

If any of the tables need to be repaired, you replace 'check' with 'repair'. Example:

logmysql -e "repair table zimbra_logger.amavis"

/tmp/logprocess.out Shows MySQL Can't Connect

Please follow these instructions:

Issues_with_mysql_and_logmysql_passwords#For_logger_database

Logger Not Working , No Stats, Services Show Not Running In Admin Console

ZCS 6 And Above

On The Logger Host Server

On the Logger monitor host [zmprov gacf |grep zimbraLogHostname] start with the following. Note: run below steps in non-peak hours.

su - zimbra
zmcontrol stop ; exit

Run as root user:

/opt/zimbra/libexec/zmfixperms -e -v

On the logger monitor host, you must enable syslog [if your running rsyslog, see below] to log statistics from remote machines. Run the following as root also :

Edit the /etc/sysconfig/syslog file, add -r to the SYSLOGD_OPTIONS setting, SYSLOGD_options=”-r -m 0”
Run : /opt/zimbra/libexec/zmsyslogsetup
Stop the syslog daemon. Type : /etc/init.d/syslogd restart
- Or by: service syslog restart
- Use stop / start if restart isn't available.

If you are using rsyslog instead of syslog, then you will need the following changes in rsyslog config. First, enable the module for remote logging on the logger host. Edit the rsyslog config file "/etc/rsyslog.conf" and enable "imudp" module by uncommenting the same:

$ModLoad imudp
$UDPServerRun 514

After the above changes to the following if you are running rsyslog: restart rsyslog service.

Run : /opt/zimbra/libexec/zmsyslogsetup
Restart rsyslog with : /etc/init.d/rsyslog restart
- Or by: service rsyslog restart
- Use stop / start if restart isn't available.

Your last steps are to then run:

su - zimbra 
zmcontrol start 
/opt/zimbra/libexec/zmloggerinit 
/opt/zimbra/bin/zmsshkeygen
/opt/zimbra/bin/zmupdateauthkeys

On The Other Servers

On all other servers (if you have multiple zimbra servers in your setup). Verify the LogHostname (it should be set to name of Logger monitor Host on all servers) if its different, change the same to Logger monitor Host using below command.

su - zimbra
zmprov gacf |grep zimbraLogHostname

If it is set wrong, you can correct it by doing:

zmprov mcf zimbraLogHostname <Logger monitor Hostname>

Then run the following after the zimbraLogHostname variable is confirmed or set correctly:

/opt/zimbra/bin/zmsshkeygen
/opt/zimbra/bin/zmupdateauthkeys ; exit
/opt/zimbra/libexec/zmsyslogsetup (run as root user)
Restart either syslogd or rsyslog, depending on what your server is using
service syslog restart or service rsyslog restart
su - zimbra
zmcontrol restart

Reference Link:

http://www.zimbra.com/docs/ne/latest/administration_guide/wwhelp/wwhimpl/js/html/wwhelp.htm (Pleaserefer Chapter 15: Monitoring ZCS Servers)

This Section Written For ZCS5

Check the following log files first to gather some information:

Make sure the /etc/hosts details make the actual hostname and the zmhostname output
- If someone recently changed the hostname variables for this box, logger can be effected if done improperly and will not log the obvious issue.
/tmp/logprocess.out
/tmp/gengraphs.out
/opt/zimbra/logger/db/data/<hostname>.err
/var/log/zimbra.log (logswatch monitors this file)
- grep'ing for things such as:
  - ```
   egrep 'postfix|amavis|STATUS|DISK|QUEUE' /var/log/zimbra.log
```
You might also want to look at the following directory:
- /opt/zimbra/logger/db/work/
- You should see data in here, gif and rrd files.

Then proceed with the following wiki pages, as ordered:

Logger
- If you find that tables are in need of repair, also consult the following wiki:
  - Repair_Logger_Data_Corruption
If all else fails, you can clear the old logger database and reinitalize it. Please see:
- Ajcody-Logger-Issues#Reinitializing_Logger_Database_From_Scratch

Large Logger Database Killing Performance Of ZCS

Review the following to see what might be best option for you:

King0770-Notes#Logger_Issues_-_Improve_Disk_Utilization_for_the_logger_service
- Adjusting the [ zimbraLogRawLifetime: 31d ] & [ zimbraLogSummaryLifetime: 730d ] variables.
Ajcody-Logger-Issues#Reinitializing_Logger_Database_From_Scratch

Reinitializing Logger Database From Scratch

This will effectively blow away your old logger database, allowing you to start over.

zmloggerctl stop
cd /opt/zimbra/logger/db

Remove the old database:

mv db db-old

or data , if db doesn't exist

mv data data-old

You can remove (rm -rf db or data) the directory if your not worried about old directory being perserved

Reinitialize a new logger database:

/opt/zimbra/libexec/zmloggerinit

Logger is probably running after this, but to make sure

zmloggerctl start

You'll need to manually start this one though

zmlogswatchctl start

You'll have to allow some time to pass for there to be enough data for the graphs to be built in the web interface.

Turning Off Logger

To not have logger startup [ use `hostname` or type in hostname] :

zmprov ms `hostname` -zimbraServiceEnabled logger

To manually shutdown logger

zmloggerctl stop

Customization Or Modification Of Swatch Config

The swatch config file is :

/opt/zimbra/conf/swatchrc.in

If you make a customization, you'll need to do the following for it to take effect:

zmsnmpinit
zmlogswatchctl stop
zmlogswatchctl start
zmswatchctl stop
zmswatchctl start