Ajcody-Logger-Issues

Ajcody Logger Issues

   KB 2706        Last updated on 2016-06-20  




0.00
(0 votes)
24px ‎  - This is Zeta Alliance Certified Documentation. The content has been tested by the Community.

You might want to cross review these other sections as well:

Actual Logger Issues Homepage

Please see Ajcody-Logger-Issues

Logger and ZCS 6.x+

Please see Logger(GnR) , as logger has changed dramatically in 6.x+ as compared to 5.x.

Missing Stats

Are You Using rsyslog, syslog-ng, or syslog

See this forum thread: upgrade to 6.0.2 stats and status no longer working

See the bugs listed below as well.

Confirm All hosts Are Still Present

Follow the instructions at Ajcody-Logger-Issues#What_Tables_And_To_View_Them and confirm all the hosts displayed are still present in your environment. You'll probably also see an error about this in the output from the instructions given at Logger(GnR)#Statistics_show_no_data_available

Logger Not Running

You might want to check /opt/zimbra/log/zmlogswatch.out and see if it mentions a missing file or directory. For example:

tail -f log/zmlogswatch.out
   Error opening /var/log/zimbra-stats.log: No such file or directory at
   /opt/zimbra/data/tmp/.swatch_script.21119 line 92

Here's the ownership as that file should be:

ls -la /var/log/zimbra-stats.log
-rw-r--r-- 1 zimbra zimbra 2500220 Feb 17 05:37 /var/log/zimbra-stats.log

Logger 6.x Bugs To Review

Some bugs to review:

sqlite3 and Logger

External Reference: Command Line Shell For SQLite

Login To DB and Quit

How to log into the logger db and to quit:

[zimbra@mail3 data]$ pwd
/opt/zimbra/logger/db/data
[zimbra@mail3 data]$ sqlite3 logger.sqlitedb
SQLite version 3.3.6
Enter ".help" for instructions
sqlite> .quit
[zimbra@mail3 data]$
What Tables And To View Them
$ pwd
 /opt/zimbra/logger/db/data

$ sqlite3 logger.sqlitedb
    SQLite version 3.3.6
    Enter ".help" for instructions
    sqlite> .mode list
    sqlite> .tables
      config           hosts            rrd_column_type  rrds
    sqlite> select * from hosts;
      1|mail3|mail3.zimbra.REMOVED.com
      2|mail3.zimbra.REMOVED.com|mail3.zimbra.REMOVED.com
    sqlite>.quit

$
Delete A Host

This is the command to delete a host, id is the number shown on the left of the line where the hostname is display from "select * from hosts;"

 sqlite>  DELETE  from hosts WHERE id=1;

That would delete the first hostname [id 1] .

Query Of ZCS Service Status History Via zmrrdfetch

Example were done on ZCS 8.0.6 and a single ZCS server setup. First, if you need a converter for unix time see : http://www.epochconverter.com/ . Secondy, note the columns are described below. Some of my examples will not include this first row. :

timestamp,opendkim,zmconfigd,mailbox,proxy,spell,logger,antivirus,archiving,snmp,ldap,memcached,convertd,stats,antispam,mta

The below example checks to see when the archiving service was reported as up. Archiving is the 9th column.

/opt/zimbra/libexec/zmrrdfetch -f zmstatuslog -h zcs806.us.DOMAIN.com -s 1388534400 -e 1405618603 | awk -F, '$9 == 1'
[cut]
1405454400,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
1405458000,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
1405461600,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
1405465200,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
1405468800,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
1405472400,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
1405476000,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
1405479600,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
[cut]

The below example checks to see when the archiving service was reported as unavailable 100%. Archiving is the 9th column. [zimbra@zcs806 data]$ /opt/zimbra/libexec/zmrrdfetch -f zmstatuslog -h zcs806.us.DOMAIN.com -s 1388534400 -e 1405618603 | awk -F, '$9 == ""'

1405069200,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
1405072800,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
1405076400,1,0.964354991388889,1,1,1,1,1,,1,1,1,1,1,1,1
1405080000,1,0.946143796944445,1,1,1,1,1,,1,1,1,1,1,1,1
1405083600,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
1405087200,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
1405090800,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
1405094400,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
1405098000,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
1405101600,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
1405105200,1,0.987032061666667,1,1,1,1,1,,1,1,1,1,1,1,1
1405108800,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
1405112400,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
1405116000,1,0.982143184166667,1,1,1,1,1,,1,1,1,1,1,1,1
1405119600,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
1405123200,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
1405126800,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
1405130400,1,0.967078515555556,1,1,1,1,1,,1,1,1,1,1,1,1
1405134000,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
1405137600,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
1405141200,1,0.920520648333333,1,1,1,1,1,,1,1,1,1,1,1,1
1405144800,1,0.9539186725,1,1,1,1,1,,1,1,1,1,1,1,1
1405148400,1,0.981826185277778,1,1,1,1,1,,1,1,1,1,1,1,1
1405152000,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
1405155600,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
1405159200,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
1405162800,1,0.996330343333333,1,1,1,1,1,,1,1,1,1,1,1,1
1405166400,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
1405170000,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
1405173600,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
1405177200,1,1,1,1,1,1,1,,1,1,1,1,1,1,1
[cut]

The below example checks to see when the mailbox service was reported as NOT available 100% or the time but was NOT 100% unavailable either. Mailbox is the 3rd column.

[zimbra@zcs806 data]$ /opt/zimbra/libexec/zmrrdfetch -f zmstatuslog -h zcs806.us.DOMAIN.com -s 1388534400 -e 1405618603 | awk -F, '$3 ~ 0'
[cut]
1405076400,1,0.964354991388889,1,1,1,1,1,,1,1,1,1,1,1,1
1405080000,1,0.946143796944445,1,1,1,1,1,,1,1,1,1,1,1,1
1405105200,1,0.987032061666667,1,1,1,1,1,,1,1,1,1,1,1,1
1405116000,1,0.982143184166667,1,1,1,1,1,,1,1,1,1,1,1,1
1405130400,1,0.967078515555556,1,1,1,1,1,,1,1,1,1,1,1,1
1405141200,1,0.920520648333333,1,1,1,1,1,,1,1,1,1,1,1,1
1405144800,1,0.9539186725,1,1,1,1,1,,1,1,1,1,1,1,1
1405148400,1,0.981826185277778,1,1,1,1,1,,1,1,1,1,1,1,1
1405162800,1,0.996330343333333,1,1,1,1,1,,1,1,1,1,1,1,1
[cut]

An example of a zmcontrol restart and what zmrrdfetch will show. Note I adjusted the ending flag to be : -e `date +%s --date="1 minute ago"  : as compared to the above examples :

[zimbra@zcs806 data]$ /opt/zimbra/libexec/zmrrdfetch -f zmstatuslog -h zcs806.us.DOMAIN.com -s `date +%s --date="15 minute ago"` -e `date +%s --date="1 minute ago"`
timestamp,opendkim,zmconfigd,mailbox,proxy,spell,logger,antivirus,archiving,snmp,ldap,memcached,convertd,stats,antispam,mta
1405621920,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
1405621950,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
1405621980,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
1405622010,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
1405622040,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
1405622070,0.105026416666667,0.1050323,1,1,0.105029991666667,1,0.105016,0.105017466666667,0.105028733333333,1,1,1,0.10503115,0.105014483333333,0.105025008333333
1405622100,0.105026416666667,0.1050323,1,1,0.105029991666667,1,0.105016,0.105017466666667,0.105028733333333,1,1,1,0.10503115,0.105014483333333,0.105025008333333
1405622130,0.105026416666667,0.1050323,1,1,0.105029991666667,1,0.105016,0.105017466666667,0.105028733333333,1,1,1,0.10503115,0.105014483333333,0.105025008333333
1405622160,0.105026416666667,0.1050323,1,1,0.105029991666667,1,0.105016,0.105017466666667,0.105028733333333,1,1,1,0.10503115,0.105014483333333,0.105025008333333
1405622190,,0,,,,,,,,,,,,,
1405622220,,0,,,,,,,,,,,,,
1405622250,,0,,,,,,,,,,,,,
1405622280,,0,,,,,,,,,,,,,
1405622310,,0,,,,,,,,,,,,,
1405622340,,0.893621444444444,,,,,,,,,,,,,
1405622370,,0.893621444444444,,,,,,,,,,,,,
1405622400,,0.893621444444444,,,,,,,,,,,,,
1405622430,,1,,,,,,,,,,,,,
1405622460,,1,,,,,,,,,,,,,
1405622490,,1,,,,,,,,,,,,,
1405622520,,1,,,,,,,,,,,,,
1405622550,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
1405622580,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1

Logger Related Bugs

5.0.11 - Fixed in 5.0.12 But See Below Before Upgrading

UPDATE: From release notes: kill all zmlogger processes before upgrading to 5.0.12 from 5.0.10+

Steps To Resolve
  1. Edit /opt/zimbra/conf/my.logger.cnf, and modify this line for 20 connections.
    • max_connections = 20
  2. Review the following bug and download the zmlogswatchctl script listed in the attachments section.
  3. Shutdown logger.
    • zmloggerctl stop
    • find any orphaned logger or swatch processes, and kill them
      •  ps -ef | grep logg
      •  ps -ef | grep watch
    • Replace the logswatch control file
      • mv /opt/zimbra/bin/zmlogswatchctl /opt/zimbra/bin/original.zmlogswatchctl
    • Add the new zmlogswatchctl file from the above bug to /opt/zimbra/bin
      • as root
        chown root:root /opt/zimbra/bin/zmlogswatchctl
      • as root
        chmod 755 /opt/zimbra/bin/zmlogswatchctl
  4. Start up the logger process.
    • zmloggerctl start

You can be checking the output in the following as well to see if any errors are still being reported:

  • /tmp/logprocess.out
  • /tmp/gengraphs.out
  • /opt/zimbra/logger/db/data/<hostname>.err
Logger Table Check And Repair

Other items to check:

Also, check the logger tables to see if any of the tables need to be repaired

logmysql -e "check table zimbra_logger.amavis"
logmysql -e "check table zimbra_logger.amavis_aggregate"
logmysql -e "check table zimbra_logger.config"
logmysql -e "check table zimbra_logger.disk_aggregate"
logmysql -e "check table zimbra_logger.disk_status"
logmysql -e "check table zimbra_logger.mta"
logmysql -e "check table zimbra_logger.mta_aggregate"
logmysql -e "check table zimbra_logger.processing_history"
logmysql -e "check table zimbra_logger.raw_logs"
logmysql -e "check table zimbra_logger.service_status"

If any of the tables need to be repaired, you replace 'check' with 'repair'. Example:

logmysql -e "repair table zimbra_logger.amavis"
/tmp/logprocess.out Shows MySQL Can't Connect

Please follow these instructions:

Logger Not Working , No Stats, Services Show Not Running In Admin Console

ZCS 6 And Above

On The Logger Host Server

On the Logger monitor host [zmprov gacf |grep zimbraLogHostname] start with the following. Note: run below steps in non-peak hours.

su - zimbra
zmcontrol stop ; exit

Run as root user:

/opt/zimbra/libexec/zmfixperms -e -v

On the logger monitor host, you must enable syslog [if your running rsyslog, see below] to log statistics from remote machines. Run the following as root also :

  • Edit the /etc/sysconfig/syslog file, add -r to the SYSLOGD_OPTIONS setting, SYSLOGD_options=”-r -m 0”
  • Run : /opt/zimbra/libexec/zmsyslogsetup
  • Stop the syslog daemon. Type : /etc/init.d/syslogd restart
    • Or by: service syslog restart
    • Use stop / start if restart isn't available.

If you are using rsyslog instead of syslog, then you will need the following changes in rsyslog config. First, enable the module for remote logging on the logger host. Edit the rsyslog config file "/etc/rsyslog.conf" and enable "imudp" module by uncommenting the same:

$ModLoad imudp
$UDPServerRun 514

After the above changes to the following if you are running rsyslog: restart rsyslog service.

  • Run : /opt/zimbra/libexec/zmsyslogsetup
  • Restart rsyslog with : /etc/init.d/rsyslog restart
    • Or by: service rsyslog restart
    • Use stop / start if restart isn't available.

Your last steps are to then run:

su - zimbra 
zmcontrol start 
/opt/zimbra/libexec/zmloggerinit 
/opt/zimbra/bin/zmsshkeygen
/opt/zimbra/bin/zmupdateauthkeys
On The Other Servers

On all other servers (if you have multiple zimbra servers in your setup). Verify the LogHostname (it should be set to name of Logger monitor Host on all servers) if its different, change the same to Logger monitor Host using below command.

su - zimbra
zmprov gacf |grep zimbraLogHostname  

If it is set wrong, you can correct it by doing:

zmprov mcf zimbraLogHostname <Logger monitor Hostname>

Then run the following after the zimbraLogHostname variable is confirmed or set correctly:

/opt/zimbra/bin/zmsshkeygen
/opt/zimbra/bin/zmupdateauthkeys ; exit
/opt/zimbra/libexec/zmsyslogsetup (run as root user)
Restart either syslogd or rsyslog, depending on what your server is using
service syslog restart or service rsyslog restart
su - zimbra
zmcontrol restart

Reference Link:

This Section Written For ZCS5

Check the following log files first to gather some information:

  • Make sure the /etc/hosts details make the actual hostname and the zmhostname output
    • If someone recently changed the hostname variables for this box, logger can be effected if done improperly and will not log the obvious issue.
  • /tmp/logprocess.out
  • /tmp/gengraphs.out
  • /opt/zimbra/logger/db/data/<hostname>.err
  • /var/log/zimbra.log (logswatch monitors this file)
    • grep'ing for things such as:
      •  egrep 'postfix|amavis|STATUS|DISK|QUEUE' /var/log/zimbra.log
  • You might also want to look at the following directory:
    • /opt/zimbra/logger/db/work/
    • You should see data in here, gif and rrd files.

Then proceed with the following wiki pages, as ordered:

Large Logger Database Killing Performance Of ZCS

Review the following to see what might be best option for you:

Reinitializing Logger Database From Scratch

This will effectively blow away your old logger database, allowing you to start over.

zmloggerctl stop
cd /opt/zimbra/logger/db

Remove the old database:

mv db db-old

or data , if db doesn't exist

mv data data-old

You can remove (rm -rf db or data) the directory if your not worried about old directory being perserved

Reinitialize a new logger database:

/opt/zimbra/libexec/zmloggerinit

Logger is probably running after this, but to make sure

zmloggerctl start

You'll need to manually start this one though

zmlogswatchctl start

You'll have to allow some time to pass for there to be enough data for the graphs to be built in the web interface.

Turning Off Logger

To not have logger startup [ use `hostname` or type in hostname] :

zmprov ms `hostname` -zimbraServiceEnabled logger

To manually shutdown logger

zmloggerctl stop

Customization Or Modification Of Swatch Config

The swatch config file is :

/opt/zimbra/conf/swatchrc.in

If you make a customization, you'll need to do the following for it to take effect:

zmsnmpinit
zmlogswatchctl stop
zmlogswatchctl start
zmswatchctl stop
zmswatchctl start
Verified Against: Zimbra Collaboration 7.0, 6.0 Date Created: 04/16/2014
Article ID: https://wiki.zimbra.com/index.php?title=Ajcody-Logger-Issues Date Modified: 2016-06-20



Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »




Jump to: navigation, search