Logger high CPU utilization
Logger high CPU utilization
How to fix high cpu utilization caused by the zmlogger service (Ubuntu 14)?
Issue symptoms:
- AdminUI and WebUI perform very slow. Every click takes significantly long time to complete.
- top shows 99% CPU utilization for the zmlogger service.
Troubleshooting:
1. logrotate is a service that rotates different logs (including zimbra), based on different criteria. The configuration files for logrotate is in /etc/logrotate.conf and /etc/logrotate.d. Zimbra specific information can be modified by editing /etc/logrotate.d/zimbra file.
Example of the zimbra file:
/var/log/zimbra.log {
daily
missingok
notifempty
create 0644 zimbra zimbra
postrotate
/usr/sbin/service rsyslog restart >/dev/null || true
endscript
compress
}
2. The first thing to do to find out if the logrotate is causing any sort of troubles is to run the following command:
logrotate -vf /etc/logrotate.conf
That command forces the logrotate to go through the config file, to rotate any logs that need to be rotated., and to check for errors. That is very useful as in verbose mode (-v) we can see if there are any errors during the process. In our case, there were couple of errors:
rotating pattern: /var/log/zimbra.log forced from command line (4 rotations) empty log files are not rotated, old logs are removed switching euid to 0 and egid to 104 error: error switching euid to 0 and egid to 104: Operation not permitted - rotating pattern: /var/log/zimbra-stats.log forced from command line (no old logs will be kept) empty log files are not rotated, old logs are removed switching euid to 0 and egid to 104 error: error switching euid to 0 and egid to 104: Operation not permitted
The above errors were pointing at two zimbra files, and the error show that switching the effective uid and gid was not possible.
3. Next, we look at these two files in /var/log directory: (We see other files that have been rotated too. )
-rw-r--r-- 1 syslog adm 158075445 janv. 21 16:01 zimbra.log -rw-r--r-- 1 syslog adm 8136923 janv. 20 06:43 zimbra.log.1.gz -rw-r--r-- 1 syslog adm 8644314 janv. 19 06:50 zimbra.log.2.gz -rw-r--r-- 1 syslog adm 3589749 janv. 18 06:46 zimbra.log.3.gz -rw-r--r-- 1 syslog adm 5584471 janv. 17 06:40 zimbra.log.4.gz -rw-r--r-- 1 syslog adm 78538930 janv. 21 16:01 zimbra-stats.log
What we see here is that the ownership of these files is wrong. Instead of owner syslog and group adm, we should see zimbra & zimbra. From here, the only place we can go is to check the /etc/logrotate.d/zimbra file.
/var/log/zimbra.log {
daily
missingok
notifempty
create 0644 syslog adm
postrotate
/usr/sbin/service rsyslog restart >/dev/null || true
endscript
compress
}
-
/var/log/zimbra-stats.log {
daily
missingok
notifempty
create 0644 syslog adm
postrotate
/usr/sbin/service rsyslog restart >/dev/null || true
endscript
rotate 0
}
We now know that the wrong permissions have been assigned from here.
Resolution
In short, changing the above values, from syslog adm to zimbra zimbra fixed the issue.
Post tasks
After changing the values we need to do couple of things to make sure its going to work:
as root: # service rsyslog restart # /opt/zimbra/libexec/zmfixperms -e -v
as zimbra: $ zmloggerctl stop $ zmloggerctl start $ zmlogswatchctl stop $ zmlogswatchctl start
