Server Monitoring

From Zimbra :: Wiki

Jump to: navigation, search
Admin Article

Article Information

This article applies to the following ZCS versions.
  ZCS 5.0 Article  ZCS 5.0

Contents

Stats Collection

One of the coolest things about working with ZCS is that it exposes you to many technologies such as Java, Postfix, OpenLDAP, and MySQL. An administrator of a ZCS system should have a working knowledge of these technologies, in order to monitor the system and solve performance problems.

The ZCS server collects many performance-related statistics. The data is stored in the following CSV files in /opt/zimbra/zmstat:

  • cpu.csv: CPU utilization
  • fd.csv: file descriptor count
  • mailboxd.csv: ZCS server and JVM statistics
  • mtaqueue.csv: Postfix queue
  • proc.csv: disk utilization
  • soap.csv: SOAP request processing time
  • threads.csv: JVM thread counts
  • vm.csv: Linux VM statistics (from the vmstat command)

These files are in a standard CSV format that can be loaded into Excel for viewing and charting. They are archived to subdirectories of /opt/zimbra/zmstat every day at midnight.

The zmstat-chart Utility

Zimbra provides a command-line utility called zmstat-chart that is used to generate charts from the CSV data. The following command:

$ zmstat-chart -s /opt/zimbra/zmstat/2008-04-03 -d ~/zmstat/2008-04-03/charts

will read data from CSV files in /opt/zimbra/2008-04-03/ and write HTML and PNG files to the ~/zmstat/2008-04-03charts/ directory (which would be /opt/zimbra/zmstat/2008-04-03/charts/ if run as the zimbra user). Default chart parameters are specified in /opt/zimbra/conf/zmstat-chart.xml. If you'd like to skip certain charts or add charts that aren't generated by default, you can specify an alternate chart conf file with the -c option.

The zmstat-chart configuration file may need generated prior to running zmstat-chart. To generate, run:

$ zmstat-chart-config > /opt/zimbra/conf/zmstat-chart.xml

Chart Analysis

CPU utilization

CPU utilization is tracked both at the server level and the process level. Here's a sample process CPU graph:
ProcCpu.png
This chart shows that server CPU increases in the morning as users come to work, followed by a spike at 9:00AM. To further investigate the problem, you could look at other charts or the server logs to determine the cause of the spike.

Disk Utilization

Disk utilization is tracked for each disk partition:
DiskUtil.png
This chart shows that disk activity also goes up along with the increased CPU utilization. It also shows that the sda partition is experiencing more load than the others. When laying out disk partitions for a ZCS installation, it's a good idea to put different system components (/opt/zimbra/store, /opt/zimbra/db, /opt/zimbra/index) on separate partitions. This makes it much easier to determine which system component is performing more I/O.

Memory Consumption

ZCS stats track the amount of memory used by each process in the system:
ProcTotalMem.png
This information can be used to determine how system memory is being allocated between the various processes.

JVM Garbage Collection

ZCS tracks the percentage of time that the Java Virtual Machine spends on garbage collection:
GcTime.png
If the JVM is spending more than a few percent of its time on garbage collection, consider increasing the amount of memory allocated to the server Java process.

InnoDB Buffer Pool Hit Rate

This chart tracks the buffer pool hit rate for the InnoDB storage engine in MySQL:
MysqlBufpoolHitRate.png
Higher numbers indicate that MySQL is able to get data from memory instead of going to disk. If your hit rate is below 990, MySQL is hitting the disk harder than it should. Investigate the following issues:

  • Consider increasing the buffer pool size in my.cnf.
  • Run EXPLAIN on some of the SQL statements in /opt/zimbra/log/myslow.log to see if they are causing InnoDB to read a large amount of data into memory.

Summary

This article describes just a few of the statistics tracked and charted by ZCS. For more details, spend some time looking at your server's performance data either in the zmstat-chart output or Excel.

Related Articles

Verified Against: 5.0.x (not 6.0) Date Created: 4/4/2008
Article ID: http://wiki.zimbra.com/index.php?title=Server_Monitoring Date Modified: 11/15/2011
Personal tools