From Zimbra :: Wiki

Jump to: navigation, search
Admin Article

Article Information

This article applies to the following ZCS versions.
  ZCS 5.0 Article  ZCS 5.0



Zmstats is how Zimbra exposes its performance metrics and statistics to the world. The information covers a wide array of data: disk usage, cpu utilization, java statistics, zimbra counters and beyond.

Zmstats consists of the following components and scripts:

  • zmstatctl (all zmstat collection scripts are located in /opt/zimbra/libexec)
    • zmstat-fd
    • zmstat-df
    • zmstat-cpu
    • zmstat-proc
    • zmstat-io
    • zmstat-vm
    • zmstat-convertd
    • zmstat-mtaqueue
    • zmstat-mysql
  • zmstat-chart
    • zmstat-chart-config

Running zmstats

  • On the server where the stats were produced, make sure that the zmstat-chart.xml is provided. When running zmdiaglog, the zmstats and zmstat-chart-config are automatically produced]
# su - zimbra
$ zmstat-chart-config > /tmp/zmstat-chart-`zmhostname`.xml
  • Make a charts directory:
$ mkdir ~/zmstat/2010-06-01/charts
  • Then produce the stats:
$ zmstat-chart -c /tmp/zmstat-chart.xml -s ~/zmstat/2010-06-01 -d ~/zmstat/2010-06-01/charts


zmstatctl is used to start and stop and various zmstat-* data logging scripts


zmstat-chart reads an XML configuration file generated by zmstat-chart-config and generates a set of HTML and PNG graph images suitable for rapidly diagnosing problems and load issues.


Used to control what is graphed and how


 <chart title="Mailboxd: JVM Heap Used"
   <plot data="heap_used" legend="total" divisor="1m"/>

The above defines a chart that reads the counter "heap_used" out of mailboxd.csv. It takes that counter and graphs it to a file "mboxd-heap-used.png". There will be a resulting graph with a yAxis labelled "MB" and "heap_used" divided by 1million (megabytes) will be graphed over time.

Multiple plots can be placed onto a single chart through the use of additional <plot> elements.

See /opt/zimbra/conf/zmstat-chart.xml for more examples

Individual stat scripts and their related counters


Contains the following statistics written by the mailbox server. Counters that are not cumulative are reset once per minute.

account_cache_hit_rate LDAP account cache hit rate
account_cache_size LDAP account cache size
acl_cache_hit_rate LDAP ACL cache hit rate
bis_read Number of times that the file descriptor cache read message data from disk
bis_seek_rate Percentage of file descriptor cache disk reads that required a seek
calcache_hit Hit rate of calendar summary cache, counting cache hit from both memory and file
calcache_lru_size Number of calendars (folders) in the calendar summary cache LRU in Java heap
calcache_mem_hit Hit rate of calendar summary cache, counting cache hit from memory only
cos_cache_hit_rate LDAP COS cache hit rate
cos_cache_size LDAP COS cache size
db_conn_count Number of times that the server got a database connection from the pool
db_conn_ms_avg Average latency (ms) of getting a database connection from the pool
db_pool_size Number of database connections in use
domain_cache_hit_rate LDAP domain cache hit rate
domain_cache_size LDAP domain cache size
fd_cache_hit_rate File descriptor cache hit rate
fd_cache_size Number of open file descriptors that reference message content
gc_copy_count Number of times that copy GC was invoked
gc_copy_ms Time (ms) spent on copy GC
gc_major_count Number of times that major GC was invoked
gc_major_ms Time (ms) spent on major GC
gc_marksweepcompact_count Number of times that marksweepcompact GC was invoked
gc_marksweepcompact_ms Time (ms) spent on marksweepcompact GC
gc_minor_count Number of times that minor GC was invoked
gc_minor_ms Time (ms) spent on minor GC
group_cache_hit_rate LDAP group cache hit rate
group_cache_size LDAP group cache size
heap_free Number of bytes free in the entire JVM heap
heap_used Number of bytes used in the entire JVM heap
idx_bytes_read Accumulated bytes read by Lucene
idx_bytes_read_avg Average of idx_bytes_read
idx_bytes_written Accumulated bytes written by Lucene
idx_bytes_written_avg Average of idx_bytes_written
idx_wrt_avg Average number of concurrent index writers
idx_wrt_opened Accumulated number of index writers opened
idx_wrt_opened_cache_hit Accumulated number of cache hits when opening an index writer
imap_conn Number of cleartext IMAP connections
imap_count Number of IMAP requests received
imap_ms_avg Average processing time (ms) of IMAP requests
imap_ssl_conn Number of SSL IMAP connections
innodb_bp_hit_rate InnoDB buffer pool hit rate
ldap_dc_count Number of times that the server got an LDAP directory context
ldap_dc_ms_avg Average latency (ms) of getting an LDAP directory context
lmtp_dlvd_bytes Number of bytes of data delivered to mailboxes as a result of LMTP delivery
lmtp_dlvd_msgs Number of messages delivered to mailboxes as a result of LMTP delivery
lmtp_rcvd_bytes Number of bytes received over LMTP
lmtp_rcvd_msgs Number of messages received over LMTP
lmtp_rcvd_rcpt Number of LMTP recipients
mbox_add_msg_count Number of messages that were added to a mailbox
mbox_add_msg_ms_avg Average latency (ms) of adding a message to a mailbox
mbox_cache Mailbox cache hit rate
mbox_cache_size Number of mailboxes cached in memory
mbox_get_count Number of times that the server got a mailbox from the cache
mbox_get_ms_avg Average latency (ms) of getting a mailbox from the cache
mbox_item_cache Item cache hit rate
mbox_msg_cache Message cache hit rate
mpool_code_cache_free Number of bytes free in the code cache memory pool
mpool_code_cache_used Number of bytes used in the code cache memory pool
mpool_eden_space_free Number of bytes free in the eden space memory pool
mpool_eden_space_used Number of bytes used in the eden space memory pool
mpool_perm_gen_free Number of bytes free in the perm gen memory pool
mpool_perm_gen_used Number of bytes used in the perm gen memory pool
mpool_survivor_space_free Number of bytes free in the survivor space memory pool
mpool_survivor_space_used Number of bytes used in the survivor space memory pool
mpool_tenured_gen_free Number of bytes free in the tenured gen memory pool
mpool_tenured_gen_used Number of bytes used in the tenured gen memory pool
msg_cache_size Number of message structures cached in memory
pop_conn Number of cleartext POP3 connections
pop_count Number of POP3 requests received
pop_ms_avg Average processing time (ms) of POP3 requests
pop_ssl_conn Number of SSL POP3 connections
server_cache_hit_rate LDAP server cache hit rate
server_cache_size LDAP server cache size
soap_count Number of SOAP requests received
soap_ms_avg Average processing time (ms) of SOAP requests
soap_sessions Number of SOAP sessions
xmpp_cache_hit_rate LDAP XMPP cache hit rate
xmpp_cache_size LDAP XMPP cache size
zimlet_cache_hit_rate LDAP zimlet cache hit rate
zimlet_cache_size LDAP zimlet cache size

zmstat-fd: fd.csv

Captures file descriptor usage on the system

  • fd_count: current number of open file descriptors

zmstat-df: df.csv

Captures disk usage

  • path: mount point
  • disk: device
  • disk_use: space used
  • disk_pct_used: percentage used
  • disk_space: total space

zmstat-cpu: cpu.csv

  • cpu:user: total user time
  • cpu:nice: total nice process time
  • cpu:sys: total system time
  • cpu:idle: total idle time
  • cpu:iowait: total time in iowait
  • cpu:irq: total time in irq
  • cpu:softirq: total time in softirq
  • cpuN:XXX: same as above, but per individual core/cpu

zmstat-io: io.csv and io-x.csv

  • dev:tps: transactions per second
  • dev:kB_read/s: read rate
  • dev:kB_wrtn/s: write rate
  • dev:kB_read: bytes read
  • dev:kB_wrtn: bytes written
  • dev:rrqm/s: read requests merged per second queued to device
  • dev:wrqm/s: write requests merged per second queued to device
  • dev:r/s: reads per second
  • dev:w/s: writes per second
  • dev:rkB/s read rate
  • dev:wkB/s: write rate
  • dev:avgrq-sz: average size (sectors) of requests
  • dev:avgqu-sz: average queue length
  • dev:await: average wait time for requests to be served
  • dev:svctm: average time to service requests
  • dev:%util: percentage of CPU time / bandwidth utilization of device

zmstat-proc: proc.csv

  • user: total user time
  • sys: total system time
  • idle: total idle time
  • iowait: total iowait time
  • PROC-total-cpu: total cpu usage for PROC
  • PROC-utime: usertime for PROC
  • PROC-stime: system time for PROC
  • PROC-totalMB: total memory footprint for PROC
  • PROC-rssMB: resident-set-size of PROC
  • PROC-sharedMB: shared memory of PROC
  • PROC-process-count: number of threads/subprocesses

zmstat-vm: vm.csv

The output of vmstat(1) is recorded for reviewable statistics.

  • r:
  • b:
  • swpd:
  • free:
  • buff:
  • cache:
  • si:
  • so:
  • bi:
  • bo:
  • in:
  • cs:
  • us:
  • sy:
  • id:
  • wa:
  • st:
  • ...: other counters reported by vmstat

zmstat-mtaqueue: mtaqueue.csv

  • KBytes: kilobytes queued by the mta
  • requests: number of items queued by the mta

zmstat-convertd: convertd.csv

Collects CPU statistics for the convertd process (NE only).

  • utime: user time for convertd
  • stime: system time for convertd
  • cputime: user + system time total
  • rss: resident-set-size memory usage
  • threads: number of threads for convertd
  • processes: number of processes for convertd

Only if IO Accounting is enabled in the Linux kernel

  • rchar: bytes read
  • wchar: bytes written
  • read_bytes: bytes read from disk
  • write_bytes: bytes written to disk

zmstat-mysql: mysql.csv

Columns for mysql.csv are derived from the values of the query "SHOW GLOBAL STATUS". Refer to the mysql administration manual for further elaboration on the meanings of all its counters.

Related: Server Monitoring

Verified Against: unknown Date Created: 3/07/2009
Article ID: Date Modified: 10/22/2013
Personal tools