Performance Tuning Guidelines for Large Deployments: Difference between revisions
No edit summary |
|||
Line 1: | Line 1: | ||
{{ZC}}{{Article Infobox|{{admin}}|{{ZCS 6.0}}|{{ZCS 5.0 | {{ZC}}{{Article Infobox|{{admin}}|{{ZCS 6.0}}|{{ZCS 5.0}}}}=Hardware and Operating System= | ||
==RAM and CPU== | ==RAM and CPU== |
Revision as of 18:46, 24 May 2011
Article Information | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
This article applies to the following ZCS versions.
Hardware and Operating SystemRAM and CPUZCS, like all messaging and collaboration systems, is an IO bound application. Three core components of ZCS (a) the Java based message store (aka mailbox server, tomcat in 4.5.x and jetty 5.0 onwards), (b) MySQL instances used for metadata, and (c) LDAP servers (master and replica) - all rely heavily on caching data in RAM to provide better performance and reduce disk IO. For all large installations we recommend at least 8GB of RAM. Our testing shows that a system with the same CPUs and disk is able to support more users when upgraded from 8GB to 16GB of RAM. We recommend an x86_64 dual-dual core CPU, of a speed that is not too low or too high on the price/performance ratio. Disable hyper-threading if that feature is present in your CPU (performance monitoring data is unreliable). At this time, we have not tested on dual-quad cores (coming soon). Almost all recent 'x86' server CPUs support installing a 32-bit/i686 version of Linux or a 64-bit/x86_64 version. We strongly recommend installing the 64-bit version if you have more than 4GB of RAM. If you have 4GB of RAM or less, it is unclear that the 64-bit version will boost performance, but you shouldn't really be running a large install on a system with < 8GB of RAM. If you anticipate adding more RAM in the near future during a maintenance window, do install the 64-bit version now - upgrade from 32-bit to 64-bit is possible, but a lot of work. A word of caution is in order around 32-bit kernel and 8GB of RAM. In 32-bit mode, the CPU can address only 4GB of RAM even if you paid for 8GB worth of memory sticks, and, by default, a 32-bit Linux kernel only allows each process to address 2GB of space. Through PAE (Process Address Extension), a feature available in some CPUs, and a special 32-bit kernel that supports large address space for processes, it is possible to get a 32-bit mode kernel that really uses > 4GB of RAM, and get a per process 3-4GB address range. Please avoid this hassle. Given there is plenty of RAM, the CPU performs better in 64-bit mode, more CPU registers are available, there is no segment addressing overhead introduced by PAE, and you get a tested platform. Monitor for swap activity as swapping very adversely affects Zimbra performance. Make sure you have not over-configured memory settings for ZCS components (details in mailbox server section below). Consider setting swappiness to 0. You can have it take effect immediately with sysctl -w. For a permanent change that takes effect at next reboot, add the following to /etc/sysctl.conf: vm.swappiness=0 DiskZimbra mailbox servers are read/write intensive, and even with enough RAM/cache, the message store will generate a lot of disk activity. LDAP is read heavy and light on writes, is able to use caches a lot more effectively, and does not generate the type of disk activity that mailbox servers do. In a mailbox server, the greatest source IO activity is generated by these three sources, in decreasing order of load generated:
MySQL, Lucene and blob stores generate random IO and therefore have to be serviced by a fast disk subsystem. Below are some guidelines around selecting a disk system. Contact pre-sales support for more detailed guidance.
Services to DisableLinux distributions tend to enable services by default that are not really required in a production ZCS server. We recommend identifying and disabling such services to reduce risk of exposure to vulnerabilities in services that are enabled but not really needed/used, and to avoid any unintended performance interference.
The table below lists a few examples of services that may be installed by your Linux distribution that you might consider disabling. Use it as a guide - it is not an exhaustive or prescriptive list. Some services maybe required for the proper functioning of your system, so exercise caution when disabling services.
Services to EnableCertain services are either required or useful when running ZCS:
Diagnostic ToolsPlease install and be familiar with the use of at least the following operating system monitoring tools.
Some of these tools are part of the procps and sysstat packages. The zmstat service shipped since ZCS 4.5.9 requires atleast the IO/CPU/memory monitoring tools to record performance data periodically. It is a good idea to make sure all your servers have the 'zmstats' service enabled. Open File Descriptors LimitThe mailbox server (specially the Lucene search index) might need to operate on a large number of files at the same time. The Zimbra installer modifies /etc/security/limits.conf to set the maximum number of file descriptors that the 'zimbra' UNIX user is allowed to concurrently open. Until ZCS 5.0.2, the installer used to set this limit to 10,000, and this wiki page used to advice that large installs modify this to 100,000. As of ZCS 5.0.2, the installer sets the max file descriptor limit to 524,288 (2^19). See bug 23211 for details. Installations upgrading from earlier releases of ZCS should verify that their /etc/security/limits.conf contains the following lines after the upgrade to ZCS 5.0.2: zimbra soft nofile 524288 zimbra hard nofile 524288 File SystemWe recommend the ext3 file system for Linux deployments (tried and true, performance for random IO is a wash, gains only in blob store for other file systems). Mount your file systems with the noatime option. By default, the atime option is enabled and updates the last access time data for files. Mounting your file systems with the noatime option reduces write load on the disk subsystem. You should enable dirsync for ext3 file systems (or equivalent method for a non-ext3 file system). Zimbra mailbox server uses fsync(2) as necessary to ensure that data in files are flushed from buffers to disk. Eg, when an incoming message is received, fsync is called on the blob store message file before the MTA is given an acknowledgment that the message was received/delivered by the MBS. However, when Zimbra mailbox server or MTA creates new files, such as during message delivery, the update to the directory containing the file must be flushed to disk. Even if the data in the file is flushed with fsync, the file entry in the directory might be lost if server crashes before the directory update is flushed to disk. With the ext3 file system, to have directory updates be written to disk automatically and atomically, you can update directory attributes by running chattr +D dir on all the relevant directories. If you are doing this for the first time, you should consider running chattr -R +D to recursively update a whole tree of directories; future sub-directories will inherit the attribute from parent directory. We recommend dirsync be enabled for all blob stores, Lucene search index directories, and MTA queues. With the ext3 file system, you can also add dirsync as a mount option, but you will have to exercise caution to not enable this on the root filesystem - we've received reports of the boot loader failing in the presence of dirsync as a mount option on the root file system. We suggest the following options as a guideline for when creating an ext3 file system with the mke2fs command. Consult ext3 documentation. Caution: Running mke2Fs will wipe all data from the partition. Make sure that you create the file system in the correct partition.
Network PortsPerform a portscan of your servers from a remote host and localhost (eg, use nmap). Only ports that you need to have open should be open. The following ports are used by ZCS. If you have any other services running on these ports, turn them off.
For a multi-server deployment, see also Ports. Network StackThe Linux kernel makes TCP/IP network tunables available in the /proc/sys/net/ipv4 directory. These files can be modified directly or with the sysctl command to make kernel configuration changes on the fly. But changes made this way do not persist across reboots. We recommend editing the file /etc/sysctl.conf and adding the settings below so they will be permanent. If you need your edits to sysctl.conf to take effect right away, use the sysctl -p option. net.ipv4.tcp_fin_timeout=15 net.ipv4.tcp_tw_reuse=1 net.ipv4.tcp_tw_recycle=1 The above settings allow for ZCS servers to handle a lot of short lived connections. When TCP/IP was designed, networks were lossy and had high latency. With today's modern networks, it is common practice to configure the above options so port numbers are not stuck in TIME_WAIT state. Various RFCs specify how long to wait, and even kernel docs caution you against changing the defaults, but it can pay to be aggressive. Documentation for these settings is in the kernel-doc package, in the file networking/ip-sysctl.txt. Zimbra Mailbox ServerConnection HandlingEach ZCS mailbox server is a HTTP, IMAP and POP3 server, rolled into one process. The server is highly multi-threaded and uses pool of threads to service incoming connections for these services. An important part of connection handling configuration is sizing these thread pools. Moderns JVMs and kernels are able to support a lot of threads (we have tested as high as 3000), however too many threads can cause memory pressure on the server. These thread pool sizes can be configured on a per server basis. However, if you have or will have a multi-node install and all your servers will have a similar configuration, you can set the size in global config, so any new servers you add will get the right defaults for your environment. You can always override the global config setting on any server, by setting the value in the server object. We'll identify any attributes that can be set in both global config or server with a comment below, but show the example as modifying server. Use zmprov modifyServer to modify server attributes, and zmprov modifyConfig to modify global config. Add note about server restart. Add note about having to apply this on each server. Both these should be higher level comments, probably along with the global config vs server blurb. HTTPIn ZCS 5.0, the HTTP stack used by the mailbox server is provided by Jetty (a Java application container). Earlier releases used Apache Tomcat. In both cases, a thread is dedicated to the servicing a HTTP request. Jetty also offers support for idle but long lived HTTP connections without a dedicated thread (see Zimbra blog). Since HTTP connections are not usually long lived, you must size the HTTP thread pool to accommodate concurrent connections at any instant during at the busiest time of the day for the server. You can examine access_log from Jetty or Tomcat to look at your concurrent connections in peak second, and add a 50% padding to that. For most installations, we have found 250 threads each for HTTP and HTTPS to be sufficient. # ZCS 5.0 has a single thread pool for both HTTP and HTTPS # global config or server OK $ zmprov ms this.server.name zimbraHttpNumThreads 500 # ZCS 4.5.x and earlier had distinct thread pools for HTTP and HTTPS # global config or server OK $ zmprov ms this.server.name zimbraHttpNumThreads 250 $ zmprov ms this.server.name zimbraHttpSSLNumThreads 250 POP3POP3 connections are in general short lived like HTTP connections. The size of the POP3 thread pool should be derived similarly, but the log file to use is audit.log. We have found that a setting of 300 is able to support a few 10s of thousands of users logging in and checking mail every 8 minutes. # global config or server OK $ zmprov ms this.server.name zimbraPop3NumThreads 300 Most POP3 connections do login, download mail, delete downloaded mail on server, logout. Mailboxes are small and contain the most recent mail. POP3 users who use download but keep on server cause higher load on server for large Inbox folders. Users with large mailboxes seldom use POP3, so this is not a common case. IMAPIMAP thread pool sizing is very different from HTTP/POP3 thread pool sizing. IMAP clients connect and leave the connections open for long periods of time. Some IMAP clients create as many as 4 simultaneous connections to the server. IMAP protocol, by nature, also places a lot of load on servers. Until bug 9470 is resolved, a single thread needs to be dedicated for each IMAP connection at peak time. Too many concurrent IMAP connections can two types of failures - the Java server can get an out of memory error or the disk system will not be able to keep up with the load placed by IMAP. In our tests we have set IMAP thread pool as high as 2500. We recommend closely monitoring your IMAP load and distributing IMAP users across more mailbox servers. See also bug 24096. # global config or server OK $ zmprov ms this.server.name zimbraImapNumThreads 500 LMTPLMTP is the protocol through which mailbox servers receive messages from the Postfix MTA. When possible, Postfix performs multiple LMTP transactions on the same connection. Message delivery is an expensive operation, so a handful of message delivery threads can keep the server busy, unless the message delivery threads become blocked on some resource. While it is tempting to increase the LMTP threads (and the corresponding Postfix LMTP concurrency setting) when MTA queues are behind and latency on message delivery is high, adding more concurrent load is unlikely to speed delivery - you will likely bottleneck your IO subsystem and risk making throughput lower because of contention. If you do experience mail queue backup because LMTP deliveries are slow, then do thread dumps on the mailbox server to see why the LMTP threads are unable to make progress. Another risk of a high LMTP concurrency is that is the event there is a bulk mailing, the server may become unresponsive because it is so busy with message deliveries. The default postfix LMTP concurrency and mailbox server LMTP threads is 20. # global config or server OK $ zmprov ms <localservername> zimbraLmtpNumThreads 40 # on each MTA server... $ zmlocalconfig -e postfix_lmtp_destination_concurrency_limit=20 Memory AllocationBoth the Java mailboxd and MySQL processes that are part of the mailbox service benefit from more memory, with a few caveats. Allocation of memory to these two processes is done through the local config variable mailboxd_java_heap_memory_percent and the my.cnf variable innodb_buffer_pool_size (details on both are in the following sections). In a new install, ZCS tries to allocate 30% of system memory to the Java heap, and 25% of system memory to innodb buffer pool. Please make sure that you do not configure these values too high for your system. If these values are too high, your system will swap and swapping is extremely detrimental to ZCS performance. Check the JVM Options section to see if you should reduce your heap memory percent. If you believe your system has unused RAM, you can allocate this memory to innodb buffer pool, only after you have monitored your system and made sure your system in not swapping.
Never run memory hungry processes like rsync or imapsync along side a mailbox server. JVM OptionsYou can tune the Java virtual machine (JVM) that runs the mailbox server application by making changes to a few local config variables. Java runtimes have an automatically garbage collected heap and ZCS maintains caches in the Java heap, and therefore selecting the garbage collector and adjusting the heap size/options are critical to good performance. ZCS by default tries to provide best settings for your system. However, we strongly recommend that all installations double check their JVM settings based on reviewing and understanding this section. Local Config VariablesThe following local config variables control the options provided to the mailbox server JVM. Changes to these local config variables are preserved across upgrades. For your changes to take effect, you must restart the mailbox service.
$ zmlocalconfig -e zimbra_require_interprocess_security=0 Recommended OptionsZCS 6.0 enables the following options by default, and all of these are recommended regardless of your ZCS version:
You should consider enabling the following options which are not on by default:
ZCS 4.5.x and EarlierIn ZCS 4.5.x and older releases, the local config variables mailboxd_java_options and mailboxd_java_heap_memory_percent were called tomcat_java_options and tomcat_java_heap_memory_percent. There was no local config variable for thread stack size, and thread stack size should be specified in tomcat_java_options. CMS GC and Young Gen SizeWhen enabling concurrent mark sweep GC (-XX:+UseConcMarkSweepGC), it is important to make sure you specify a bigger young generation for ZCS. The default young generation size of 10MB causes too many objects get promoted to the older generation resulting in more frequent older generation collections.
Hints and ExamplesWhen modifying mailboxd_java_options, do not change -server from being the first option on the list. After changing Java options, when restarting the mailbox service, if the mailbox Java process does not start up, you should check /var/log/zimbra.log and /opt/zimbra/log/zmmailboxd.out for any errors. If you have a typo or error in the options, the Java process will not start. zmlocalconfig does not have an append option. You should exercise care to not blow away existing value of local config variables. Follow a pattern of seeing what's set now and then appending to it. The use of the fmt command and multi-line strings (ie, \ terminated lines) in the following examples is just to make this page more readable. Enable HeapDumpOnOutOfMemoryError# Check current value $ zmlocalconfig mailboxd_java_options | fmt mailboxd_java_options = -server -Djava.awt.headless=true -XX:+UseConcMarkSweepGC -XX:NewRatio=2 -XX:PermSize=128m -XX:MaxPermSize=128m -XX:SoftRefLRUPolicyMSPerMB=1 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime # Set new value $ zmlocalconfig -e \ mailboxd_java_options="-server -Djava.awt.headless=true \ -XX:+UseConcMarkSweepGC -XX:NewRatio=2 -XX:PermSize=128m \ -XX:MaxPermSize=128m -XX:SoftRefLRUPolicyMSPerMB=1 \ -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps \ -XX:+PrintGCApplicationStoppedTime \ -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/zimbra/log" Enable CMS GC in ZCS 6.0# This made up example has parallel GC set and a very small # young generation size $ zmlocalconfig mailboxd_java_heap_new_size_percent mailboxd_java_heap_new_size_percent = 5 $ zmlocalconfig mailboxd_java_options mailboxd_java_options = -server -Djava.awt.headless=true -XX:+UseParallelGC -XX:NewRatio=2 -XX:PermSize=128m -XX:MaxPermSize=128m -XX:SoftRefLRUPolicyMSPerMB=1 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime # Set young gen size and change parallel to CMS $ zmlocalconfig -e mailboxd_java_heap_new_size_percent=25 $ zmlocalconfig -e \ mailboxd_java_options="-server -Djava.awt.headless=true \ -XX:+UseConcMarkSweepGC -XX:NewRatio=2 -XX:PermSize=128m \ -XX:MaxPermSize=128m -XX:SoftRefLRUPolicyMSPerMB=1 \ -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps \ -XX:+PrintGCApplicationStoppedTime" Enable CMS GC in ZCS 5.0# This server has parallel GC and ZCS 5.0 doesn't have # a local config to directly control young gen size $ zmlocalconfig mailboxd_java_options + fmt mailboxd_java_options = -server -Djava.awt.headless=true -XX:+UseParallelGC -XX:NewRatio=2 -XX:PermSize=128m -XX:MaxPermSize=128m -XX:SoftRefLRUPolicyMSPerMB=1 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime # Change parallel to CMS and set young gen size directly # in mailboxd_java_options (eg assumes heap size is 2GB) $ zmlocalconfig -e \ mailboxd_java_options="-server -Djava.awt.headless=true \ -XX:+UseConcMarkSweepGC -Xmn500m -XX:NewRatio=2 -XX:PermSize=128m \ -XX:MaxPermSize=128m -XX:SoftRefLRUPolicyMSPerMB=1 \ -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps \ -XX:+PrintGCApplicationStoppedTime" Unrestricted LauncherThe Java process that runs the mailbox server application is launched by a manager process. This manager process does two things: make sure certain JVM options are disallowed (for setuid purposes), and to watch and restart the Java process if it died (due a bug, out of memory error, etc). In releases prior to ZCS 5.0.6, the standard launcher program was very strict about JVM options allowed that you could set in mailboxd_java_options, eg, -XX:HeapDumpPath and -XX:ErrorFile were not allowed and required the user of an unrestricted launcher. If you are on these very old releases and you need to add these restricted options, perform the followings before restarting the mailbox service (and revert to the standard launcher once you have diagnosed your problem): (as root) # cd /opt/zimbra/libexec For 4.5.x: # mv zmtomcatmgr zmtomcatmgr.orig # ln -s zmtomcatmgr.unrestricted zmtomcatmgr For 5.0.x: # mv zmmailboxdmgr zmmailboxdmgr.orig # ln -s zmmailboxdmgr.unrestricted zmmailboxdmgr MySQLZCS stores metadata about the content of mailboxes in a MySQL database. ZCS uses MySQL's innodb storage engine. Innodb caches data from disk, and performs best when load doesn't cause it to constantly evict pages from its cache and read new ones. Every mailbox store server has its own instance of MySQL so ZCS can scale horizontally. Inside each MySQL server instance, there are 100 mailbox groups each with its own database to avoid creating very large tables that store data for all users. 100 was somewhat arbitrary, but works well. MySQL configuration for the mailbox server is stored in /opt/zimbra/conf/my.cnf which is not rewritten by a config rewriter, but is not preserved across upgrades. Configure the following tunables. All settings below should be in the [mysqld] section.
table_cache = 1200 innodb_open_files = 2710
innodb_buffer_pool_size = 3435973840
innodb_max_dirty_pages_pct = 10
innodb_flush_method = O_DIRECT We are often shown http://bugs.mysql.com/bug.php?id=21947 as a reason why O_DIRECT should not be used. There is very little information or evidence in that bug. Our own testing that shown that O_DIRECT makes ZCS database performance better. Do NOT change innodb_log_file_size. Change this setting is non-trivial, and requires certain procedures to be followed which are documented in the MySQL manual. Lucene IndexZCS creates and maintains a Lucene search index for every mailbox. As messages arrive, they are added to the Lucene index, and Lucene merges these additions frequently (which results in IO). If multiple additions to a mailbox can be performed together in RAM and flushed at once, write load will be lower. ZCS tries to perform this optimization by keeping open a certain number of mailboxes' index writers (local config zimbra_index_lru_size, default 100), and flushes any open index writers periodically (local config zimbra_index_idle_flush_time, default 10 minutes). Eg, a single mailbox gets two messages in a 10 minute window between flushes, and its index writer was in cache in that time between the two deliveries, then the index update writes to disk less. However, increasing number of index writers is bad under at least these conditions:
We have found that setting index writer cache size to more than 500-1000 on 8GB can result in high GC times and/or out of memory errors, depending on your mailbox usage. If you need to disable the index writer cache entirely (because you are seeing out of memory errors, or you have determined that your message delivery rate is so even across many mailboxes that the cache doesn't reduce IO), do this: # need to restart mailbox service for this to take effect $ zmlocalconfig -e zimbra_index_max_uncommitted_operations=0 The value of 0 for zimbra_index_max_uncommitted_operations overrides any value in zimbra_index_lru_size, ie, 0 uncommitted ops disables the index writer cache use. See also bug 24074 (too many recipients should bypass index writer cache). In ZCS 5.0.3 "batched indexing" capability (bug 19235) for Lucene indexes was added. In a future release, this will become the default mode (bug 27913: make batched indexing the only indexing mode). To take advantage of this performance enhancement and reduce overall Lucene I/O on a ZCS system you can use a command similar to the following (Note: this is a per COS setting): $ for cos in `zmprov gac`; do zmprov mc $cos zimbraBatchedIndexingSize 20; # see below for size recommendations done The zimbraBatchedIndexingSize value depends on a number of factors including, typical access methods, average size of messages/attachments and CPU/IO capabilities of the server. However, the primary factor currently looked at is the access method for retrieving/viewing email (HTTP vs IMAP vs POP). A zimbraBatchedIndexingSize of 20 should be a good starting point for almost any type of user access patterns, but when the usage for a particular COS is strictly IMAP(S) and/or POP(S) the batch size can go even higher (zimbraBatchedIndexingSize == 100, for example). Backup and RecoveryThe Network Edition of ZCS includes full backup and restore functionality. When ZCS is installed, a backup schedule is automatically added to the cron table. You can change the schedule, but you should not disable it. Backing up the server on a regular basis can help you restore your mail service if an unexpected crash occurs. The default full backup is scheduled for 1:00 a.m. every Sunday and the default incremental backups are scheduled for 1:00 a.m. Monday through Saturday. Backups are stored in /opt/zimbra/backup. You will need to make sure that this backup is on a different disk and partition than your data and set up the process to automatically copy the zmbackups offsite or to a different machine or tape backup to minimize the possibility of unrecoverable data loss in the event that the backup disk fails. Backup and restore is documented in the Administrator’s Guide and more information can be found elsewhere in the Zimbra wiki. TODO: add a note about auto-grouped backups in 5.0. Max message sizeZCS 5.0 has much better support for larger messages. In earlier versions large messages caused increased memory pressure and we recommend using the default 10MB max message size. Even with ZCS 5.0 do not increase max message size arbitrarily - large messages do caused increased IO load on the system (by nature), and external mail servers will like not accept large messages. Message cache sizePlease see Message_Cache for the purpose of the message cache along with details on changes in the behavior and configuration of the message cache between ZCS 6 and ZCS 5. As of ZCS 6.0, the message cache is an in-memory cache that stores the MIME structures of recently-accessed messages. In ZCS 5.0, the message cache stored not just the message structure, but also the content of messages less than 1MB. This cache speeds up retrieval of message content for mail clients such as Mail.app, which repeatedly access the same message in a short time window. ZCS 6: on large installs, increase the number entries in the message cache to 10000 (the maximum allowed): # NOTE: this is a ZCS 6 specific change! # can be set on global config or server: 10000 entries in cache zmprov ms `zmhostname` zimbraMessageCacheSize 10000 ZCS 5, on large installs, set this cache size to at least 100MB: # NOTE: this is a ZCS 5 specific change! # can be set on global config or server: 104857600 == 100MB zmprov ms `zmhostname` zimbraMessageCacheSize 104857600 The message cache hit rate is tracked in /opt/zimbra/zmstat/mailboxd.csv in the mbox_msg_cache column. Cache hit rate stats are charted by zmstat-chart in the "Blob Cache Hit Rate" graph. TroubleshootingIf HTTP, IMAP or POP3 clients get connection refused errors from the server, and if the server appears to be running OK, initiate a thread dump on the Java mailbox server process. You can do this by using either ~zimbra/libexec/zmtomcatmgr threaddump command pre-5.0, or ~zimbra/libexec/zmmailboxdmgr threaddump command in 5.0 or later. The thread dump should show what all the thread pool threads are doing. If they are just idling (usually blocked on a monitor in the thread pool, waiting), this is not a thread pool problem. If all threads are busy doing something else, then either (a) you have hit a bug where the process has wedged itself, or (b) the threads are all busy doing disk IO. Report (a) to us, and for (b) consider better disks or adding RAM for your load or another server. Zimbra OpenLDAP ServerCurrently only one master LDAP server can be set up; this server is authoritative for user information, server configuration, etc. Replica LDAP servers can be defined to improve performance and to reduce the load on the master server. All updates are made to the master server and these updates are copied to the replica servers. For tuning OpenLDAP with ZCS 6.0 and later, please see OpenLDAP_Performance_Tuning_6.0 For tuning OpenLDAP with ZCS 5.0 and previous, please see OpenLDAP_Performance_Tuning_5.0 Zimbra MTAPlease review the Postfix tuning guide, but double check that your system's values are not already higher before implementing recommendations from that guide. As of this writing, recent Linux kernel defaults are much higher than the values for file-max (16384) and threads-max (2048) recommended in the Postfix tuning guide. (Section needs more detail.)
bayes_auto_learn 0
# loadplugin Mail::SpamAssassin::Plugin::AWL
lock_method flock
|