View source for OpenLDAP Performance Tuning

{{BC|Certified}}
__FORCETOC__
<div class="col-md-12 ibox-content">
=OpenLDAP Performance Tuning=
{{KB|{{ZC}}|{{ZCS 7.0}}|{{ZCS 6.0}}|}}
{{WIP}}
=Zimbra OpenLDAP Server=
Currently only one master LDAP server can be set up; this server is authoritative for user information, server configuration, etc. Replica LDAP servers can be defined to improve performance and to reduce the load on the master server. All updates are made to the master server and these updates are copied to the replica servers.

'''For ZCS 8.0 and later, use [http://wiki.zimbra.com/wiki/OpenLDAP_Performance_Tuning_8.0 OpenLDAP performance tuning for ZCS 8] instead.'''

Upcoming on radar: [http://bugzilla.zimbra.com/show_bug.cgi?id=27872 RFE 27872 - Support multi-master LDAP deployment]

You may also wish to read over the entire set of keys available on the [http://wiki.zimbra.com/wiki/OpenLDAP_Tuning_Keys OpenLDAP Tuning Keys Wiki].

== OpenLDAP Server Tuning ==
For best performance, the following settings should be modified.  These settings apply to both the master LDAP server and the replica servers.

* Set the '''threads''' count to an appropriate level. The ZCS default is 8, which is fine for 1 or 2 CPU servers.  The general rule of thumb is 4 threads for every real core.

 zmlocalconfig -e ldap_common_threads=8

* Set the '''toolthreads''' count to an appropriate level.  The ZCS default is 1.  It should be set to the number of real cores you have available.  The purpose of the toolthreads setting is to decrease the amount of time it takes to slapadd a database.  '''WARNING:  If you are on ZCS 8, this can ONLY be set to 2 at this time.  Please read the [http://wiki.zimbra.com/wiki/OpenLDAP_Performance_Tuning_8.0 ZCS8 Performance Tuning Wiki].'''

 zmlocalconfig -e ldap_common_toolthreads=4

* Change the entry '''cachesize''' for the primary database. The number set should be the number of configured active accounts and the number of configured active domains. The default is 10000.
 zmlocalconfig -e ldap_db_cachesize=50000

* Set '''idlcachesize''' for the primary database. The number set should optimally be three times the '''cachesize''' setting.

 zmlocalconfig -e ldap_db_idlcachesize=50000

* Change the '''dn entry cachseize''' for the primary database.  It is highly recommended to leave it at the default value of zero, which means an unlimited DN cache.  If it is set to a non-zero value, it should be no less than two times the entry cache.

 zmlocalconfig -e ldap_db_dncachesize=0

If you have more than 100 domains, we suggest adjusting the following local config setting:

* '''ldap_cache_domain_maxsize'''. This sets the cache of the number of domains in the server. The default is 100. If more than 100 domains are configured, you should adjust this to the lower of the number of domains you have configured and 30,000. For example, with 45,000 domains, set as this to 30000.

 # Apply this to all mailbox servers!
 zmlocalconfig -e ldap_cache_domain_maxsize=30000

==Configuring the Berkeley DB subsystem to increase LDAP server performance==

Berkeley DB is the underlying high-performance transactional database used to store the LDAP data.  Proper configuration of this database is essential to maintaining a performant LDAP service.  There are several parameters involved in tuning the underlying Berkeley DB database.  

===Berkeley DB and Shared memory keys===
Starting with ZCS 6.0 and later, it is possible to instruct slapd to allocate a shared memory key for usage with the Berkeley DB cache backing files.  Prior to ZCS 6.0, a disk-based backing cache was always used.  Using a shared memory key rather than using a disk-based cache shows a 3x improvement over the disk-based method IF and ONLY IF the size of your database is over 8GB, and results in the same amount of memory usage as the disk-based backing cache.  To configure the LDAP server to use a shared memory key takes multiple steps.  You will need to use the value of your BDB cachesize (See Berkeley DB DB_CONFIG section below) to set the correct values.

* Get the current configured pagesize for the system

 getconf PAGE_SIZE

* Divide the amount of RAM you will need for your shared memory key by the value of PAGE_SIZE, and assign this to the kernel.shmall sysctl value.  For example, if I have a 14GB BDB cachesize and the PAGE_SIZE result is 4096, the setting would be 3801088.  Which is 14.5*1024*1024*1024/4096.  The extra .5 GB is for other OS processes that will also need to use shared memory.
 
 # For Linux add to /etc/sysctl.conf:
 kernel.shmall=3801088
 # Then update the currently used value:
 /sbin/sysctl -e kernel.shmall=3801088

 # For OSX 10.5 add to/etc/sysctl.conf:
 kern.sysv.shmall=3801088
 # Then update the currently used value:
 /usr/sbin/sysctl -w kern.sysv.shmall=3801088

 # For OSX 10.4, modify the current listed value for kernel.sysv.shmall in the /etc/rc file
 # if necessary.  The line will look similar to:
 #sysctl -w kern.sysv.shmmax=4194304 kern.sysv.shmmin=1 kern.sysv.shmmni=32 kern.sysv.shmseg=8 kern.sysv.shmall=3801088

* Configure the maximum size allowed for an individual shared memory key.  For example, if I have a 14GB BDB cachesize, the value would be 15032385536 (14*1024*1024*1024).

 # For Linux add to /etc/sysctl.conf:
 kernel.shmmax=15032385536
 # Then update the currently used value:
 /sbin/sysctl -e kernel.shmmax=15032385536

 # For OSX 10.5 add to /etc/sysctl.conf:
 kernel.sysv.shmmax=15032385536
 # Then update the currently used value:
 /usr/sbin/sysctl -w kern.sysv.shmmax=15032385536

 # For OSX 10.4, modify the current listed value for kernel.sysv.shmmax in the /etc/rc file
 # if necessary.  The line will look similar to:
 #sysctl -w kern.sysv.shmmax=15032385536 kern.sysv.shmmin=1 kern.sysv.shmmni=32 kern.sysv.shmseg=8 kern.sysv.shmall=3801088
 # Also on OSX 10.4, you will now want to run your fully modified sysctl line:
 /usr/sbin/sysctl -w kern.sysv.shmmax=15032385536 kern.sysv.shmmin=1 kern.sysv.shmmni=32 kern.sysv.shmseg=8 kern.sysv.shmall=3801088

* For all OSes, as the zimbra user, you will now need to tell slapd to use a shared memory key by setting it in localconfig.  slapd and zmmtaconfig must both be running.  On a master, it is also possible to tell the accesslog database to use a shared memory key.  If a shared memory key is going to be used for both the accesslog and main db, then the shmmax value needs to be the larger of the two BDB cachesizes (this will most likely always be the main db value), and the shmall value needs to be the total of the two databases BDB cachesizes plus .5.  So, if I had an 8 GB main DB and a 2GB accesslog DB cachesize, I would want to use 10.5 as the value for determining shmall size.  Shared memory key values must be unique inside slapd, and they can be set to any number.  For example, to set the main DB to use a shared memory key with the value 5:

 zmlocalconfig -e ldap_db_shmkey=5

* If it was then desired to set the accesslog DB to use a shared memory key, it would need a different unique value:

 zmlocalconfig -e ldap_accesslog_shmkey=7

* Wait several minutes (it can take up to at least 5 minutes for zmmtaconfig to react to the change) and verify that the modification has occurred in the LDAP configuration database.  The value for olcDbShmKey should match what you set ldap_db_shmkey to be.

 # On a replica or stand-alone server
 ldapsearch -LLL -s base-x -H ldapi:/// -D cn=config -w `zmlocalconfig -s -m nokey ldap_root_password` -b olcDatabase={2}hdb,cn=config olcDbShmKey
 
 # On a master with replication enabled
 ldapsearch -LLL -s base-x -H ldapi:/// -D cn=config -w `zmlocalconfig -s -m nokey ldap_root_password` -b olcDatabase={3}hdb,cn=config olcDbShmKey

* To check the SHM key value for the accesslog database on a master:

 # On a master with replication enabled, to check the accesslog DB setting:
 ldapsearch -LLL -s base-x -H ldapi:/// -D cn=config -w `zmlocalconfig -s -m nokey ldap_root_password` -b olcDatabase={2}hdb,cn=config olcDbShmKey

* Once you have verified the settings have been applied, restart slapd as the zimbra user

  ldap stop
  ldap start

===Berkeley DB DB_CONFIG tuning===
Most of the tuning for BDB involves editing the '''DB_CONFIG''' file.  Modifications to the '''DB_CONFIG''' file require a restart of the LDAP server before they are picked up, and should be made to both master and replica servers.  The proper way to do this for Zimbra is documented at [[Managing_Customized_BDB_configurations]]

You can increase LDAP server performance by adjusting the BDB backend cache size to be at or near the size of your data set. This is subject to the limit of 4 GB for 32 bit and 10 TB for 64 bit, and the amount of RAM you have.  The size of the data set is the sum of the Berkeley DataBase (BDB) files in '''/opt/zimbra/data/ldap/hdb/db'''.  To increase the cache size, add (or replace) the following line to the '''DB_CONFIG''' file in '''/opt/zimbra/data/ldap/hdb/db'''.  The following would set the database in-memory cachesize to 500MB:

 set_cachesize 0 524288000 1

The format for the set_cachesize command is &lt;gigabytes&gt; &lt;bytes&gt; &lt;segments&gt;

It is possible to check that the cache setting has taken effect by using the '''/opt/zimbra/bdb/bin/db_stat -m -h /opt/zimbra/data/ldap/hdb/db&#x01c0;head -n 16''' command can be used to see the current cache setting, as well as to find other important information. Example output:

 500MB Total cache size
 1 Number of caches
 1 Maximum number of caches
 500MB Pool individual cache size
 0 Maximum memory-mapped file size
 0 Maximum open file descriptors
 0 Maximum sequential buffer writes
 0 Sleep after writing maximum sequential buffers
 0 Requested pages mapped into the process' address space
 3437M Requested pages found in the cache (100%)
 526125 Requested pages not found in the cache
 47822 Pages created in the cache
 526125 Pages read into the cache
 27M Pages written from the cache to the backing file
 0 Clean pages forced from the cache
 0 Dirty pages forced from the cache
 
The above output shows that there is a 500MB total cache in a single segment all of which is allocated to the cache pool.  The other important data to evaluate are the '''Requested pages found in the cache''', the '''Clean pages forced from the cache''' and the '''Dirty pages forced from the cache'''.  For optimal performance, the '''Requested pages found in the cache''' should be above '''95%''', and the '''pages forced from the cache''' should be '''0'''.

As part of the transaction interface, BDB uses a number of locks, lockers, and lock objects.  The default value for each of these parameters is 1000.  How many of each are being used depends on the number of entries and indices in the BDB database.  The '''/opt/zimbra/bdb/bin/db_stat -c -h /opt/zimbra/data/ldap/hdb/db&#x01c0;head -n 16''' command can be used to determine current usage:

 5634 Last allocated locker ID
 0x7fffffff Current maximum unused locker ID
 9 Number of lock modes
 3000 Maximum number of locks possible
 1500 Maximum number of lockers possible
 1500 Maximum number of lock objects possible
 93 Number of lock object partitions
 93 Number of current locks
 1921 Maximum number of locks at any one time
 6 Maximum number of locks in any one bucket
 0 Maximum number of locks stolen by for an empty partition
 0 Maximum number of locks stolen for any one partition
 483 Number of current lockers
 485 Maximum number of lockers at any one time
 93 Number of current lock objects
 1011 Maximum number of lock objects at any one time

The above output shows that there are a maximum of '''3000 locks''', '''1500 lockers''', and '''1500 lock objects''' available to the BDB database located in '''/opt/zimbra/data/ldap/hdb/db'''.  Of those settings, there are currently '''93 locks''' in use, '''483 lockers''' in use, and '''93 lock objects''' in use.  Over the course of the lifetime of the database, the highest recorded values have been '''1921 locks''' used, '''485 lockers''' used, and '''1011 lock objects''' used.  As long as usage is within '''85%''' of the configured value(s), it should not be necessary to modify the settings.

The following entries in '''DB_CONFIG''' would increase the number of locks to 3000, the lock objects to 1500, and the lockers to 1500:

 set_lk_max_locks 3000
 set_lk_max_objects 1500
 set_lk_max_lockers 1500

{{Article_Footer|ZCS 7.0, ZCS 6.0|9/3/2009}}

[[Category:LDAP]]
[[Category:Performance and Tuning]]
[[Category:ZCS 7.0]]
[[Category:ZCS 6.0]]