LDAP Multi Master Replication

Zimbra Multi-Master Replication

   KB 15143        Last updated on 2016-08-17  




0.00
(0 votes)

The ability to have more than a single functioning master has been added with ZCS 8.0 and later. This document details the steps required for setting up a Multi-Master configuration, as well as how to perform administrative tasks such as promote an existing replica to be part of the multi-master pool. Multi-Master Replication will be shortened to MMR for the rest of this document.

Two example hostnames will be used throughout this document.

master1.example.com (Primary master) master2.example.com (Secondary master that is being added)

Data to have before starting

  • Zimbra Admin LDAP Password
  • LDAP replication password
  • NGINX LDAP password
  • Amavis LDAP password
  • Postfix LDAP password
  • BES LDAP password

Enabling Multi-Master replication on an existing Single node master

This assumes you already have an installed single-master ZCS LDAP server that is not already enabled for MMR. Enabling MMR is a manual one-time step. You will need to know the hostname of the secondary master you plan on adding into the multi-master pool before you can enable MMR on your standalone master.

On the single-node master as the zimbra user:

$ ./libexec/zmldapenable-mmr -s 1 -m ldap://master2.example.com:389/
$ zmlocalconfig -e ldap_master_url="ldap://master1.example.com:389 ldap://master2.example.com:389"
$ zmlocalconfig -e ldap_url="ldap://master1.example.com:389 ldap://master2.example.com:389"
$ zmcontrol restart

Executing these commands will do the following:

  • Set this master's Server ID (sid) to 1. See the section on server IDs.
  • Tell the master that it will be in a pool with a secondary master named master2.example.com that is listening to LDAP on port 389
  • It will use the default Replication ID (rid) of 100 for the secondary master. See the section on replication IDs.
  • Writes initiated from this server will go to ldap://master1.example.com by default. If it is down, they will move to ldap://master2.example.com:389
  • Reads initiated from this server will go to ldap://master1.example.com by default. If it is down, they will move to ldap://master2.example.com:389
  • Restarts the services that cache the values for ldap_master_url and ldap_url so they are utilized correctly.

After you bring up the secondary master, make sure that the localconfig keys ldap_master_url and ldap_url are up to date on all nodes.

Installing a Secondary master

To install a brand new secondary master, install ZCS as you normally would for a multi-node installation. zcs-ldap must be one of the packages chosen for installation.

  • On the installation menu, choose "1" for common configuration
    • Change the ldap master hostname to be that of your primary master (Ex. master1.example.com)
    • Change the admin password to be that of the Zimbra admin password on the primary master
  • On the installation menu, choose "2" for LDAP configuration
    • Choose "4" to change the replication type to mmr instead of replica
    • The Server ID for this secondary master will default to 2. Leave it unchanged if this is the second master. See more in server IDs.
    • Change "7" to match the replica password from the primary master
    • Change "8" to match the postfix password from the primary master
    • Change "9" to match the amavis password from the primary master
    • Change "10" to match the nginx password from the primary master
  • After installation finishes, update the ldap_master_url to contain both masters, preferring this master.
$ zmlocalconfig -e ldap_master_url="ldap://master2.example.com:389 ldap://master1.example.com:389"
  • Verify that both servers show up in ldap_url: zmlocalconfig -s ldap_url

Make sure that the localconfig keys ldap_master_url and ldap_url are up to date on all nodes.

BUG NOTE for 8.6 and prior

Due to bug 101054 it is strongly advised that after installation completes on new MMR nodes, that its primary database reloaded from the primary master via slapcat/slapadd: LDAP_data_import_export. The accesslog DB on the secondary node should be erased (rm -f /opt/zimbra/data/ldap/accesslog/db/*) while slapd is offline.

For 8.6, there are updated OpenLDAP builds available upon request that include the fix for this issue. It is recommended to deploy the updated builds on any 8.6 LDAP deployment.

Note for when the accesslog DB is cleared

After slapd is back online, it is advised to do a no-op (no real update) change on the master where the accesslog database was reset:

ldapmodify -x -H ldapi:/// -D cn=config -w `zmlocalconfig -s -m nokey ldap_root_password`
dn: cn=admins,cn=zimbra
changetype: modify
replace: description
description: admin accounts

Then press Control-D to submit the change. This will write a single entry to the accesslog, allowing replication between the two nodes to resume.

Promoting an existing replica to be a multi-master

In an existing ZCS setup where there is already a single master and multiple replicas, it is very simple to promote an existing replica to become a secondary master. First make sure you have enabled MMR on the primary master as noted above. Assuming the hostname for this replica is master2.example.com, then:

  • Obtain the existing values for the following localconfig values on the current primary master: ldap_amavis_password, ldap_bes_searcher_password, ldap_nginx_password, ldap_postfix_password, ldap_replication_password
  • Update these passwords in localconfig on the replica to be promoted so that they match the values on the current master
$ /opt/zimbra/libexec/zmldappromote-replica-mmr -s 2
$ zmlocalconfig -e ldap_master_url="ldap://master2.example.com:389 ldap://master1.example.com:389"
$ zmcontrol restart

This will update the replica to be MMR enabled with a Server ID of 2. See server ID section. It will automatically be configured to be a paired master with the master it was previously replicating from.

Make sure that the localconfig keys ldap_master_url and ldap_url are up to date on all nodes.

MMR utilities

Given the critical nature of the Server ID and Replication ID values in the multi-master configuration, Zimbra ships a utility to easily query this information from the local MMR node. When executed, it will print out the Server ID for this master, plus all multi-master servers it is configured to replicate against and the Replication ID values for those masters for this server.

8.0.6 and previous:

$ /opt/zimbra/libexec/zmldapquery-mmr

8.0.7 and later:

$ /opt/zimbra/libexec/zmldapmmrtool -q

In addition, the zmldapmmrtool can be used to modify the RID, TLS, or provider settings of the replication agreements.

Server ID information (sid)

Server ID's are unique identifiers used in a multi-master cluster. Each master in the cluster must have a different Server ID. If you have two masters with the same Server ID, MMR will break and chaos will ensue.

Replication ID information (rid)

Replication ID's are unique identifiers that are unique internally to a server. A given master may use the same Replication IDs as another master, but internal to a given master's database, those replication IDs must be unique. This only matters if you plan on having more than two masters.

Configuring more than two masters

Theoretically, you can have as many masters as desired in a master pool. Zimbra does not advise having more than 4 masters at this time.

To add additional masters to an existing MMR pool, the following steps should be taken:

On all existing masters in the MMR pool:

  • Run /opt/zimbra/libexec/zmldapquery-mmr (8.0.6 and previous or /opt/zimbra/libexec/zmldapmmrtool -q (8.0.7 and later) to find out the RID values for the existing replication agreements
  • Run /opt/zimbra/libexec/zmldapenable-mmr -r <new rid> -m ldap://<new master>:389/ to add a replication agreement for the new MMR master

Example:

 zimbra@ldap1.example.com:~$ ./libexec/zmldapquery-mmr
 Server information
 ServerID: 1
 Master replication information
 Master replica 1
 rid: 100 URI: ldap://ldap2.example.com:389/
 zimbra@ldap1.example.com:~$ ./libexec/zmldapenable-mmr -r 101 -m ldap://ldap3.example.com:389/
 zimbra@ldap2.example.com:~$ ./libexec/zmldapquery-mmr
 Server information
 ServerID: 2
 Master replication information
 Master replica 1
 rid: 100 URI: ldap://ldap1.example.com:389/
 zimbra@ldap2.example.com:~$ ./libexec/zmldapenable-mmr -r 101 -m ldap://ldap3.example.com:389/

Now start ZCS installation on the new server you wish to add to the pool, as described above in the section Installing a Secondary master. Be sure that when the Ldap replication type is changed to mmr that the Server ID for this new server is set to something unique.

Example:

 Ldap configuration
   3) Ldap replication type:                   mmr
   4) Ldap Server ID:                          3

After installation of the additional MMR server is complete:

  • Add replication agreements to the new master with the the masters it doesn't yet have an agreement with.

Example: Assuming that ldap3.example.com was given ldap1.example.com as its master during installation time, we must add an agreement with ldap2.example.com

zimbra@ldap3.example.com:~$ ./libexec/zmldapenable-mmr -r 101 -m ldap://ldap2.example.com:389/
  • Remember to update the localconfig keys ldap_master_url and ldap_url on all servers.

Example:

zimbra@ldap1.example.com:~$ zmlocalconfig -e ldap_master_url="ldap://ldap1.example.com:389 ldap://ldap2.example.com:389 ldap://ldap3.example.com:389"
zimbra@ldap1.example.com:~$ zmlocalconfig -e ldap_url="ldap://ldap1.example.com:389 ldap://ldap2.example.com:389 ldap://ldap3.example.com:389"
zimbra@ldap2.example.com:~$ zmlocalconfig -e ldap_master_url="ldap://ldap2.example.com:389 ldap://ldap3.example.com:389 ldap://ldap1.example.com:389"
zimbra@ldap2.example.com:~$ zmlocalconfig -e ldap_url="ldap://ldap2.example.com:389 ldap://ldap3.example.com:389 ldap://ldap1.example.com:389"
zimbra@ldap3.example.com:~$ zmlocalconfig -e ldap_master_url="ldap://ldap3.example.com:389 ldap://ldap1.example.com:389 ldap://ldap2.example.com:389"
zimbra@ldap3.example.com:~$ zmlocalconfig -e ldap_url="ldap://ldap3.example.com:389 ldap://ldap1.example.com:389 ldap://ldap2.example.com:389"
  • Restart services on all of the servers.

Example:

zimbra@ldap1.example.com:~$ zmcontrol restart
zimbra@ldap2.example.com:~$ zmcontrol restart
zimbra@ldap3.example.com:~$ zmcontrol restart

Make sure that the localconfig keys ldap_master_url and ldap_url are up to date on all nodes.

Multi-Node environments

In a multi-node environmentit is required to update the localconfig keys for ldap_master_url and ldap_url appropriately as well, so that LDAP failover works correctly on all nodes. Once you have updated the localconfig keys on those servers, the services will need to be restarted via zmcontrol restart

Deleting MMR nodes

Attention.png WARNING: Configuring MMR is a one-way trip! Once you have configured MMR, you must not remove all nodes from the MMR configuration! If you're removing nodes, you must retain at least one replication agreement on your MMR nodes.

To delete an MMR node, the following steps should be taken. This is only possible in ZCS 8.0.7 and later.:

  1. Update ldap_master_url and ldap_url on every node, removing the ldap MMR node that will be shut down.
  2. Wait 5-10 minutes to ensure this modification is in place.
  3. Monitor /var/log/zimbra.log on the MMR node that will be shut down and confirm it is no longer receiving modification traffic
  4. run ldap stop on the MMR node that is being shut down.
  5. Log into the remaining MMR nodes and do the following:
    1. /opt/zimbra/libexec/zmldapmmrtool -q
    2. Find the matching RID for the MMR node you shut down
    3. /opt/zimbra/libexec/zmldapmmrtool -d -o RID

Example of using zmldapmmrtool: Given 3 MMR servers ldap1.example.com, ldap2.example.com, ldap3.example.com and ldap3.example.com is being shut down.

zimbra@ldap1:/opt/zimbra$ /opt/zimbra/libexec/zmldapmmrtool -q
Master replication information
Master replica 1
rid: 100 URI: ldap://ldap2.example.com:389/ TLS: critical
Master replica 2
rid: 101 URI: ldap://ldap3.example.com:389/ TLS: critical

So the RID being used ldap3.example.com is 101.  This agreement can be deleted with:

zimbra@ldap1:/opt/zimbra$ /opt/zimbra/libexec/zmldapmmrtool -d -o 101

Then confirm it worked with:

zimbra@ldap1:/opt/zimbra$ /opt/zimbra/libexec/zmldapmmrtool -q
Master replication information
Master replica 1
rid: 100 URI: ldap://ldap2.example.com:389/ TLS: critical
zimbra@ldap1:/opt/zimbra$

Repeat on the remaining node(s)
 

Fixed Bug 95847 in Zimbra Collaboration 8.7

In Zimbra Collaboration 8.7 is not possible delete the last remaining replication LDAP.

In a MMR system consisting of two nodes

zimbra@zqa-xx:~$ /opt/zimbra/libexec/zmldapmmrtool -q
Master replication agreement: 1
rid: 100 URI: ldap://zqa-yy.eng.zimbra.com:389/ TLS: critical
zimbra@zqa-xx:~$ /opt/zimbra/libexec/zmldapmmrtool -d -o 100

Error: It is illegal to delete the last remaining replication agreement.
You must define a new replication agreement prior to deleting the final one.

MMR and Replica considerations

Prior to ZCS 8.7, replicas can only pull updates from a single master. This means that in the event of an outage affecting that master, the replicas will remain out of date until it is put back into service. With ZCS 8.7 and later, it is possible to add additional replication agreements to a replica so that it can pull updates from additional masters. See bug 95200 for further details.


Verified Against: ZCS 8.0 Date Created: 1/23/2014
Article ID: https://wiki.zimbra.com/index.php?title=LDAP_Multi_Master_Replication Date Modified: 2016-08-17



Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »

Jump to: navigation, search