Rolling Upgrades Overview

  1. Zimbra Tech Center
  2. Community Sandbox
  3. Rolling Upgrades Overview

Rolling upgrades Overview

   KB 20835        Last updated on 2016-06-7  




0.00
(0 votes)

Important: This is not official documentation. The information on this page has not been confirmed by Zimbra.

Upgrading ZCS 7.x to ZCS 8.x

RollUpgr2.png


Rolling Upgrade Process

  • Ordered upgrade to limit downtime:
    • LDAP Master (firewall all servers from connecting Master during upgrade)
    • LDAP Replicas (one at a time)
    • MTAs (one at a time)
    • Proxies (one at a time)
    • Mailstores (one at a time)
  • Can be extended over a period of time (if upgrading from ZCS7)
    • For example to allow one mailstore to be on ZCS8 for a beta period
  • Depending on scenario, there may be variations:
    • Standard in-place upgrades
    • Low-downtime upgrade (zmmboxmove)
    • OS Upgrade (e.g., RHEL5 to RHEL6)
    • ZCS to ZCS Migration
    • Physical to Virtual Migration
    • Introduction of LDAP Multi-Master

zmmboxmove and zmdedupe

  • zmmboxmove in 7.1.3, http://bugzilla.zimbra.com/show_bug.cgi?id=58159 - decrease downtime during moves and allow moves between ZCS7 and ZCS8
    • Destination server manages the overall move process, sending various commands to the source server along the way
    • rsync command is used to pull blobs and Lucene index files (repeated)
    • Then account is locked, mailbox is flushed and unloaded on the source server to quiesce it, one more rsync is done and MySQL data is exported/imported.
    • Runs in the background by default, with --sync option to run synchronously
  • One ramification of these moves is loss of single-instance blob store (per store filesystem) - the fix is zmdedupe: 
 zmdedupe - makes use of above SOAP API
 zmdedupe start - schedules a dedupe and returns immediately
 zmdedupe status - returns the current status of the dedupe request 
    (status, number of links created, approximate size saved)
 zmdedupe stop - Stops the currently running dedupe request

Zmmboxmove – Known and Fixed Issues

Zimlets by COS

  • Due to the new Serenity theme and related functions, some Zimlets may not function as expected
  • Each Zimlet must be specifically tested with ZCS8
  • Upon upgrading, you may want to disable all non-distributed Zimlets
  • Some Zimlets will be automatically disabled by the ZCS8 upgrade process, such as Social Zimlet
  • In the case of a Rolling Upgrade, you may want to have some Zimlets enabled on ZCS7 mailstores, and disabled on ZCS8 mailstores - in order to do this, the recommended approach is to setup one or more COSs for those users who are already on ZCS8
  • zmsetup.pl - affects even LDAP-only upgrades:
 # remove deprecated zimlets on upgrades
 if (!$newinstall) {
   progress("Checking for deprecated zimlets...");
   progress((zimletCleanup()) ? "failed.\n" : "done.\n");
 }

GalSync Considerations

  • Particularly useful during Rolling Upgrades to ZCS8, as new galsync accounts can be created on additional mailstores is on ZCS8
    • ZCS8 allows one galsync account per mailstore per domain
    • Reduces bottlenecks or performance issues related to heavy galsync usage on only one mailstore
  • Should be used in combination with ZCO 7.2.3 or 8.0.3 or later
  • Support for multiple GAL sync accounts:

Memory Allocation to ZCS8 Mailstores

  • Modifications may include:
  • Increase PermGen space https://bugzilla.zimbra.com/show_bug.cgi?id=78661
    • -XX:PermSize=192m -XX:MaxPermSize=350m
  • Increasing mailboxd_java_heap_size to 10240 or larger (10+GB)
  • Increasing mailboxd_java_heap_new_size_percent to 40% to reduce minor GCs and retain larger active allocation
  • Reduce Full GCs by increasing CMS collections more regularly -XX:CMSInitiatingOccupancyFraction=50 (default is 92%)

RHEL5 to RHEL6 Upgrade

Red Hat does not support in-place upgrades between any major versions of Red Hat Enterprise Linux.

  • These methods should be considered:
    • In-place OS Upgrade
    • rsync of data to RHEL6 mailstore
    • zmrestore/zmrestoreoffline of data to RHEL6 mailstore
    • Mounting of data partitions on RHEL6 box, then upgrade
    • Rolling upgrade with zmmboxmove of users to RHEL6 (or other OS) mailstore
  • Note: CentOS will be officially supported as of 8.0.4 https://bugzilla.zimbra.com/show_bug.cgi?id=23487

Rolling Upgrade Example – RHEL5 to RHEL6

  1. Create a RHEL6 guest template
  2. Install ZCS 7.2 LDAP replica on RHEL 6 VM
  3. Promote RHEL6 LDAP 7.2 replica to master (requires modifying all other nodes to point to the new master)
  4. Move LDAP Replicas to new RHEL6 VMs
  5. Move MTAs to new RHEL6 VMs
  6. Do a Rolling Upgrade on LDAP nodes to ZCS8 [1]
  7. Optionally: Promote one or more LDAP replicas to LDAP Multi-Master https://wiki.zimbra.com/wiki/LDAP_Multi_Master_Replication#Promoting_an_existing_replica_to_be_a_multi-master
  8. Do a Rolling Upgrade on MTA nodes to ZCS8
  9. Do a Rolling Upgrade on Proxy nodes to ZCS8
  10. Create X new ZCS 8 mailstores on RHEL6
  11. zmmboxmove users from ZCS7.2 mailstores to ZCS8
  12. Decommission RHEL5 and ZCS7.2 systems

Rolling Upgrades – Fixed & Known Issues

Known Issues – Provisioning

The problem is the way in which "ra" functions in ZCS7, as it is fundamentally different than the way it functions in ZCS8. In ZCS7, "ra" makes a *copy* of the account, which duplicates all attributes, including attributes required to be unique, such as zimbraID. This operation will be blocked by the ZCS8 LDAP server. In ZCS8, the entry is simply updated with the new information (modified), rather than (copy, delete) as is done by ZCS7. The solution is to issue "zmprov ra" commands from ZCS8 servers only.

  • A Domain Admin Distribution List in a Rolling Upgrade mode from ZCS7 to ZCS8 must have an additional attribute added for zimbraMailHost. See https://bugzilla.zimbra.com/show_bug.cgi?id=91029
    • The workaround is to add a zimbraMailHost to the DL. It can be any of the ZCS8 mailstores, for example:
zmprov mdl listname@example.com zimbraMailHost mail5.example.com
  • Rolling Upgrade ZCS 7 > 8 Broken Admin Console Functionality - https://bugzilla.zimbra.com/show_bug.cgi?id=86709
    • The grant domainAdminAccountRights was not necessary in ZCS7 to have access these features, but apparently that changed in Z8. A number of the domain admin accounts did not have this right, and those were the ones who had issues with account edit and view mail within the Z8 console.
    • Workaround is to go through all domain admin groups and accounts and updated this right for them. This seems to have restored full function to the accounts.

Known Issues – Web Login Page

  • For a consistent web user experience at login time, the login page request for all the users (irrespective of zcs version of the mailstore they are hosted on) will be routed by the proxy to only ZCS 8.5+ mailstores so as to display a consistent newer login page (take a look at zimbraReverseProxyUpstreamLoginServers)

Known Issues – LDAP Fixes for 8.0.4

http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=shortlog;h=refs/heads/OPENLDAP_REL_ENG_2_4

Miscellaneous Solutions and Troubleshooting

This section contains solutions to some issues you might experience during your upgrade process.

1. If authenticating users in a Rolling Upgrade Mixed-version platform, the login page for all users should be set to the same version for a consistent user experience.

Solution:

For a consistent web user experience, may want to consider routing all users to either only ZCS7 web login pages, or only ZCS8 Either should work, it is really a Service/Admin preference Use zimbraReverseProxyLookupTarget or zimbraReverseProxyAvailableLookupTargets to set the zmprov garpu and garpb lists appropriately 

  <attr id="504" name="zimbraReverseProxyLookupTarget" type="boolean" cardinality="single"
  optionalIn="globalConfig,server" flags="serverInherited" requiresRestart="nginxproxy">
    <globalConfigValue>FALSE</globalConfigValue>
    <desc>whether this server is a reverse proxy lookup target</desc> 
  </attr>

  <attr id="1379" name="zimbraReverseProxyAvailableLookupTargets" type="string" cardinality="multi"
  optionalIn="server,globalConfig" flags="serverInherited" since="8.0.0">
    <desc>The servers to be included in the proxy lookup hanlders list. Proxy will only use the
  servers specified here to do the lookup. Leaving empty means using all the servers whose
  zimbraReverseProxyLookupTarget is TRUE.</desc> </attr>

2. If users are sharing calendars/mailboxes, they should be on ZCS8 or ZCS7, but not mixed. The normal method of addressing this is to migrate groups of users at once - for example, if IT is sharing a lot of mailboxes between them, then zmmboxmove the IT users at one time, so that those shares all continue to function.

3. All provisioning and account administration should be performed on the new version only. For example, if running a mixed-version between ZCS8 and ZCS7, then a ZCS8 mailstore should be used for all administration UI functions.

The problem is the way in which "ra" functions in ZCS7, as it is fundamentally different than the way it functions in ZCS8.

  • In ZCS7, "ra" makes a *copy* of the account, which duplicates all attributes, including attributes required to be unique, such as zimbraID. This operation will be blocked by the ZCS8 LDAP server.
  • In ZCS8, the entry is simply updated with the new information (modified), rather than (copy, delete) as is done by ZCS7.

The solution is to issue "zmprov ra" commands from ZCS8 servers only.

Upgrading ZCS5/6 to ZCS7

Important: The following is not official documentation. The data included here has been gathered from the field and provided only as a basis for discussion. These concepts must be tested prior to implementation, and results may vary in different environments.

No zmmboxmove

  • Since zmmboxmove was added in 7.1.3 and designed only for migrations from ZCS7 to ZCS8, it cannot be used for older versions:
  • The official QA’d and most well-known/reliable method of upgrading from ZCS5/6 mailstores to ZCS7 continues to be the standard in-place upgrade
  • However, some customers have reported using migration methods and have reported general success
    • None of these methods are officially supported
    • They may not be perfect in migrating all data, but should be sufficient if needed
    • Some areas of likely problem may include Shared Objects (mailboxes, calendar, Briefcase, etc.) and tags
    • Please be sure to test

zmbackup/zmrestore

  • zmbackup/zmrestore individual users from one system to the other
  • Advantages:
    • Generally should work
    • Although there may be differences in versions that cause problems in some metadata areas, such as tags
  • Disadvantages:
    • Slow
    • Cumbersome (there may not be a simple method to move these backups to another server)
    • Need to test if any former hostname entries remain
    • Does not move LDAP

Tar Export/Import

  • Tar Export/Import
  • Advantages:
    • Data-independent method for importing into any ZCS version
  • Disadvantages:
    • Includes no User Tag table
    • May break Shared Objects (for sure, if moving to different ZCS platform with separate LDAP Master and new zimbraIDs)
    • Known Issues:

Migrate Data by Protocol-based Methods

  • Migrate data by protocol-based methods (e.g., imapsync, iCal export/import, Contacts export/import)
  • Advantages:
    • Data reliability
    • Accuracy
  • Disadvantages:
    • Time consuming
    • Labor intensive
  • Can also use a combination of methods:
    • imapsync email – largest data set, presync asynchronously (reduced user downtime, in background)
    • tar export/import other data – smaller data set, performed during user downtime


Verified Against: ZCS Date Created: 2/20/2014
Article ID: https://wiki.zimbra.com/index.php?title=Rolling_Upgrades_Overview Date Modified: 2016-06-07



Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Other help Resources

User Help Page »
Official Forums »
Zimbra Documentation Page »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »


Retrieved from "http://wiki.zimbra.com/index.php?title=Rolling_Upgrades_Overview&oldid=61895"
Jump to: navigation, search