Rolling Upgrades Overview
Rolling upgrades Overview
Important: This is not official documentation. The information on this page has not been confirmed by Zimbra.
Upgrading ZCS 7.x to ZCS 8.x
Rolling Upgrade Process
- Ordered upgrade to limit downtime:
- LDAP Master (firewall all servers from connecting Master during upgrade)
- LDAP Replicas (one at a time)
- MTAs (one at a time)
- Proxies (one at a time)
- Mailstores (one at a time)
- Can be extended over a period of time (if upgrading from ZCS7)
- For example to allow one mailstore to be on ZCS8 for a beta period
- Depending on scenario, there may be variations:
- Standard in-place upgrades
- Low-downtime upgrade (zmmboxmove)
- OS Upgrade (e.g., RHEL5 to RHEL6)
- ZCS to ZCS Migration
- Physical to Virtual Migration
- Introduction of LDAP Multi-Master
zmmboxmove and zmdedupe
- zmmboxmove in 7.1.3, http://bugzilla.zimbra.com/show_bug.cgi?id=58159 - decrease downtime during moves and allow moves between ZCS7 and ZCS8
- Destination server manages the overall move process, sending various commands to the source server along the way
- rsync command is used to pull blobs and Lucene index files (repeated)
- Then account is locked, mailbox is flushed and unloaded on the source server to quiesce it, one more rsync is done and MySQL data is exported/imported.
- Runs in the background by default, with --sync option to run synchronously
- One ramification of these moves is loss of single-instance blob store (per store filesystem) - the fix is zmdedupe:
- https://bugzilla.zimbra.com/show_bug.cgi?id=77597
- Get the blob digests from volume_blobs table and dedupe the blobs having the same digest. Add new SOAP API DedupeBlobsRequest/DedubeBlobResponse. Add CLI
zmdedupe - makes use of above SOAP API zmdedupe start - schedules a dedupe and returns immediately zmdedupe status - returns the current status of the dedupe request (status, number of links created, approximate size saved) zmdedupe stop - Stops the currently running dedupe request
Zmmboxmove – Known and Fixed Issues
- Known Issues:
- zmproxypurge is not able to delete alias https://bugzilla.zimbra.com/show_bug.cgi?id=76512 - Fixed 7.2.4, 8.0.4
- memcached not reporting alias account when being accessed via Activesync https://bugzilla.zimbra.com/show_bug.cgi?id=79940 - Fixed 7.2.4, 8.0.4
- Reminder data vanishes on iPhone after moving users data from ZCS7 to ZCS8 box with zmmboxmove https://bugzilla.zimbra.com/show_bug.cgi?id=81553 - Fixed 8.0.4
- Fixed Issues:
- Tags are not restored after mailbox move https://bugzilla.zimbra.com/show_bug.cgi?id=78254 – Fixed 8.0.2
- zmmboxmove should account for volume differences https://bugzilla.zimbra.com/show_bug.cgi?id=67056 – Fixed 7.1.4, 8.0.0
- zmmboxmove should abort when it can't fetch blobs https://bugzilla.zimbra.com/show_bug.cgi?id=68040 - Fixed 7.2.0, 8.0.0
- zmmailboxmove prematurely unlocks mailbox if target server is restarted https://bugzilla.zimbra.com/show_bug.cgi?id=67831 - Fixed 7.2.0, 8.0.0
- Unable to stop a move when using zmmboxmove https://bugzilla.zimbra.com/show_bug.cgi?id=78074 - Fixed 7.2.2, 8.0.2
Zimlets by COS
- Due to the new Serenity theme and related functions, some Zimlets may not function as expected
- Each Zimlet must be specifically tested with ZCS8
- Upon upgrading, you may want to disable all non-distributed Zimlets
- Some Zimlets will be automatically disabled by the ZCS8 upgrade process, such as Social Zimlet
- In the case of a Rolling Upgrade, you may want to have some Zimlets enabled on ZCS7 mailstores, and disabled on ZCS8 mailstores - in order to do this, the recommended approach is to setup one or more COSs for those users who are already on ZCS8
- zmsetup.pl - affects even LDAP-only upgrades:
# remove deprecated zimlets on upgrades if (!$newinstall) { progress("Checking for deprecated zimlets..."); progress((zimletCleanup()) ? "failed.\n" : "done.\n"); }
- Additional References:
GalSync Considerations
- Particularly useful during Rolling Upgrades to ZCS8, as new galsync accounts can be created on additional mailstores is on ZCS8
- ZCS8 allows one galsync account per mailstore per domain
- Reduces bottlenecks or performance issues related to heavy galsync usage on only one mailstore
- Should be used in combination with ZCO 7.2.3 or 8.0.3 or later
- Support for multiple GAL sync accounts:
Memory Allocation to ZCS8 Mailstores
- Modifications may include:
- Increase PermGen space https://bugzilla.zimbra.com/show_bug.cgi?id=78661
- -XX:PermSize=192m -XX:MaxPermSize=350m
- Increasing mailboxd_java_heap_size to 10240 or larger (10+GB)
- Increasing mailboxd_java_heap_new_size_percent to 40% to reduce minor GCs and retain larger active allocation
- Reduce Full GCs by increasing CMS collections more regularly -XX:CMSInitiatingOccupancyFraction=50 (default is 92%)
RHEL5 to RHEL6 Upgrade
- Due to RHEL5 packaging issues, only RHEL6 is available https://bugzilla.zimbra.com/show_bug.cgi?id=75191
- gcc version is 4.1.2. We require gcc 4.2 or later.
- perl version is 5.8. We require perl 5.10 or later.
- Red Hat discourages in-place upgrades of OS from RHEL5 to RHEL6 https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Installation_Guide/ch-upgrade-x86.html
Red Hat does not support in-place upgrades between any major versions of Red Hat Enterprise Linux.
- These methods should be considered:
- In-place OS Upgrade
- rsync of data to RHEL6 mailstore
- zmrestore/zmrestoreoffline of data to RHEL6 mailstore
- Mounting of data partitions on RHEL6 box, then upgrade
- Rolling upgrade with zmmboxmove of users to RHEL6 (or other OS) mailstore
- Note: CentOS will be officially supported as of 8.0.4 https://bugzilla.zimbra.com/show_bug.cgi?id=23487
Rolling Upgrade Example – RHEL5 to RHEL6
- Create a RHEL6 guest template
- Install ZCS 7.2 LDAP replica on RHEL 6 VM
- Promote RHEL6 LDAP 7.2 replica to master (requires modifying all other nodes to point to the new master)
- Move LDAP Replicas to new RHEL6 VMs
- Move MTAs to new RHEL6 VMs
- Do a Rolling Upgrade on LDAP nodes to ZCS8 [1]
- Optionally: Promote one or more LDAP replicas to LDAP Multi-Master https://wiki.zimbra.com/wiki/LDAP_Multi_Master_Replication#Promoting_an_existing_replica_to_be_a_multi-master
- Do a Rolling Upgrade on MTA nodes to ZCS8
- Do a Rolling Upgrade on Proxy nodes to ZCS8
- Create X new ZCS 8 mailstores on RHEL6
- zmmboxmove users from ZCS7.2 mailstores to ZCS8
- Decommission RHEL5 and ZCS7.2 systems
Rolling Upgrades – Fixed & Known Issues
- Known Issues:
- zmrestore creates faulty mailbox when using HSM https://bugzilla.zimbra.com/show_bug.cgi?id=80610 – Fixed 7.2.4, 8.0.4
- Login page after store1 upgrade in multinode environment gets corrupted https://bugzilla.zimbra.com/show_bug.cgi?id=76863 – Fixed 7.2.4, 8.0.4
- zmupdateauthkeys does not update all servers during installation of a new mailbox in a mix environment between 7.x and 8.x https://bugzilla.zimbra.com/show_bug.cgi?id=77452 – Not Fixed
- SOAP Exception during create COS https://bugzilla.zimbra.com/show_bug.cgi?id=82141 - Assigned 7.2.5, 8.0.5
- Rolling Upgrade : Accounts/aliases/dls are not displayed in list view https://bugzilla.zimbra.com/show_bug.cgi?id=75552 - Assigned JudasPriest
- Rolling Upgrade : Contact Group members not displayed in HTML client when 7.x user shares AB with 8.x user https://bugzilla.zimbra.com/show_bug.cgi?id=75523 - Not Assigned
- Fixed Issues:
- When moving users from 7.x to 8.0 Active sync no longer works https://bugzilla.zimbra.com/show_bug.cgi?id=78232 – Fixed 8.0.2
- ZCO should auto-update from a user's own mailstore, in order to ensure correct version when mixed versions in use https://bugzilla.zimbra.com/show_bug.cgi?id=79472 – Fixed: 7.2.3, 8.0.3
- Unable to rename user account in a split environment https://bugzilla.zimbra.com/show_bug.cgi?id=78046 – Fixed: 7.2.2, 8.0.0
Known Issues – Provisioning
- Some Admin commands may require being run from a ZCS8 mailstore:
- It is important to use a ZCS8 Admin Console for all provisioning activities
- Provisioning should generally be performed on a mailstore with the most recent ZCS version
- Reference: Unable to rename user account in a split environment http://bugzilla.zimbra.com/show_bug.cgi?id=78046 – Fixed 7.2.2
- Reference: https://bugzilla.zimbra.com/show_bug.cgi?id=81934#c4
The problem is the way in which "ra" functions in ZCS7, as it is fundamentally different than the way it functions in ZCS8. In ZCS7, "ra" makes a *copy* of the account, which duplicates all attributes, including attributes required to be unique, such as zimbraID. This operation will be blocked by the ZCS8 LDAP server. In ZCS8, the entry is simply updated with the new information (modified), rather than (copy, delete) as is done by ZCS7. The solution is to issue "zmprov ra" commands from ZCS8 servers only.
- A Domain Admin Distribution List in a Rolling Upgrade mode from ZCS7 to ZCS8 must have an additional attribute added for zimbraMailHost. See https://bugzilla.zimbra.com/show_bug.cgi?id=91029
- The workaround is to add a zimbraMailHost to the DL. It can be any of the ZCS8 mailstores, for example:
zmprov mdl listname@example.com zimbraMailHost mail5.example.com
- Rolling Upgrade ZCS 7 > 8 Broken Admin Console Functionality - https://bugzilla.zimbra.com/show_bug.cgi?id=86709
- The grant domainAdminAccountRights was not necessary in ZCS7 to have access these features, but apparently that changed in Z8. A number of the domain admin accounts did not have this right, and those were the ones who had issues with account edit and view mail within the Z8 console.
- Workaround is to go through all domain admin groups and accounts and updated this right for them. This seems to have restored full function to the accounts.
Known Issues – Web Login Page
- For a consistent web user experience at login time, the login page request for all the users (irrespective of zcs version of the mailstore they are hosted on) will be routed by the proxy to only ZCS 8.5+ mailstores so as to display a consistent newer login page (take a look at zimbraReverseProxyUpstreamLoginServers)
Known Issues – LDAP Fixes for 8.0.4
- OpenLDAP accesslog purge can trigger an assert https://bugzilla.zimbra.com/show_bug.cgi?id=80851 – Fixed 8.0.4
- OpenLDAP: invalid page split can cause db corruption https://bugzilla.zimbra.com/show_bug.cgi?id=82179 - Fixed 8.0.4
- Cumulatively, includes list of fixes in this list:
http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=shortlog;h=refs/heads/OPENLDAP_REL_ENG_2_4
Miscellaneous Solutions and Troubleshooting
This section contains solutions to some issues you might experience during your upgrade process.
1. If authenticating users in a Rolling Upgrade Mixed-version platform, the login page for all users should be set to the same version for a consistent user experience.
Solution:
For a consistent web user experience, may want to consider routing all users to either only ZCS7 web login pages, or only ZCS8 Either should work, it is really a Service/Admin preference Use zimbraReverseProxyLookupTarget or zimbraReverseProxyAvailableLookupTargets to set the zmprov garpu and garpb lists appropriately
<attr id="504" name="zimbraReverseProxyLookupTarget" type="boolean" cardinality="single" optionalIn="globalConfig,server" flags="serverInherited" requiresRestart="nginxproxy"> <globalConfigValue>FALSE</globalConfigValue> <desc>whether this server is a reverse proxy lookup target</desc> </attr> <attr id="1379" name="zimbraReverseProxyAvailableLookupTargets" type="string" cardinality="multi" optionalIn="server,globalConfig" flags="serverInherited" since="8.0.0"> <desc>The servers to be included in the proxy lookup hanlders list. Proxy will only use the servers specified here to do the lookup. Leaving empty means using all the servers whose zimbraReverseProxyLookupTarget is TRUE.</desc> </attr>
2. If users are sharing calendars/mailboxes, they should be on ZCS8 or ZCS7, but not mixed. The normal method of addressing this is to migrate groups of users at once - for example, if IT is sharing a lot of mailboxes between them, then zmmboxmove the IT users at one time, so that those shares all continue to function.
3. All provisioning and account administration should be performed on the new version only. For example, if running a mixed-version between ZCS8 and ZCS7, then a ZCS8 mailstore should be used for all administration UI functions.
- Provisioning should generally be performed on a mailstore with the most recent ZCS version
- Reference: Unable to rename user account in a split environment http://bugzilla.zimbra.com/show_bug.cgi?id=78046 – Fixed 7.2.2
- Reference: https://bugzilla.zimbra.com/show_bug.cgi?id=81934#c4
The problem is the way in which "ra" functions in ZCS7, as it is fundamentally different than the way it functions in ZCS8.
- In ZCS7, "ra" makes a *copy* of the account, which duplicates all attributes, including attributes required to be unique, such as zimbraID. This operation will be blocked by the ZCS8 LDAP server.
- In ZCS8, the entry is simply updated with the new information (modified), rather than (copy, delete) as is done by ZCS7.
The solution is to issue "zmprov ra" commands from ZCS8 servers only.
Upgrading ZCS5/6 to ZCS7
Important: The following is not official documentation. The data included here has been gathered from the field and provided only as a basis for discussion. These concepts must be tested prior to implementation, and results may vary in different environments.
No zmmboxmove
- Since zmmboxmove was added in 7.1.3 and designed only for migrations from ZCS7 to ZCS8, it cannot be used for older versions:
- http://bugzilla.zimbra.com/show_bug.cgi?id=58159 - decrease downtime during moves and allow moves between ZCS7 and ZCS8
- The official QA’d and most well-known/reliable method of upgrading from ZCS5/6 mailstores to ZCS7 continues to be the standard in-place upgrade
- However, some customers have reported using migration methods and have reported general success
- None of these methods are officially supported
- They may not be perfect in migrating all data, but should be sufficient if needed
- Some areas of likely problem may include Shared Objects (mailboxes, calendar, Briefcase, etc.) and tags
- Please be sure to test
zmbackup/zmrestore
- zmbackup/zmrestore individual users from one system to the other
- Advantages:
- Generally should work
- Although there may be differences in versions that cause problems in some metadata areas, such as tags
- Disadvantages:
- Slow
- Cumbersome (there may not be a simple method to move these backups to another server)
- Need to test if any former hostname entries remain
- Does not move LDAP
Tar Export/Import
- Tar Export/Import
- Advantages:
- Data-independent method for importing into any ZCS version
- Disadvantages:
- Includes no User Tag table
- May break Shared Objects (for sure, if moving to different ZCS platform with separate LDAP Master and new zimbraIDs)
- Known Issues:
- unable to import .tgz using ImportUI through the Proxy https://bugzilla.zimbra.com/show_bug.cgi?id=74622 – Fixed: 7.2.1, 8.0.0
- Conversation merge deletes messages containing the same ID as a virtual conversation [2] – Fixed: 8.0.3
Migrate Data by Protocol-based Methods
- Migrate data by protocol-based methods (e.g., imapsync, iCal export/import, Contacts export/import)
- Advantages:
- Data reliability
- Accuracy
- Disadvantages:
- Time consuming
- Labor intensive
- Can also use a combination of methods:
- imapsync email – largest data set, presync asynchronously (reduced user downtime, in background)
- tar export/import other data – smaller data set, performed during user downtime