Zimbra Next Generation Modules/Zimbra NG Backup/Incremental migration with Zimbra NG Backup
Zimbra NG Backup - Incremental migration with Zimbra NG Backup
Zimbra NG Backup - Incremental migration with Zimbra NG Backup
- This guide describes how to perform an Incremental Migration using Zimbra NG Backup.
- It's specifically designed for the migration of a production environment, minimizing the downtime and aiming to be transparent for the users.
- If correctly planned and executed, your mail system won't suffer any downtime and the impact on the users will be close to zero.
- All the CLI commands in this guide must be executed as the Zimbra user unless otherwise specified.
What will be migrated
- Emails and Email Folders.
- Contacts and Address Books.
- Appontments and Calendars.
- Tasks and Tasklists.
- Files and Briefcases.
- Share informations.
- User Preferences.
- User Settings.
- Class of Service Settings.
- Domain Settings.
What will NOT be migrated
- Server settings (migrated for reference but not restored).
- Global Settings (migrated for reference but not restored).
- Customizations (Postfix, Jetty, etc...).
- Items moved or deleted during the process will not be moved or deleted on the destination server.
- Preferences (e.g. passwords) changed during the process will be reset upon each import.
The Incremental Migration is not designed to set up a server-to-server mirroring. Using multiple imports to create a mirrored copy of the source server won't create a mirrored copy at all, since no deletions are performed by the Import process
FAQs and Troubleshootings
- The source server: any Zimbra server can be the source of your migration, as long as the running Zimbra version is 6.0.7 or higher.
- The destination server: any Zimbra server can be the destination of your migration, as long as the running Zimbra version is 6.0.7 or higher.
Be sure to have root access to both servers, and have a look at "Zimbra NG Backup: External Restore - Before you start" for some import performance tips.
- On the Source server: If Zimbra NG Backup is not currently enabled on the source server, make sure you have an amount of free disk space comparable to the size of the /opt/zimbra/store/ folder (the exported data is compressed through the gzip algorythm, and all zimbra items are deduplicated, usually reducing the size of exported to the 70% of the original size).
- On the Destination server: Make sure you have an amount of free space greater than the size of the /opt/zimbra/store/ and of the "export" folders on the source server combined.
While you can choose to transfer the data in any other way, rsync is our method of choice as it's a good compromise between speed and convenience.
The main data transfer is executed while the source server is still active and functional. However, since the transfer is performed via network, carefully plan your transfer in advance so that you'll have transfered all of your data before migrating.
Alternative Ways to transfer your data
Anything spanning from remote mount to physical move of the drive is ok as long as it suits your needs.
Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway. --Tanenbaum, Andrew S. (1996). Computer Networks. New Jersey: Prentice-Hall. p. 83. ISBN 0-13-349945-6.
Set the TTL value of your MX record to 300 on your "real" DNS. This will allow a fast switch between source and destination server.
Step 1: Coherency checks
In order to avoid any possible data-related issue, run the following checks on the source server:
- zmblobchk: this command checks the consistency between Zimbra's metadata and BLOBs.
- zmdbintegrityreport: this command checks the integrity of the Zimbra's database.
Repair any error found as described in Zimbra's official documentation.
Running a reindex of all mailboxes is also suggested.
Step 2: Zimbra Next Generation Modules setup
Only for servers on which Zimbra NG Backup is not already initialized and running Install Zimbra Next Generation Modules on both servers:
Disable the Real Time Scanner on both servers:
zxsuite backup setProperty ZxBackup_RealTimeScanner false
A dedicated device for the data export is strongly recommended in order to improve the export performance and to lower the impact on the performances of the running system.
Such device must be mounted on the /opt/zimbra/backup/ path and the Zimbra user must have r/w permissions on it
Step 3: Data Export (SmartScan)
Run a SmartScan on the source server:
zxsuite backup doSmartScan
All your data will be exported to the default backup path (/opt/zimbra/backup/zextras/).
Pro-Tip: Single Domains Export
You can also choose to only migrate one or more domains instead of all of them. To do so, run the following command instead of the SmartScan:
zxsuite backup doExport /path/to/export/folder/ domains yourdomain.com,yourdomain2.com[..]
Mind that if you start with the "SmartScan" method you'll have to carry on the migration with such method, and if you start with the "Single Domains" method you'll have to carry on the migration with this one. The two methods cannot be mixed.
Data export (SmartScan) via the Zimbra Next Generation Modules Administration Zimlet
You can also choose to export your data using the Zimbra Next Generation Modules Administration Zimlet following THIS guide.
Step 4: Data Synchronization
When you move the exported data to the destination server make sure that the destination folder is not Zimbra NG Backup's backup path on the destination server in order to avoid any nuisiances if you already use Zimbra NG Backup or plan to do so on the destination server.
(You can skip this step if you choose to transfer your data by other means than rsync.)
Using rsync, copy the data contained in the /opt/zimbra/backup/zextras/ on a directory in the destination server (make sure the Zimbra user has r/w permissions on such folder). Use a terminal multiplexer like screen or tmux, this process command might need A LOT of time depending on network speed and amount of data involved.
[run this command as Root] rsync -avH /opt/zimbra/backup/zextras/ root@desinationserver:/path/for/the/data/
Alternate synchronization method
While the suggested method is great for high-bandwidth situations, the first synchronization can involve a lot of data. If you feel that the rsync method is too slow, you might consider a physical move of the device (or the proper disk file if running on a virtual environment).
After moving the disk, you can remotely mount it back to the source server (e.g. via SSHFS), as the additional synchronizations needed for the migration will involve much less data. In this case, be sure to remount the device on the source server as /opt/zimbra/backup/zextras/ with all due permissions.
Step 5: First import
Import all exported data to the destination server:
zxsuite backup doExternalRestore /path/for/the/data/
Now sit back and relax while Zimbra Next Generation Modules imports your data on the destination server.
Warning: Do not edit nor delete the Map Files after this point. Doing so will cause item duplication issues and will almost surely disrupt your migration process.
First import via the Zimbra Next Generation Modules Administration Zimlet
You can also choose to import your data using the Zimbra Next Generation Modules Administration Zimlet following THIS guide. While importing via the Zimbra Next Generation Modules Administration Zimlet be sure to remove all System Accounts (like GalSync, Ham, Spam, Quarantine etc.) from the imported account list.
Step 5 (alternate): First import for large migrations [ADVANCED users only]
If you are to migrate a very large infrastructure where an export/import lasts for hours or even days, there is an alternative way to handle the migration from this point forward.
Instead of importing all of your data to the destination server, you can run a "Provisioning Only" import that will only create Domains, Classes of Service and Accounts on the destination server, skipping all mailbox contents.
zxsuite backup doExternalRestore /path/for/the/data/ provisioning_only TRUE
After doing this, switch the mailflow to the new server and, when the switch is completed, start the "real" import.
zxsuite backup doExternalRestore /path/for/the/data/
This way, your users will now connect to the new server where new emails will be delivered while old emails are being restored.
This approach has it's pros and cons, namely:
- Since items are only imported once and never modified or deleted afterwards, using this method will result in less discrepancies than the "standard" incremental migration.
- This is the option that has less impact on the source server (e.g. good if you are in a hurry to decommission it).
- Depending on the timing of the operation, this method has a higher impact on your users due to the fact that items are restored WHILE they work on their mailbox.
- Since the import is done on a running system, you might notice some slowdowns.
Be aware that THERE IS NO WAY BACK!
Once the firewall and DNS settings are changed, going back to the old server is pretty much impossible without extensive handiwork.
Thus, make sure that things like:
- Disk Sizing
- Filesystem health
- Zimbra BLOB and DB health
- Proper Zimbra JVM memory settings (heap size and PermGen/MaxPermGen)
are properly dealt with before starting the import.}}
The situation so far
Right now the vast majority of the data has already been imported to the destination server. The source server is still active and functional, and you are ready to perform the actual migration.
Step 6: Pre-migration checks
Before switching the mail flow, ALWAYS make sure that the new server is ready to become active (check your firewall, your DNS settings, your security systems etc.)
Step 7: The Switch
This is it, the migration moment has come! At the end of this step the destination server will be active and functional.
- Repeat step 3, step 4 and step 5 (only new data will be exported and synchronized)
- Switch the mail flow to the new server.
- Once NO MORE EMAILS arrive to the source server, repeat step 3, step 4 and step 5.
The Destination server is now active and functional.
Step 8: Post-migration checks
Run the following command to check for shares inconsistencies.
zxsuite backup doCheckShares
Should this command report any inconsistency, the
zxsuite backup doFixShares
command will parse the import mapfile used as the first argument and fix any broken share.
Mapfiles can be found in the Backup Path of the destination server as "map_[source_serverID]".
Step 9: Galsync
Delete all the imported GalSync accounts from the Zimbra Administration Console, create new GalSync accounts on all the imported domains and then re-initialize all the galsync accounts with the following command:
zmgsautil forceSync -a firstname.lastname@example.org -n [resourcename]
Step 10: Message Deduplication
The migration is completed, you have 2 choices:
- Uninstall Zimbra Next Generation Modules
- Initialize Zimbra NG Backup and start using this awesome piece of software known as Zimbra Next Generation Modules ;)
Incremental Migration FAQ
Q: Do I need a valid license in order to perform an incremental migration?
Yes. It can be either a Trial License or a purchased one.
Q: What will be migrated?
Everything except for the server configuration. This includes:
- User Data
- User Preferences
- Classes of Service configuration
- Domain configurations
Q: How should I transfer the exported data between my servers?
Again, anything that suits your needs is ok. You just need to be very sure about what your needs are ;)
Do you need to move the data very fast? Physically moving an USB disk between your servers might not be a good idea...
Do you need to move the data in a very reliable way? Mounting the export folder via SSHFS to the destination server might not be a good idea if your internet connection is sloppy...
All in all, each case is different... Pick your own poison!