View source for Zimbra NG Modules/Zimbra NG Backup/Incremental migration with Zimbra NG Backup

<div class="col-md-12"><br></div>
<div class="col-md-12"><br></div>
<ol class="breadcrumb">
  <li>[[Main Page|Zimbra Wiki]]</li>
  <li>[[Zimbra_NG_Modules]]</li>
  <li>[[Zimbra_NG_Backup]]</li>
  <li class="active"> Incremental migration with Zimbra NG Backup</li>
</ol>
__NOTOC__
<div class="col-md-12"><br /></div>
<div class="col-md-9">
    <h2 class="title-header" style="padding-bottom: 9px; border-bottom: 4px solid #0087c3;">Zimbra NG Backup -  Incremental migration with Zimbra NG Backup</h2>
    <div class="col-md-12">
        <div class="ibox-content">
            <div class="post animated fadeInLeft animation-delay-8" style="padding-top:5px">
                <div class="panel panel-default">
                    <div class="panel-body">
                        <h5 class="post-title">Zimbra NG Backup -  Incremental migration with Zimbra NG Backup</h5>
                        <div class="row">

= Description =

* This guide describes how to perform an Incremental Migration using Zimbra NG Backup.
* It's specifically designed for the migration of a production environment, minimizing the downtime and aiming to be transparent for the users.
* If correctly planned and executed, your mail system won't suffer any downtime and the impact on the users will be close to zero.
* ''' All the CLI commands in this guide must be executed as the Zimbra user unless otherwise specified.'''

== What will be migrated ==
* Emails and Email Folders.
* Contacts and Address Books.
* Appontments and Calendars.
* Tasks and Tasklists.
* Files and Briefcases.
* Share informations.
* User Preferences.
* User Settings.
* Class of Service Settings.
* Domain Settings.

== What will NOT be migrated ==
* Server settings (migrated for reference but not restored).
* Global Settings (migrated for reference but not restored).
* Customizations (Postfix, Jetty, etc...).
* Items moved or deleted during the process will not be moved or deleted on the destination server.
* Preferences (e.g. passwords) changed during the process will be reset upon each import. 

<div class="col-md-8"> 
    <div class="alert alert-warning fade in"> <p> <i class="fa  fa-warning"></i> <strong>warning</strong> </p>
    <p class="text-justify">The Incremental Migration is not designed to set up a server-to-server mirroring. Using multiple imports to create a mirrored copy of the source server won't create a '''mirrored''' copy at all, since no deletions are performed by the Import process</p> 
    </div> 
</div>
<div class="clearfix"></div>

= Pre-migration checks =
== FAQs and Troubleshootings ==
Please carefully read the [[Zimbra_NG_Modules/Zimbra_NG_Backup/Troubleshooting|Zimbra NG Backup Troubleshooting]] and the [[Zimbra_NG_Modules/Zimbra_NG_Backup/FAQ|Zimbra NG Backup FAQs]] pages before running an incremental migration

== Servers ==
* The source server: any Zimbra server can be the source of your migration, as long as the running Zimbra version is 6.0.7 or higher.
* The destination server: Network Edition installed running 8.8.x or higher or FOSS server running Zimbra Plus can be the destination. 


Be sure to have root access to both servers, and have a look at "[[Zimbra_NG_Modules/Zimbra_NG_Backup/External_Restore#Before_you_start|Zimbra NG Backup: External Restore - Before you start]]" for some import performance tips.

== Storage ==
* On the Source server: If Zimbra NG Backup is not currently enabled on the source server, make sure you have an amount of free disk space ''comparable'' to the size of the ''/opt/zimbra/store/'' folder (the exported data is compressed through the gzip algorythm, and all zimbra items are deduplicated, usually reducing the size of exported to the 70% of the original size).
* On the Destination server: Make sure you have an amount of free space greater than the size of the ''/opt/zimbra/store/'' and of the "export" folders on the source server combined.

== Data Transfer ==
While you can choose to transfer the data in any other way, rsync is our method of choice as it's a good compromise between speed and convenience.

The main data transfer is executed while the source server is still active and functional. However, since the transfer is performed via network, carefully plan your transfer in advance so that you'll have transfered '''all of your data''' before migrating.

===Alternative Ways to transfer your data===
Anything spanning from remote mount to physical move of the drive is ok as long as it suits your needs.
 Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.
 --Tanenbaum, Andrew S. (1996). Computer Networks. New Jersey: Prentice-Hall. p. 83. ISBN 0-13-349945-6.

== DNS ==
Set the TTL value of your MX record to 300 on your "real" DNS. This will allow a fast switch between source and destination server.

= The Set-up =

== Step 1: Coherency checks ==
In order to avoid any possible data-related issue, run the following checks on the source server:
* [http://wiki.zimbra.com/wiki/Ajcody-Notes-No-Such-Blob#Zmblobchk_for_5.0.6.2B_Systems zmblobchk]: this command checks the consistency between Zimbra's metadata and BLOBs.
* [http://wiki.zimbra.com/wiki/Zmdbintegrityreport zmdbintegrityreport]: this command checks the integrity of the Zimbra's database.

Repair any error found as described in Zimbra's official documentation.

Running a reindex of all mailboxes is also suggested.

== Step 2: Zimbra NG Modules setup ==
'''Only for servers on which Zimbra NG Backup is not already initialized and running'''
Install Zimbra NG Modules on both servers:

Disable the Real Time Scanner on both servers:
<pre>
zxsuite backup setProperty ZxBackup_RealTimeScanner false
</pre>

''' A dedicated device for the data export is strongly recommended in order to improve the export performance and to lower the impact on the performances of the running system.

Such device must be mounted on the /opt/zimbra/backup/ path and the Zimbra user must have r/w permissions on it'''
== Step 3: Data Export (SmartScan) ==
Run a SmartScan on the source server:
<pre>
zxsuite backup doSmartScan
</pre> 
All your data will be exported to the default backup path (/opt/zimbra/backup/zextras/).

==== Pro-Tip: Single Domains Export ====
You can also choose to only migrate one or more domains instead of all of them. To do so, run the following command '''instead''' of the SmartScan:
<pre>
zxsuite backup doExport /path/to/export/folder/ domains yourdomain.com,yourdomain2.com[..]
</pre>
Mind that if you start with the "SmartScan" method you'll have to carry on the migration with such method, and if you start with the "Single Domains" method you'll have to carry on the migration with this one. The two methods cannot be mixed.

==== Data export (SmartScan) via the Zimbra NG Modules Administration Zimlet ====
You can also choose to export your data using the Zimbra NG Modules Administration Zimlet following [[Zimbra_NG_Modules/Zimbra_NG_Backup/Smartscan#Starting_the_scan_via_the_Zimbra_NG_Modules_Administration_Zimlet | THIS guide]].

== Step 4: Data Synchronization ==

<div class="col-md-8"> 
    <div class="alert alert-warning fade in"> <p> <i class="fa  fa-warning"></i> <strong>warning</strong> </p>
    <p class="text-justify">When you move the exported data to the destination server make sure that the destination folder is not Zimbra NG Backup's backup path on the destination server in order to avoid any nuisiances if you already use Zimbra NG Backup or plan to do so on the destination server.</p> 
    </div> 
</div>
<div class="clearfix"></div>

''(You can skip this step if you choose to transfer your data by other means than rsync.)''

Using ''rsync'', copy the data contained in the /opt/zimbra/backup/zextras/ on a directory in the destination server (make sure the Zimbra user has r/w permissions on such folder). Use a terminal multiplexer like ''screen'' or ''tmux'', this process command might need A LOT of time depending on network speed and amount of data involved.
<pre>
[run this command as Root]
rsync -avH /opt/zimbra/backup/zextras/ root@desinationserver:/path/for/the/data/
</pre>

=== Alternate synchronization method ===
While the suggested method is great for high-bandwidth situations, the first synchronization can involve a lot of data.
If you feel that the rsync method is too slow, you might consider a physical move of the device (or the proper disk file if running on a virtual environment).

After moving the disk, you can remotely mount it back to the source server (e.g. via SSHFS), as the additional synchronizations needed for the migration will involve much less data. In this case, be sure to remount the device on the source server as /opt/zimbra/backup/zextras/ with all due permissions.

== Step 5: First import ==
Import all exported data to the destination server:
<pre>
zxsuite backup doExternalRestore /path/for/the/data/
</pre>

Now sit back and relax while Zimbra NG Modules imports your data on the destination server.

''Warning: Do not edit nor delete the [[Zimbra_NG_Modules/Zimbra_NG_Backup/Store#Map_Files|Map Files]] after this point. Doing so will cause item duplication issues and will almost surely disrupt your migration process.''

==== First import via the Zimbra NG Modules Administration Zimlet ====
You can also choose to import your data using the Zimbra NG Modules Administration Zimlet following [[Zimbra_NG_Modules/Zimbra_NG_Backup/Migration_Tool_Import |THIS guide]].
While importing via the Zimbra NG Modules Administration Zimlet be sure to remove all System Accounts (like GalSync, Ham, Spam, Quarantine etc.) from the imported account list.


== Step 5 (alternate): First import for large migrations [ADVANCED users only] ==
If you are to migrate a very large infrastructure where an export/import lasts for hours or even days, there is an alternative way to handle the migration from this point forward. 

Instead of importing all of your data to the destination server, you can run a "Provisioning Only" import that will only create Domains, Classes of Service and Accounts on the destination server, skipping all mailbox contents. 
<pre>
zxsuite backup doExternalRestore /path/for/the/data/ provisioning_only TRUE
</pre>

After doing this, switch the mailflow to the new server and, when the switch is completed, start the "real" import.
<pre>
zxsuite backup doExternalRestore /path/for/the/data/
</pre>

This way, your users will now connect to the new server where new emails will be delivered while old emails are being restored. 


This approach has it's pros and cons, namely:

'''Pros'''
* Since items are only imported once and never modified or deleted afterwards, using this method will result in less discrepancies than the "standard" incremental migration.
* This is the option that has less impact on the source server (e.g. good if you are in a hurry to decommission it).

'''Cons'''
* Depending on the timing of the operation, this method has a higher impact on your users due to the fact that items are restored WHILE they work on their mailbox.
* Since the import is done on a running system, you might notice some slowdowns.

<div class="col-md-8"> 
    <div class="alert alert-warning fade in"> <p> <i class="fa  fa-warning"></i> <strong>warning</strong> </p>
    <p class="text-justify">Be aware that THERE IS NO WAY BACK!</p> 
    </div> 
</div>
<div class="clearfix"></div>

Once the firewall and DNS settings are changed, going back to the old server is pretty much impossible without extensive handiwork.

Thus, make sure that things like:

* Disk Sizing

* Filesystem health

* Zimbra BLOB and DB health

* Proper Zimbra JVM memory settings (heap size and PermGen/MaxPermGen)

are properly dealt with before starting the import.}}

= The situation so far =
Right now the vast majority of the data has already been imported to the destination server. The source server is still active and functional, and you are ready to perform the actual migration.

= The Migration =
== Step 6: Pre-migration checks ==
Before switching the mail flow, ALWAYS make sure that the new server is ready to become active (check your firewall, your DNS settings, your security systems etc.)

== Step 7: The Switch ==
This is it, the migration moment has come! At the end of this step the destination server will be active and functional.

* Repeat step 3, step 4 and step 5 (only new data will be exported and synchronized)
* Switch the mail flow to the new server.
* Once NO MORE EMAILS arrive to the source server, repeat step 3, step 4 and step 5.

The Destination server is now active and functional.

== Step 8: Post-migration checks ==
Run the following command to check for shares inconsistencies.
<pre>
zxsuite backup doCheckShares
</pre>

Should this command report any inconsistency, the 
<pre>
zxsuite backup doFixShares
</pre>
command will parse the import mapfile used as the first argument and fix any broken share. 

Mapfiles can be found in the Backup Path of the destination server as "map_[source_serverID]".

== Step 9: Galsync ==
Delete all the imported GalSync accounts from the Zimbra Administration Console, create new GalSync accounts on all the imported domains and then re-initialize all the galsync accounts with the following command:
<pre>
zmgsautil forceSync -a galsync@domain.com -n [resourcename]
</pre>

== Step 10: Message Deduplication ==
Running a [[Zimbra_NG_Modules/Zimbra_NG_HSM/Item_Deduplication|Message Deduplication]] operation through [[Zimbra_NG_Modules/Zimbra_NG_HSM|Zimbra NG HSM]] is highly suggested in order to save precious storage space.

== What now? ==
The migration is completed, you have 2 choices:

* Uninstall Zimbra NG Modules
* Initialize Zimbra NG Backup and start using this awesome piece of software known as Zimbra NG Modules ;)

= Incremental Migration FAQ =
== Q: Do I need a valid license in order to perform an incremental migration? ==
Yes. It can be either a Trial License or a purchased one.

== Q: What will be migrated? ==
Everything except for the server configuration.
This includes: 
* User Data
* User Preferences
* Classes of Service configuration
* Domain configurations

== Q: Will I lose my shares? Will I need to re-configure all my shares? ==
Absolutely not!

== Q: How should I transfer the exported data between my servers? ==
Again, anything that suits your needs is ok. You just need to be very sure about what your '''needs''' are ;)

Do you need to move the data very fast? Physically moving an USB disk between your servers might not be a good idea...

Do you need to move the data in a very reliable way? Mounting the export folder via SSHFS to the destination server might not be a good idea if your internet connection is sloppy... 

All in all, each case is different... Pick your own poison!
                        </div>
                    </div>
                    <div class="col-md-9">
                        <div class="panel-footer">
                            <p><i class="fa fa-clock-o"></i> Aug 25, 2016 - [https://www.zimbra.com/email-server-software/ Know more »]</p>
                        </div>
                    </div>
                </div>
            </div>
        </div>
    </div>
</div>
<div class="col-md-3"><br /></div>
<div class="col-md-3">
    <div class="panel panel-zimbrared-light-border"> 
        <div class="panel-heading"> 
            <h3 class="panel-title"><i class="fa fa-gear pull-left"></i> Zimbra NG Modules</h3> 
        </div>
        <div class="panel-body">
            {{ZNG}}
        </div>
    </div>
</div>
<div class="col-md-3">
    <div class="panel panel-primary-light-border"> 
        <div class="panel-heading"> 
            <h3 class="panel-title"><i class="fa fa-info-circle pull-left"></i> Zimbra NG Modules Resources</h3> 
        </div>
        <div class="panel-body">
            {{ZNGL}}
        </div>
    </div>
</div>
<div class="clearfix"></div>
<div class="col-md-12"><br></div>
{{FH}}