View source for Network Edition Disaster Recovery

{{BC|Certified}}
__FORCETOC__
<div class="col-md-12 ibox-content">
=Network Edition Disaster Recovery=
{{KB|{{ZC}}|{{ZCS 8.0}}|{{ZCS 7.0}}|{{ZCS 6.0}}}}
{{WIP}}
'''Discussion on this document is located here:
* "Network_Edition_Disaster_Recovery  -  Needed Updates / Discussion"
** http://bugzilla.zimbra.com/show_bug.cgi?id=84562

This article describes the steps to replace a failed server in a version 4.5.x, 5x, 6x, 7x and 8x network edition single-server ZCS configuration. Any distinctions between versions will be handled within the article when needed.

'''Important: '''<font size="3" color="#000000" face="Arial">''T''</font><font color="#000000">''he ZCS release you install on the new server '''must be the same release '''as installed on the old server. The server can have a different operating system.''</font>

<font color="#000000">The new server hardware must meet the requirements described in the Installation Prerequisites section of the ZCS Single Server Installation Guide. Install the new operating system, making any necessary OS configuration modifications as described in the installation guide. </font>

==<font size="4" color="#00007f" face="Arial">'''Other References'''</font>==

===<font size="4" color="#00007f" face="Arial">'''Other Documentation And Options'''</font>===

* The Official Administration Guide For the Various ZCS Versions:
** http://www.zimbra.com/support/documentation/
*** Got the html based Administration Guide and search for : '''Disaster Recovery for Specific Situations'''
* As this page is exclusively for a single ZCS server deployment, you might need to also reference the following for your situation:
** [[Ajcody-Notes-Multi-Server-Restore-DR]]
** [[5.0.x_Network_Edition_Backup_and_Restore#Disaster_Recovery_for_Specific_Situations]]
* Some "notes" on specific disaster recovery situations, bugs, or other issues:
** [[Ajcody-Disaster-Recovery-Specific-Notes]]
** [[5.0.x_Network_Edition_Backup_and_Restore#Disaster_Recovery_for_Specific_Situations]]
* For more focus DR restores against 'types' of data rather than a FULL DR like this wiki handles.
** [[Ajcody-Backup-Restore-Issues#Restore_For_Disaster_Recovery]]
* This process below uses rsync between two servers or points and bypasses the need for zmrestoreoffline/zmrestore:
** [[Ajcody-Notes-Server-Move]]
* In case someone is wanting the older version of this wiki page prior to the latest major update, see:
** http://wiki.zimbra.com/index.php?title=Network_Edition_Disaster_Recovery&oldid=15765
* Steps involved when you are using '''snap shots''' :
** From the current 7.2.0 Administration Guide under the Backup section.
** http://www.zimbra.com/docs/ne/7.2.0/administration_guide/wwhelp/wwhimpl/js/html/wwhelp.htm#href=ZCS_Admin_Guide_7_NEUsing_snapshots_to_backup_and_restore.html

===<font size="4" color="#00007f" face="Arial">'''Outstanding RFE's That Will Effect This Document'''</font>===

* "RFE: zmrestore & zmrestoreoffline have option to set account status"
** http://bugzilla.zimbra.com/show_bug.cgi?id=52460

=====<font size="4" color="#00007f" face="Arial">'''RFE For Live DR Restore Option'''</font>=====

* "Live or minimal downtime DR option [ need it tested/QA'ed/dev sign off for fully supported option ]"
** https://bugzilla.zimbra.com/show_bug.cgi?id=94606
* "First-come, First-served On-demand Live DR"
** https://bugzilla.zimbra.com/show_bug.cgi?id=94608
* "zmrestore / zmrestoreoffline have options to --skipDeleteMailboxes - Provides Live DR option"
** https://bugzilla.zimbra.com/show_bug.cgi?id=94368
* "RFE: zmplayredo option for --frommailboxId --tomailboxId or --pre"
** https://bugzilla.zimbra.com/show_bug.cgi?id=52642
*** see comment 7

===<font size="4" color="#00007f" face="Arial">'''When Samba, Posix, And LDAP Customizations Were Used'''</font>===

====ZCS 6.0.7 And After====

* "zmrestoreldap should restore config database"
** https://bugzilla.zimbra.com/show_bug.cgi?id=46623
** Summary
*** With 6.0.7, I've added the config database to the nightly backups.  We should have zmrestoreldap restore this database as well, so that the entire configuration of the server as it was (including any and all customizations, like additional schema, etc) are restored at the same time as the database is.

====Before ZCS 6.0.7====

* [[UNIX_and_Windows_Accounts_in_Zimbra_LDAP_and_Zimbra_Admin_UI_6.0]]
* "DR wiki addition to Samba/Posix extensions" - which depends on bug 46401 below.
** http://bugzilla.zimbra.com/show_bug.cgi?id=46400
* "LDAP config database should be backed up"
** http://bugzilla.zimbra.com/show_bug.cgi?id=46401

==<font size="4" color="#00007f" face="Arial">'''Disaster Recovery - Changing Servers '''</font>==

<font color="#000000">You do the following to restore to a new server:</font>

*Prepare the new server
*Block client access to the old server’s IP address with firewall rules
*Mount any volumes that were in use on the older server
** Volume information can be found in the full backup session your using:
*** Example
**** cat ~/backup/sessions/full-20120905.080010.492/sys/volume.dat
**** 1,1,"message1","/opt/zimbra/store",12,8,12,8,0,4096
**** 2,10,"index1","/opt/zimbra/index",12,8,12,8,0,4096
*Delete the MySQL data that is set up in the initial installation of ZCS
*Copy the backup files to the new server
*Run zmrestoreldap to restore the global LDAP data
*Run zmrestoreoffline to restore account data from the backup sessions
*Prepare and run a new backup

==<font size="3" color="#00007f" face="Arial">'''Old Server Status'''</font>==

Two scenarios for disaster recovery are the server has died and the ZCS files cannot be accessed, or ZCS is still running, but the server needs to be replaced.

'''If the server is not running:''' 

#Block client access to the server IP address with firewall rules.
#Find the latest full ZCS backup session to use.

'''If ZCS is still running, to prepare the move to the new server:'''

#Block client access to the server’s IP address with firewall rules.
#Run a full backup of the old service, or if the backup is recent, run an incremental backup to get the most current incremental backup session.
#Run <font color="#000000">'''zmcontrol stop'''</font>, to stop ZCS. In order to restore to the most current state, no new mail should be received after the last incremental backup has run. 
#Change the hostname and IP address on the old server to something else. Do not turn off the server.

==<font size="3" color="#00007f" face="Arial">'''Preparing the New Server'''</font>==

Before you begin, make sure that the new server is correctly configured with the IP address and hostname and that ZCS is installed and configured with the same domain, hostname, passwords, etc. as the previous server. See the Single-Server Installation Guide for more information about preparing the server. Before you begin to install ZCS, note the information you need from the old server including: admin account name and password, spam training and non-spam training user account names, exact domain name, and the global document account name.

==<font size="3" color="#00007f" face="Arial">'''Before You Doing Anything, Copy Your Redologs If Possible'''</font>==

Before you attempt anything, you should copy the redologs from /opt/zimbra/redolog/* so they will be available for replay on the new server or on this one after you have rebuilt from the backups. The redologs will have data that occurred after your last backup.

==<font size="3" color="#00007f" face="Arial">'''Installing ZCS on new server'''</font>==

# If your doing a restore exclusively from a backup and don't recall the exact version, you can find out the version of ZCS within the full backup session.
#* '''cd /opt/zimbra/backup/sessions/full-[YOUR LABEL]/'''
#* '''grep  zcsRelease session.xml'''
#* You'll see a line like this - notice the version is given:
#:* <backupSet label="full-20100330.201810.579" '''''zcsRelease="6.0.4_GA_2038.RHEL5 20091214172206 20091214-1723 NETWORK"''''' startTime="1269980290579" endTime="1269980316645" minRedoSeq="44" maxRedoSeq="44" sharedBlobsZipped="true" sharedBlobsZipNameDigestChars="1" sharedBlobsDirectoryDepth="5" sharedBlobsCharsPerDirectory="2" type="full" accountsDirectoryDepth="2">
#'''Make sure your TIME and TIMEZONE is set right! See [[:Time_Zones_in_ZCS#The_server_OS]]'''
#Ensure that the old hostname and MX DNS records resolve to the new server
#* [[Ajcody-Hostname-DNS]] has the commands you can run to confirm resolution and also basic setup instructions if you need to configure BIND on this DR server for DNS resolution issues.
#If the new server was used for testing in the past then there may be remnants of previous installations that need to be removed.
#: '''''Warning - the following commands will completely destroy any existing Zimbra installation:'''''
#:* '''sudo -u zimbra /opt/zimbra/bin/zmcontrol stop<br>
#:* '''If''' you still have the tarball extracted directory available, you can do the following to uninstall zimbra, '''as root''':
#:**  '''zcs/install.sh -u'''
#:* '''If not''', remove all the zimbra packages manually. '''As root''' :
#:** '''rpm -qa zimbra*'''
#:** Now, paste just in the package names for removal. '''''For example''''':
#:** '''rpm -e zimbra-spell zimbra-ldap zimbra-mta zimbra-logger zimbra-snmp zimbra-apache zimbra-store zimbra-core'''
#:** And move aside the old and unneeded zimbra directory (assuming enough space in the destination location):
#:** '''mkdir /var/tmp/zimbra-backup; mv -f /opt/zimbra /var/tmp/zimbra-backup'''
#:''Non-rpm based OS's can review [[UnInstall_Zimbra]]''
# Have the ZCS installation software available on the new server and extracted. The tarball that is; for example : zcs-NETWORK-6.0.4_GA_2038.RHEL5.20091214172206.tgz .
##* '''Make Sure Your Using The Same ZCS VERSION THAT WAS USED FOR THE BACKUP''' It is unsupported to attempt a ZCS system DR restore using a ZCS version that is newer than what was used for the backup. The zmrestoreoffline [-sys] is not able to handle the various db [ldap & mysql] and schema changes during the restore. This is normally done via the upgrade installer script.
##** '''OS TYPE/VERSION''' [RHEL5 for example] changes ''might'' cause issues, but generally shouldn't . If you end up needing assistance from Zimbra Support, please include and changes from the old to the new server in regards to the three variables mentioned here.
##** If the OS TYPE or PLATFORM are going to be different, you should review the following to see if it better suits your needs:
##*** [[Ajcody-Notes-Server-Move]]
##*** [[Network_Edition:_Moving_from_32-bit_to_64-bit_Server]]
##* Latest Version of ZCS Network Edition can be downloaded from : 
##** [http://www.zimbra.com/downloads/ne-downloads.html http://www.zimbra.com/downloads/ne-downloads.html]
##* Older Versions of ZCS Network Edition can be downloaded from : 
##** [http://www.zimbra.com/downloads/ne-downloads-previous.html http://www.zimbra.com/downloads/ne-downloads-previous.html]
#Run the zcs/install.sh from the ZCS tar ball of '''the same version of ZCS'' that was running on your old server.
## '''Please make sure your "backups" are not located under /opt/zimbra/* at this time since the installer may remove everything under /opt/zimbra as it does the installation.'''
##Allow it to install all modules.
#When the configuration menu appears open up a new terminal window.
## License: Copy your ZCSLicense.xml file to /opt/zimbra/conf and then change ownership and rights:
##:* '''cp ZCSLicense.xml /opt/zimbra/conf<br>
##:* '''chown zimbra:zimbra /opt/zimbra/conf/ZCSLicense.xml<br>
##:* '''chmod 444 /opt/zimbra/conf/ZCSLicense.xml<br>
## Gather Variables From Old Server Config File To Use To Configure New Server During Setup:
### You have access to the old servers /opt/zimbra/conf directory and can copy the localconfig.xml from the old server or old location to /tmp on the new server:
###* '''/opt/zimbra/bin/zmlocalconfig -c /tmp/localconfig.xml -s > /tmp/OLD-localconfig.xml'''
### If you only have access to full backup sessions, you'll be able to find a localconfig.xml in the subdirectory of the full backup. 
###* ''For example:''
###* '''/opt/zimbra/bin/zmlocalconfig -c /opt/<old-zimbra-dir>/backup/sessions/full-<latest-label>/sys/localconfig.xml -s > /tmp/OLD-localconfig.xml'''
### You now have a text file [ /tmp/OLD-localconfig.xml ] that will have all the variables you need for the next part. It will also display the needed passwords for the various accounts. 
###* '''Please delete this file once your finished with it.'''
#Returning to the configuration menu, which should be showing you 10 main categories that you can configure. 
::A.  Select 1 for '''Common Configuration''' , you'll have the following sub-options.
::# Confirm '''hostname''' [ zimbra_server_hostname ], '''ldap master host''' [ ldap_host ] , and '''timezone'''.
::# Set '''Ldap Admin password''' using the information from the /tmp/OLD-localconfig.xml you made. 
::#* Variable that has password is : [ ldap_root_password ]
::B. Select 2 for '''zimbra-ldap''' , you'll have the following sub-options.
::# Confirm that '''Create Domain = Yes'''
::# Confirm that '''Domain to create''' is correct. '''''Note: Common Mistake Warning''''' This value is your old initial domain rather than a subdomain of it. If it's wrong, revisit the above section for '''Common Configuration'''.
::#* For example: mail3.zimbra.DOMAIN.com [bad] vs. zimbra.DOMAIN.com [correct]
::#* A number of variables from the OLD-localconfig.xml can be double checked for this, one that probably changes the least is [ av_notify_domain ] .
::# Set the following passwords using the variables from the OLD-localconfig.xml
::#* Password Asked For  ||  Variable Name From OLD-localconfig.xml
::#* '''Ldap root password''' = ldap_root_password
::#* '''Ldap replication password''' = ldap_replication_password
::#* '''Ldap postfix password''' = ldap_postfix_password
::#* '''Ldap amavis password''' = ldap_amavis_password
::#* '''Ldap nginx password''' = ldap_nginx_password
::C. Select 3 for '''zimbra-store''', you'll have the following sub-options.
::# '''Create Admin User''' should be '''yes'''. 
::# Double the value for '''Admin user to create''' , confirming it's not an improper subdomain. 
::#* Old variables to look at: [ av_notify_users ] , [ smtp_destination ] , [ smpt_source ] , [ zimbra_ldap_userdn ]
::# Set '''Admin Password''' : [ this is the admin password you would use to login to admin console ]
::# '''Spam training user''' and the '''Non-spam(Ham) training users''' user values from the old server can be found in the ldap.bak files from your full session backups. 
::#* Example of path location for ldap.bak: 
::#: '''cd /opt/<old-zimbra-dir>/backup/sessions/full-<latest-label>/ldap/'''
::#: To get values : '''egrep -i 'zimbraSpamIsNotSpamAccount|zimbraSpamIsSpamAccount' ldap.bak'''
::#* Example output : 
::#*: ''zimbraSpamIsNotSpamAccount: ham.ic9f_54eb7@zimbra.DOMAIN.com''
::#*: ''zimbraSpamIsSpamAccount: spam.rujjnikc@zimbra.DOMAIN.com''
::# '''Global Documents Accounts''' equals the [ wiki_user ] variable plus your domain.
::#* For example: [ wiki_user = wiki ] so I would set this option to wiki@zimbra.DOMAIN.com using the domain example above.
::# '''SMTP host:''' can be confirmed by checking the ldap.bak like we did above.
::#* Example of path location for ldap.bak: 
::#: '''cd /opt/<old-zimbra-dir>/backup/sessions/full-<latest-label>/ldap/'''
::#: To get value : '''grep zimbraSmtpHostname ldap.bak'''
::# Adjust the remaining variables if you need to, most customer will not. '''Exception might be''' those using the '''proxy services''' and needing to enable the '''Instant Messaging Feature'''.
::D. Change any other settings on the new server to match the configuration on the original server.
::E. Disable auto-backup and starting of servers after configuration in the configuration menu:
:::* '''Enable default backup schedule:         no'''                            
::F. Apply configuration changes, there will be one or two warnings that say - it's fine to ignore them: 
:::* '''''"WARNING: Document and Zimlet initialization skipped because Application Server was not configured to start."'''''
:::* '''''"WARNING: Convertd version 2 migration skipped because Application Server was not configured to start".'''''

==<font size="3" color="#00007f" face="Arial">'''Restoring to the new server'''</font>==

#Confirm the new server isn't running zimbra. It shouldn't be if you selected the right option above during the installation. '''As zimbra''' type:
#* '''zmcontrol stop'''
#* confirm there's no 'zimbra services' processes running [you might see some related to swatch, which you can ignore].
#* '''ps -ef | grep zimbra'''
# '''Create Any Addition Storage Volumes''' 
## '''Note: Common Problem''' : This step is skipped/forgotten by many customers that were using '''additional storage volumes''' on their old server.
##* Configure your extra disk/san/nfs mounts.
##** Review your /etc/fstab file from your old server if you have access to it.
##** Double check your old server, if possible, for any symbolic links going outside of /opt/zimbra . Common reason is sym linking the backup directory < /opt/zimbra/backup > to some other location < /other/location/backup >.
##** '''ls -la /opt/zimbra/'''
#Move aside the old mysql data and re-initialize an empty data directory. If you do not do this, zmrestoreoffline will have errors. '''As zimbra''', type:
#:* '''mkdir /var/tmp/mysql-data-backup; mv -f /opt/zimbra/db/data/* /var/tmp/mysql-data-backup'''
#:* '''/opt/zimbra/libexec/zmmyinit'''   
#:* '''Note:''' You might see events of '''"* Failed to connect to mysql...retrying"''', this is normal. If it goes over 10 occurrences, this might be an indication of something being wrong. Check the following logs:
#:** /var/log/zimbra.log
#:** '''ls -latr /opt/zimbra/log/''' 
#:** If you think there's an issue, did you confirm your passwords? Look for something like the following in /var/log/zimbra.log :
#:*** ''Oct 20 17:39:33 mail zimbramon[9745]: 9745:info: zmmtaconfig: gacf ERROR: service.FAILURE (system failure: unable to get config) (cause: javax.naming.AuthenticationException [LDAP: error code 49 - Invalid Credentials])''
#:*** See the section below to fix: "9. Double check your LDAP password"
#:*** Review from the most recent and then check older ones. most likely: zmmyinit.log , myslow.log , mysql_error.log.
#:* The MySQL service should now be running, a simple check would be:
#::* '''ps -ef | grep mysql'''
# '''Create Any Additional Zimbra Volumes Now Or Adjust Default Paths'''
## '''Note: Common Problem''' : This step is skipped/forgotten by many customers that were using additional '''[[CLI_zmvolume|zimbra volumes]]''' or had changed the paths to the defaults on their old server.
##* Create your additional '''[[CLI_zmvolume|zimbra volumes]]''' now.
##** Common reason for additional zimbra volumes is for [[Ajcody-HSM-Notes|HSM]] , [[Ajcody-Notes-Archive-Discovery|Archive & Discover [A&D]]], or large mailstores because of sizing issues.
##** Volume information can be found in the full backup session your using:
##** Example
##*** cat ~/backup/sessions/full-20120905.080010.492/sys/volume.dat
##*** 1,1,"message1","/opt/zimbra/store",12,8,12,8,0,4096
##*** 2,10,"index1","/opt/zimbra/index",12,8,12,8,0,4096
##** Types are: primaryMessage, secondaryMessage, or index
##** '''Important zmvolume related bug'''
##*** "zmrestoreoffline fails for most DR recovery situation for volume differences"
##**** https://bugzilla.zimbra.com/show_bug.cgi?id=77257
# Copy the backup files from the old server or from an archive location to /'''opt/zimbra/backup/sessions/''' or use the '''''-t /PATH/TO/backup''''' option for the zmrestore* commands below.
# Set the ownership of the copied backup files: '''chown -R zimbra:zimbra /opt/zimbra/backup'''
#To find the LDAP session label to restore, '''as zimbra''' type:
#*  '''zmrestoreldap –lbs'''
#: or using the -t /PATH/TO/backup option
#* ''Example'': '''zmrestoreldap -t /opt/<old-zimbra-dir>/backup/ -lbs'''
#** I see that this is my latest one:  incr-20100315.050022.428  
#To restore the LDAP data, run one of the zmrestoreldap commands below '''as zimbra''' :
#:* '''zmrestoreldap -lb <latest_label>'''
#:: or with the '''nohup''' option if you are restoring large number of accounts:
#:* ''Example'': '''cd /tmp ; nohup zmrestoreldap -t /opt/<old-zimbra-dir>/backup/ -lb <latest_label> &'''
#:** Output is redirected to the nohup.out file that is created in the directory you run the command in. 
#:** ''Note: You'll [ zimbra ] need write permission for that directory the nohup.out file is written to.''
#:A. '''Note:''' Observe whether the LDAP server has started successfully after the restore, it must be running for the next steps. Simple checks are:
#::* Confirm output to the screen or to the nohup.out file ended with: 
#:::* '''''Starting ldap...Started slapd: pid ####''''''
#:::* '''''done.'''''
#::* Confirm ldap process is still running: '''ps -ef | grep ldap'''
#:B. '''Note:''' The '''zmrestoreldap''' script included in ZCS 4.5.7 through ZCS 4.5.10 and ZCS 5.0 through ZCS 5.0.1 is '''broken'''. 
#::* This is being tracked as '''Bug 23644''': zmrestoreldap not taking accesslog db into consideration. The fix will be included in ZCS 4.5.11 and ZCS 5.0.2.  You can also download an updated script with the fix from these links:
#::** ZCS 4.5.x: http://files.zimbra.com/downloads/4.5.10_GA/zmrestoreldap_4511
#::** ZCS 5.0.x: http://files.zimbra.com/downloads/5.0.1_GA/zmrestoreldap_502
#:D. '''Note:''' On zimbra 5.0.7 this failed with the error "ERROR: Failed to move existing ldap data: 256"
#::* This is because the directory /opt/zimbra/openldap-data/ is empty and the script is trying to backup the contents to /opt/zimbra/openldap-data/.priv and failing.
#::* As a work around for this I placed a text file in that directory and the restore proceeded fine.
#:E. '''Can't Do zmrestoreldap?''' - an alternative, depending on your situation and zcs version, would be to follow [[LDAP_data_import_export]] .
# This is required before running the zmrestoreoffline step below, which will error if convertd isn't running. '''As zimbra''', type:
#* '''zmconvertctl start''' 
#** '''Note:''' Skip the above step if your running ZCS 4.x or 5.x on Mac. Convertd isn't supported on Mac until ZCS 6.x - [http://bugzilla.zimbra.com/show_bug.cgi?id=29453 bug 29453]
#Double check your LDAP password [ zimbra_ldap_password ] from the old servers localconfig.xml or the /tmp/OLD-localconfig.xml if you made it in the proceeding steps to the new servers LDAP config.
#* Run the following to see the new servers zimbra_ldap_password:
#:* '''zmlocalconfig -s zimbra_ldap_password'''
#* '''If you have to change it''' , run the following below and replacing <password> with the needed password:
#:* '''zmlocalconfig -f -e zimbra_ldap_password=<password>'''
# '''Note, A Common Problem Described Here And A Change In The Order Of The Old DR Steps:''' 
#* To avoid the '''"No appenders could be found for logger (zimbra.misc) / Please initialize log4j"''' and other related problems that happened during the zmrestoreoffline. We'll start and then stop the mailboxd service:
#:* '''zmmailboxdctl start'''
#:* This could take a couple of minutes before it comes up. Monitor activity by :  '''tail -f /var/log/zimbra.log'''
#:** If you think there's an issue, did you confirm your passwords? Look for something like the following in /var/log/zimbra.log :
#:*** ''Oct 20 17:39:33 mail zimbramon[9745]: 9745:info: zmmtaconfig: gacf ERROR: service.FAILURE (system failure: unable to get config) (cause: javax.naming.AuthenticationException [LDAP: error code 49 - Invalid Credentials])''
#:*** See the section above to fix: "9. Double check your LDAP password"
#:* '''zmmailboxdctl stop'''
#:* This will configure the log4j.properties and create the necessary log files, such as mailbox.log . 
#Last check before doing system [-sys] and user data restores [-a all]
#: Before you attempt the zmrestoreoffline below; make sure mailboxd isn't running and that only the necessary services are running. (ldap, mysql.server, convertd [unless MAC w/ ZCS 4.x or 5.x]). To confirm [as zimbra user] :
#:** '''zmcontrol status''' note running and those services that aren't.
#:** '''mysql.server status'''
#:** '''ldap status'''
#:** '''zmconvertctl status'''
#To start the offline restore, we have two methods:
#:'''Option A - Traditional''' : This syntax will do a full restore from your latest FULL backup session label and then proceeds through your incremental sessions. 
#::A1. Type the following to start your restore or with the nohup options:
#::: '''Note: ''' ''Use –c on the command line so that accounts will be restored even if some accounts encounter errors during the offline restore process.''
#:::* ''' zmrestoreoffline -sys -a all -c -br'''
#::: or with the '''nohup''' option and the -t /PATH/TO/backup . Remember to adjust for your session label:
#:::* Get latest full session label you want:  '''zmrestoreldap -t /opt/<old-zimbra-dir>/backup/ -lbs | grep full'''
#:::* ''Example'' : '''cd /tmp ; nohup zmrestoreoffline -t /opt/<old-zimbra-dir>/backup/ -lb full-20100314.060050.268 -sys -a all -c -br &'''
#::A2. To watch the progress, '''tail -f /opt/zimbra/log/mailbox.log''' [if it exists] and your nohup.out file if you used the nohup command. 
#::: '''Note''': There is a current bug about the output from zmrestoreoffline showing jvm gc events, this would explain the odd log events you might be witnessing. [http://bugzilla.zimbra.com/show_bug.cgi?id=41516 bug 41516]
#::A3. Proceed to next step, skipping the '''Option B''' entry below.
#: 
#:'''Option B - Faster & More Incremental In Restore Steps''' : This syntax will do a full restore reading from the specified FULL backup session label you give it. Then you'll use zmplayredo to play 'incremental' data you have that came after that full. This is to avoid the performance issues described in [http://bugzilla.zimbra.com/show_bug.cgi?id=33606 bug 33606 : Improve zmrestore & zmrestoreoffline performance] . 
#::B1. We'll first restore using '''ONLY''' the data in the full backup session. Type:
#::: '''Note:''' ''Use –c on the command line so that accounts will be restored even if some accounts encounter errors during the offline restore process. ''
#::* '''zmrestoreoffline -sys -a all -c -rf -lb <full backup session label>'''
#::: or with the '''nohup''' option: 
#::* '''cd /tmp ; nohup zmrestoreoffline -sys -a all -c -rf -lb <full backup session label> &'''
#::B2. To watch the progress, tail '''/opt/zimbra/log/mailbox.log''' and your nohup.out file if you used the nohup command. 
#::: '''Note''': There is a current bug about the output from zmrestoreoffline showing jvm gc events, this would explain the odd log events you might be witnessing. [http://bugzilla.zimbra.com/show_bug.cgi?id=41516 bug 41516]
#::B3. Restore ldap to the last incremental you have:  zmrestoreldap -lb <latest_label>
#::B4. Now we'll restore the data from the redologs you have from your incremental sessions '''/opt/zimbra/backup/sessions/incr-''<label>''/redologs/''' that are '''after''' the full session you used. An advanced option would be to also use the redologs from '''/opt/zimbra/redolog''' & '''/opt/zimbra/redolog/archives''' directories from the old zimbra installation. This would restore data after your last backup. When replaying redologs, you play from the oldest log to the newest.
#::* '''zmplayredo --logfiles <arg> Replay these logfiles, in order''' ''As shown from --help output.''
#::** For example: <pre>zmplayredo --logfiles ./incremental-XXXXX/redolog/redo.xxxxxx.seq1.log ./incremental-XXXXX/redolog/redo.xxxxxx.seq2.log</pre>
#::*** Oldest incremental to newest. If multiple incremental logs, seq1 first and then to next number in order.
#:'''Possible Problems With The Offline Restore''' - updated Aug 29, 2012
#:: Permission error on the backup tmp directory
#::* zmlocalconfig | grep -i backup
#::* look for a var that also mentions the tmp path
#:: You have customized zmvolumes and/or had multiple volumes on your old server and the new one isn't configured the same 
#::* su - zimbra
#::* mysql -e "select * from zimbra.volume"
#:: Also note, the zmrestoreoffline options above will not restore deleted accounts. So if you see issues about not being able to play redolog events against some mailbox id's, it might be because they weren't restored. From the help, "--skipDeletedAccounts 		Do not restore if named accounts were deleted or did not exist at backup time. (This option is always enabled with -a all) "
# If your have redologs to play now that you copied from the "old" server that occurred after the last backup you have, now is the time to play them.
#: To restore the data from the redologs you have, use the redologs you copied from '''/opt/zimbra/redolog''' & '''/opt/zimbra/redolog/archives''' directories from the old zimbra installation. When replaying redologs, you play from the oldest log to the newest.
#::* '''zmplayredo --logfiles <arg> Replay these logfiles, in order''' ''As shown from --help output.''
#::** For example: <pre>zmplayredo --logfiles ./redolog/redo.xxxxxx.seq1.log ./redolog/redo.xxxxxx.seq2.log</pre>
#Because some ZCS services are running at this point and we'll want to stop them before proceeding, type:
#:* '''zmcontrol stop'''
#Now start ZCS, type :
#:* '''zmcontrol start''' 
#:* You might want to monitor the following log files in various shells/terminals:
#::* '''tail -f /var/log/zimbra.log'''
#::* '''tail -f /opt/zimbra/log/zmmailboxd.out'''
#::* '''tail -f /opt/zimbra/log/mailbox.log'''
# Check and confirm status of all zimbra services with:
#:* '''zmcontrol status'''

'''Something go wrong? See [[Ajcody-Disaster-Recovery-Specific-Notes]] for current issues being investigated or documented.'''

==<font size="3" color="#00007f" face="Arial">'''Post Restore And Confirmation Steps'''</font>==

# Here are some fairly fast and simple commands from the CLI to get a quick status of the situation. You can grep for or include a single user's account if you're confident of what it should be reported as.
##Confirm your accounts and domains are present and show ''active'', '''as zimbra''' run:
##* '''zmaccts'''
##Confirm accounts show mailbox quota usage appropriately:
##* '''zmprov gqu `zmhostname`'''
##** Columns are for : User, Quota, Used
##Confirm data is in the various zimbra volumes and all that are needed are present:
##*  '''zmvolume -l'''
##* Confirm all needed zimbra volumes are present. Some customers might have different volumes for older primaryMessage stores, older indexes, HSM, A&D and so forth.
##*  Run the following command against your volume paths and ctrl+c once your confident things look right:
##* '''ls -laR /opt/zimbra/store'''  [or some other volume path that you have]
# To test from the shell/terminal basic html client usability, you could install and run lynx.
## If you need to install lynx do the following '''as root''':
##* [rhel] '''yum install lynx'''
##* [ubuntu/debian] '''apt-get install lynx'''
## To test, do the following below. Your arrow and tab keys are used to navigate you on the page. '''As root''' , because the zimbra $HOME directory isn't writable for zimbra at the top level :
##* <pre>lynx http://<zimbra servername></pre>
# Special Items To Confirm:
## Proxy Operations
## Self-Signed and Commercial Certificates
### Commercial Certs, currently [bug 62515 is RFE to do so], are not saved as part of the backup. If you don't have a copy, then you'll have to get them again. If you do have a copy, then copy /opt/zimbra/ssl/zimbra/commercial/* from the old box and the commercial.key to that directory on the new server. Then put the commercial-ca.crt and commercial.crt in say /tmp and run [as root] /opt/zimbra/bin/zmcertmgr deploycrt comm /tmp/commercial.crt /tmp/commercial-ca.crt
## Confirm any custom configuration changes made to /opt/zimbra/conf/salocal.cf.in are still intact.
## If in the admin console, there is no "Certificates" listing under tools.
##* The zimlet needed to be reinstalled.
##* zmzimletctl install /opt/zimbra/zimlets/com_zimbra_cert_manager.zip
##* zmprov fc zimlet com_zimbra_cert_manager
## Convertd version 1 or version 2 - might need to reinitialize convert v2
##* on all mailstores
##* /opt/zimbra/convertd/bin/upgrade_v2
##* zimbra restart required afterwards
## Double check any '''postfix customizations''' you've might of had
## [[Ajcody-HSM-Notes|HSM]]
## [[Ajcody-Notes-Archive-Discovery|A&D]] [Archive and Discovery]
## Backup Method And Cron Schedule
### Review the output from [special note to the zimbraBackupTarget path and zimbraBackupMode variable]. '''As zimbra''':
###* '''zmprov gs `zmhostname` | grep -i backup'''
### Confirm what the schedule is for the backups. '''As zimbra''':
###* '''crontab -l | grep -i backup'''
### If empty, this will set up the default backup schedule that is normally set and confirm crontab for it. '''As zimbra''':
###* '''zmschedulebackup -D; crontab -l | grep backup '''
#Remove any old backup sessions because these sessions are no longer valid. Either by deleting or moving out of the way :
## To delete:
##* '''rm -rf /opt/zimbra/redolog/* /opt/zimbra/backup/*'''
## To move [make sure target location has space]:
##* '''rm -rf /opt/zimbra/redolog/*
##* '''mv /opt/zimbra/backup /TMP/PATH/'''  
##** ''Of course you'll need to adjust /TMP/PATH above for your situation.''
##* '''mkdir -p /tmp/zimbra/backup/{sessions,tmp} ; chown -R zimbra:zimbra /opt/zimbra/backup'''
#If your situation allows; you should probably get a full backup of the system as well.
#Remove the firewall rules and allow client access to the new server.
===ZCO and Outlook After A DR===

After a disaster recovery, the ZCO connector could have more data than what's on the ZCS server for that user.  This occurs because of the recovery method used, usually when you restore only to a full or incremental backup and did not or could not restore the redologs [ /opt/zimbra/redolog/ ] that were not in the backup sessions.  ZCO was not design to recover or resync data in a disaster recovery situation. It is possible, though time consuming, to recover the data in the "old" profile and resync it to the server. The steps are:

* Identify the existing profile and path by selecting Mail within the Control Panel > show Profiles > select the existing profile then click properties > Select Data Files.  Under the location, it contains the path and file name of the profile.
*** It's normally in the following location:
**** For XP :  \Documents and Settings\<username>\Local Settings\Application Data\Microsoft\Outlook
**** For 7  :  \user\<username>\Local Settings\Application Data\Microsoft\Outlook
* Create a new profile within the same location and set it as "Always use this profile"
* Start Outlook and allow the new profile to sync the data from the ZCS server
* Open Window Explore and go to the directory that contains the old profile and change the extension of the zdb file to pst.   
* Once the new profile has finished syncing, select File > Open > Open Outlook Data File and select the file that you modified to be a pst extension.  (This  will mount the old profile to the new profile as an external profile)
* You can now drag and drop missing messages to the folder within the new profile. The adding of the message to the new profile will cause the messages to  sync to the ZCS server.
* Once you've 'restored' the data into the new profile account, you can remove old profile from Mail within the Control Panel > show Profiles.

{{Article Footer|Zimbra Collaboration 8.0, 7.0, 6.0|4/12/2007}}

[[Category:Backup and Restore]]
[[Category:Pending Certification]]
[[Category:Disaster Recovery]]