Network Edition Disaster Recovery: Difference between revisions

Line 82: Line 82:
==<font size="3" color="#00007f" face="Arial">'''Restoring to the new server:'''</font>==
==<font size="3" color="#00007f" face="Arial">'''Restoring to the new server:'''</font>==


As user '''zimbra''' do the following...
#Stop the new server, '''as zimbra''' type:
#Stop the new server, type <font color="#000000">'''zmcontrol stop'''
#* '''zmcontrol stop'''
#If the old server had additional storage volumes configured, mount any additional volumes now.</font>
#If the old server had additional storage volumes configured, mount any additional volumes now.
#Delete the mysql data and re initialize an empty data directory. If you do not do this, zmrestoreoffline will have errors. As zimbra, type
#Delete the mysql data and re initialize an empty data directory. If you do not do this, zmrestoreoffline will have errors. '''As zimbra''', type:
#:* '''rm -rf /opt/zimbra/db/data/*'''
#:* '''rm -rf /opt/zimbra/db/data/*'''
#:* '''/opt/zimbra/libexec/zmmyinit'''
#:* '''/opt/zimbra/libexec/zmmyinit'''
#:* The mySQL service should now be running.
#:* The mySQL service should now be running.
# Copy the backup files from the old server or from an archive location to /'''opt/zimbra/backup'''.
# Copy the backup files from the old server or from an archive location to /'''opt/zimbra/backup'''.
#To find the LDAP session label to restore, type '''zmrestoreldap –lbs'''.
#To find the LDAP session label to restore, '''as zimbra''' type:
#To restore the LDAP, type <font color="#000000">'''zmrestoreldap -lb <latest_label>'''</font>.
#*  '''zmrestoreldap –lbs'''.
#To restore the LDAP, '''as zimbra''' type:
#* '''zmrestoreldap -lb <latest_label>'''
#:a. If you are restoring large number of accounts, you may want to run this command with '''nohup''' so that the session does not terminate.
#:a. If you are restoring large number of accounts, you may want to run this command with '''nohup''' so that the session does not terminate.
#::* <font color="#000000">'''nohup zmrestoreldap -lb <latest_label> &'''</font>
#::* <font color="#000000">'''nohup zmrestoreldap -lb <latest_label> &'''</font>
#::* Output goes to the nuhup.out file that will be in the directory you run the command in. [You'll been write permission for that directory]
#::* Output goes to the nuhup.out file that will be in the directory you run the command in. [You'll need write permission for that directory]
#:b. '''Note:''' Observe whether the LDAP server is started successfully after the restore, it must be running for the next steps.
#:b. '''Note:''' Observe whether the LDAP server is started successfully after the restore, it must be running for the next steps.
#:c. '''Note:''' The '''zmrestoreldap''' script included in ZCS 4.5.7 through ZCS 4.5.10 and ZCS 5.0 through ZCS 5.0.1 is '''broken'''.  
#:c. '''Note:''' The '''zmrestoreldap''' script included in ZCS 4.5.7 through ZCS 4.5.10 and ZCS 5.0 through ZCS 5.0.1 is '''broken'''.  
Line 104: Line 106:
#::* This is because the directory /opt/zimbra/openldap-data/ is empty and the script is trying to backup the contents to /opt/zimbra/openldap-data/.priv and failing.
#::* This is because the directory /opt/zimbra/openldap-data/ is empty and the script is trying to backup the contents to /opt/zimbra/openldap-data/.priv and failing.
#::* As a work around for this I placed a text file in that directory and the restore proceeded fine.
#::* As a work around for this I placed a text file in that directory and the restore proceeded fine.
#Type <font color="#000000">'''zmconvertctl start'''</font>. This is required before running zmrestoreoffline step below, which will error if convertd isn't running.
# This is required before running the zmrestoreoffline step below, which will error if convertd isn't running. '''As zimbra''', type:
## Skip the above step if your running ZCS 4.x or 5.x on Mac. Convertd isn't supported on Mac until ZCS 6.x - [http://bugzilla.zimbra.com/show_bug.cgi?id=29453 bug 29453]
#* '''zmconvertctl start'''
#** Skip the above step if your running ZCS 4.x or 5.x on Mac. Convertd isn't supported on Mac until ZCS 6.x - [http://bugzilla.zimbra.com/show_bug.cgi?id=29453 bug 29453]
#Sync your LDAP password from backup directory to the new production servers LDAP config.  Run :
#Sync your LDAP password from backup directory to the new production servers LDAP config.  Run :
#:a. <font color="#000000">'''zmlocalconfig -f -e zimbra_ldap_password=<password>'''</font>.
#:a. <font color="#000000">'''zmlocalconfig -f -e zimbra_ldap_password=<password>'''</font>.
#To start the offline restore, type<font color="#000000">''' zmrestoreoffline -sys -a all -c -br'''</font>.
#To start the offline restore, type:
#* ''' zmrestoreoffline -sys -a all -c -br'''
#:a. You may want to run '''nohup''' here also.  
#:a. You may want to run '''nohup''' here also.  
#::* <font color="#000000">'''nohup zmrestoreoffline -sys -a all -c -br &'''</font>
#::* <font color="#000000">'''nohup zmrestoreoffline -sys -a all -c -br &'''</font>

Revision as of 21:43, 15 March 2010

This article describes the steps to replace a failed server in a version 4.5.x network edition single-server ZCS configuration, (should be mostly applicable to version 5.x as well but this has not yet been confirmed).

Important: The ZCS release you install on the new server must be the same release as installed on the old server. The server can have a different operating system.

The new server hardware must meet the requirements described in the Installation Prerequisites section of the ZCS Single Server Installation Guide. Install the new operating system, making any necessary OS configuration modifications as described in the installation guide.

Disaster Recovery - Changing Servers

You do the following to restore to a new server:

  • Prepare the new server
  • Block client access to the old server’s IP address with firewall rules
  • Mount any volumes that were in use on the older server
  • Delete the MySQL data that is set up in the initial installation of ZCS
  • Copy the backup files to the new server
  • Run zmrestoreldap to restore the global LDAP data
  • Run zmrestoreoffline to restore account data from the backup sessions
  • Prepare and run a new backup

Old Server Status

Two scenarios for disaster recovery are the server has died and the ZCS files cannot be accessed, or ZCS is still running, but the server needs to be replaced.

If the server is not running:

  1. Block client access to the server IP address with firewall rules.
  2. Find the latest full ZCS backup session to use.

If ZCS is still running, to prepare the move to the new server:

  1. Block client access to the server’s IP address with firewall rules.
  2. Run a full backup of the old service, or if the backup is recent, run an incremental backup to get the most current incremental backup session.
  3. Run zmcontrol stop, to stop ZCS. In order to restore to the most current state, no new mail should be received after the last incremental backup has run.
  4. Change the hostname and IP address on the old server to something else. Do not turn off the server.

Preparing the New Server

Before you begin, make sure that the new server is correctly configured with the IP address and hostname and that ZCS is installed and configured with the same domain, hostname, passwords, etc. as the previous server. See the Single-Server Installation Guide for more information about preparing the server. Before you begin to install ZCS, note the information you need from the old server including: admin account name and password, spam training and non-spam training user account names, exact domain name, and the global document account name.

Installing ZCS on new server

Make sure your TIME is set right! See Time_Zones_in_ZCS#The_server_OS

  1. Ensure that the old hostname and MX DNS records resolve to the new server
  2. If the new server was used for testing in the past then there may be remnants of previous installations that need to be removed.

Warning - the following commands will completely destroy any existing Zimbra installation:
sudo -u zimbra /opt/zimbra/bin/zmcontrol stop
rpm -e zimbra-spell
rpm -e zimbra-ldap
rpm -e zimbra-mta
rpm -e zimbra-logger
rpm -e zimbra-snmp
rpm -e zimbra-apache
rpm -e zimbra-store
rpm -e zimbra-core
rm -Rf /opt/zimbra
Non-rpm based OS's can review UnInstall_Zimbra

  1. Run the zcs/install.sh and allow it to install all modules
  2. When the configuration menu appears open up a new terminal window and copy your ZCSLicense.xml file to /opt/zimbra/conf then change ownership and rights:

cp ZCSLicense.xml /opt/zimbra/conf
chown zimbra:zimbra /opt/zimbra/conf/ZCSLicense.xml
chmod 444 /opt/zimbra/conf/ZCSLicense.xml

  1. Returning to the configuration menu make sure that you configure the same domain, hostname, passwords as on the old server. During ZCS install, the following settings must be changed to match the original server settings:
a. Zimbra LDAP Server – For Domain to create – identify the same default domain as on the original server.
b. Zimbra Mailbox Server – An administrator’s account is automatically created.
  • Make sure that the account name for Admin user to create is the same name as on the original server.
  • Set the admin password the same as on the old server.
  • Set the ldap password the same as in the old server.
  • Set the Postfix user and Amavis user passwords the same as on the old server (usually identical).
  • Change the Spam training user and the Non-spam (HAM) training user account names to be the same as the spam account names on the original server.
  • Global Document Account – This account name is automatically generated. Change the Global Document Account name to be the same account name as on the original server.
c. Change any other settings on the new server to match the configuration on the original server.
d. Disable auto-backup and starting of servers after configuration in the configuration menu:
  11) Enable default backup schedule:          no                            
   r) Start servers after configuration        no
  1. Apply configuration changes, there will be one warning:

"WARNING: Document and Zimlet initialization skipped because Application Server was not configured to start."

Restoring to the new server:

  1. Stop the new server, as zimbra type:
    • zmcontrol stop
  2. If the old server had additional storage volumes configured, mount any additional volumes now.
  3. Delete the mysql data and re initialize an empty data directory. If you do not do this, zmrestoreoffline will have errors. As zimbra, type:
    • rm -rf /opt/zimbra/db/data/*
    • /opt/zimbra/libexec/zmmyinit
    • The mySQL service should now be running.
  4. Copy the backup files from the old server or from an archive location to /opt/zimbra/backup.
  5. To find the LDAP session label to restore, as zimbra type:
    • zmrestoreldap –lbs.
  6. To restore the LDAP, as zimbra type:
    • zmrestoreldap -lb <latest_label>
    a. If you are restoring large number of accounts, you may want to run this command with nohup so that the session does not terminate.
    • nohup zmrestoreldap -lb <latest_label> &
    • Output goes to the nuhup.out file that will be in the directory you run the command in. [You'll need write permission for that directory]
    b. Note: Observe whether the LDAP server is started successfully after the restore, it must be running for the next steps.
    c. Note: The zmrestoreldap script included in ZCS 4.5.7 through ZCS 4.5.10 and ZCS 5.0 through ZCS 5.0.1 is broken.
    d. Note: On zimbra 5.0.7 this failed with the error "ERROR: Failed to move existing ldap data: 256"
    • This is because the directory /opt/zimbra/openldap-data/ is empty and the script is trying to backup the contents to /opt/zimbra/openldap-data/.priv and failing.
    • As a work around for this I placed a text file in that directory and the restore proceeded fine.
  7. This is required before running the zmrestoreoffline step below, which will error if convertd isn't running. As zimbra, type:
    • zmconvertctl start
      • Skip the above step if your running ZCS 4.x or 5.x on Mac. Convertd isn't supported on Mac until ZCS 6.x - bug 29453
  8. Sync your LDAP password from backup directory to the new production servers LDAP config. Run :
    a. zmlocalconfig -f -e zimbra_ldap_password=<password>.
  9. To start the offline restore, type:
    • zmrestoreoffline -sys -a all -c -br
    a. You may want to run nohup here also.
    • nohup zmrestoreoffline -sys -a all -c -br &
    • To watch the progress, tail /opt/zimbra/log/mailbox.log and your nohup.out file.
    b. Note: Use –c on the command line so that accounts will be restored even if some accounts encounter errors during the offline restore process.
    c. If you encounter "No appenders could be found for logger (zimbra.misc) / Please initialize log4j" you may need to do a zmmailboxdctl start/stop to configure log4j.properties and/or touch log files such as mailbox.log before the zmrestoreoffline; make sure you stop mailboxd so that only the necessary services are running. (ldap, mysql.server, zmconvertdctl [unless MAC w/ ZCS 4.x or 5.x]).
  10. Because some ZCS services are running at this point and we'll want to stop them before proceeding, type:
    • zmcontrol stop
  11. Remove any old backup sessions because these sessions are no longer valid. Either by deleting or moving out of the way :
    1. To delete:
      • rm -rf /opt/zimbra/redolog/* /opt/zimbra/backup/*
    2. To move [make sure target location has space]:
      • mv /opt/zimbra/backup /TMP/PATH/
        • Of course you'll need to adjust /TMP/PATH above for your situation.
      • mkdir -p /tmp/zimbra/backup/{sessions,tmp} ; chown -R zimbra:zimbra /opt/zimbra/backup
  12. To start ZCS, type :
    • zmcontrol start
  13. Remove the firewall rules and allow client access to the new server.


Something go wrong? See Ajcody-Disaster-Recovery-Specific-Notes.

-------------------------------------------------------------------------------

Rev 1.1 731/2008

Verified Against: unknown Date Created: 4/12/2007
Article ID: https://wiki.zimbra.com/index.php?title=Network_Edition_Disaster_Recovery Date Modified: 2010-03-15



Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »

Jump to: navigation, search