Network Edition Disaster Recovery: Difference between revisions

Line 145: Line 145:
#*  '''zmrestoreldap –lbs'''
#*  '''zmrestoreldap –lbs'''
#: or using the -t /PATH/TO/backup option
#: or using the -t /PATH/TO/backup option
#* ''Example'': '''zmrestoreldap -t /opt/zimbra-604/backup/ -lbs'''
#* ''Example'': '''zmrestoreldap -t /opt/<old-zimbra-dir>/backup/ -lbs'''
#** I see that this is my latest one:  incr-20100315.050022.428   
#** I see that this is my latest one:  incr-20100315.050022.428   
#To restore the LDAP data, run one of the zmrestoreldap commands below '''as zimbra''' :
#To restore the LDAP data, run one of the zmrestoreldap commands below '''as zimbra''' :
#:* '''zmrestoreldap -lb <latest_label>'''
#:* '''zmrestoreldap -lb <latest_label>'''
#:: or with the '''nohup''' option if you are restoring large number of accounts:
#:: or with the '''nohup''' option if you are restoring large number of accounts:
#:* ''Example'': '''cd /tmp ; nohup zmrestoreldap -t /opt/zimbra-604/backup/ -lb <latest_label> &'''
#:* ''Example'': '''cd /tmp ; nohup zmrestoreldap -t /opt/<old-zimbra-dir>/backup/ -lb <latest_label> &'''
#:** Output is redirected to the nohup.out file that is created in the directory you run the command in.  
#:** Output is redirected to the nohup.out file that is created in the directory you run the command in.  
#:** ''Note: You'll [ zimbra ] need write permission for that directory the nohup.out file is written to.''
#:** ''Note: You'll [ zimbra ] need write permission for that directory the nohup.out file is written to.''
Line 177: Line 177:
#::* ''' zmrestoreoffline -sys -a all -c -br'''
#::* ''' zmrestoreoffline -sys -a all -c -br'''
#::: or with the '''nohup''' option and the -t /PATH/TO/backup . Remember to adjust for your session label:
#::: or with the '''nohup''' option and the -t /PATH/TO/backup . Remember to adjust for your session label:
#::* Get latest full session label you want:  '''zmrestoreldap -t /opt/zimbra-604/backup/ -lbs | grep full'''
#::* Get latest full session label you want:  '''zmrestoreldap -t /opt/<old-zimbra-dir>/backup/ -lbs | grep full'''
#::* ''Example'' : '''nohup zmrestoreoffline -t /opt/zimbra-604/backup/ -lb full-20100314.060050.268 -sys -a all -c -br &'''
#::* ''Example'' : '''nohup zmrestoreoffline -t /opt/<old-zimbra-dir>/backup/ -lb full-20100314.060050.268 -sys -a all -c -br &'''
#:A2. To watch the progress, '''tail -f /opt/zimbra/log/mailbox.log''' [if it exists] and your nohup.out file if you used the nohup command.  
#:A2. To watch the progress, '''tail -f /opt/zimbra/log/mailbox.log''' [if it exists] and your nohup.out file if you used the nohup command.  
#::* '''Note, Common Problem :''' If you encounter '''"No appenders could be found for logger (zimbra.misc) / Please initialize log4j"''' log event then you should do:
#::* '''Note, Common Problem :''' If you encounter '''"No appenders could be found for logger (zimbra.misc) / Please initialize log4j"''' log event then you should do:

Revision as of 07:31, 16 March 2010

This article describes the steps to replace a failed server in a version 4.5.x network edition single-server ZCS configuration, (should be mostly applicable to version 5.x as well but this has not yet been confirmed).

Important: The ZCS release you install on the new server must be the same release as installed on the old server. The server can have a different operating system.

The new server hardware must meet the requirements described in the Installation Prerequisites section of the ZCS Single Server Installation Guide. Install the new operating system, making any necessary OS configuration modifications as described in the installation guide.

Other References

Disaster Recovery - Changing Servers

You do the following to restore to a new server:

  • Prepare the new server
  • Block client access to the old server’s IP address with firewall rules
  • Mount any volumes that were in use on the older server
  • Delete the MySQL data that is set up in the initial installation of ZCS
  • Copy the backup files to the new server
  • Run zmrestoreldap to restore the global LDAP data
  • Run zmrestoreoffline to restore account data from the backup sessions
  • Prepare and run a new backup

Old Server Status

Two scenarios for disaster recovery are the server has died and the ZCS files cannot be accessed, or ZCS is still running, but the server needs to be replaced.

If the server is not running:

  1. Block client access to the server IP address with firewall rules.
  2. Find the latest full ZCS backup session to use.

If ZCS is still running, to prepare the move to the new server:

  1. Block client access to the server’s IP address with firewall rules.
  2. Run a full backup of the old service, or if the backup is recent, run an incremental backup to get the most current incremental backup session.
  3. Run zmcontrol stop, to stop ZCS. In order to restore to the most current state, no new mail should be received after the last incremental backup has run.
  4. Change the hostname and IP address on the old server to something else. Do not turn off the server.

Preparing the New Server

Before you begin, make sure that the new server is correctly configured with the IP address and hostname and that ZCS is installed and configured with the same domain, hostname, passwords, etc. as the previous server. See the Single-Server Installation Guide for more information about preparing the server. Before you begin to install ZCS, note the information you need from the old server including: admin account name and password, spam training and non-spam training user account names, exact domain name, and the global document account name.

Installing ZCS on new server

  1. Make sure your TIME and TIMEZONE is set right! See Time_Zones_in_ZCS#The_server_OS
  2. Ensure that the old hostname and MX DNS records resolve to the new server
    • Ajcody-Hostname-DNS has the commands you can run to confirm resolution and also basic setup instructions if you need to configure BIND on this DR server for DNS resolution issues.
  3. If the new server was used for testing in the past then there may be remnants of previous installations that need to be removed.
    Warning - the following commands will completely destroy any existing Zimbra installation:
    • sudo -u zimbra /opt/zimbra/bin/zmcontrol stop
    • If you still have the tarball extracted directory available, you can do the following to uninstall zimbra, as root:
      • zcs/install.sh -u
    • If not, remove all the zimbra packages manually. As root :
      • rpm -qa zimbra*
      • Now, paste just in the package names for removal. For example:
      • rpm -e zimbra-spell zimbra-ldap zimbra-mta zimbra-logger zimbra-snmp zimbra-apache zimbra-store zimbra-core
      • And remove the old and unneeded zimbra directory:
      • rm -Rf /opt/zimbra
    Non-rpm based OS's can review UnInstall_Zimbra
  4. Have the ZCS installation software available on the new server. The tarball you used to initially installed zimbra on the old server, for example : zcs-NETWORK-6.0.4_GA_2038.RHEL5.20091214172206.tgz .
  5. Run the zcs/install.sh from the ZCS tar ball of 'the same version of ZCS that was running on your old server.
    1. Allow it to install all modules.
  6. When the configuration menu appears open up a new terminal window.
    1. License: Copy your ZCSLicense.xml file to /opt/zimbra/conf and then change ownership and rights:
      • cp ZCSLicense.xml /opt/zimbra/conf
      • chown zimbra:zimbra /opt/zimbra/conf/ZCSLicense.xml
      • chmod 444 /opt/zimbra/conf/ZCSLicense.xml
    2. Gather Variables From Old Server Config File To Use To Configure New Server During Setup:
      1. You have access to the old servers /opt/zimbra/conf directory and can copy the localconfig.xml from the old server or old location to /tmp on the new server:
        • /opt/zimbra/bin/zmlocalconfig -c /tmp/localconfig.xml -s > /tmp/OLD-localconfig.xml
      2. If you only have access to full backup sessions, you'll be able to find a localconfig.xml in the subdirectory of the full backup.
        • For example:
        • /opt/zimbra/bin/zmlocalconfig -c /opt/<old-zimbra-dir>/backup/sessions/full-20100314.060050.268/sys/localconfig.xml -s > /tmp/OLD-localconfig.xml
      3. You now have a text file [ /tmp/OLD-localconfig.xml ] that will have all the variables you need for the next part. It will also display the needed passwords for the various accounts.
        • Please delete this file once your finished with it.
  7. Returning to the configuration menu, which should be showing you 10 main categories that you can configure.
A. Select 1 for Common Configuration , you'll have the following sub-options.
  1. Confirm hostname [ zimbra_server_hostname ], ldap master host [ ldap_host ] , and timezone.
  2. Set Ldap Admin password using the information from the /tmp/OLD-localconfig.xml you made.
    • Variable that has password is : [ ldap_root_password ]
B. Select 2 for zimbra-ldap , you'll have the following sub-options.
  1. Confirm that Create Domain = Yes
  2. Confirm that Domain to create is correct. Note: Common Mistake Warning This value is your old initial domain rather than a subdomain of it. If it's wrong, revisit the above section for Common Configuration.
    • For example: mail3.zimbra.DOMAIN.com [bad] vs. zimbra.DOMAIN.com [correct]
    • A number of variables from the OLD-localconfig.xml can be double checked for this, one that probably changes the least is [ av_notify_domain ] .
  3. Set the following passwords using the variables from the OLD-localconfig.xml
    • Password Asked For || Variable Name From OLD-localconfig.xml
    • Ldap root password = ldap_root_password
    • Ldap replication password = ldap_replication_password
    • Ldap postfix password = ldap_postfix_password
    • Ldap amavis password = ldap_amavis_password
    • Ldap nginx password = ldap_nginx_password
C. Select 3 for zimbra-store, you'll have the following sub-options.
  1. Create Admin User should be yes.
  2. Double the value for Admin user to create , confirming it's not an improper subdomain.
    • Old variables to look at: [ av_notify_users ] , [ smtp_destination ] , [ smpt_source ] , [ zimbra_ldap_userdn ]
  3. Set Admin Password : [ zimbra_ldap_password ]
  4. Spam training user and the Non-spam(Ham) training users user values from the old server can be found in the ldap.bak files from your full session backups.
    • Example of path : /opt/<old-zimbra-dir>/backup/sessions/full-<some-label>/ldap/ldap.bak
    • To get values : egrep -i 'zimbraSpamIsNotSpamAccount|zimbraSpamIsSpamAccount' ldap.bak
    • Example output :
      • zimbraSpamIsNotSpamAccount: ham.ic9f_54eb7@zimbra.DOMAIN.com
      • zimbraSpamIsSpamAccount: spam.rujjnikc@zimbra.DOMAIN.com
  5. Global Documents Accounts equals the [ wiki_user ] variable plus your domain.
    • For example: [ wiki_user = wiki ] so I would set this option to wiki@zimbra.DOMAIN.com using the domain example above.
  6. SMTP host: can be confirmed by checking the ldap.bak like we did above.
    • Example of path : /opt/zimbra/backup/sessions/full-<some-label>/ldap/ldap.bak
    • grep zimbraSmtpHostname ldap.bak
  7. Adjust the remaining variables if you need to, most customer will not. Exception might be those using the proxy services and needing to enable the Instant Messaging Feature.
D. Change any other settings on the new server to match the configuration on the original server.
E. Disable auto-backup and starting of servers after configuration in the configuration menu:
  • Enable default backup schedule: no
  • Start servers after configuration no
F. Apply configuration changes, there will be one warning that says:
  • "WARNING: Document and Zimlet initialization skipped because Application Server was not configured to start."

Restoring to the new server:

  1. Confirm the new server isn't running zimbra. It shouldn't be if you selected the right option above during the installation. As zimbra type:
    • zmcontrol stop
  2. If the old server had additional storage volumes configured, mount any additional volumes now.
    • Note: Need to find various ways customers can track down the old information about volumes and include it here.
  3. Delete the mysql data and re initialize an empty data directory. If you do not do this, zmrestoreoffline will have errors. As zimbra, type:
    • rm -rf /opt/zimbra/db/data/*
    • /opt/zimbra/libexec/zmmyinit
    The MySQL service should now be running, a simple check would be:
    • ps -ef | grep mysql
  4. Copy the backup files from the old server or from an archive location to /opt/zimbra/backup or use the -t /PATH/TO/backup option for the zmrestore* commands below.
  5. To find the LDAP session label to restore, as zimbra type:
    • zmrestoreldap –lbs
    or using the -t /PATH/TO/backup option
    • Example: zmrestoreldap -t /opt/<old-zimbra-dir>/backup/ -lbs
      • I see that this is my latest one: incr-20100315.050022.428
  6. To restore the LDAP data, run one of the zmrestoreldap commands below as zimbra :
    • zmrestoreldap -lb <latest_label>
    or with the nohup option if you are restoring large number of accounts:
    • Example: cd /tmp ; nohup zmrestoreldap -t /opt/<old-zimbra-dir>/backup/ -lb <latest_label> &
      • Output is redirected to the nohup.out file that is created in the directory you run the command in.
      • Note: You'll [ zimbra ] need write permission for that directory the nohup.out file is written to.
    A. Note: Observe whether the LDAP server has started successfully after the restore, it must be running for the next steps. Simple checks are:
    • Confirm output to the screen ended with: "Starting ldap...Started slapd: pid ####"'
    • Confirm ldap process is still running: ps -ef | grep ldap
    B. Note: The zmrestoreldap script included in ZCS 4.5.7 through ZCS 4.5.10 and ZCS 5.0 through ZCS 5.0.1 is broken.
    D. Note: On zimbra 5.0.7 this failed with the error "ERROR: Failed to move existing ldap data: 256"
    • This is because the directory /opt/zimbra/openldap-data/ is empty and the script is trying to backup the contents to /opt/zimbra/openldap-data/.priv and failing.
    • As a work around for this I placed a text file in that directory and the restore proceeded fine.
  7. This is required before running the zmrestoreoffline step below, which will error if convertd isn't running. As zimbra, type:
    • zmconvertctl start
      • Note: Skip the above step if your running ZCS 4.x or 5.x on Mac. Convertd isn't supported on Mac until ZCS 6.x - bug 29453
  8. Double check your LDAP password [ zimbra_ldap_password ] from the old servers localconfig.xml to the new servers LDAP config.
    • Run the following to see the new servers zimbra_ldap_password:
    • zmlocalconfig -s zimbra_ldap_password
    • If you have to change it , run the following below and replacing <password> with the needed password:
    • zmlocalconfig -f -e zimbra_ldap_password=<password>
  9. To start the offline restore, we have two methods:
    Option A - Traditional : This syntax will do a full restore from your latest FULL backup session label and then proceeds through your incremental sessions.
    A1. Type the following to start your restore or with the nohup options:
    Note: Use –c on the command line so that accounts will be restored even if some accounts encounter errors during the offline restore process.
    • zmrestoreoffline -sys -a all -c -br
    or with the nohup option and the -t /PATH/TO/backup . Remember to adjust for your session label:
    • Get latest full session label you want: zmrestoreldap -t /opt/<old-zimbra-dir>/backup/ -lbs | grep full
    • Example : nohup zmrestoreoffline -t /opt/<old-zimbra-dir>/backup/ -lb full-20100314.060050.268 -sys -a all -c -br &
    A2. To watch the progress, tail -f /opt/zimbra/log/mailbox.log [if it exists] and your nohup.out file if you used the nohup command.
    • Note, Common Problem : If you encounter "No appenders could be found for logger (zimbra.misc) / Please initialize log4j" log event then you should do:
    • zmmailboxdctl start
    • This could take a couple of minutes before it comes up. Monitor activity by : tail -f /var/log/zimbra.log
    • zmmailboxdctl stop
    • This will configure the log4j.properties and create the necessary log files, such as mailbox.log . Before you attempt the zmrestoreoffline again; make sure mailboxd isn't running and that only the necessary services are running. (ldap, mysql.server, zmconvertdctl [unless MAC w/ ZCS 4.x or 5.x]). A zmcontrol status should give you enough information to confirm.
    • If you ran into this problem, you can now rerun your zmrestoreoffline now.
    A2. Proceed to next step, skipping the Option B entry below.
    Option B - Faster & More Incremental In Restore Steps : This syntax will do a full restore reading from the specified FULL backup session label you give it. Then you'll use zmplayredo to play 'incremental' data you have that came after that full. This is to avoid the performance issues described in bug 33606 : Improve zmrestore & zmrestoreoffline performance .
    Note - WORK IN PROGRESS NOW FOR OPTION B : Mar 15,2010 - AJCODY
    B1. We'll first restore using ONLY the data in the full backup session. Type:
    Note: Use –c on the command line so that accounts will be restored even if some accounts encounter errors during the offline restore process.
    • zmrestoreoffline -sys -a all -c -rf -lb <full backup session label>
    or with the nohup option:
    • nohup zmrestoreoffline -sys -a all -c -rf -lb <full backup session label> &
    B2. To watch the progress, tail /opt/zimbra/log/mailbox.log and your nohup.out file if you used the nohup command.
    • If you encounter "No appenders could be found for logger (zimbra.misc) / Please initialize log4j" you may need to do a zmmailboxdctl start/stop to configure log4j.properties and/or touch log files such as mailbox.log before the zmrestoreoffline; make sure you stop mailboxd so that only the necessary services are running. (ldap, mysql.server, zmconvertdctl [unless MAC w/ ZCS 4.x or 5.x]).
    B3. Now we'll restore the data from the redologs you have from your incremental sessions /opt/zimbra/backup/sessions/incr-<label>/redologs/ that are after the full session you used. An advanced option would be to also use the redologs from /opt/zimbra/redolog & /opt/zimbra/redolog/archives directories from the old zimbra installation. This would restore data after your last backup. When replaying redologs, you play from the oldest log to the newest.
    • zmplayredo -logfiles <arg> Replay these logfiles, in order As shown from --help output.
  10. Because some ZCS services are running at this point and we'll want to stop them before proceeding, type:
    • zmcontrol stop
  11. Now start ZCS, type :
    • zmcontrol start
    • You might want to monitor the following log files in various shells/terminals:
    • tail -f /var/log/zimbra.log
    • tail -f /opt/zimbra/log/zmmailboxd.out
    • tail -f /opt/zimbra/log/mailbox.log
  12. Check and confirm status of all zimbra services with:
    • zmcontrol status

Something go wrong? See Ajcody-Disaster-Recovery-Specific-Notes for current issues being investigated or documented.

Post Restore And Confirmation Steps:

  1. Here are some fairly fast and simply commands form the CLI to get a quick status of the situation. You can grep for or include a single users account if your confident of what it should be reported as.
    1. Confirm your accounts and domains are present and show active, as zimbra run:
      • zmaccts
    2. Confirm accounts show mailbox quota usage appropriately:
      • zmprov gqu `zmhostname`
        • Columns are for : User, Quota, Used
    3. Confirm data is the various volumes and all that are needed are present:
      • zmvolume -l
      • Confirm all needed volumes are present. Some customers might have different volumes for older primaryMessage stores, older indexes, HSM, A&D and so forth.
      • Run the following command against your volume paths and ctrl+c once your confident things look right:
      • ls -laR /opt/zimbra/store [or some other volume path that you have]
  2. To test from the shell/terminal basic html client usability, you could install and run lynx.
    1. If you need to install lynx do the following as root:
      • [rhel] yum install lynx
      • [ubuntu/debian] apt-get install lynx
    2. To test, do the following below. Your arrow and tab keys are used to navigate you on the page :
      • lynx http://<zimbra servername>'''
  3. Special Items To Confirm:
    1. Proxy Operations
    2. Self-Signed and Commercial Certificates
    3. HSM
    4. A&D [Archive and Discovery]
    5. Mobile Phones and BES
    6. Backup Method And Cron Schedule
      1. Review the output from [special note to the zimbraBackupTarget path and zimbraBackupMode variable]. As zimbra:
        • zmprov gs `zmhostname` | grep -i backup
      2. Confirm what the schedule is for the backups. As zimbra:
        • crontab -l | grep -i backup
      3. If empty, this will set up the default backup schedule that is normally set and confirm crontab for it. As zimbra:
        • zmschedulebackup -D; crontab -l | grep backup
  4. Remove any old backup sessions because these sessions are no longer valid. Either by deleting or moving out of the way :
    1. To delete:
      • rm -rf /opt/zimbra/redolog/* /opt/zimbra/backup/*
    2. To move [make sure target location has space]:
      • rm -rf /opt/zimbra/redolog/*
      • mv /opt/zimbra/backup /TMP/PATH/
        • Of course you'll need to adjust /TMP/PATH above for your situation.
      • mkdir -p /tmp/zimbra/backup/{sessions,tmp} ; chown -R zimbra:zimbra /opt/zimbra/backup
  5. Remove the firewall rules and allow client access to the new server.


-------------------------------------------------------------------------------

Rev 1.1 731/2008

Verified Against: 6.0.4 using Option B [3/16/2010] Date Created: 4/12/2007
Article ID: https://wiki.zimbra.com/index.php?title=Network_Edition_Disaster_Recovery Date Modified: 2010-03-16



Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »

Jump to: navigation, search