5.0.x Network Edition Backup and Restore

From Zimbra :: Wiki

Jump to: navigation, search
Admin Article

Article Information

This article applies to the following ZCS versions.
  ZCS 5.0 Article  ZCS 5.0

Backing up the ZCS mailbox server on a regular basis can help you quickly restore your mail service, if an unexpected crash occurs. The backup process writes a consistent snapshot of mailboxes to a designated backup directory. ZCS mailboxes can be restored from the following:

  • Full backup files that contain all the information needed to restore mailboxes
  • Incremental backup files that contain the LDAP data files and all the redo logs written since the last backup
  • Redo logs that contain current and archived transactions processed by the mailbox server since the last incremental backup

The following figure shows the sequence of a full recovery. When a system is restored, the last full backup is restored, each incremental backup since the last backup is restored, and the archived and current redo logs are restored.

Sample backup timeline.png

This article describes how data is backed up and restored and how to use the CLI tools to backup or restore your ZCS mailbox server. In addition, this chapter also provides information and general guidelines for disaster recovery.

Contents

Zimbra Backup Methods

Two distinct backup methods are available on ZCS.

  • The standard backup method is to run a weekly full backup session and daily incremental backup sessions to back up all mailboxes daily. The standard backup method is appropriate for enterprise deployments where full backups are run during non-working days.
  • The auto-grouped backup method is recommended for large ZCS environments where running a full backup of all accounts at one time would take too long.The auto-grouped backup method runs a full backup session for a different group of mailboxes at each scheduled backup. The system administrator configures the interval that backups should run and configures the number of groups that backups are made up of. ZCS then automatically backs up mailboxes in groups over the interval specified.

Standard Backup Method

A full backup process backs up all the information needed to restore mailboxes, including the LDAP directory server, database, index directory, and message directory for each mailbox.

When backing up shared messages, the backup process looks to see whether a file representing a message already exists in the backup. If it does, it flags this object as such and does not copy its content again.

An incremental backup process backs up the LDAP data and gathers all the redo logs written since the last incremental backup. If the incremental backup process finds no previous full backup for a mailbox, a full backup is performed on that mailbox.

Incremental backups move the redo logs to the backup directory. The redo logs are a journal of every activity that has taken place. They contain a full copy of all messages delivered, as well as metadata such as tags, contacts, and conversations.

These backup files can be used to restore the complete mailbox server or individual mailboxes so that account and message data is completely restored.

The LDAP directory is backed up as part of either the full or incremental backup process. All accounts, domains, servers, COS, and other data are backed up.

Each mailbox server generates redo logs that contain every transaction processed by that server. If an unexpected shutdown occurs to the server, the redo logs are used for the following:

  • To ensure that no uncommitted transactions remain, the server reads the current redo log upon startup and re-executes and completes any uncommitted transactions.
  • To recover data written since the last full backup in the event of a server failure.

When the server is restored, after the backed up files are fully restored, any redo logs in the archive and the current redo log in use are replayed to bring the system to the point before the failure.

Note: The Zimbra MTA is not backed up, as the data is only on the server for a very short time. Custom configurations, such as mailboxd’s server.xml, are not backed up.

Auto-Grouped Backup Method

The auto-grouped backup method is designed for very large ZCS environments where backing up all accounts can take a long time. Autogrouped backups combine full and incremental backup functions. This eliminates the need for incremental backups. Each auto-grouped session runs a full backup of the targeted group of mailboxes.

Directory Structure for Backup Files

The backup destination is known as a backup target. To the backup system, it is a path in the file system of the mail server. The Zimbra default backup directory is /opt/zimbra/backup.

The backup directory structure created by the standard backup process is shown in Figure 11. You can run regularly scheduled backups to the same target area without overwriting previous backup sessions.

The accounts.xml file lists all accounts that are in all the backups combined. For each account, this file shows the account ID, the email address, and the label of the latest full backup for that account. If you save your backup sessions to another location, you must also save the latest accounts.xml file to that location. The accounts.xml file is used to look up the latest full backup for an account during restore. If the accounts.xml file is missing you must specify the backup label to restore from.

All incremental and auto-grouped backup sessions must be saved to the same directory as all the redo logs must be found in the same backup target. Standard full backup sessions can use a different target directory.

Standard Backup directory structure.png

Note: For auto-grouped backups, the directory structure saves the redo log files to the full backup session. There are no incremental backup sessions.

Backup and Restore Using the Administration Console

Many of the backup and restore procedures can be run directly from the Administration Console. In the Navigation pane, Monitoring>Backup lists each of the servers.

Standard Backup Method

You can perform the following backup and restore tasks:

  • Immediately start a full or incremental backup
  • Restore all accounts or specific accounts and restore to a new account or another server
  • Abort a full backup that is in progress
  • View the backup session labels, start time, end time, and status
  • Find a specific backup

The standard backup schedule is set by default during install, you can change this schedule from the command line.

Auto-grouped Backup Method

You can only run full backups in the auto-grouped method. It is not recommended to run auto-grouped backups manually since they are scheduled from the CLI and run automatically at the scheduled times. You can perform the following backup and restore tasks:

  • Configure the server to run in auto-grouped backup mode
  • Find a specific backup
  • Abort a backup that is in progress
  • Restore all or specific accounts and restore to a new account or another server
  • View backup session labels, start and end times, and the status

The auto-grouped backup schedule can only be set from the CLI using the zmschedulebackup command.

Backup and Restore Using the Command Line Interface

The Zimbra backup and restore procedures can be run as CLI commands. The following utilities are provided to create backup schedules, perform full and incremental backups, restore the mail server, or restore the LDAP server.

  • zmschedulebackup. This command is used to schedule full backups, incremental backups, and deletion of old backups.
  • zmbackup. This command executes full or incremental backup of the mail server. This is run on a live server, while the mailboxd process and the mailbox server are running. This command also has an option to manually delete old backups when they are no longer needed.
  • zmbackupabort. This command stops a full backup that is in process.
  • zmbackupabort -r. This command stops an ongoing restore.
  • zmbackupquery. This command lists the information about ongoing and completed backups, including labels and dates.
  • zmrestore. This command executes a full or incremental restore to the Zimbra mail server. The zmrestore command is performed on a server that is running.
  • zmrestoreoffline. This command restores the Zimbra mail server when the mailboxd process is stopped.
  • zmrestoreldap. This command restores the complete LDAP directory server, including accounts, domains, servers, COS and other data.

Refer to Category:Command Line Interface for usage and definitions for each of these commands.

Backing up Using the Standard Method

When you initiate a backup, you can issue the command from the same server being backed up, run the command remotely and specify the target server on the command line, or use the administration console to start a backup session.

Scheduling Backups

When ZCS was installed, the backup schedule for the standard method of full and incremental backups was added to the crontab. Under the default schedule, the full backup is scheduled for 1:00 a.m., every Saturday. The incremental backups are scheduled for 1:00 a.m., Sunday through Friday.

By default, backups older than a month are deleted every night at 12 a.m.

You can change the backup schedule using the zmschedulebackup command.

Specify the fields as follows, separate each field with a blank space:

  • minute — 0 through 59
  • hour — 0 through 23
  • day of month — 1 through 31
  • month — 1 through 12
  • day of week — 0 through 7 (0 or 7 is Sunday, or use names)

Type an asterisk (*) in the fields you are not using.

Example of zmschedulebackup options

  • Replace the existing full backup, incremental backup and delete backup schedule. When you use -R, the complete backup schedule is replaced. If you use this command, remember to set the delete schedule, if you want backup sessions to be scheduled for automatic deletion. This example replaces the existing schedule to have full backups run on Sunday at 1 a.m., incremental backups to run Monday through Saturday at 1 a.m., and backups older than 1 month deleted at 12:00 a.m. every day.
zmschedulebackup -R f '0 1 * * 7' i '0 1 * * 1-6' d 1m '0 0 * * *'
  • Add an additional full backup time to your current schedule. This example adds a full backup on Thursday at 1 a.m.
zmschedulebackup -A f '0 1 * * 4'
  • Review your backup schedule. The schedule is displayed.
zmschedulebackup -q
  • Save the schedule command to a text file. This would allow you to easily recreate the same schedule after reinstall or upgrade
zmschedulebackup -s

Note: If you change the default schedule and want to return to it, enter the command zmschedulebackup -D.

The default backup schedule looks like this in the cron table:

BACKUP BEGIN :

0 1 * * 6 /opt/zimbra/bin/zmbackup -f - all
0 1* * 0-5 /opt/zimbra/bin/zmbackup -i
0 0 * * * /opt/zimbra/bin/zmbackup -del 1m

Read as follows:

  • 0 1 * * * 6 /opt/zimbra/bin/zmbackup -f - all means that the full backup runs on 1 a.m. on Saturdays.
  • 0 1* * 0-5 /opt/zimbra/bin/zmbackup -i means that an incremental backup runs at 1 a.m. from Sunday through Friday.
  • 0 0 * * * /opt/zimbra/bin/zmbackup -del 1m means that backup sessions are deleted at midnight 1 month after they were created.

How to read the crontable

Each cron tab entry contains six fields that appear in this order:

Field

1 2 3 4 5 6

0 1 * * 6 /opt/zimbra/bin/zmbackup -f -all

1 - minute (0-59 allowed)
2 - hour (0-23)
3 - day of month (1-31)
4 - month (1-12 or names)
5 - day of week (0-7 or names allowed, with both 0 and 7 representing Sunday
6 - string to be executed

The asterisk character works as a wild card, representing every occurrence of the field’s value.

Backup Completion Email Notification

A backup report is sent to the admin mailbox when full and incremental backups are performed. This report shows the success or failure of the backup and includes information about when the backup started and ended, the number of accounts backed up and redo log sequence range.

If the backup failed, additional error information is included.

You can add additional recipient addresses or change the notification email address in the Administration Console Global Settings, Backup/Restore tab.

Full Backup Process

The full backup process goes through the following steps to backup the mailbox, the database, the indexes, and the LDAP directory:

  1. Backs up the global system data including system tables and the local config.xml file.
  2. Iterates through each account to be backed up and backs up the LDAP entries for those accounts.
  3. Places the account’s mailbox in maintenance mode to temporarily block mail delivery and user access to that mailbox.
  4. Backs up the mailbox.
    1. Creates MySQL dump for all data related to that mailbox.
    2. Backs up the message directory for that mailbox.
    3. Creates a backup of the index directory for that mailbox.
  5. Returns that account’s mailbox to active mode and moves on to the next one.
  6. Backs up the LDAP directory.

Full backup is usually run asynchronously. When you begin the full backup, the label of the ongoing backup process is immediately displayed. The backup continues in the background. You can use the zmbackupquery command to check the status of the running backup at any time.

Incremental Backup Process

Incremental backups are run using the CLI command, zmbackup. The process for incremental backup is as follows:

  1. Backs up the global system data including system tables and the local config.xml.
  2. Iterates through each account to be backed up and backs up the LDAP entries for those accounts.
  3. Moves the archive redo logs, created since the last backup, to the <backup target>/redologs directory. If no full backup for this account is found, the backup process performs a full backup on this account, even if only an incremental backup was specified.
  4. Backs up the LDAP directory.

Example Backup Commands

Note: -zip can be added to the command line to zip the message files during backup. Zipping these can save backup storage space.

  • Perform a full backup of all mailboxes on server1
zmbackup -f -s server1.domain.com -a all
  • Perform incremental backup of all mailboxes on server1 since last full backup
zmbackup -i -s server1.domain.com -a all
  • Perform full backup of only user1’s mailbox on server1
zmbackup -f -s server1.domain.com -a user1@domain.com
  • Delete backup sessions either by label or by date. Deleting by label deletes that session and all backup sessions before that session. Deleting by date deletes all backup session prior to the specified date. zmbackup -del 7d deletes backups older than 7 days from now. You can specify day (d), month (m), or year (y).

Finding Specific Backups

Each run of full or incremental backup creates a backup session, also known as the backup set.

The zmbackupquery command is used to find full backup sets. Each backup session is automatically labeled by date and time. For example, the label full- 20070712.155951.123 says this is a backup from July 12, 2007 at 3:59:51.123.

Note: The times set in the session label are GMT, not the local time. GMT is used rather than local time to preserve visual ordering across daylight savings transitions.

The command can be used to find the following sessions:

  • A specific full backup set
zmbackupquery -lb full-20070712.155951.123
  • Full backup sets since a specific date
zmbackupquery --type full --from “2007/01/01 12:45:45”
  • All full backup sets in the backup directory
zmbackupquery --type full
  • Best point in time to restore for an account by specifying a time window
zmbackupquery -a user1@example.com --type full --from “2007/07/05 12:01:15” --to “2007/07/12 17:01:45”

Note: If a backup session is interrupted because the server crashes during backup (not aborted), the backup session that was interrupted was saved as a temporary session. The temporary backup session can be found at <backup target>/sessions_tmp directory. You can use the rm command to delete the directory.

Aborting Full Backup In Progress

You can use the CLI command, zmbackupabort to stop a backup that is in progress. The backup is immediately stopped and becomes a partially successful backup.

But before you can abort a backup, you must know the backup session label. This label is displayed when zmbackup first starts. If you do not know the full backup label, use zmbackupquery to find the label.

Example

  • Stop the backup, if you know the label name
zmbackupabort -lb full-20070712.155951.123 -s server1
  • Stop the backup, if you do not know the label
    1. zmbackupquery
    2. zmbackupabort -s server1 -lb full-20070712.155951.123

Backing up Using the Auto-Grouped Method

The auto-grouped backup method is configured either from the Admin Console User Interface or from the CLI.

Configure Auto-Grouped Backup from the Admin Console UI

To set up Auto-Grouped Backup from the Admin Console, select the appropriate server under Configuration > Servers and open the server information to edit. Under the Backup/Restore tab, select Auto-Grouped as the Backup mode and save the changes.

Configure Autogrouped.png

Auto-grouped backup has to be scheduled from the CLI. Once you select the mode in the Admin Console UI, access the CLI to schedule the backup.

Configure Auto-Grouped Backup from the CLI

Set the backup method in the global configuration, and you can override the configuration on a per server basis if you do not want a server to use the autogrouped backup method.

To set up auto-grouped backup, you modify LDAP attributes using the zmprov CLI. Type the command as

zmprov mcf <ldap_attribute> <arg>

The following LDAP attributes are modified:

  • zimbraBackupMode. Set it to be Auto-Grouped. The default is Standard.
  • zimbraBackupAutoGroupedInterval. Set this to the interval in either days or weeks that backup sessions should run for a group. The default is 1d. Backup intervals can be 1 or more days, entered as xd (1d); or 1 or more weeks, entered as xw (1w).
  • zimbraBackupAutoGroupedNumGroups. This the number of groups to spread mailboxes over. The default is 7 groups.

Scheduling Backups

The standard backup is the default and is automatically scheduled. To run the auto-grouped backup you must manually configure the backup schedule. Run zmschedulebackup -D to set the default schedule for auto-grouped backups based on your zimbraBackupAutoGroupedInterval setting.

One group is backed up each interval. The auto-grouped backup automatically adjusts for changes in the number of mailboxes on the server. Each backup session backs up the following:

  • All mailboxes that have never been backed up before. These are newly provisioned mailboxes.
  • All mailboxes that have not been backed within the number of scheduled backup days. For example, if backups are scheduled to run over six days, mailboxes that have not been backed up in the past 5 days are backed up.
  • More mailboxes, the oldest backup first. This is done so that the daily autogrouped backup load is balanced.
Example - if you configured the auto-grouped backup interval to be daily (1d) and the number of groups to be 7, the first time auto-grouped backup runs, all accounts are backed up. After the initial backup, auto-grouped backup runs again the next day. This time accounts that have been newly provisioned and a percentage of accounts close to 1/7 of the total are backed up again, accounts with the oldest backup date are backed up first. This would continue with newly provisioned account and approximately 1/7

of accounts being backed up daily over seven days.

As with the standard backup method, when backing up shared messages, the backup process looks to see whether a file representing a message already exists in the backup. If it does, it flags this object as such and does not copy its content again.

These backup files can be used to restore the complete ZCS system or individual mailboxes so that account and message data is completely restored. Archived redo logs are moved to the backup session as part of the full backup. When the server is restored from an auto-grouped backup, redo logs are replayed to bring the system to the point before the failure.

Restoring Data

Three types of restore procedures can be run:

  • The zmrestore command is used to restore the mailboxes while the ZCS mailbox server is running.
  • The zmrestoreoffline is used to restore the mail server when the mail server is down. This command is run for disaster recovery.
  • The zmrestoreldap is used to restore the content of the LDAP directory server.

The restore process allows all accounts or individual accounts to be specified.

Restore Process

The zmrestore process goes through the following steps to restore the mailbox, the database, the indexes, and the LDAP directory.

  1. Retrieves specified accounts to be restored, or specify all for all accounts that have been backed up.
  2. Iterates through each mailbox:
    1. Deletes the mailbox on the server to clear any existing data
    2. Restores the last full backup of the MySQL data, the index directory, and the message directory for that mailbox
    3. Replays redo logs in all incremental backups since last full backup
    4. Replays all archived redo logs for that mailbox, from the redo log archive area on the mailbox server
    5. Replays the current redo log

Important: Users using the ZCS Connector for Outlook must perform an initial sync on the Outlook client when they log on after the Zimbra server is restored.

Example

  • Perform a full restore of all accounts on server1, including last full backup and any incremental backups since last full backup
zmrestore -a all
  • Perform a single account restore on server1
zmrestore -a account@company.com
Note: A single account can be restored from the Administration Console as well.
  • Restore to a specific point in time (PIT). The following restore options affect redo log replay. If you do not specify one of these options, all redo logs since the full backup you're restoring from are replayed
Important: After you perform any of the following point-in-time restores, you should immediately run a complete backup for those accounts to avoid future restore problems with those accounts.
A restore that is run using any of the following options is a point-in-time restore:
    • -restoreToTime <arg> - Replay the redo logs until the time specified.
    • -restoreToIncrLabel <arg> - Replay redo logs up to and including this incremental backup.
    • -restoreToRedoSeq <arg> - Replay up to and including this redo log sequence.
    • -br - Replays the redo logs in backup only, therefore excluding archived and current redo logs of the system.
    • -rf - Restores to the full backup only. This does not include any incremental backups at all.
  • Specify an exact time, the incremental backup label, or the redo log sequence to restore to. Restore stops at the earliest possible point in time if more than one point in time restore options are specified.
zmrestore -a account@company.com-restoreToTime <arg>
Two common ways to write the <timearg> are
    • “YYYY/MM/DD hh:mm:ss”
    • YYYYMMDD.hhmmss
  • Perform an incremental restore only to last full backup, excluding incremental backups since then, for all accounts
zmrestore -rf --a all
  • Restore mailbox and LDAP data for an account
zmrestore -ra -a account@company.com
  • Restore to a new target account. A prefix is prepended to the original account names
zmrestore -ca -a account@company.com -pre restore
The result from the above example would be an account called restoreaccount@company.com.
  • Restore system tables in the database (db) and the local config
zmrestore -sys
  • Include --contineOnError (-c) to the command so that the restore process continues if an error is encountered.
zmrestore -a all -c
When -c is designated, accounts that could not be restored are displayed when the restore process is complete
  • To restore a specific account. Can also be used to restore deleted accounts
zmrestore -a account@company.com
  • To avoid restoring accounts that were deleted
zmrestore -a account@company.com -skipDeletedAccounts

Note: In order to restore an account that was deleted, find the last full backup that included the account. Specify the backup label (--lb) to restore the account.

Stopping a Restore Process

The zmbackupabort -r command interrupts a restore that is in process. The restore process stops after the current account finishes being restored. The command displays a message showing which accounts were not restored.

To stop the restore type:

zmbackupabort -r

Offline Restore Process

The offline restore process can only be run when the mailboxd server is not running. In general, offline restore is run under the following circumstances:

  • Certain components of the Zimbra server are corrupted, and the server cannot be started. For example, the data in LDAP or the database are corrupted.
  • A disaster requires the Zimbra software to be reinstalled on the server.

In a disaster recovery when the Zimbra software is reinstalled, if mailboxd is started before the backup files are restored, the mail server would begin to accept email messages and perform other activities, producing redo logs in the process. Since the pre-disaster data have not been restored to the server, the redo logs would be out of sequence. Once mailboxd is running, it would be too late to restore the pre-disaster data. For this reason, the offline restore must be run before the ZCS mailbox store server is started.

The offline restore process goes through the following steps.

  1. Specified accounts to be restored are retrieved. If the command-line does not specify any mailbox address, the list of all mailboxes on the specified mail host are retrieved from Zimbra LDAP directory server.
  2. Iterates through each mailbox:
    1. Deletes the mailbox on the server to clear any existing data
    2. Restores the last full backup of the MySQL data, the index directory, and the message directory for that mailbox
    3. Replays redo logs in all incremental backups since last full backup
    4. Replays all archived redo logs for that mailbox, from the redo log archive area on the mailbox server
    5. Replays the current redo log

You must start mailboxd after the offline restore is complete. From the command line, type zmcontrol startup to start mailboxd.

Example

  • Restore all accounts on server1 when mailboxd is stopped
zmrestoreoffline -a all
 or maybe better:
zmrestoreoffline -rf [last full backup label]

then do

zmplayredo 

once for all logs. The latter method has been shown to have a roughly 60x improvement over "-a all," which can be extremely slow, according to bug 33606. [1]http://bugzilla.zimbra.com/show_bug.cgi?id=33606

Restoring Individual Accounts on a Live System

Use the zmrestore command to restore one or more selected accounts. In the event that a user’s mailbox has become corrupted, you might want to restore that user from the last full and incremental backup sets.

Note': You can also restore one account at a time from Accounts on the Administration Console.

To restore using the CLI command

1. For each account to be restored, put the account into maintenance mode. From the command line, type

zmprov ma <account> zimbraAccountStatus maintenance
The maintenance mode prevents delivery of new emails during the restore. Otherwise, the emails would be overwritten during the restore process.

2. Run the zmrestore command to restore the accounts.

zmrestore -a (account@abc.com account@abc.com)

3. For each account that was restored, put the account back into active mode. From the command line, type

zmprov ma <account> zimbraAccountStatus active

Important: If an user account is restored and the COS that the account was assigned no longer exists, the default COS is assigned to the account.

Restoring the LDAP Server

In a disaster recovery where you need to restore not just one server, but the entire system, you should restore your LDAP directory server first. The zmrestoreldap command restores the global LDAP data including COS, distribution lists, etc. You can restore the complete LDAP server, which recreates the entire schema or you can restore specific accounts. You specify the session to restore. The restore command has to be run on the LDAP server being restored.

Examples

  • To find the LDAP session labels type -lbs.
zmrestoreldap -lbs
  • Restore the complete LDAP directory server
zmrestoreldap -lb full20061130135236
  • Restore LDAP data for specific accounts
zmrestoreldap -lb full20061130135236 -a tac@abc.com jane@abc.com

Disaster Recovery for Specific Situations

This section provides general guidelines for disaster recovery.

General Steps for Disaster Recovery

The sequence of events to restore your mailbox store server in a general disaster scenario involving multiple machines would be as follows:

Preparation

  1. Restore your LDAP directory server to a known good state before doing anything with the mailbox store server.
  2. Put all mailboxes into maintenance mode, to prevent mail delivery and user login while restoring the mailboxes.
  3. Stop the mailbox store server, if it is running.

Recovery

  1. Reinstall the ZCS software on the mailbox server, if necessary.
  2. Restore mailboxes.
  3. Start the Zimbra server.
  4. Put all Zimbra mailboxes back in active mode.
  5. Run a full backup of the server.

Crash Recovery Server Startup

When your system is unexpectedly stopped and then restarted, on startup, the server automatically searches the redo log for any uncommitted transactions, and replays any that it finds. Replaying the redo logs brings the system to a consistent state.

Restore the Zimbra Collaboration Suite Servers

This direction would be in the case of complete machine failure.

Important': The ZCS version you install on the new server must be the same version' as installed on the old server. The server can have a different operating system.

The new server hardware must meet the requirements described in the Installation Prerequisites section of the ZCS Single Server Installation Guide. Install the new operating system, making any necessary OS configuration modifications as described in the installation guide.

Disaster Recovery Changing Servers

You do the following to restore to a new server:

  1. Prepare the new server
  2. Block client access to the old server’s IP address with firewall rules
  3. Mount any volumes that were in use on the older server
  4. Delete the MySQL data that is set up in the initial installation of ZCS
  5. Copy the backup files to the new server
  6. Run zmrestoreldap to restore the global LDAP data
  7. Run zmrestoreoffline to restore account data from the backup sessions
  8. Prepare and run a new backup

Old Server Status

Two scenarios for disaster recovery are the server has died and the ZCS files cannot be accessed, or ZCS is still running, but the server hardware needs to be replaced.

If the server is not running:

  1. Block client access to the server IP address with firewall rules.
  2. Find the latest full ZCS backup session to use.

If ZCS is still running, to prepare the move to the new server:

  1. Block client access to the server’s IP address with firewall rules.
  2. Run a full backup of the old service, or if the backup is recent, run an incremental backup to get the most current incremental backup session.
  3. Run zmcontrol stop, to stop ZCS. In order to restore to the most current state, no new mail should be received after the last incremental backup has run.
  4. Change the hostname and IP address on the old server to something else. Do not turn off the server.

Preparing the New Server

Before you begin, make sure that the new server is correctly configured with the IP address and hostname and that ZCS is installed and configured with the same domain, hostname, passwords, etc. as the previous server. See the ZCS installation guide for more information about preparing the server. Before you begin to install ZCS, note the information you need from the old server including: admin account name and password, LDAP, Amavis, and Postfix passwords, spam training and non-spam training user account names, exact domain name, and the global document account name.

Installing ZCS on new server

Note: Make sure the computer time is set to the same time as the old server. Verify that the old hostname and MX DNS records resolve to the new server.

  1. Copy your ZCSLicense.xml file to a directory on the new server. You will not be able to complete the ZCS installation if the license is not on the new server.
  2. Run ./install.sh and follow the directions in the installation guide to install ZCS. Make sure that you configure the same domain, hostname, passwords as on the old server. During ZCS install, the following settings must be changed to match the original server settings:
    1. Zimbra LDAP Server. For Domain to create, identify the same default domain as on the old server.
    2. Zimbra Mailbox Server. An administrator’s account is automatically created.
      • Make sure that the account name for Admin user to create is the same name as on the original server.
      • Set the admin password to be the same as on the old server.
      • Set the LDAP password to be the same as on the old server.
      • Set the Postfix user and Amavis user passwords to be the same as on the old server
      • Change the Spam training user and the Non-spam (HAM) training user account names to be the same as the spam account names on the old server.
      • Global Document Account – This account name is automatically generated and is usually named wiki. If you changed this, change the Global Document Account name to be the same account name as on the original server.
    3. Change any other settings on the new server to match the configuration on the original server.
    4. In the main menu, set the default backup schedule and the automatic starting of servers after the configuration is complete to NO.


Simpler Alternative to the above:

  1. At the end of "./install.sh --platform-override", do not configure anything but quit.
  2. cp ZCSLicense.xml /opt/zimbra/conf/
  3. /opt/zimbra/libexec/zmsetup.pl -c <setup config file saved in /opt/zimbra/config.xxx of old server>

Restoring to the new server

1. Stop the new server, type zmcontrol stop.

2. If the old server had additional storage volumes configured, mount any additional volumes now.

3. Delete the mysql data and re initialize an empty data directory. If you do not do this, zmrestoreoffline will have errors. As zimbra, type

a. rm -rf /opt/zimbra/db/data/*
b. /opt/zimbra/libexec/zmmyinit
The mySQL service is now running.

4. Copy all the files in the /backup directory from the old server or from an archive location to /opt/zimbra/backup.

5. To restore the LDAP, type

zmrestoreldap -lb <latest_label>.
If you are restoring large number of accounts, you may want to run a command such as the UNIX command, nohup, so that the session does not terminate before the restore is complete.

Note: To find the LDAP session label to restore, type zmrestoreldap –lbs.

6. Because some ZCS services are running at this point, type zmconvertctl start. This is required before running zmrestoreoffline.

7. Sync your LDAP password from backup directory to the new production servers LDAP config. type

zmlocalconfig -e zimbra_server_hostname=<hostname> # To set zmhostname and prevent errors in restore
zmlocalconfig -f -e ldap_url=ldap://<hostname>:389 # Optional to ensure "ldap start" works
zmlocalconfig -f -e ldap_master_url=ldap://<hostname>:389 # Optional to ensure "ldap start" works
zmlocalconfig -f -e ldap_host=<hostname> # Optional to ensure "ldap start" works
zmlocalconfig -f -e zimbra_ldap_password=<password>.

8. To start the offline restore, type

zmhostname # To verify zmhostname and prevent errors in restore
zmrestoreoffline -sys -a all -br -lb <backup-session-label> # Use of –lb is optional, and fallsback to last backup label.

Note: You may want to run a command such as nohup or screen here also. To watch the progress, tail /opt/zimbra/log/mailbox.log.

Note: Use –c on the zmrestoreoffline so that accounts will be restored even if some accounts encounter errors during the offline restore process.

9. Because some ZCS services are running at this point, type zmcontrol stop to stop all services.

10. Optional: Only for advanced users who wants to restore till latest state, scp or rsync /opt/zimbra/redolog from old server to new server, and then run: zmplayredo --fromTime <backup-session label-time: e.g. 20090605.043403.512>

11.Remove any old backup sessions because these sessions are no longer valid. Type

rm -rf /opt/zimbra/redolog/* /opt/zimbra/backup/*

12.To start ZCS, type zmcontrol start.

13.Now run a full backup, type zmbackup -f -a all.

14.Remove the firewall rules and allow client access to the new server.

Restoring from Different Failure Scenarios

The restoration steps are similar for most server failures you may encounter. If a failure occurs, review the disaster recovery section to understand the process and then follow the steps below for the specific type of failure.

Zimbra LDAP server is corrupted

  1. Reinstall the LDAP server. See the Zimbra Installation Guide.
  2. Find the label for the LDAP session to restore. Run the zmrestoreldap -lb <label> command, with no arguments to restore all accounts, domains, servers, COS, etc. for the LDAP server.
  3. Make sure that all accounts are in active mode. From the command line, type zmprov ma zimbraAccountStatus active.

Partitions become corrupted

If any partition becomes corrupted, replace the failed disk(s), then run zmrestore, to restore the latest full and incremental backup files. The zmrestore process automatically retrieves the list of all mailboxes on the specified mail host from the backup date and iterates through each mailbox to restore the mailboxes to the last known good state.

Redo log is corrupted or unreadable

If the redo log becomes unreadable for any reason, the mailboxd service stops and cannot restart. If this happens, inspect the hardware and software to find the source of the problem before proceeding.

Without the latest redo log, the Zimbra mailbox server cannot be returned to the most current state. The Zimbra mailbox data can be restored to the latest archived redo log state. A new redo log for current transactions is created after the Zimbra mailbox server is restored.

Important: The mailboxd service must not be running and all accounts must be in maintenance mode before beginning.

1. To put all accounts into maintenance mode, from the command line, type

zmprov ma zimbraAccountStatus maintenance

2. With the mailboxd service not running, type zmrestoreoffline.

The offline restore process begins by retrieving the list of all mailboxes on the specified mail host from the backup.

The offline restore than iterates through each mailbox to:

  • Delete the mailboxes on the server
  • Restore the last full backup from the backup area
  • Restore all incremental backups for that mailbox in order, since the last full backup. This involves replaying the redo logs from the backup target area
  • Replay all archived redo logs

Since the redo log for current transactions is not available, the mailbox server is returned to the state of the last archived redo log.

3. Start mailboxd, after the offline restore is complete. From the command line, type zmcontrol startup.

4. Once the Zimbra mailbox server is up, run a full backup of the Zimbra server. The full backup must be run immediately to have the latest data backed up, as the latest redo log is not available.

Changing Local Configuration Files after Restoring Zimbra

The localconfig.xml file, located in the /opt/zimbra/conf directory, includes the core Zimbra server configuration, such as paths and passwords. This file is backed up in full and incremental backups. When you run an incremental or full restore, the backed-up version of the localconfig.xml is renamed localconfig.xml.restore and is copied to the /opt/zimbra/conf directory.

If you have made changes since the last backup, you may need to replace the localconfig.xml file with the restored copy. Compare these files, and if the .restore file has the latest local configuration data, delete the localconfig.xml file and rename the file with the .restore extension to localconfig.xml.

Using snapshots to backup and restore

You can also backup and restore your server using the snapshot feature provided by the storage layer, rather than using Zimbra’s backup and restore feature. Using snapshots, you can maintain a standby site to use if your primary site fails at some point of time and reroute users to the standby site to keep operations running.

Snapshots are taken for all volumes of data and are transferred to the standby site periodically. Data volumes that are backed up using snapshots include mysql, blobs, lucene index, and redologs.

When the primary site is down, the zmplayredo command is used

  • to bring consistency to the snapshots
  • to reapply any changes in data to minimize data loss across volumes

Example

There are four volumes of data:

  • Mysql
  • Blob
  • Lucene index
  • Redologs

Sets of snapshots are taken every hour and transferred to the remote standby site. However, all snapshots are not taken at one instant and could be a second to a minute apart from each other. Also, snapshots of redologs may be taken more frequently. The sequence of events could look like:

8:00:00 - snapshot mysql
8:00:01 - snapshot blob
8:00:02 - snapshot index
8:00:03 - snapshot redolog
8:05:00 - transfer the snapshot set to remote site completed
---
8:15:00 - snapshot redolog
8:15:05 - transfer of redolog snapshot to remote site completed
---
8:30:00 - snapshot redolog
8:30:05 - transfer of redolog snapshot to remote site completed
---
8:35:00 - primary site fails

On the remote site, there are snapshots from the 8:00 set of data as well as subsequent snapshots of the redologs. They all have to be brought together so that the most recent information is available on the standby site once users are rerouted to it.

You can now run the zmplayredo command to replay changes from 8:00:00. All data is brought forward to 8:30 and the standby site is set up and running. Data from 8:30:00 to 8:35:00 is lost but that is expected when the restore process is being carried out.


Verified Against: ZCS 5.0.10 Date Created: 10/13/2008
Article ID: http://wiki.zimbra.com/index.php?title=5.0.x_Network_Edition_Backup_and_Restore Date Modified: 03/29/2012
Personal tools