5.0.x Network Edition Backup and Restore

 - This article is a Work in Progress, and may be unfinished or missing sections.

Backing up the ZCS mailbox server on a regular basis can help you quickly restore your mail service, if an unexpected crash occurs. The backup process writes a consistent snapshot of mailboxes to a designated backup directory. ZCS mailboxes can be restored from the following:

• Full backup files that contain all the information needed to restore mailboxes
• Incremental backup files that contain the LDAP data files and all the redo logs written since the last backup
• Redo logs that contain current and archived transactions processed by the mailbox server since the last incremental backup

The following figure shows the sequence of a full recovery. When a system is restored, the last full backup is restored, each incremental backup since the last backup is restored, and the archived and current redo logs are restored.

This article describes how data is backed up and restored and how to use the CLI tools to backup or restore your ZCS mailbox server. In addition, this chapter also provides information and general guidelines for disaster recovery.

Zimbra Backup Methods

Two distinct backup methods are available on ZCS.

• The standard backup method is to run a weekly full backup session and daily incremental backup sessions to back up all mailboxes daily. The standard backup method is appropriate for enterprise deployments where full backups are run during non-working days.
• The auto-grouped backup method is recommended for large ZCS environments where running a full backup of all accounts at one time would take too long.The auto-grouped backup method runs a full backup session for a different group of mailboxes at each scheduled backup. The system administrator configures the interval that backups should run and configures the number of groups that backups are made up of. ZCS then automatically backs up mailboxes in groups over the interval specified.

Standard Backup Method

A full backup process backs up all the information needed to restore mailboxes, including the LDAP directory server, database, index directory, and message directory for each mailbox.

When backing up shared messages, the backup process looks to see whether a file representing a message already exists in the backup. If it does, it flags this object as such and does not copy its content again.

An incremental backup process backs up the LDAP data and gathers all the redo logs written since the last incremental backup. If the incremental backup process finds no previous full backup for a mailbox, a full backup is performed on that mailbox.

Incremental backups move the redo logs to the backup directory. The redo logs are a journal of every activity that has taken place. They contain a full copy of all messages delivered, as well as metadata such as tags, contacts, and conversations.

These backup files can be used to restore the complete mailbox server or individual mailboxes so that account and message data is completely restored.

The LDAP directory is backed up as part of either the full or incremental backup process. All accounts, domains, servers, COS, and other data are backed up.

Each mailbox server generates redo logs that contain every transaction processed by that server. If an unexpected shutdown occurs to the server, the redo logs are used for the following:

• To ensure that no uncommitted transactions remain, the server reads the current redo log upon startup and re-executes and completes any uncommitted transactions.
• To recover data written since the last full backup in the event of a server failure.

When the server is restored, after the backed up files are fully restored, any redo logs in the archive and the current redo log in use are replayed to bring the system to the point before the failure.

Note: The Zimbra MTA is not backed up, as the data is only on the server for a very short time. Custom configurations, such as mailboxd’s server.xml, are not backed up.

Auto-Grouped Backup Method

The auto-grouped backup method is designed for very large ZCS environments where backing up all accounts can take a long time. Autogrouped backups combine full and incremental backup functions. This eliminates the need for incremental backups. Each auto-grouped session runs a full backup of the targeted group of mailboxes.

Directory Structure for Backup Files

The backup destination is known as a backup target. To the backup system, it is a path in the file system of the mail server. The Zimbra default backup directory is /opt/zimbra/backup.

The backup directory structure created by the standard backup process is shown in Figure 11. You can run regularly scheduled backups to the same target area without overwriting previous backup sessions.

The accounts.xml file lists all accounts that are in all the backups combined. For each account, this file shows the account ID, the email address, and the label of the latest full backup for that account. If you save your backup sessions to another location, you must also save the latest accounts.xml file to that location. The accounts.xml file is used to look up the latest full backup for an account during restore. If the accounts.xml file is missing you must specify the backup label to restore from.

All incremental and auto-grouped backup sessions must be saved to the same directory as all the redo logs must be found in the same backup target. Standard full backup sessions can use a different target directory.

Note: For auto-grouped backups, the directory structure saves the redo log files to the full backup session. There are no incremental backup sessions.

Backup and Restore Using the Administration Console

Many of the backup and restore procedures can be run directly from the Administration Console. In the Navigation pane, Monitoring>Backup lists each of the servers.

Standard Backup Method

You can perform the following backup and restore tasks:

• Immediately start a full or incremental backup
• Restore all accounts or specific accounts and restore to a new account or another server
• Abort a full backup that is in progress
• View the backup session labels, start time, end time, and status
• Find a specific backup

The standard backup schedule is set by default during install, you can change this schedule from the command line.

Auto-grouped Backup Method

You can only run full backups in the auto-grouped method. It is not recommended to run auto-grouped backups manually since they are scheduled from the CLI and run automatically at the scheduled times. You can perform the following backup and restore tasks:

• Configure the server to run in auto-grouped backup mode
• Find a specific backup
• Abort a backup that is in progress
• Restore all or specific accounts and restore to a new account or another server
• View backup session labels, start and end times, and the status

The auto-grouped backup schedule can only be set from the CLI using the zmschedulebackup command.

Backup and Restore Using the Command Line Interface

The Zimbra backup and restore procedures can be run as CLI commands. The following utilities are provided to create backup schedules, perform full and incremental backups, restore the mail server, or restore the LDAP server.

• zmschedulebackup. This command is used to schedule full backups, incremental backups, and deletion of old backups.
• zmbackup. This command executes full or incremental backup of the mail server. This is run on a live server, while the mailboxd process and the mailbox server are running. This command also has an option to manually delete old backups when they are no longer needed.
• zmbackupabort. This command stops a full backup that is in process.
• zmbackupabort -r. This command stops an ongoing restore.
• zmbackupquery. This command lists the information about ongoing and completed backups, including labels and dates.
• zmrestore. This command executes a full or incremental restore to the Zimbra mail server. The zmrestore command is performed on a server that is running.
• zmrestoreoffline. This command restores the Zimbra mail server when the mailboxd process is stopped.
• zmrestoreldap. This command restores the complete LDAP directory server, including accounts, domains, servers, COS and other data.

Refer to Category:Command Line Interface for usage and definitions for each of these commands.

Backing up Using the Standard Method

When you initiate a backup, you can issue the command from the same server being backed up, run the command remotely and specify the target server on the command line, or use the administration console to start a backup session.

Scheduling Backups

When ZCS was installed, the backup schedule for the standard method of full and incremental backups was added to the crontab. Under the default schedule, the full backup is scheduled for 1:00 a.m., every Saturday. The incremental backups are scheduled for 1:00 a.m., Sunday through Friday.

By default, backups older than a month are deleted every night at 12 a.m.

You can change the backup schedule using the zmschedulebackup command.

Specify the fields as follows, separate each field with a blank space:

• minute — 0 through 59
• hour — 0 through 23
• day of month — 1 through 31
• month — 1 through 12
• day of week — 0 through 7 (0 or 7 is Sunday, or use names)

Type an asterisk (*) in the fields you are not using.

Example of zmschedulebackup options

• Replace the existing full backup, incremental backup and delete backup schedule. When you use -R, the complete backup schedule is replaced. If you use this command, remember to set the delete schedule, if you want backup sessions to be scheduled for automatic deletion. This example replaces the existing schedule to have full backups run on Sunday at 1 a.m., incremental backups to run Monday through Saturday at 1 a.m., and old backups deleted at 12:00 a.m. every day.

zmschedulebackup -R f “0 1 * * 7” i “0 1 * * 1-6” d “0 0 * * *”

• Add an additional full backup time to your current schedule. This example adds a full backup on Thursday at 1 a.m.

zmschedulebackup -A f “0 1 * * 4”

• Review your backup schedule. The schedule is displayed.

zmschedulebackup -q

• Save the schedule command to a text file. This would allow you to easily recreate the same schedule after reinstall or upgrade

zmschedulebackup -s

Note: If you change the default schedule and want to return to it, enter the command zmschedulebackup -D.

The default backup schedule looks like this in the cron table:

BACKUP BEGIN :

0 1 * * 6 /opt/zimbra/bin/zmbackup -f - all
0 1* * 0-5 /opt/zimbra/bin/zmbackup -i
0 0 * * * /opt/zimbra/bin/zmbackup -del 1m

Read as follows:

• 0 1 * * * 6 /opt/zimbra/bin/zmbackup -f - all means that the full backup runs on 1 a.m. on Saturdays.
• 0 1* * 0-5 /opt/zimbra/bin/zmbackup -i means that an incremental backup runs at 1 a.m. from Sunday through Friday.
• 0 0 * * * /opt/zimbra/bin/zmbackup -del 1m means that backup sessions are deleted at midnight 1 month after they were created.

How to read the crontable

Each cron tab entry contains six fields that appear in this order:

Field

1 2 3 4 5 6

0 1 * * 6 /opt/zimbra/bin/zmbackup -f -all

1 - minute (0-59 allowed)

2 - hour (0-23)

3 - day of month (1-31)

4 - month (1-12 or names)

5 - day of week (0-7 or names allowed, with both 0 and 7 representing Sunday

6 - string to be executed

The asterisk character works as a wild card, representing every occurrence of the field’s value.

Backup Completion Email Notification

A backup report is sent to the admin mailbox when full and incremental backups are performed. This report shows the success or failure of the backup and includes information about when the backup started and ended, the number of accounts backed up and redo log sequence range.

If the backup failed, additional error information is included.

You can add additional recipient addresses or change the notification email address in the Administration Console Global Settings, Backup/Restore tab.

Full Backup Process

The full backup process goes through the following steps to backup the mailbox, the database, the indexes, and the LDAP directory:

1. Backs up the global system data including system tables and the local config.xml file.
2. Iterates through each account to be backed up and backs up the LDAP entries for those accounts.
3. Places the account’s mailbox in maintenance mode to temporarily block mail delivery and user access to that mailbox.
4. Backs up the mailbox.
   1. Creates MySQL dump for all data related to that mailbox.
   2. Backs up the message directory for that mailbox.
   3. Creates a backup of the index directory for that mailbox.
5. Returns that account’s mailbox to active mode and moves on to the next one.
6. Backs up the LDAP directory.

Full backup is usually run asynchronously. When you begin the full backup, the label of the ongoing backup process is immediately displayed. The backup continues in the background. You can use the zmbackupquery command to check the status of the running backup at any time.

Incremental Backup Process

Incremental backups are run using the CLI command, zmbackup. The process for incremental backup is as follows:

1. Backs up the global system data including system tables and the local config.xml.
2. Iterates through each account to be backed up and backs up the LDAP entries for those accounts.
3. Moves the archive redo logs, created since the last backup, to the <backup target>/redologs directory. If no full backup for this account is found, the backup process performs a full backup on this account, even if only an incremental backup was specified.
4. Backs up the LDAP directory.

Example Backup Commands

Note: -zip can be added to the command line to zip the message files during backup. Zipping these can save backup storage space.

• Perform a full backup of all mailboxes on server1

zmbackup -f -s server1.domain.com -a all

• Perform incremental backup of all mailboxes on server1 since last full backup

zmbackup -i -s server1.domain.com -a all

• Perform full backup of only user1’s mailbox on server1

zmbackup -f -s server1.domain.com -a user1@domain.com

• Delete backup sessions either by label or by date. Deleting by label deletes that session and all backup sessions before that session. Deleting by date deletes all backup session prior to the specified date. zmbackup -del 7d deletes backups older than 7 days from now. You can specify day (d), month (m), or year (y).

Finding Specific Backups

Each run of full or incremental backup creates a backup session, also known as the backup set.

The zmbackupquery command is used to find full backup sets. Each backup session is automatically labeled by date and time. For example, the label full- 20070712.155951.123 says this is a backup from July 12, 2007 at 3:59:51.123.

Note: The times set in the session label are GMT, not the local time. GMT is used rather than local time to preserve visual ordering across daylight savings transitions.

The command can be used to find the following sessions:

• A specific full backup set

zmbackupquery -lb full-20070712.155951.123

• Full backup sets since a specific date

zmbackupquery --type full --from “2007/01/01 12:45:45”

• All full backup sets in the backup directory

zmbackupquery --type full

• Best point in time to restore for an account by specifying a time window

zmbackupquery -a user1@example.com --type full --from “2007/07/05 12:01:15” --to “2007/07/12 17:01:45”

Note: If a backup session is interrupted because the server crashes during backup (not aborted), the backup session that was interrupted was saved as a temporary session. The temporary backup session can be found at <backup target>/sessions_tmp directory. You can use the rm command to delete the directory.

Aborting Full Backup In Progress

You can use the CLI command, zmbackupabort to stop a backup that is in progress. The backup is immediately stopped and becomes a partially successful backup.

But before you can abort a backup, you must know the backup session label. This label is displayed when zmbackup first starts. If you do not know the full backup label, use zmbackupquery to find the label.

Example

• Stop the backup, if you know the label name

zmbackupabort -lb full-20070712.155951.123 -s server1

• Stop the backup, if you do not know the label
  1. zmbackupquery
  2. zmbackupabort -s server1 -lb full-20070712.155951.123

Backing up Using the Auto-Grouped Method

The auto-grouped backup method is configured either from the Admin Console User Interface or from the CLI.

Configure Auto-Grouped Backup from the Admin Console UI

To set up Auto-Grouped Backup from the Admin Console, select the appropriate server under Configuration > Servers and open the server information to edit. Under the Backup/Restore tab, select Auto-Grouped as the Backup mode and save the changes.

Auto-grouped backup has to be scheduled from the CLI. Once you select the mode in the Admin Console UI, access the CLI to schedule the backup.

Configure Auto-Grouped Backup from the CLI

Set the backup method in the global configuration, and you can override the configuration on a per server basis if you do not want a server to use the autogrouped backup method.

To set up auto-grouped backup, you modify LDAP attributes using the zmprov CLI. Type the command as

zmprov mcf <ldap_attribute> <arg>

The following LDAP attributes are modified:

• zimbraBackupMode. Set it to be Auto-Grouped. The default is Standard.
• zimbraBackupAutoGroupedInterval. Set this to the interval in either days or weeks that backup sessions should run for a group. The default is 1d. Backup intervals can be 1 or more days, entered as xd (1d); or 1 or more weeks, entered as xw (1w).
• zimbraBackupAutoGroupedNumGroups. This the number of groups to spread mailboxes over. The default is 7 groups.

Scheduling Backups

The standard backup is the default and is automatically scheduled. To run the auto-grouped backup you must manually configure the backup schedule. Run zmschedulebackup -D to set the default schedule for auto-grouped backups based on your zimbraBackupAutoGroupedInterval setting.

One group is backed up each interval. The auto-grouped backup automatically adjusts for changes in the number of mailboxes on the server. Each backup session backs up the following:

• All mailboxes that have never been backed up before. These are newly provisioned mailboxes.
• All mailboxes that have not been backed within the number of scheduled backup days. For example, if backups are scheduled to run over six days, mailboxes that have not been backed up in the past 5 days are backed up.
• More mailboxes, the oldest backup first. This is done so that the daily autogrouped backup load is balanced.

Example - if you configured the auto-grouped backup interval to be daily (1d) and the number of groups to be 7, the first time auto-grouped backup runs, all accounts are backed up. After the initial backup, auto-grouped backup runs again the next day. This time accounts that have been newly provisioned and a percentage of accounts close to 1/7 of the total are backed up again, accounts with the oldest backup date are backed up first. This would continue with newly provisioned account and approximately 1/7

of accounts being backed up daily over seven days.

As with the standard backup method, when backing up shared messages, the backup process looks to see whether a file representing a message already exists in the backup. If it does, it flags this object as such and does not copy its content again.

These backup files can be used to restore the complete ZCS system or individual mailboxes so that account and message data is completely restored. Archived redo logs are moved to the backup session as part of the full backup. When the server is restored from an auto-grouped backup, redo logs are replayed to bring the system to the point before the failure.

Restoring Data

Three types of restore procedures can be run:

• The zmrestore command is used to restore the mailboxes while the ZCS mailbox server is running.
• The zmrestoreoffline is used to restore the mail server when the mail server is down. This command is run for disaster recovery.
• The zmrestoreldap is used to restore the content of the LDAP directory server.

The restore process allows all accounts or individual accounts to be specified.

Restore Process

The zmrestore process goes through the following steps to restore the mailbox, the database, the indexes, and the LDAP directory.

1. Retrieves specified accounts to be restored, or specify all for all accounts that have been backed up.
2. Iterates through each mailbox:
   1. Deletes the mailbox on the server to clear any existing data
   2. Restores the last full backup of the MySQL data, the index directory, and the message directory for that mailbox
   3. Replays redo logs in all incremental backups since last full backup
   4. Replays all archived redo logs for that mailbox, from the redo log archive area on the mailbox server
   5. Replays the current redo log

Important: Users using the ZCS Connector for Outlook must perform an initial sync on the Outlook client when they log on after the Zimbra server is restored.

Example

• Perform a full restore of all accounts on server1, including last full backup and any incremental backups since last full backup

zmrestore -a all

• Perform a single account restore on server1

zmrestore -a account@company.com

Note: A single account can be restored from the Administration Console as well.

• Restore to a specific point in time (PIT). The following restore options affect redo log replay. If you do not specify one of these options, all redo logs since the full backup you're restoring from are replayed

Important: After you perform any of the following point-in-time restores, you should immediately run a complete backup for those accounts to avoid future restore problems with those accounts.

A restore that is run using any of the following options is a point-in-time restore:

• • -restoreToTime <arg> - Replay the redo logs until the time specified.
  • -restoreToIncrLabel <arg> - Replay redo logs up to and including this incremental backup.
  • -restoreToRedoSeq <arg> - Replay up to and including this redo log sequence.
  • -br - Replays the redo logs in backup only, therefore excluding archived and current redo logs of the system.
  • -rf - Restores to the full backup only. This does not include any incremental backups at all.

• Specify an exact time, the incremental backup label, or the redo log sequence to restore to. Restore stops at the earliest possible point in time if more than one point in time restore options are specified.

zmrestore -a account@company.com-restoreToTime <arg>

Two common ways to write the <timearg> are

• • “YYYY/MM/DD hh:mm:ss”
  • YYYYMMDD.hhmmss

• Perform an incremental restore only to last full backup, excluding incremental backups since then, for all accounts

zmrestore -rf --a all

• Restore mailbox and LDAP data for an account

zmrestore -ra -a account@company.com

• Restore to a new target account. A prefix is prepended to the original account names

zmrestore -ca -a account@company.com -pre restore

The result from the above example would be an account called restoreaccount@company.com.

• Restore system tables in the database (db) and the local config

zmrestore -sys

• Include --contineOnError (-c) to the command so that the restore process continues if an error is encountered.

zmrestore -a all -c

When -c is designated, accounts that could not be restored are displayed when the restore process is complete

• To restore a specific account. Can also be used to restore deleted accounts

zmrestore -a account@company.com

• To avoid restoring accounts that were deleted

zmrestore -a account@company.com -skipDeletedAccounts

Note: In order to restore an account that was deleted, find the last full backup that included the account. Specify the backup label (--lb) to restore the account.