Zimbra Next Generation Modules/Zimbra NG Backup/How Backup NG Works
Zimbra 8.8 introduces a new Backup module called Backup Next Generation - or Backup NG for short - which is an entirely different system than the one available in Zimbra up until 8.7 which is now called "Legacy Backup". This document will explain the similarities and differences between the two systems and provide an organised introduction to Backup NG.
Similarities between Legacy Backup and Backup NG
Both backup systems are entirely integrated in Zimbra, and can be controlled either via the Zimbra Administration Console or through a dedicated CLI.
Differences between Legacy Backup and Backup NG
While the Legacy Backup was designed as a series of scheduled operations safeguarding several different Zimbra components, Backup NG is designed as a high-level consistence based backup system which relies on a realtime backup engine called "RealTime Scanner" to safeguard your data by putting it into a dedicated and self-consistent directory called "Backup Path" (by default,
/opt/zimbra/backup/ng/). Backup NG still relies on some scheduled operations for data safety and configuration backups, detailed below, but does not rely on scheduling for backing up data itself.
This means that with Backup NG, all items and their changes are backed up at the exact moment those happen while configurations are backed up periodically during the daily checks and cleanup maintenance operations. Not relying on a Full/Incrementals cycle, its retention is managed directly by the admin who can choose for how long to keep deleted items and accounts in the backup (default: 30 days).
Another substantial difference is that Backup NG does not rely on Maintenance Mode anymore, always allowing user access during any backup and restore operation.
Finally, Backup NG includes 6 built-in restore modes which allow a broader range of restore options - from single items to complex Disaster Recovery scenarios.
On an administrative point of view, cron backup scheduling and grouped backups are a thing of the past: on top of the RealTime Scanner mentioned above, Backup NG features its own scheduling engine and works at mailboxd level, meaning that on a Multiserver environment each mailbox server will be running its own independent instance of the backup module.
On a user's point of view, with Backup NG service downtime given to users is limited to a bare minimum and only happens in *some* Disaster Recovery scenarios: in any other case, users will always have access to their mailbox even if a backup/restore operation is running on their very own mailbox.
How Backup NG works
The workflow of Backup NG is pretty simple, and can be easily recapped as follows:
- The RealTime Scanner constantly watches for new items and for item changes, backing up anything it detects into the Backup Path.
- For each item, Backup NG saves both the BLOB data and the related Item Metadata: when a new item is created, a new BLOB and a new Metadata file are created, while when an item is updated the relevant Metadata file is updated allowing to retain the entire history of an item from deletion to creation.
- Daily, the SmartScan checks the coherency and the consistency of the backup and then backs up settings, customizations and LDAP data.
- The SmartScan is also used for other tasks, which are detailed in the dedicated section below.
- Weekly, the Backup Purge cleans up the Backup Path removing any deleted item that is out of the defined Backup Retention period.
- *Only deleted items* are purged. Items existing in the system will always exist in the backup.
The Backup Path
The Backup Path is a directory containing all of Backup NG's backup information and data. It's compressed and deduplicated by default to ensure optimised disk space usage, with an average size of 50-60% of the nominal quota of all items on the system (roughly 1/3rd of the disk space used by the Legacy Backup).
Any directory can be used as a Backup Path, as long as:
- It has a mountpoint on the server.
- It's stored in a case-sensitive filesystem.
Backup NG does not use hardlinks.
Sample Backup Path content
The most common content of a Backup Path is the following:
tree /opt/zimbra/backup/ng/ -L 1 /opt/zimbra/backup/ng/ ├── accounts ├── items ├── map_[server_id] └── [server]
As you can see, the first level of the folder tree contains 3 main directories:
- accounts - contains all the accounts' data and informations (Metadata)
- items - contains all items BLOBs (Data)
- server - contains server configurations
The map_[server_id] is created when importing accounts with the External Import feature, and contains the mapping between the account IDs in the Old and Current server. Using this file's name as an argument for the
zxsuite backup doFixShares will fix the consistency of any share broken during the import procedure.
WARNING: being self-consistent and self-coherent, manually adding/removing/editing files in the Backup Path has a high chance of making it invalid. Data restores from an invalid or tampered backup path are not guaranteed to work.
Components of Backup NG
The Core is the beating heart of Backup NG: it takes care to run the RealTime Scanner, manage operations and the operation queue, run scheduled operations and perform every action related to this module.
One instance of the Core is running on each mailbox server, thus:
- Backup NG is only functional when the Mailbox service is running.
- In a multistore environment, every mailbox server has its own independent backup.
Backup NG can be managed via the Zimbra Administration Console, in the new "Network NG" section.
Backup NG has its own CLI toolset, which can be accessed by running
- Running this command without any further argument will display all available operations along with a small description for each one.
- The help system can be accessed by running
zxsuite help backup [operation name].
Backup NG Operations
Main backup Operations
The Real Time Scanner is the big innovation in Backup NG. Each event on the system is recorded live to Zimbra’s RedoLog and saved by Backup NG, which means that is always possible to rollback an account to a previous state. Thanks to the Real Time Scanner all the restore modes work with a split-second precision.
It reads all the events of the mail server almost real-time by following the flow of informations provided by the RedoLog, then it 'replicates' the same operations on it’s own data structure creating items or updating their metadata. No information is ever overwritten in the backup, so that every item has its own complete history.
The RealTime Scanner is enabled by default right after the backup's initialization, and while it's possible to disable it both via the Admin Console and the CLI, there are very few scenarios where this is needed: even if the system is under heavy load, the RealTime Scanner's impact on the system's resources is neglectible.
Secondary backup Operations
The External Backup creates a snapshot of all current items, mailboxes and configs in the mail system ready to be used for a migration or disaster recovery. Exported data is deduplicated and compressed to optimize disk utilization, transfer times and I/O rates.
Due to the self-consistent nature of Backup NG's Backup Path, taking External Backups is not extremely common other than when doing small migrations where Backup NG is not initialized in the source server, as in any other case a simple rsync of the Backup Path will give the same results with an additional layer of flexibility since External Restores cannot be changed in any way while a Backup Path can be rsync'ed many times over the same destination directory to keep it up to date.
This section provides a quick description of the Restore Modes available in Backup NG, for more specific information and usage information please refer to the Zimbra Admin Guide or to the CLI help.
This CLI-only restore mode restores a single item from the backup to the owner’s account. Any type of item can be restored this way.
The Undelete Restore allows an administrator to restore all items deleted from a mailbox in a period of time and and put them in a dedicated Zimbra folder inside the mailbox itself.
During an Undelete Restore the Backup NG engine searches the backup datastore for items flagged as DELETED and restores them in a dedicated folder in the selected Mailbox.
WARNING: In order to allow to deal with IMAP-deleted emails in a more comfortable way for the user: the deleted IMAP flag will now be stripped from any restored item so that the item itself is visible in the Zimbra WebClient.
Restore on New Account
The Restore on New Account procedure allows you to restore a the contents and preferences of a mailbox as it was in a moment in time, in a completely new account. The source account is not changed in any way, so that is possible to recover one or more deleted items in a user’s account without actually rolling back the whole mailbox. When you run this kind of restore you can choose to hide the newly created account from the GAL as a security measure.
When a Restore on New Account starts a new account is created (the Destination Account) and all the items existing in the Source Account at the moment selected are recreated in the Destination Account, including the folder structure and all the user’s data. All restored items will be created in the Current Primary Store unless the "Obey HSM Policy" box is checked.
Restore Deleted Account
The Restore Deleted Account procedure allows you to restore a the contents and preferences of a mailbox as it was when said mailbox was deleted, in a completely new account.
When a Restore Deleted Account starts a new account is created (the Destination Account) and all the items existing in the Source Account at the moment of the deletion are recreated in the Destination Account, including the folder structure and all the user’s data. All restored items will be created in the Current Primary Store unless the "Obey HSM Policy" box is checked.
The External Restore adds to the current Zimbra server all the COS, domains, users and items stored in a Backup Path or in an External Backup.
This restore method is used for Disaster Recovery and for Migrations, and has little-to-no use in daily backup usage.
This advanced CLI-only restore mode is used in case of corruption complete loss of a volume in order to restore all missing and broken BLOBs.
The Smart Scan is the main coherency check for the health of your backup system. It’s Smart because it operates only on accounts modified since the last SmartScan, hence improving system performance and decreasing scan time exponentially.
By default a SmartScan is scheduled to be executed each night (if Scan Operation Scheduling is enabled in the Backup NG section of the Administration Zimlet). Once a week, on a day set by the user, a Purge is executed together with the SmartScan to clear Backup NG’s datastore from any deleted item that exceeded the retention period.
How does it work?
The SmartScan scans through all the items on the Zimbra Datastore looking for items modified after the last SmartScan updating any outdated entry and creating any item not yet present in the backup while flagging as deleted any item found in the backup and not in the Zimbra Datastore.
Finally, it updates all Configuration metadata in the backup, so that Domains, Accounts, COSs, LDAP, Global and Server Configurations are backed up.
When is a Smart Scan executed?
- To initialise Backup NG (called "Backup Initialisation")
- When the Backup NG module is started.
- Daily, if the Scan Operation Scheduling is enabled in the Administration Zimlet.
- When the Real Time Scanner is re-enabled via the Administration Zimlet after being previously disabled.
The Backup Purge is a cleanup operation that removes from the Backup Path any deleted item which exceeded the retention time defined by the Data Retention Policy, scanning through the metadata of all deleted items, and removing any item whose last update (deletion) timestamp is higher than the retention time.
Should an item BLOB still be referenced by one or more valid metadata files - due to Backup NG’s built-in deduplication - the BLOB itself will not be deleted.
When is a Backup Purge executed?
- Weekly, if the Scan Operation Scheduling is enabled in the Administration Zimlet.
- When manually started either via the Administration Console or the CLI.
The Coherency Check is a CLI-only feature which performs a deeper check of a Backup Path than the one done by the SmartScan.
While the SmartScan works incrementally, by only checking items which have been modified since the last SmartScan, the Coherency Check performs a throughout check of all metadata and BLOBs in the backup path, and it’s specifically designed to detect corrupted metadata and BLOBs.
What is backed up
Backup NG's RealTime Scanner and External Backup will safeguard:
- Global Config
- Server Config
- Postfix Customizations
- Classes of Service
- User Accounts
- User settings and preferences
- Items (folders, emails, contacts, appointments, tasks, briefcase files).
What is restored
All restore modes will restore the appropriate data among the following:
- Classes of Service
- User Accounts
- User settings and preferences
What is not restored
No restore mode will ever automatically restore the following:
- Global Config
- Server Config
- Postfix Customizations
Backed up Global Config, Server Config and Postfix Customizations can be viewed and searched through by using the
zxsuite backup getServerConfig command.
Taking additional, offsite copies of your backup
Backup NG's Backup Path stores its data according to the ACID paradigm:
- Atomicity: any transaction is committed and written to the disk only when completed.
- Consistency: any commited transaction is valid and no invalid transaction will be committed and written to the disk.
- Isolation: all transactions are executed sequentially so that no more than 1 transaction can affect the same item at once.
- Durability: once a transaction is committed, it will stay so even in case of a crash (e.g. power loss or hardware failure).
Due to this, it’s very easy to make a backup of it. The best (and easiest) way to do so is by using rsync. Specific options and parameters depend on many factors, such as the amount of data to be synced and the storage in use, while connecting to an rsync daemon instead of using a remote shell as a transport as it’s usually much faster in transferring the data.
You won’t need to stop neither Zimbra nor the Real Time Scanner in order to make an additional backup of Backup NG’s datastore using rsync, and you will be always able to stop the sync at any time and reprise it afterwards if needed.
Storing your Backup NG’s datastore backup offsite
As seen in the previous paragraph, making a backup of Backup NG’s Datastore is very easy, and the use of rsync makes it just as easy to store your backup in a remote location.
In order to optimize your backup strategy when dealing to this kind of setup, the following best practices might help a lot:
- If you schedule your rsync backups, make sure that you leave enough time between an rsync instance and the next one in order for the transfer to be completed.
- Use the --delete options so that files that have been deleted in the source server are deleted in the destination one to avoid inconsistencies.
- If you notice that using the --delete option takes too much time, schedule two different rsync instances: one with the --delete to be ran after the weekly Purge and one without such option.
- Make sure you transfer the whole folder tree recursively starting from Backup NG’s Backup Path. This includes Server Config backups and mapfiles.
- Make sure the destination filesystem is case sensitive (just as Backup NG’s Backup Path must be).
If you plan to restore directly from the remote location, make sure that the zimbra user on your server has read and write permissions on the transferred data and expect to experience slowlynesses if your transfer speed is much higher than your storage throughtput (or vice versa).