HOT Backup with rdiff-backup

From Zimbra :: Wiki

Jump to: navigation, search
   Article-alert.png  - This article is a community contribution and may include unsupported customizations for network edition customers.

Admin Article

Article Information

This article applies to the following ZCS versions.
  ZCS 6.0 Article  ZCS 6.0
  ZCS 5.0 Article  ZCS 5.0


Full backup vs. User Data backup

This method describes backing up user data. If you are looking for a method to fully restore the complete server at once, you'll have to search for other tools and scripts.

Tales of the backup journey

The RestAPI gives you an interface to extract parts or all of a users data that is stored on the Zimbra server. The usage is described in various articles.

However one problem with using this mechanism for a daily backup is space efficiency. Imagine having a couple of users with a 1Gb quota and their mailboxen well filled. You'd have to backup several gigabytes each day, even though most of the data has not changed at all.

Investigating into the rsync hardlink feature, inspired by dirvish and rsnapshot, one can indeed download and unpack the blob and then store it against yesterdays backup with hardlinks. The problem is: restore might be incomplete. It took me a while to figure this out. The tgz formatter uses the group of the files to store information about the item category:

-rw-r--r-- 0/folder        391 2010-04-26 16:35 Zweiter Kalender.meta
-rw-r--r-- 0/remote folder 427 2010-01-19 11:22 IMAK-Testkalender.meta
-r--r--r-- 0/message       551 2010-01-19 12:13 Inbox/Freigabe geändert.eml.meta
-r--r--r-- 0/message      3485 2010-01-19 12:13 Inbox/Freigabe geändert.eml
-rw-r--r-- 0/appointment  1469 2010-04-26 16:37 Zweiter Kalender/Termin mit Anhang.appt.meta

And there are even groups with two words (remote folder) which makes it impossible to reproduce these groups on the local system.

Another issue is the filename encoding. Depending on what locales are configured on your systems, filenames with umlauts might get messed up. Even though this can be managed by setting the right locales. But still this is something you don't have to deal with, if you don't unpack the tgz data.

So the second shot was not to unpack the tgz blob and seach for a binary diff solution. xdelta3 was one candidate and rdiff-backup another. Experimenting with rdiff-backup the observed effect was that the increments were roundabout the same size as the complete tgz blob, thus giving you no space efficiency at all. Further digging revealed, that gzip is per default not very rsync friendly.

But gzip can be trimmed to be rsync friendly by using the --rsyncable switch. So all you have to do is to re-encrunch the package while downloading the users tgz from your Zimbra server. Simply pipe it through zcat and into gzip --rsyncable.

/usr/bin/curl -s -k -u admin:password https://zimbraserver:7071/home/${ACCOUNT} | /bin/zcat | /bin/gzip -9 --rsyncable > ${TEMPDIR}/${ACCOUNT}/${ACCOUNT}.tgz
(cd ${TEMPDIR} && /usr/bin/rdiff-backup ${ACCOUNT} ${BACKUPDIR}/${ACCOUNT} )

If you have sufficient backup space, you might want to download all the users data first and then use rdiff-backup only once. If you don't have that much space, iterate over the users and "download, rdiff-backup, delete the blob, rinse, repeat".


Simply throw the desired tgz of a user at the Zimbra server as described here. Make sure you know the different meanings of the resolve parameter (resolve=[modfy|replace|reset|skip]). Briefly described here.

Drawbacks and loose ends

You are extracting the complete mailbox of a user for each backup. So this backup method is space efficient, but not bandwidth efficient. If you need remote backups, better have the rdiff-backup locally on the Zimbra server and then mirror the backup to the remote location.

Things you won't be able to backup and restore this way are the Jabber/XMPP rosters. In Zimbra 7 the internal Jabber server will be dropped anyway, therefore the jabber server migration topic will surely come up with the introduction of Zimbra 7.

If you use this backup method to migrate user data from one Zimbra server to another, you will lose the sharing infomation if you simply recreated new user accounts on the target server. If you migrated the user accounts with some LDAP magic, the shared items should be ok.

One remaining question is what happens when mails are incoming into the users account, while the data is being extracted from the Zimbra server. Is there any problem with that? If yes, then maybe a solution would be to set the account into maintenance mode before the backup. Is this possible with the RestAPI?

Verified Against: Jay Silence Date Created: 10/29/2010
Article ID: Date Modified: 06/7/2011
Personal tools