Troubleshooting Course Content Rough Drafts-Recover Missing Data - User: Difference between revisions

mNo edit summary
 
(23 intermediate revisions by 2 users not shown)
Line 1: Line 1:
==Need To Do==
__FORCETOC__


* review training materials for sys admin course
==Bug & RFE's I Made While Researching This Draft Write Up [Remove For Course]==
* determine what should be 'video' and how it can all be built upon preceding steps in a hands on training manner
 
==Bug & RFE's I Made While Researching This Draft Write Up==


* [story] Ability to search data within backup and do "item" restores or identify locations of search results
* [story] Ability to search data within backup and do "item" restores or identify locations of search results
Line 39: Line 36:
* ''Remember that this won't be obvious for all audience members.''
* ''Remember that this won't be obvious for all audience members.''


==General Overview Of Backup Environment==
==User Issues==


===Backup Directory Partition & Filesystem===
===Questions To Ask & Address Prior To Doing A User Restore===


====Confirm You Are Using Support Partition & Filesystem====
* Can data be restored from the user's dumpster [if enabled] ?
* Is there a copy of the data in a third party client ?  [POP email client that downloaded it, for example]
* Does the data maybe exist in another user's account ? [For example, the senders account or others in the To/CC]
* Does the data reside in a redolog/s ?
* Finally, restore from backups ?
** Do I need to overwrite the existing account with the restore or should I restore to a new account name?
*** Using the option -ca [create account] and -pre restore_ [prefix to the new account name, restore_user@ ]
** What are your retention policies and how it might determine the backup set you need?


It's recommended to have a dedicated partition for backups and to use only a supported partition/filesystem type:
===User Restores===


* No CIFS
* Admin Console
* No unsupported 'cloud' based computers or filesystems - the various Amazon offerings, Google Computing, etc.
** Restore/Backup
** [[Ajcody-Notes-ServerPlanning#Amazon_S3_-_And_Amazon_EC2_Information]]
** Redirected restore : -ca -pre restored_
** [[Ajcody-Notes-ServerPlanning#Google_Computing]]
** View Mail > TGZ export/import
* NFS has limited support
* Crossmailboxsearch
** [[Ajcody-Backup-Restore-Issues#NFS_Use_For_Backups]]
* Importing Restored Data Into Parent Account


====NFS Use For Backups====
====Restoring A User To A Specific Full Backup, Incremental Backup, Or Point In Time====


Please see this RFE:
One of the more common support issues around restoring a user account is the complaint that the restore didn't roll back the account to the desired state. Almost always, this is because the proper options were not passed to the zmrestore command for the restore to stop at the desired time frame. If you don't give the proper options to zmrestore, it will restore the account all the way to the most recent redolog operations on the server. For example, it replays the delete operations that caused the user to request to restore in the first place.


* "Need clarity on supporting nfs mounted zimbra directories - report error/msg if nfs mount is present"
You '''MUST''' use the -lb [full backup label name] option when your trying to restore anything that ISN'T meant to include the latest information for the mailbox. '''''The -lb argument should specify a full backup that took place prior to the time of the backup you wish to restore.'''''
** http://bugzilla.zimbra.com/show_bug.cgi?id=33221


Statement that was included in the release notes following the RFE:
=====Find Out What Backup Session Labels You Need First=====


: ZCS & NFS:
To find out what backups are associated with a particular account, you would do the following :
: Zimbra will support customers that store backups (e.g. /opt/zimbra/backup) on an NFS-mounted partition. Please note that this does not relieve the customer from the responsibility of providing a storage system with a performance level appropriate to their desired backup and restore times. In our experience, network-based storage access is more likely to encounter latency or disconnects than is equivalent storage attached by fiber channel or direct SCSI.


:Zimbra continues to view NFS storage of other parts of the system as unsupported. Our testing has shown poor read and write performance for small files over NFS implementations, and as such we view it unlikely that this policy will change for the index and database stores. We will continue to evaluate support for NFS for the message store as customer demand warrants.
  zmbackupquery -a user@domain


:When working with Zimbra Support on related issues, the customer must please disclose that the backup storage used is NFS.
You'll want to note what is the first full that occurs before the point in time you want to restore. And then the incremental that follows right after your point in time.


=====Things To Check When Using NFS=====
Backup labels (-lb) for fulls for a particular user can be found by doing :


* Check the /var/log/messages on both the zimbra server and the nfs server for nfs related errors during the time frame of your backup.
  zmbackupquery -a user@domain --type full
* Check /opt/zimbra/log/mailbox.log for error messages about folders/files not being able to be written or missing directory errors.
* Is root_squash configured on the nfs server? If it's changed to no_root_squash , does the behavior of the backup change?
* Is the */backup directory owned by zimbra:zimbra with at least 750 or 755 permissions?
** This parent directory as given in:
*** zmprov gs `zmhostname` zimbraBackupTarget
* Does zimbraBackupTarget have at least the subdirectories of : sessions and tmp  : and are owned by zimbra:zimbra with 750 or 755 permissions?
** If not, try manually creating them and then running a test backup.
* '''IF USING A NAS - MAKE SURE YOUR NOT USING EXTENDED ACLS OR THAT YOU HAVE THEM CONFIGURED PROPERLY'''
** If your backup session directory shows a + sign in the description section like the example below, it's a sign your using extend acls.
*** <pre> drwxrwx---+  2 zimbra zimbra  4096 Sep 14 00:00 TO_DELETE-full-XXXXXX</pre>
*** ''The example above is from a customer reporting that their backups weren't being deleted and it was filling up their backup partitions. The old backup sessions were being renamed to TO_DELETE-full-XXXXXX directory naming scheme but weren't being  deleted. The root cause of the problem was because extended acls were being used on their NAS server and it was preventing the delete operation from completing.''


=====Troubleshooting Example When Using NFS=====
Backup labels (-restoreToIncrLabel) for incrementals can be found by doing :


Steps to setup your ZCS server to use a NFS mount and to confirm you can save a backup to the NFS partition mount:
zmbackupquery -a user@domain --type incremental


<pre>
=====Command Syntax Example For Restores On The CLI=====
1. make a test partition on nfs server - /nfs-test


2. mount on zimbra server
Example to restore ONLY the data in the full backup session and stop :
2A. mkdir /nfs-test
2B. chmod 755 /nfs-test
2C. mount nfs-server:/nfs-test /nfs-test
2D. ls -la /nfs-test
2E. mkdir /nfs-test/backup
2F. chown zimbra:zimbra /nfs-test/backup
2G. chmod 755 /nfs-test/backup
2H. su - zimbra ; touch /nfs-test/backup/testfile
2I. ls -laR /nfs-test/
2J. rm /nfs-test/backup/testfile


  3. Set zimbraBackupTarget
  zmrestore -a USER@DOMAIN.com -lb full-20080726.050017.306 -rf -ca -pre restore_
3A. zmprov ms `zmhostname` zimbraBackupTarget /nfs-test/backup


4. Run a full backup against one account
Example to restore only the data in the full backup session and all the incremental data after the full to a specific incremental label and stop :
4A. ex. zmbackup -f -a user@domain.com


  5. ls -laR /nfs-test/
  zmrestore -a USER@DOMAIN.com -restoreToIncrLabel incr-20080731.060007.644 -lb full-20080726.050017.306 -br -ca -pre restore_


6. If you again, run into the same problem. You could also repeat the backup after increasing the backup
Example to restore to a specific time and stop :
    logging variable for the account your trying to backup. If you didn't run into the same problem, it might
    had to do with the initial setup of the nfs mount and permissions being used during the directory creation.
6A. zmprov aal user@domain.com zimbra.backup debug
6B. logging will show up in /opt/zimbra/log/mailbox.log
6C. Remove account logging when your done.  zmprov ral user@domain.com zimbra.backup


  8. Change zimbraBackupTarget back to your production path.
  zmrestore -a USER@DOMAIN.com -restoreToTime 20080801011800 -lb full-20080726.050017.306 -br -ca -pre restore_
</pre>


=====Setup A Fast Test NOT Using NFS=====


A way to "avoid" the NFS issues for testing purposes would be to setup a new zimbraBackupTarget to try doing a full backup of a couple of user accounts. I DON'T recommend this if you are using auto-group for zimbraBackupMode [ zmprov gs `zmhostname` zimbraBackupMode ] , only if you are using Standard mode.
'''Important Options You Might Want Or Need To Include'''


<pre>
'''--ignoreRedoErrors''' : If you attempt a restore and you see an error about problems related to playing the redolog, you'll want to run the restore command again and include this option.
[as root]
** adjust your new "backup" directory path as needed - mine is just an example**
mkdir /mnt/usb1/backup-test
chown zimbra:zimbra /mnt/usb1/backup-test
chmod 750 /mnt/usb1/backup-test
su - zimbra
**confirm backup mode as standard**
zmprov gs `zmhostname` zimbraBackupMode
**if not Standard, please stop**
zmprov ms `zmhostname` zimbraBackupTarget /mnt/usb1/backup-test
**Find a couple of test accounts to do a full for. Make sure you'll have space to do the backup regards to need free space.**
zmbackup -f -a user1@domain.com user2@domain.com user3@domain.com
**Watch and confirm status of backup you just started.**
zmbackupquery
**Confirm files were backed up in right location**
ls /mnt/usb1/backup-test/sessions/
**Failed backups would most likely results in left over directory in tmp directory**
ls /mnt/usb1/backup-test/tmp/
</pre>


===Standard Backup Or Autogroup===


Knowing what the zimbraBackupMode variable is set to on the server is important when trouble shooting and backup/restore issues. The options for zimbraBackupMode are Standard or Auto-Grouped.
'''--skipDeletes''' : Please see http://bugzilla.zimbra.com/show_bug.cgi?id=31824#c5 for details on this.


* The '''Standard''' backup method is to run a weekly full backup session and daily incremental backup sessions to back up all mailboxes daily. The standard backup method is appropriate for enterprise deployments where full backups are run during non-working days.
* The '''Auto-Grouped''' backup method is recommended for large ZCS environments where running a full backup of all accounts at one time would take too long.The auto-grouped backup method runs a full backup session for a different group of mailboxes at each scheduled backup. The system administrator configures the interval that backups should run and configures the number of groups that backups are made up of. ZCS then automatically backs up mailboxes in groups over the interval specified.


zimbraBackupMode can be set at the Global and Server level, if set at the server level it overrides the Global setting if different. To see what it's currently set at you would do:
'''-t /path/to/backup_dir''' : If you are restoring from another backup directory besides your current default path.


zmprov gcf zimbraBackupMode
zmprov gs `zmhostname` zimbraBackupMode


Your backup scheduled and crontab should be set appropriately for the zimbraBackupMode method your using. And changes to zimbraBackupMode should also have you review the backup schedule and crontab setup.
Variables that are asking for TIME rather than LABELS should follow this syntax (from zmrestore --help):
<pre>
Specify date/time in one of these formats:


====Auto-Grouped Wants To Run A Full Backup Against All Users On The First Run====
    2008/08/06 09:55:50
    2008/08/06 09:55:50 572
    2008/08/06 09:55:50.572
    2008/08/06-09:55:50-572
    2008/08/06-09:55:50
    20080806.095550.572
    20080806.095550
    20080806095550572
    20080806095550


One issue that might happen when switching from Standard to Auto-Grouped for your backup method is for the first Auto-Grouped backup job it will attempt to backup all users. This can cause your disk partition to become full if your unprepared for it. To avoid this first full backup of all users when switching to the Auto-Grouped method you can consult the following bug for a work around.
Specify year, month, date, hour, minute, second, and optionally millisecond.
Month/date/hour/minute/second are 0-padded to 2 digits, millisecond to 3 digits.
Hour must be specified in 24-hour format, and time is in local time zone.
</pre>


* Ability to set last_backup_at to NULL or to the day before for All users, group of users, user
==Trouble-shooting Backup/Restore Issues And Other General Questions==
** https://bugzilla.zimbra.com/show_bug.cgi?id=87834


===Backup Schedule===
===Backup And Restore Compatibility Between ZCS Versions===


The backup scheduled can be set by the zmschedulebackup command or modifying the zimbra users crontab file.
* You can use backups from older versions of ZCS for account restores.
* You can '''NOT''' use backups from old versions of ZCS for system restores though. Disaster Recovery restores have to be done with the same version of ZCS as the backups were done with.


The default backup schedule for a server that is using the Standard backup method is below. The example shows what's in the crontab and also the output of zmschedulebackup query command for the current schedule, the -q option :
References:


<pre>
* Upgrades And Compatibility Of Older Backups
[zimbra@mail-172 sessions]$ crontab -l | grep backup
** "Backups must be compatible across patch releases"
0 1 * * 6 /opt/zimbra/bin/zmbackup -f -a all    --mail-report
*** http://bugzilla.zimbra.com/show_bug.cgi?id=16239
0 1 * * 0-5 /opt/zimbra/bin/zmbackup -i  --mail-report
** "Incorrect upgrade documentation regarding backups"
0 0 * * * /opt/zimbra/bin/zmbackup -del 1m --mail-report
*** http://bugzilla.zimbra.com/show_bug.cgi?id=30218
** "support for restore across major versions"
*** http://bugzilla.zimbra.com/show_bug.cgi?id=15750
** "Add conversion tool to upgrade backup versions to allow restore on later zcs versions"
*** http://bugzilla.zimbra.com/show_bug.cgi?id=14002
* From 5.0.7 - 5.0.10 You Might See minor version upgrades moving your backups into a subdirectory
** "upgrade incorrectly invalidates backups."
*** http://bugzilla.zimbra.com/show_bug.cgi?id=304


[zimbra@mail-172 ~]$ zmschedulebackup -q
===Trouble-shooting===
Current Schedule:


        f 0 1 * * 6 -a all --mail-report
Ref:
        i 0 1 * * 0-5 --mail-report
* http://wiki.zimbra.com/wiki/Ajcody-Backup-Restore-Issues#Information_To_Provide_When_Submitting_A_Support_Case_For_Backup_Issues
        d 1m 0 0 * * * --mail-report
</pre>


'''''zmschedulebackup -D''''' will set the following defaults backup scheduled based upon what zimbraBackupMode is set to on the server. The example below shows setting zimbraBackupMode to Auto-Grouped and using zmschedulebackup to set the default backup schedule in the zimbra crontab for Auto-Grouped method.
====Basic Backup Information To Submit To Support====


<pre>
=====Disk Space Usage Issues=====
[zimbra@mail-172 sessions]$ zmschedulebackup -D
Default schedule set


Current Schedule:
======Trend Data======


        f 0 1 * * 0-6
If there is concerns about disks/partitions getting full, this command would be helpful for trending data on your server. Send support the resulting df.tar file . Note - adjust the tail command if you want more than 20 day's worth of trending data, the -n 20 option.
        d 1m 0 0 * * *
[zimbra@mail-172 sessions]$ crontab -l | grep backup
0 1 * * 0-6 /opt/zimbra/bin/zmbackup -f
0 0 * * * /opt/zimbra/bin/zmbackup -del 1m
</pre>


Notice that the Auto-Grouped method runs a "full" everyday and drops the line for incremental backups. It also drops the "-a all" option since it's an inappropriate option when using Auto-Grouped. There is currently a bug, see below, that also caused the --mail-report option to be dropped. You can include it against by manually modifying the crontab file.
<pre>
[zimbra@zcs806 tmp]$ /tmp


* zmschedulebackup ignores zmbackup options in auto-grouped mode
[zimbra@zcs806 tmp]$ tar cvf /tmp/df.tar `find /opt/zimbra/zmstat -name df.cs\* | sort | tail -n 20`
** https://bugzilla.zimbra.com/show_bug.cgi?id=97154
tar: Removing leading `/' from member names
/opt/zimbra/zmstat/2014-03-10/df.csv.gz
/opt/zimbra/zmstat/2014-03-11/df.csv.gz
[cut - Ajcody]
/opt/zimbra/zmstat/2014-03-27/df.csv.gz
/opt/zimbra/zmstat/2014-03-28/df.csv.gz
/opt/zimbra/zmstat/df.csv


===Backup And Restore CLI Commands You Should Be Familiar With===
[zimbra@zcs806 tmp]$ ls -lah /tmp/df.tar
-rw-r----- 1 zimbra zimbra 80K Mar 29 06:44 /tmp/df.tar


* zmschedulebackup : This command is used to schedule full backups, incremental backups, and deletion of old backups.
[zimbra@zcs806 tmp]$ tar tvf /tmp/df.tar
* zmbackup : This command executes full or incremental backup of the mail server. This is run on a live server, while the mailboxd process and the mailbox server are running. This command also has an option to manually delete old backups when they are no longer needed.
-rw-r----- zimbra/zimbra  2566 2014-03-11 00:00 opt/zimbra/zmstat/2014-03-10/df.csv.gz
* zmbackupabort : This command stops a full backup that is in process.
-rw-r----- zimbra/zimbra  2553 2014-03-12 00:00 opt/zimbra/zmstat/2014-03-11/df.csv.gz
** zmbackupabort -r : This command stops an ongoing restore.
[cut - Ajcody]
* zmbackupquery : This command lists the information about ongoing and completed backups, including labels and dates.
-rw-r----- zimbra/zimbra  2513 2014-03-28 00:00 opt/zimbra/zmstat/2014-03-27/df.csv.gz
* zmrestore : This command executes a full or incremental restore to the Zimbra mail server. The zmrestore command is performed on a server that is running.
-rw-r----- zimbra/zimbra  2531 2014-03-29 00:00 opt/zimbra/zmstat/2014-03-28/df.csv.gz
* zmrestoreoffline : This command restores the Zimbra mail server when the mailboxd process is stopped.
-rw-r----- zimbra/zimbra  8013 2014-03-29 06:40 opt/zimbra/zmstat/df.csv
* zmrestoreldap : This command restores the complete LDAP directory server, including accounts, domains, servers, COS and other data.
</pre>
* zmplayredo : This command allows you to play/replay a redolog file/s. It requires the mailbox service to be stopped though.
* zmredodump : This command will dump the contents of a redolog file/s so you can see the transactions that were done. This is useful if your trying to confirm an action was or wasn't done, what time it was done, and so forth. This does not require the mailbox service to be stopped.


===Monitoring And Maintenance Of Backups===
======Directory Sizes In /opt/zimbra======


Monitoring backups and disk usage
Please see the following and provide the output to support. Note, even though this method is faster than doing a du it still can take awhile.


* Confirm that your zimbra crontab file for your backup runs include the --mail-report option.
* [[Ajcody-Server-Misc-Topics#Faster_Way_To_Get_Directory_Size_On_Filesytem_-_find_vs_du]]


* Make sure your monitoring your partition space appropriately. You should have the stats service installed and enabled on your servers so it can monitor your partitions and trigger an alert if they exceed a percentage used.
======Adjusting The Disk Alert Threshold======


<pre>
'''Note''' - zmlocalconfig smtp_notify  must return yes if you want to receive the notifications.
[zimbra@mail-172 ~]$ zmcontrol status | grep stats
        stats                  Running


[zimbra@mail-172 ~]$ zmlocalconfig | egrep -i 'zmstat_d|zmdisk'
If you just need to adjust the disk alert threshold, then see the following:
zmdisklog_critical_threshold = 95
 
zmdisklog_warn_threshold = 85
See current values:
zmstat_df_excludes =
zmstat_disk_interval = 600
</pre>


* Monitor the following directories as well, they often times can be the reason behind your disk partition becoming full.
  zmlocalconfig | grep zmdisklog
** Default path: /opt/zimbra/redolog/archive/  ['''zmprov gs `zmhostname` zimbraRedoLogArchiveDir'''] ; uses the following variable for the redolog archive directory.
*** This could indicate your backups are failing and therefor your archived redologs aren't being purged with a successful backup.
** Default path: /opt/zimbra/backup/tmp/ ['''zmprov gs `zmhostname` zimbraBackupTarget'''] ; uses the following variable for the tmp directory.
*** Failed backups will stay in the tmp directory and not be moved into the sessions directory.


* Monitoring and reviewing the output of zmbackupquery and confirming that the "Number of accounts:" line falls within expectations for your system - most notably, for the full backups. Do NOT assume that full backup labels in your backup directory are an indication your system is backing up all your users.
Example adjustment:
** Case in point of a real support case, if the system no longer has enough space to do a full backup of all your users it will still backup new users and the label will be a full. The system administrator has assumed full backups were being done correctly because they simply listed the directory contents of the sessions directory and saw numerous full backup directories. What they didn't realize, those were just for new users and each full backup session only held one or two accounts. The server was failing on the full backup for all users because it was running out of space and would terminate prior to completing.


<pre>
<pre>
[zimbra@mail-172 ~]$ zmbackupquery --type full
su - zimbra
Label:  full-20150228.060012.892
zmlocalconfig -e zmdisklog_critical_threshold=98
Type:    full
  zmlocalconfig -e zmdisklog_warn_threshold=95
Status: completed
zmstatctl
Started: Sat, 2015/02/28 01:00:12.892 EST
Ended:  Sat, 2015/02/28 01:00:29.751 EST
Redo log sequence range: 7 .. 7
Number of accounts: 10 out of 10 completed
 
[zimbra@mail-172 ~]$ zmprov -l gaa | wc -l
10
</pre>
</pre>


When your using the Standard backup method, your incremental backup runs will also run a full backup against any 'new' users that currently don't have a full backup. Example of "new" accounts being backed up and also seeing the incremental being ran.
To exclude a partition from the checks [example of two being excluded]:


<pre>
<pre>
drwxr-x--- 6 zimbra zimbra 4096 Aug 20 01:04 incr-20130819.193041.672
su - zimbra
drwxr-x--- 6 zimbra zimbra 4096 Aug 20 01:04 full-20130819.193420.421
zmlocalconfig -e zmstat_df_excludes="/mount/point:/mount/point2"
drwxr-x--- 6 zimbra zimbra 4096 Aug 21 01:04 incr-20130820.193044.443
zmstatctl
drwxr-x--- 6 zimbra zimbra 4096 Aug 21 01:04 full-20130820.193424.551
drwxr-x--- 6 zimbra zimbra 4096 Aug 22 01:04 incr-20130821.193039.818
drwxr-x--- 6 zimbra zimbra 4096 Aug 22 01:04 full-20130821.193427.056
drwxr-x--- 6 zimbra zimbra 4096 Aug 23 01:03 incr-20130822.193037.145
drwxr-x--- 6 zimbra zimbra 4096 Aug 23 01:04 full-20130822.193352.394
</pre>
</pre>


==Server Issues==
Note, depending on your version of ZCS, you might be hitting a bug where you'll keep getting emails until a logrotate happens.


===Questions To Ask & Address Prior To Doing A Server Restore===
* Changing Zmstat-df values do not take affect until logrotate
** https://bugzilla.zimbra.com/show_bug.cgi?id=79148


* Should I shutdown ZCS services or block client access until issue and resolution are identified?
Some things to do to confirm and share with support or in bug. As zimbra
* What data needs to really be restored - can you restore just that or does it need a full restore?


===Server Restores===
<pre>
su - zimbra


====Server Restores - Disaster Recoveries====
ls -la /var/log/zimbra.log


=====For A ZCS Server DR Restore=====
df -h
/dev/mapper/vg_rhel664-lv_root
                      5.5G  3.5G  1.7G  68% /
tmpfs                939M    0  939M  0% /dev/shm
/dev/sda1            485M  79M  381M  18% /boot
/dev/sdb1              30G  6.2G  23G  22% /opt


Please see [[Network_Edition_Disaster_Recovery]] on doing a full system restore.
date


=====To Restore Just The LDAP Date=====
zmlocalconfig | grep zmdisklog
zmdisklog_critical_threshold = 80
zmdisklog_warn_threshold = 85 


Let's say your ldap data was 'lost/destoyed' but everything else was intact, you should look at the zmrestoreldap command.
zmlocalconfig -e zmdisklog_critical_threshold=95


This section should have more precaution and background information to handle this section.
zmlocalconfig -e zmdisklog_warn_threshold=90


The basics:
zmlocalconfig | grep zmdisklog
* To find the LDAP session labels type -lbs.
zmdisklog_critical_threshold = 95
** zmrestoreldap -lbs
zmdisklog_warn_threshold = 90
* Restore the complete LDAP directory server
** zmrestoreldap -lb full20061130135236
* Restore LDAP data for specific accounts
** zmrestoreldap -lb full20061130135236 -a tac@DOMAIN.com jane@DOMAIN.com


=====To Restore Just The Mysql DB=====
zmstatctl restart


Option available from the following RFE work:
date


* "Allow backup of only primary message volume"
ps -eaf | grep zmstat-df
** https://bugzilla.zimbra.com/show_bug.cgi?id=35278
*** Options '''''to exclude types of data''''' as described in the 6.0.6 Admin Guide:
**** '''Search index''' : If you do not restore the search index data, the mailbox will have to be reindexed after the restore.
***** zmrestore <all or account> --exclude-search-index
**** '''Blobs''' : This is a useful option when all blobs for the mailbox being restored already exists.
***** zmrestore <all or account>|--exclude-blobs
**** '''HSM-blobs''' : This is useful when all HSM blobs for the mailbox being restored already exists.
***** zmrestore <all or account> --exclude-hsm-blobs


======'''Steps To Practice And Test A DR Situation - System And Mysql Data Only'''======
ls -la /var/log/zimbra.log


'''This test should be done against a non-production server. Either install a 'dummy' ZCS server or create a clone of your production server to perform this test against.'''
date ; grep "Disk warning" /var/log/zimbra* ; zmmailbox -z -m admin@`zmhostname` s -l 100 -t message "Subject: Disk and after:yesterday"


Let's say your mysql data was 'lost/destoyed' but everything else was intact. This might be the solution for the situation:
##Note - Emails by default go out every 10 minutes - for example:


The steps below are to REPRODUCE A DR situation to test.
[zimbra@zcs803 ~]$ date ; grep "Disk warning" /var/log/zimbra* ; zmmailbox -z -m admin@`zmhostname` s -l 100 -t message "Subject: Disk and after:yesterday"
# zmcontrol stop
Thu May 22 09:40:08 PDT 2014
# '''Caution''' - step to reproduce DR situation to test against
/var/log/zimbra.log:May 22 08:30:00 zcs803 zimbramon[18826]: 18826:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82%
## mv ~/db/data ~/db/data.OLD
/var/log/zimbra.log:May 22 08:40:00 zcs803 zimbramon[22970]: 22970:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82%
# Getting old mysql passwords
/var/log/zimbra.log:May 22 08:50:00 zcs803 zimbramon[22970]: 22970:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82%
#* zmlocalconfig -s mysql_root_password
/var/log/zimbra.log:May 22 09:00:00 zcs803 zimbramon[22970]: 22970:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82%
#* zmlocalconfig -s zimbra_mysql_password
  ## Note - I had readjusted the variable to not warn during this time segment ##
# [as zimbra] /opt/zimbra/libexec/zmmyinit
/var/log/zimbra.log:May 22 09:20:00 zcs803 zimbramon[8322]: 8322:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82%
#* Most likely your mysql passwords will change.
/var/log/zimbra.log:May 22 09:30:00 zcs803 zimbramon[8322]: 8322:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82%
#* See the following article to set them back:
/var/log/zimbra.log:May 22 09:40:00 zcs803 zimbramon[8322]: 8322:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82%
#** [[Resetting_LDAP_%26_MySQL_Passwords]]
num: 7, more: false
#** [[Issues_with_mysql_and_logmysql_passwords#For_Mysql_Database]]
# ldap start
# zmconvertctl start
# mysql is already running per the zmmyinit - mysql.server status - to check.
# zmrestoreoffline -a all -br --systemData --excludeSearchIndex --excludeBlobs --excludeHsmBlobs
#* Note - you most likely will want to use the -br or -rf option for this situation.
#** -br,--backedupRedologsOnly : Replays the redo logs in backup only, which excludes archived and current redo logs of the system
#** -rf,--restoreFullBackupOnly : Restores to last full backup only, which excludes incremental backups.
#* -sys,--systemData : Restores global tables and local config.
#** This restores the 'zimbra' mysql data - this stuff [[Ajcody-Mysql-Topics#zimbra_Database_Default_Example_for_ZCS5]]
#** '''Important''' - the zimbra db holds information about the volumes. This needs to be restored/existing prior to user db restoration.
#* Note the use of -a all , you could also do this for one account first to confirm operation is successful for your circumstances.
# If you used -rf or -br with your zmestoreoffline, you might also need to use zmplayredo to finish it up to get the most complete restore.
#* Improve zmrestore & zmrestoreoffline performance
#*** http://bugzilla.zimbra.com/show_bug.cgi?id=33606
#* [[Ajcody-Backup-Restore-Issues#zmplayredo_-_Replaying_Content_From_Any_Redolog_File]] and other 'redolog' sections on that page.


====Server Cloning - Server Moves/Testing Purposes====
    Id  Type  From                  Subject                                            Date
 
  ----  ----  --------------------  --------------------------------------------------  --------------
==Trouble-shooting Backup/Restore Issues And Other General Questions==
1.  328  mess  admin                Disk / at 82% on zcs803.DOMAIN.com:          05/22/14 09:40
 
2.  327  mess  admin                Disk / at 82% on zcs803.DOMAIN.com:          05/22/14 09:30
===Backup And Restore Compatibility Between ZCS Versions===
3.  326  mess  admin                Disk / at 82% on zcs803.DOMAIN.com:          05/22/14 09:20
  ## Note - I had readjusted the variable to not warn during this time segment ##
4.  325  mess  admin                Disk / at 82% on zcs803.DOMAIN.com:          05/22/14 09:00
5.  324  mess  admin                Disk / at 82% on zcs803.DOMAIN.com:          05/22/14 08:50
6.  323  mess  admin                Disk / at 82% on zcs803.DOMAIN.com:          05/22/14 08:40
7.  320  mess  admin                Disk / at 82% on zcs803.DOMAIN.com:          05/22/14 08:31
</pre>


* You can use backups from older versions of ZCS for account restores.
Continue to monitor your zmmailbox search results for an hour.
* You can '''NOT''' use backups from old versions of ZCS for system restores though. Disaster Recovery restores have to be done with the same version of ZCS as the backups were done with.


References:
=====The Basic Information Support Needs=====


* Upgrades And Compatibility Of Older Backups
as root:
** "Backups must be compatible across patch releases"
*** http://bugzilla.zimbra.com/show_bug.cgi?id=16239
** "Incorrect upgrade documentation regarding backups"
*** http://bugzilla.zimbra.com/show_bug.cgi?id=30218
** "support for restore across major versions"
*** http://bugzilla.zimbra.com/show_bug.cgi?id=15750
** "Add conversion tool to upgrade backup versions to allow restore on later zcs versions"
*** http://bugzilla.zimbra.com/show_bug.cgi?id=14002
* From 5.0.7 - 5.0.10 You Might See minor version upgrades moving your backups into a subdirectory
** "upgrade incorrectly invalidates backups."
*** http://bugzilla.zimbra.com/show_bug.cgi?id=304


===Trouble-shooting===
* cat /etc/fstab
 
** Shows us what is mounted upon boot
Ref:
* cat /proc/mounts
* http://wiki.zimbra.com/wiki/Ajcody-Backup-Restore-Issues#Information_To_Provide_When_Submitting_A_Support_Case_For_Backup_Issues
** Shows us what is currently mounted and its status - you can see if a mount is read-only here.
* df -hT
** Lists current mounts using human-readable size information and also notes the filesystem type.


====Basic Backup Information To Submit To Support====
as zimbra:


=====Disk Space Usage Issues=====
* zmprov -l gs `zmhostname` | egrep 'Back|Redo'
** Will show us a number of variables related to backup and redologs. Also tell us if your using auto-group or the default method.
* du -sh /opt/zimbra/redolog
** Will might notice your redolog logs aren't rolling over, causing a possible issue.
* ls -latr /opt/zimbra/backup
** This is the default backup target, please adjust this path here and below if you are using a different zimbraBackupTarget value.
** zmprov gs `zmhostname` zimbraBackupTarget
** We'll be able to confirm permissions are right.
* ls -latr /opt/zimbra/backup/tmp
** This will show us if you have failed backup jobs and confirm tmp is being cleaned appropriately after the backup is done.
* ls -latr /opt/zimbra/backup/sessions
** This will show us what backup sessions are available and confirm permissions are correct.
** Adjust path if your zimbraBackupTarget value is not the default path.
* Some directory sizes in the backup directory:
** Default path first
*** du -sh `find /opt/zimbra/backup -maxdepth 2 -type d`
** If your using a different backup target, check that directory also. Replace /opt/zimbra/backup above with your backup path.
* zmbackupquery
** This should match what's in the sessions directory and it will also tell us if status of each backup and how many accts were done.
* crontab -l | grep -i back
** This will show use when backups are support to run and with what options they are running with.
* zmlocalconfig | grep -i back
** This is useful to see a number of backup options not exposed in the crontab, things related to the zip options.
* zmvolume -l
** This is useful to see how many volumes are being used, if HSM is being used, and if compression is being done at the volume level.


======Trend Data======
=====Additional Log Files Support Might Need=====
 
And send the following logs:
 
* /var/log/messages
** Filesystem issues often times are noted here and also in syslog. This might explain an interruption in the backup process. Server restarts, filesystem going full, filesystem going read-only, etc.
* /var/log/syslog
* /opt/zimbra/log/mailbox.log
** The backup activity is logged here.
**  And any other mailbox.log file that would cover the event
 
====Some Specific Example Restore/Backup Issues====


If there is concerns about disks/partitions getting full, this command would be helpful for trending data on your server. Send support the resulting df.tar file . Note - adjust the tail command if you want more than 20 day's worth of trending data, the -n 20 option.
=====NO_SUCH_ACCOUNT_BACKUP=====


Example errors:
<pre>
<pre>
[zimbra@zcs806 tmp]$ /tmp
Error code: backup.NO_SUCH_ACCOUNT_BACKUP
Message: no such backup for account: Missing full backup earlier than restore-to time for account 3c397edc-c015-4d82-ae8a-1084692f8a93
Details:soap:Sender


[zimbra@zcs806 tmp]$ tar cvf /tmp/df.tar `find /opt/zimbra/zmstat -name df.cs\* | sort | tail -n 20`
Error occurred: no such backup for account: 3c397edc-c015-4d82-ae8a-1084692f8a93
tar: Removing leading `/' from member names
/opt/zimbra/zmstat/2014-03-10/df.csv.gz
/opt/zimbra/zmstat/2014-03-11/df.csv.gz
[cut - Ajcody]
/opt/zimbra/zmstat/2014-03-27/df.csv.gz
/opt/zimbra/zmstat/2014-03-28/df.csv.gz
/opt/zimbra/zmstat/df.csv


[zimbra@zcs806 tmp]$ ls -lah /tmp/df.tar
Message: no such backup for account: Missing full backup earlier than restore-to time for account 3c397edc-c015-4d82-ae8a-1084692f8a93
-rw-r----- 1 zimbra zimbra 80K Mar 29 06:44 /tmp/df.tar


[zimbra@zcs806 tmp]$ tar tvf /tmp/df.tar
3c397edc-c015-4d82-ae8a-1084692f8a93 is an example of the zimbraId of a user [ user1@mail-172.example.com ] .
-rw-r----- zimbra/zimbra  2566 2014-03-11 00:00 opt/zimbra/zmstat/2014-03-10/df.csv.gz
-rw-r----- zimbra/zimbra  2553 2014-03-12 00:00 opt/zimbra/zmstat/2014-03-11/df.csv.gz
[cut - Ajcody]
-rw-r----- zimbra/zimbra  2513 2014-03-28 00:00 opt/zimbra/zmstat/2014-03-27/df.csv.gz
-rw-r----- zimbra/zimbra  2531 2014-03-29 00:00 opt/zimbra/zmstat/2014-03-28/df.csv.gz
-rw-r----- zimbra/zimbra  8013 2014-03-29 06:40 opt/zimbra/zmstat/df.csv
</pre>
</pre>


======Directory Sizes In /opt/zimbra======
Confirm the current email address is mapped to the appropriate zimbraId. You'll want to search what is in ldap and the mailstores database for the exiting email address and then also search the reference files in your backups. Often times, this happens when an account was deleted and then created again. Account meaning the email address. When this is does, the new account will have a new zimbraId.


Please see the following and provide the output to support. Note, even though this method is faster than doing a du it still can take awhile.
<pre>
$ zmprov -l ga user1@mail-172.example.com zimbraId
# name user1@mail-172.example.com
zimbraId: 3c397edc-c015-4d82-ae8a-1084692f8a93


* [[Ajcody-Server-Misc-Topics#Faster_Way_To_Get_Directory_Size_On_Filesytem_-_find_vs_du]]
$ mysql -e 'SELECT * FROM zimbra.mailbox WHERE comment LIKE "user1@mail-172.example.com"\G'


======Adjusting The Disk Alert Threshold======
$ mysql -e 'SELECT * FROM zimbra.mailbox WHERE account_id="3c397edc-c015-4d82-ae8a-1084692f8a93"\G'


'''Note''' - zmlocalconfig smtp_notify  must return yes if you want to receive the notifications.
$ grep -C1 3c397edc-c015-4d82-ae8a-1084692f8a93 /opt/zimbra/backup/accounts.xml


If you just need to adjust the disk alert threshold, then see the following:
  [using the default path; /opt/zimbra/backup/accounts.xml]


See current values:
$ egrep "user1@mail-172.example.com|3c397edc-c015-4d82-ae8a-1084692f8a93" /opt/zimbra/backup/accounts.xml


  zmlocalconfig | grep zmdisklog
$ egrep "user1@mail-172.example.com|3c397edc-c015-4d82-ae8a-1084692f8a93" /opt/zimbra/backup/sessions/*/session.xml
</pre>


Example adjustment:
=====exception during auth=====


Example error:
<pre>
<pre>
  su - zimbra
LDAP backup failed: system failure: exception during auth
  zmlocalconfig -e zmdisklog_critical_threshold=98
  {RemoteManager: zimbramail.example.com->zimbra@zimbramail.example.com:22}
  zmlocalconfig -e zmdisklog_warn_threshold=95
  com.zimbra.common.service.ServiceException: system failure: LDAP backup failed:
  zmstatctl
  system failure: exception during auth {RemoteManager: zimbramail.example.com->
  zimbra@zimbramail.example.com:22}
</pre>
</pre>


To exclude a partition from the checks [example of two being excluded]:
This indicates the ssh keys aren't configured correctly in the environment. You should be able to do the following to resolve the issue:


<pre>
  su - zimbra
  su - zimbra
  zmlocalconfig -e zmstat_df_excludes="/mount/point:/mount/point2"
  zmsshkeygen
  zmstatctl
  zmupdateauthkeys
</pre>


Note, depending on your version of ZCS, you might be hitting a bug where you'll keep getting emails until a logrotate happens.
'''Note''' - You should also confirm that you do indeed have the ssh port available. The default is port 22. If your unable to ssh into the server using port 22, then this might be the issue. If you configured ssh to listen on a different port on this server, you'll need to adjust the following variable - zmprov gcf zimbraRemoteManagementPort . You might also have the ssh port blocked - confirm your firewall, iptables, etc is not blocking the ssh port.


* Changing Zmstat-df values do not take affect until logrotate
=====LICENSE_ERROR_Message: AccountsLimits exceeded=====
** https://bugzilla.zimbra.com/show_bug.cgi?id=79148


Some things to do to confirm and share with support or in bug. As zimbra
Example error:
LICENSE_ERROR_Message: AccountsLimits exceeded


<pre>
You should check your license and confirm it's valid and active:
su - zimbra


ls -la /var/log/zimbra.log
su - zimbra
zmlicense -p


df -h
And confirm how many licenses are used: First, flush the cache on all servers for the license information. Then do the query via soap.
/dev/mapper/vg_rhel664-lv_root
                      5.5G  3.5G  1.7G  68% /
tmpfs                939M    0  939M  0% /dev/shm
/dev/sda1            485M  79M  381M  18% /boot
/dev/sdb1              30G  6.2G  23G  22% /opt


date
zmprov fc -a license
zmsoap -z GetLicenseRequest


zmlocalconfig | grep zmdisklog
Four lines that are generally of interest are shown below. The first two state how many one is licensed for, the bottom two show how many are currently used :
  zmdisklog_critical_threshold = 80
<pre>
  zmdisklog_warn_threshold = 85 
  <attr name="AccountsLimit">1000</attr>
  <attr name="ArchivingAccountsLimit">1000</attr>


zmlocalconfig -e zmdisklog_critical_threshold=95
<attr name="TotalAccounts">3</attr>
<attr name="ArchivingAccounts">3</attr>
</pre>


zmlocalconfig -e zmdisklog_warn_threshold=90
==Hands On How-To And Reviewing Of Log Events==


zmlocalconfig | grep zmdisklog
===Full Backup And Log Events Generated===
zmdisklog_critical_threshold = 95
zmdisklog_warn_threshold = 90


zmstatctl restart
Example of manually starting a full backup using the Standard zimbraBackupMode and doing a query to confirm it's completion and status.


date
<pre>
[zimbra@mail-172 archive]$ zmprov gs `zmhostname` zimbraBackupMode
# name mail-172.example.com
zimbraBackupMode: Standard


ps -eaf | grep zmstat-df
[zimbra@mail-172 archive]$ pwd ; ls *.log
/opt/zimbra/redolog/archive
redo-20150303.060015.816-seq9.log
 
[zimbra@mail-172 archive]$ zmbackup -f -a all
full-20150304.012000.193


ls -la /var/log/zimbra.log
[zimbra@mail-172 archive]$ zmbackupquery -lb full-20150304.012000.193 -v
Label:  full-20150304.012000.193
Type:    full
Status:  completed
Started: Tue, 2015/03/03 20:20:00.193 EST
Ended:  Tue, 2015/03/03 20:20:28.630 EST
Redo log sequence range: 10 .. 10
Number of accounts: 10 out of 10 completed
Accounts:
  admin@mail-172.example.com: completed
  galsync.qva4ekog0@mail-172.example.com: completed
  ham.jgfdb3oqjf@mail-172.example.com: completed
  spam.umhx3owv3s@mail-172.example.com: completed
  user1@mail-172.example.com: completed
  user2@mail-172.example.com: completed
  user3@mail-172.example.com: completed
  user4@mail-172.example.com: completed
  user5@mail-172.example.com: completed
  virus-quarantine.gusiso1cd@mail-172.example.com: completed


date ; grep "Disk warning" /var/log/zimbra* ; zmmailbox -z -m admin@`zmhostname` s -l 100 -t message "Subject: Disk and after:yesterday"
Total space: 14020MB
Free space: 6017MB
</pre>


##Note - Emails by default go out every 10 minutes - for example:
Example of what log events are generated for a full backup in /opt/zimbra/log/mailbox.log . First, you'll see the authentication/acl request and then statements about the backup request.


[zimbra@zcs803 ~]$ date ; grep "Disk warning" /var/log/zimbra* ; zmmailbox -z -m admin@`zmhostname` s -l 100 -t message "Subject: Disk and after:yesterday"
<pre>
Thu May 22 09:40:08 PDT 2014
2015-03-03 20:19:58,499 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [] AuthProvider - Adding auth provider: zimbra com.zimbra.cs.service.ZimbraAuthProvider
/var/log/zimbra.log:May 22 08:30:00 zcs803 zimbramon[18826]: 18826:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82%
2015-03-03 20:19:59,458 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] soap - AuthRequest elapsed=834
/var/log/zimbra.log:May 22 08:40:00 zcs803 zimbramon[22970]: 22970:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82%
2015-03-03 20:19:59,751 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] account - Initialized access manager: com.zimbra.cs.account.accesscontrol.ACLAccessManager
/var/log/zimbra.log:May 22 08:50:00 zcs803 zimbramon[22970]: 22970:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82%
2015-03-03 20:19:59,770 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup request started
/var/log/zimbra.log:May 22 09:00:00 zcs803 zimbramon[22970]: 22970:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82%
2015-03-03 20:20:00,193 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Found 10 accounts on server mail-172.example.com
  ## Note - I had readjusted the variable to not warn during this time segment ##
2015-03-03 20:20:00,464 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup request finished
/var/log/zimbra.log:May 22 09:20:00 zcs803 zimbramon[8322]: 8322:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82%
2015-03-03 20:20:00,464 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] soap - BackupRequest elapsed=760
/var/log/zimbra.log:May 22 09:30:00 zcs803 zimbramon[8322]: 8322:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82%
/var/log/zimbra.log:May 22 09:40:00 zcs803 zimbramon[8322]: 8322:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82%
num: 7, more: false
 
    Id  Type  From                  Subject                                            Date
  ----  ----  --------------------  --------------------------------------------------  --------------
1. 328  mess  admin                Disk / at 82% on zcs803.DOMAIN.com:           05/22/14 09:40
2.  327  mess  admin                 Disk / at 82% on zcs803.DOMAIN.com:          05/22/14 09:30
3. 326  mess  admin                Disk / at 82% on zcs803.DOMAIN.com:           05/22/14 09:20
  ## Note - I had readjusted the variable to not warn during this time segment ##
4.  325  mess  admin                Disk / at 82% on zcs803.DOMAIN.com:           05/22/14 09:00
5. 324  mess  admin                Disk / at 82% on zcs803.DOMAIN.com:           05/22/14 08:50
6.  323  mess  admin                 Disk / at 82% on zcs803.DOMAIN.com:          05/22/14 08:40
7. 320  mess  admin                Disk / at 82% on zcs803.DOMAIN.com:          05/22/14 08:31
</pre>
</pre>


Continue to monitor your zmmailbox search results for an hour.
Your log events will now reference Backup in them. AsyncFileCopier will start and it will give the backup session label for this backup job.
<pre>
2015-03-03 20:20:01,092 INFO  [FullBackupThread] [] io - AsyncFileCopier is starting
2015-03-03 20:20:01,187 INFO  [FullBackupThread] [] backup - Full backup started for backup set; label: full-20150304.012000.193
</pre>


=====The Basic Information Support Needs=====
It proceeds to backing up your accounts. I'll list the backup log events for just two accounts on this system - user1@mail-172.example.com and admin@mail-172.example.com . I'll provide the zimbraId, email address, and mailboxID information for user1@mail-172.example.com below so you can confirm it's reported in the logs correctly as well.


as root:
<pre>
[zimbra@mail-172 redolog]$ zmprov ga user1@mail-172.example.com zimbraId
# name user1@mail-172.example.com
zimbraId: 3c397edc-c015-4d82-ae8a-1084692f8a93


* cat /etc/fstab
[zimbra@mail-172 redolog]$ zmprov gmi user1@mail-172.example.com
** Shows us what is mounted upon boot
mailboxId: 6
* cat /proc/mounts
quotaUsed: 1209
** Shows us what is currently mounted and its status - you can see if a mount is read-only here.
</pre>
* df -hT
** Lists current mounts using human-readable size information and also notes the filesystem type.


as zimbra:
Now, continuing with the log events. Notice the reference to the redo log file sequence, putting the account into and out of maintenance, number of blobs and index files backed up, and the statement it's finished backing up the account. Also within the section we can see the user's zimbraId [3c397edc-c015-4d82-ae8a-1084692f8a93] , email address [user1@mail-172.example.com] , and mailboxId [Mailbox 6].


* zmprov -l gs `zmhostname` | egrep 'Back|Redo'
<pre>
** Will show us a number of variables related to backup and redologs. Also tell us if your using auto-group or the default method.
[cut]
* du -sh /opt/zimbra/redolog
2015-03-03 20:20:06,560 INFO  [FullBackupThread] [name=user1@mail-172.example.com;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/6/index/0
** Will might notice your redolog logs aren't rolling over, causing a possible issue.
2015-03-03 20:20:06,560 INFO  [FullBackupThread] [name=user1@mail-172.example.com;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 LOADED
* ls -latr /opt/zimbra/backup
2015-03-03 20:20:06,560 INFO  [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - redo log file sequence is 10 at full backup for user1@mail-172.example.com
** This is the default backup target, please adjust this path here and below if you are using a different zimbraBackupTarget value.
2015-03-03 20:20:06,565 INFO  [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - Full backup started for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) mailbox 6
** zmprov gs `zmhostname` zimbraBackupTarget
2015-03-03 20:20:06,565 INFO  [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] mailbox - Putting mailbox 6 under maintenance.
** We'll be able to confirm permissions are right.
2015-03-03 20:20:06,818 INFO  [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - Number of blobs to backup for mailbox 6: 1
* ls -latr /opt/zimbra/backup/tmp
2015-03-03 20:20:06,818 INFO  [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] mailbox - Ending maintenance on mailbox 6.
** This will show us if you have failed backup jobs and confirm tmp is being cleaned appropriately after the backup is done.
2015-03-03 20:20:06,818 INFO  [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 AVAILABLE
* ls -latr /opt/zimbra/backup/sessions
2015-03-03 20:20:06,966 INFO  [BackupSetWorkerThread-BLOB] [name=user1@mail-172.example.com;mid=6;] backup - Stored 1 blob files from volume message1
** This will show us what backup sessions are available and confirm permissions are correct.
2015-03-03 20:20:06,966 INFO  [BackupSetWorkerThread-BLOB] [name=user1@mail-172.example.com;mid=6;] backup - Stored 1 blob files
** Adjust path if your zimbraBackupTarget value is not the default path.
2015-03-03 20:20:07,014 INFO  [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - Account user1@mail-172.example.com in backup set full-20150304.012000.193: All pending file IO completed (1 out of 1)
* Some directory sizes in the backup directory:
2015-03-03 20:20:07,016 INFO  [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] mailbox - Putting mailbox 6 under maintenance.
** Default path first
2015-03-03 20:20:07,021 INFO  [BackupSetWorkerThread-INDEX] [name=user1@mail-172.example.com;mid=6;] backup - Stored 2 index files
*** du -sh `find /opt/zimbra/backup -maxdepth 2 -type d`
2015-03-03 20:20:07,496 INFO  [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - Account user1@mail-172.example.com in backup set full-20150304.012000.193: All pending file IO completed (3 out of 3)
** If your using a different backup target, check that directory also. Replace /opt/zimbra/backup above with your backup path.
2015-03-03 20:20:07,497 INFO  [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] mailbox - Ending maintenance on mailbox 6.
* zmbackupquery
2015-03-03 20:20:07,497 INFO  [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 AVAILABLE
** This should match what's in the sessions directory and it will also tell us if status of each backup and how many accts were done.
2015-03-03 20:20:07,497 INFO  [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - Account user1@mail-172.example.com in backup set full-20150304.012000.193: All pending file IO completed (3 out of 3)
* crontab -l | grep -i back
2015-03-03 20:20:07,497 INFO  [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - Full backup finished for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) mailbox 6
** This will show use when backups are support to run and with what options they are running with.
</pre>
* zmlocalconfig | grep -i back
** This is useful to see a number of backup options not exposed in the crontab, things related to the zip options.
* zmvolume -l
** This is useful to see how many volumes are being used, if HSM is being used, and if compression is being done at the volume level.


=====Additional Log Files Support Might Need=====
Events for the admin user's backup:


And send the following logs:
<pre>
2015-03-03 20:20:09,894 INFO  [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - redo log file sequence is 10 at full backup for admin@mail-172.example.com
2015-03-03 20:20:09,914 INFO  [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - Full backup started for account admin@mail-172.example.com (7ac8de92-13fe-4d19-b9da-6872169fb5c9) mailbox 1
2015-03-03 20:20:09,915 INFO  [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] mailbox - Putting mailbox 1 under maintenance.
2015-03-03 20:20:10,176 INFO  [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - Number of blobs to backup for mailbox 1: 73
2015-03-03 20:20:10,177 INFO  [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] mailbox - Ending maintenance on mailbox 1.
2015-03-03 20:20:10,177 INFO  [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] mbxmgr - Mailbox 1 account 7ac8de92-13fe-4d19-b9da-6872169fb5c9 AVAILABLE
2015-03-03 20:20:10,343 INFO  [BackupSetWorkerThread-BLOB] [name=admin@mail-172.example.com;mid=1;] backup - Stored 73 blob files from volume message1
2015-03-03 20:20:10,353 INFO  [BackupSetWorkerThread-BLOB] [name=admin@mail-172.example.com;mid=1;] backup - Stored 73 blob files
2015-03-03 20:20:11,047 INFO  [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - Account admin@mail-172.example.com in backup set full-20150304.012000.193: All pending file IO completed (73 out of 73)
2015-03-03 20:20:11,051 INFO  [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] mailbox - Putting mailbox 1 under maintenance.
2015-03-03 20:20:11,108 INFO  [BackupSetWorkerThread-INDEX] [name=admin@mail-172.example.com;mid=1;] backup - Stored 26 index files
2015-03-03 20:20:12,450 INFO  [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - Account admin@mail-172.example.com in backup set full-20150304.012000.193: All pending file IO completed (99 out of 99)
2015-03-03 20:20:12,463 INFO  [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] mailbox - Ending maintenance on mailbox 1.
2015-03-03 20:20:12,463 INFO  [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] mbxmgr - Mailbox 1 account 7ac8de92-13fe-4d19-b9da-6872169fb5c9 AVAILABLE
2015-03-03 20:20:12,463 INFO  [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - Account admin@mail-172.example.com in backup set full-20150304.012000.193: All pending file IO completed (99 out of 99)
2015-03-03 20:20:12,463 INFO  [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - Full backup finished for account admin@mail-172.example.com (7ac8de92-13fe-4d19-b9da-6872169fb5c9) mailbox 1
</pre>


* /var/log/messages
And then a final line confirming completion and the number of accounts successfully backed up.
** Filesystem issues often times are noted here and also in syslog. This might explain an interruption in the backup process. Server restarts, filesystem going full, filesystem going read-only, etc.
* /var/log/syslog
* /opt/zimbra/log/mailbox.log
** The backup activity is logged here.
**  And any other mailbox.log file that would cover the event


====Additional Checks For Performance Specific Issues====
<pre>
2015-03-03 20:20:12,463 INFO  [FullBackupThread] [] backup - Finished backing up 10 of 10 accounts
</pre>


=====If Your Using a SAN or NFS For Your Backup Target - Please Check Your IOWait=====
One thing to point out is the log events were mentioning the redo log sequence as being number 10 but in the redolog/archive directory there is only a sequence 9 redo log. This is because the "seq 10" is actually the /opt/zimbra/redolog/redo.log file. When the redo.log file rolls over and is moved to the archive directory it will have seq10 in it's filename.


Ideally, you would compare iowait and performance data from the target backup host as well as the stats available on the ZCS servers. To get graphs and stats on this from ZCS, please see [[Ajcody-Testing-Debugging#zmstat_and_zmstat-chart]] . You should submit this data and iowait conclusions if you still need to submit a support case about backup performance issues.
<pre>
[zimbra@mail-172 archive]$ pwd ; ls -la *.log
/opt/zimbra/redolog/archive
-rw-r----- 1 zimbra zimbra 146723 Mar  3 20:23 redo-20150303.060015.816-seq9.log
</pre>


=====Is HSM Running During Your Backup Window=====
The contents of the backup session directory. I removed the sub-directories under accounts that weren't either the admin or user1 account. Notice the zimbraId is used for the sub-directory naming scheme.
* Are you running HSM? HSM should not be ran during your backup window.
** "RFE: HSM and backup should not run at the same time if initated."
*** https://bugzilla.zimbra.com/show_bug.cgi?id=33452


=====Are You Using --zipStore=====
<pre>
[zimbra@mail-172 full-20150304.012000.193]$ ls -R
.:
accounts  ldap  session.xml  shared_blobs  sys


--zipStore zips the blobs vs. keeping the blobs as individual files. --zipStore does not use compression either. For most circumstances, this will give the best performance, especially with NFS. This should be the default behavior of the backups, the following RFE is when it became the default [ZCS6+] :
./accounts:
3c3  423  434  75d  7ac  89f  a46  af0  c53  eab


* "backup: default to the zip option"
./accounts/3c3:
** https://bugzilla.zimbra.com/show_bug.cgi?id=31836#c6
97e
*** Link to comment that explains options and default behavior.


To see if zip's are being used for backups for example, in the backup/session directory you'll know if it is by seeing .zip files:
./accounts/3c3/97e:
3c397edc-c015-4d82-ae8a-1084692f8a93


mail:~/backup/sessions/full-20080820.160003.770/accounts/115/988/11598896-a89b-4b9d-bedb-1ed1afcb6c87/blobs  
./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93:
  zimbra$ ls blobs-1.zip    blobs-2.zip    blobs-3.zip    blobs-4.zip
blobs  db  index  ldap.xml  meta.xml


To see if the zip file is using compression [-Z option for unzip will indicate whether or not the archive is actually compressed] :
./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93/blobs:
blobs-1.zip


unzip -Z blobs-4.zip
./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93/db:
  293 files, 5982984 bytes uncompressed, 5982984 bytes compressed: 0.0%
db_schema.xml  mail_item.dat  mailbox.dat  mailbox_metadata.dat  open_conversation.dat tag.dat tagged_item.dat


Also, if your zmvolume has compression enabled the blobs will remain compressed within the zip also upon backup. The point being, they are uncompressed to be then put into a zip file when the backup is using --zipStore.
./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93/index:
0


====Some Specific Example Restore/Backup Issues====
./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93/index/0:
segments.gen  segments_1


=====NO_SUCH_ACCOUNT_BACKUP=====
[cut]


Example errors:
./accounts/7ac:
<pre>
8de
Error code: backup.NO_SUCH_ACCOUNT_BACKUP
Message: no such backup for account: Missing full backup earlier than restore-to time for account 3c397edc-c015-4d82-ae8a-1084692f8a93
Details:soap:Sender


Error occurred: no such backup for account: 3c397edc-c015-4d82-ae8a-1084692f8a93
./accounts/7ac/8de:
7ac8de92-13fe-4d19-b9da-6872169fb5c9


Message: no such backup for account: Missing full backup earlier than restore-to time for account 3c397edc-c015-4d82-ae8a-1084692f8a93
./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9:
blobs  db  index  ldap.xml  meta.xml


3c397edc-c015-4d82-ae8a-1084692f8a93 is an example of the zimbraId of a user [ user1@mail-172.example.com ] .
./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9/blobs:
</pre>
blobs-1.zip  blobs-2.zip  blobs-3.zip  blobs-4.zip


Confirm the current email address is mapped to the appropriate zimbraId. You'll want to search what is in ldap and the mailstores database for the exiting email address and then also search the reference files in your backups. Often times, this happens when an account was deleted and then created again. Account meaning the email address. When this is does, the new account will have a new zimbraId.
./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9/db:
db_schema.xml  mail_item.dat  mailbox.dat  open_conversation.dat  tag.dat  tagged_item.dat


<pre>
./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9/index:
$ zmprov -l ga user1@mail-172.example.com zimbraId
0
# name user1@mail-172.example.com
zimbraId: 3c397edc-c015-4d82-ae8a-1084692f8a93


$ mysql -e 'SELECT * FROM zimbra.mailbox WHERE comment LIKE "user1@mail-172.example.com"\G'
./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9/index/0:
_0.fdt  _0.fnm  _0.nrm  _0.tii  _1.fdt  _1.fnm  _1.nrm  _1.tii  _2.fdt  _2.fnm  _2.nrm  _2.tii  segments.gen
_0.fdx  _0.frq  _0.prx  _0.tis  _1.fdx  _1.frq  _1.prx  _1.tis  _2.fdx  _2.frq  _2.prx  _2.tis  segments_3


$ mysql -e 'SELECT * FROM zimbra.mailbox WHERE account_id="3c397edc-c015-4d82-ae8a-1084692f8a93"\G'
[cut]


$ grep -C1 3c397edc-c015-4d82-ae8a-1084692f8a93 /opt/zimbra/backup/accounts.xml
./ldap:
ldap-config.bak.gz  ldap.bak.gz


  [using the default path; /opt/zimbra/backup/accounts.xml]
./shared_blobs:


$ egrep "user1@mail-172.example.com|3c397edc-c015-4d82-ae8a-1084692f8a93" /opt/zimbra/backup/accounts.xml
./sys:
config.dat  current_volumes.dat  db_schema.xml  localconfig.xml ssh  ssl  volume.dat


$ egrep "user1@mail-172.example.com|3c397edc-c015-4d82-ae8a-1084692f8a93" /opt/zimbra/backup/sessions/*/session.xml
./sys/ssh:
</pre>
authorized_keys  zimbra_identity  zimbra_identity.pub


=====exception during auth=====
./sys/ssl:
zimbra


Example error:
./sys/ssl/zimbra:
<pre>
ca commercial jetty.pkcs12 server
LDAP backup failed: system failure: exception during auth
{RemoteManager: zimbramail.example.com->zimbra@zimbramail.example.com:22}
  com.zimbra.common.service.ServiceException: system failure: LDAP backup failed:
  system failure: exception during auth {RemoteManager: zimbramail.example.com->
  zimbra@zimbramail.example.com:22}
</pre>


This indicates the ssh keys aren't configured correctly in the environment. You should be able to do the following to resolve the issue:
./sys/ssl/zimbra/ca:
ca.key  ca.pem  ca.srl  ca.srl.old  index.txt  index.txt.attr  index.txt.old  newcerts  zmssl.cnf


su - zimbra
./sys/ssl/zimbra/ca/newcerts:
  zmsshkeygen
1424523586.pem 1424523591.pem  1424523597.pem 1424523603.pem
  zmupdateauthkeys


'''Note''' - You should also confirm that you do indeed have the ssh port available. The default is port 22. If your unable to ssh into the server using port 22, then this might be the issue. If you configured ssh to listen on a different port on this server, you'll need to adjust the following variable - zmprov gcf zimbraRemoteManagementPort . You might also have the ssh port blocked - confirm your firewall, iptables, etc is not blocking the ssh port.
./sys/ssl/zimbra/commercial:


=====LICENSE_ERROR_Message: AccountsLimits exceeded=====
./sys/ssl/zimbra/server:
server.crt  server.csr  server.key
</pre>


Example error:
===Incremental Backup And Log Events Generated===
LICENSE_ERROR_Message: AccountsLimits exceeded


You should check your license and confirm it's valid and active:
Example of manually starting an incremental backup using the Standard zimbraBackupMode and doing a query to confirm it's completion and status.


su - zimbra
<pre>
zmlicense -p
[zimbra@mail-172 archive]$ zmprov gs `zmhostname` zimbraBackupMode
# name mail-172.example.com
zimbraBackupMode: Standard


And confirm how many licenses are used: First, flush the cache on all servers for the license information. Then do the query via soap.
[zimbra@mail-172 archive]$ ls
redo-20150303.060015.816-seq9.log  redo-20150304.070025.476-seq10.log


zmprov fc -a license
[zimbra@mail-172 archive]$ zmbackup -i
zmsoap -z GetLicenseRequest
incr-20150304.073543.151


Four lines that are generally of interest are shown below. The first two state how many one is licensed for, the bottom two show how many are currently used :
[zimbra@mail-172 archive]$ zmbackupquery -lb incr-20150304.073543.151 -v
<pre>
Label:   incr-20150304.073543.151
  <attr name="AccountsLimit">1000</attr>
Type:    incremental
<attr name="ArchivingAccountsLimit">1000</attr>
Status: completed
Started: Wed, 2015/03/04 02:35:43.151 EST
Ended:  Wed, 2015/03/04 02:35:51.981 EST
Redo log sequence range: 10 .. 11
Number of accounts: 10 out of 10 completed
Accounts:
  admin@mail-172.example.com: completed
  galsync.qva4ekog0@mail-172.example.com: completed
  ham.jgfdb3oqjf@mail-172.example.com: completed
  spam.umhx3owv3s@mail-172.example.com: completed
  user1@mail-172.example.com: completed
  user2@mail-172.example.com: completed
  user3@mail-172.example.com: completed
  user4@mail-172.example.com: completed
  user5@mail-172.example.com: completed
  virus-quarantine.gusiso1cd@mail-172.example.com: completed


<attr name="TotalAccounts">3</attr>
Total space: 14020MB
  <attr name="ArchivingAccounts">3</attr>
  Free space: 6032MB
</pre>
</pre>


====Performance Issues And Time To Complete====
Notice how the log events are soap requests and differ from the full backup above and lack the INFO [*Backup*] formatting. Important to note this if your attempting to grep for backup/restore activity. Just like with the full, it starts with an authentication/acl request and states it's starting, how many accounts found, AsyncFileCopier starting, and the label name for this backup session.


RFE's for increase backup performance:
From /opt/zimbra/log/mailbox.log


* "Improve zmrestore & zmrestoreoffline performance"
<pre>
** http://bugzilla.zimbra.com/show_bug.cgi?id=33606
2015-03-04 02:35:42,356 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] soap - AuthRequest elapsed=94
* "Improvement to backup performance"
2015-03-04 02:35:42,481 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup request started
** http://bugzilla.zimbra.com/show_bug.cgi?id=36220
2015-03-04 02:35:42,917 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Found 10 accounts on server mail-172.example.com
** Above marked as a dupe of "Reduce duration of maintenance mode during backup"
2015-03-04 02:35:43,254 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] io - AsyncFileCopier is starting
*** http://bugzilla.zimbra.com/show_bug.cgi?id=33583
2015-03-04 02:35:43,255 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup started for backup set; label: incr-20150304.073543.151
</pre>


=====What Might Be Wrong?=====
The log events are again trimmed below just for the user1 and admin account for the user backup section of the log file. Notice the absence of any statements of the accounts being put into maintenance mode.


'''Need to expand this section or refer to the part above about all the log/data gathering that support would normally ask for'''
<pre>
 
2015-03-04 02:35:44,118 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/6/index/0
First thing to check is the log file and see if anything notable stands out.
2015-03-04 02:35:44,119 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 LOADED
2015-03-04 02:35:44,120 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup is started for account user1@mail-172.example.com
2015-03-04 02:35:44,120 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Saving account information from LDAP
2015-03-04 02:35:44,127 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup has ended
2015-03-04 02:35:44,127 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Account user1@mail-172.example.com in backup set incr-20150304.073543.151: All pending file IO completed (0 out of 0)
[cut]
2015-03-04 02:35:44,212 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup is started for account admin@mail-172.example.com
2015-03-04 02:35:44,212 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Saving account information from LDAP
2015-03-04 02:35:44,215 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup has ended
2015-03-04 02:35:44,217 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Account admin@mail-172.example.com in backup set incr-20150304.073543.151: All pending file IO completed (0 out of 0)
</pre>


grep -i backup /opt/zimbra/log/mailbox.log
The incremental will then end with log events related to copying the redologs into the backup directory and then purging the redologs that can be deleted. This example states redo log sequence 9 can be deleted. We'll confirm it is later.


=====Use Auto-Group Backups Rather Than Default Style Backup=====
<pre>
2015-03-04 02:35:44,618 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] redolog - waiting for FileLogWriter.FsyncThread-1425452599560 to finish.
2015-03-04 02:35:44,628 INFO  [FileLogWriter.FsyncThread-1425452599560] [] redolog - fsync thread exiting
2015-03-04 02:35:44,628 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] redolog - FileLogWriter.FsyncThread-1425452599560 finished
2015-03-04 02:35:44,757 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] redolog - Redo log rollover took 540ms
2015-03-04 02:35:44,758 INFO  [FileLogWriter.FsyncThread-1425454544757] [] redolog - Starting fsync thread with interval 10
2015-03-04 02:35:44,776 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Found 3 redo logs to backup
2015-03-04 02:35:44,861 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Last backed-up redo log sequence = 9
2015-03-04 02:35:44,943 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Marking redo log /opt/zimbra/redolog/archive/redo-20150303.060015.816-seq9.log for deletion
2015-03-04 02:35:44,943 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Copying redo log /opt/zimbra/redolog/archive/redo-20150304.070025.476-seq10.log to /opt/zimbra/backup/tmp/incr-20150304.073543.151/redologs/redo-20150304.070025.476-seq10.log
2015-03-04 02:35:45,053 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Retaining recent redo log /opt/zimbra/redolog/archive/redo-20150304.070025.476-seq10.log
2015-03-04 02:35:45,053 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Copying redo log /opt/zimbra/redolog/archive/redo-20150304.073544.755-seq11.log to /opt/zimbra/backup/tmp/incr-20150304.073543.151/redologs/redo-20150304.073544.755-seq11.log
2015-03-04 02:35:45,067 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Retaining recent redo log /opt/zimbra/redolog/archive/redo-20150304.073544.755-seq11.log
</pre>


Having trouble completing that entire full backup during off-hours? Enter the hybrid auto-grouped mode, which combines the concept of full and incremental backup functions - you’re completely backing up a target number of accounts daily rather than running incrementals.  
The last limes confirm the backup finished successfully.


Auto-grouped mode automatically pulls in the redologs since the last run so you get incremental backups of the remaining accounts; although the incremental accounts captured via the redologs are not listed specifically in the backup account list. This still allows you to do a point in time restore for any account.
<pre>
 
2015-03-04 02:35:51,935 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup set incr-20150304.073543.151: All pending file IO completed (0 out of 0)
Please see the following for more detailed information:
2015-03-04 02:35:51,981 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] io - AsyncFileCopier is shut down
 
2015-03-04 02:35:51,981 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup finished for backup set; label: incr-20150304.073543.151
* [[Ajcody-Backup-Restore-Issues#Auto-Group_Backups_Rather_Than_Default_Method_Topics]]
2015-03-04 02:35:51,982 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Deleting redo log /opt/zimbra/redolog/archive/redo-20150303.060015.816-seq9.log
2015-03-04 02:35:52,909 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup request finished
2015-03-04 02:35:52,909 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] soap - BackupRequest elapsed=10451
</pre>


=====Need To Write Fewer Files - Add The Zip Option To Your Backup Commands=====
For an incremental, the account sub-directories will have ldap data saved also for each user but blobs and indexes aren't. That data resides in the redologs from the system under the redologs directory.


Using the zip option will compress all those thousands of single files that exist under a user's backup, decreasing performance issues that arise from writing out thousands of small files as compared to large ones. This is often seen when one is :
<pre>
[zimbra@mail-172 incr-20150304.073543.151]$ ls -R
.:
accounts  ldap  redologs  session.xml  sys


* Using nfs for the backup directory
./accounts:
* Copying/rsyncing backups to a remote server
3c3  423  434  75d  7ac  89f  a46  af0  c53  eab
* Are using some third party backup software (to tape) to archive/backup the zimbra backup sessions.


Please see the following for more information about using the Zip option:
./accounts/3c3:
97e


* [[Ajcody-Backup-Restore-Issues#The_Zip_Option_For_Backups]]
./accounts/3c3/97e:
3c397edc-c015-4d82-ae8a-1084692f8a93


=====SAN Snapshots For Backups=====
./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93:
ldap.xml


Please see:
[cut]


* [[Ajcody-Backup-Restore-Issues#SAN_Snapshot]]
./accounts/7ac:
8de


=====Cloud Backups=====
./accounts/7ac/8de:
7ac8de92-13fe-4d19-b9da-6872169fb5c9


Please see:
./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9:
ldap.xml


* [[Ajcody-Backup-Restore-Issues#Cloud_Backups]]
[cut]


=====Tape Backups=====
./ldap:
ldap-config.bak.gz  ldap.bak.gz


Use a 'secondary' network card to run your backups over if possible. This way the backup activity doesn't effect ZCS network performance over the default network card. Also, please review the following section on Zimbra and hard links and confirm if your third party backup software or process will handle restoring hard links correctly.
./redologs:
redo-20150304.070025.476-seq10.log  redo-20150304.073544.755-seq11.log


Please see:
./sys:
config.dat  current_volumes.dat  db_schema.xml  localconfig.xml  ssh  ssl  volume.dat
 
./sys/ssh:
authorized_keys  zimbra_identity  zimbra_identity.pub


* [[Ajcody-Backup-Restore-Issues#Backing_Up_Backups_-_3rd_Party_Tools_And_Software_-_Dealing_With_Directories_With_Hard_Links]]
./sys/ssl:
zimbra


===Extended How-To And Logging Events===
./sys/ssl/zimbra:
ca  commercial  jetty.pkcs12  server


====Full Backup And Log Events Generated====
./sys/ssl/zimbra/ca:
ca.key  ca.pem  ca.srl  ca.srl.old  index.txt  index.txt.attr  index.txt.old  newcerts  zmssl.cnf


Example of manually starting a full backup using the Standard zimbraBackupMode and doing a query to confirm it's completion and status.
./sys/ssl/zimbra/ca/newcerts:
1424523586.pem  1424523591.pem  1424523597.pem  1424523603.pem


<pre>
./sys/ssl/zimbra/commercial:
[zimbra@mail-172 archive]$ zmprov gs `zmhostname` zimbraBackupMode
# name mail-172.example.com
zimbraBackupMode: Standard


[zimbra@mail-172 archive]$ pwd ; ls *.log
./sys/ssl/zimbra/server:
/opt/zimbra/redolog/archive
server.crt  server.csr  server.key
redo-20150303.060015.816-seq9.log
</pre>


[zimbra@mail-172 archive]$ zmbackup -f -a all
Notice that redo log sequence 9 was indeed deleted from the redolog/archive directory.
full-20150304.012000.193


[zimbra@mail-172 archive]$ zmbackupquery -lb full-20150304.012000.193 -v
<pre>
Label:  full-20150304.012000.193
[zimbra@mail-172 archive]$ pwd ; ls
Type:    full
/opt/zimbra/redolog/archive
Status:  completed
redo-20150304.070025.476-seq10.log  redo-20150304.073544.755-seq11.log  redo-20150304.074024.058-seq12.log
Started: Tue, 2015/03/03 20:20:00.193 EST
Ended:  Tue, 2015/03/03 20:20:28.630 EST
Redo log sequence range: 10 .. 10
Number of accounts: 10 out of 10 completed
Accounts:
  admin@mail-172.example.com: completed
  galsync.qva4ekog0@mail-172.example.com: completed
  ham.jgfdb3oqjf@mail-172.example.com: completed
  spam.umhx3owv3s@mail-172.example.com: completed
  user1@mail-172.example.com: completed
  user2@mail-172.example.com: completed
  user3@mail-172.example.com: completed
  user4@mail-172.example.com: completed
  user5@mail-172.example.com: completed
  virus-quarantine.gusiso1cd@mail-172.example.com: completed
 
Total space: 14020MB
Free space: 6017MB
</pre>
</pre>


Example of what log events are generated for a full backup in /opt/zimbra/log/mailbox.log . First, you'll see the authentication/acl request and then statements about the backup request.
If you had created new users after your last full backup, you would see that there was also a full backup started as well. For example :


<pre>
<pre>
2015-03-03 20:19:58,499 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [] AuthProvider - Adding auth provider: zimbra com.zimbra.cs.service.ZimbraAuthProvider
[zimbra@mail-172 log]$ zmbackup -i
2015-03-03 20:19:59,458 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] soap - AuthRequest elapsed=834
incr-20150304.085810.220
2015-03-03 20:19:59,751 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] account - Initialized access manager: com.zimbra.cs.account.accesscontrol.ACLAccessManager
full-20150304.085818.889
2015-03-03 20:19:59,770 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup request started
 
2015-03-03 20:20:00,193 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Found 10 accounts on server mail-172.example.com
[zimbra@mail-172 log]$ zmbackupquery -a user6@mail-172.example.com
2015-03-03 20:20:00,464 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup request finished
Account: user6@mail-172.example.com
2015-03-03 20:20:00,464 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] soap - BackupRequest elapsed=760
 
    Label:  full-20150304.085818.889
    Type:   full
    Started: Wed, 2015/03/04 03:58:18.889 EST
    Ended:  Wed, 2015/03/04 03:58:25.997 EST
    Acct ID: 4e898a68-1886-4a80-b3dc-ac2ab650bf8c
 
[zimbra@mail-172 log]$ zmbackupquery -lb full-20150304.085818.889 -v
Label:   full-20150304.085818.889
Type:   full
Statuscompleted
Started: Wed, 2015/03/04 03:58:18.889 EST
Ended:  Wed, 2015/03/04 03:58:25.997 EST
Redo log sequence range: 14 .. 14
Number of accounts: 1 out of 1 completed
Accounts:
  user6@mail-172.example.com: completed
 
Total space: 14020MB
  Free space: 6038MB
</pre>
</pre>


Your log events will now reference Backup in them. AsyncFileCopier will start and it will give the backup session label for this backup job.
===Contents Of accounts.xml And sessions.xml From The Full And Incremental Backup Examples===
<pre>
2015-03-03 20:20:01,092 INFO  [FullBackupThread] [] io - AsyncFileCopier is starting
2015-03-03 20:20:01,187 INFO  [FullBackupThread] [] backup - Full backup started for backup set; label: full-20150304.012000.193
</pre>


It proceeds to backing up your accounts. I'll list the backup log events for just two accounts on this system - user1@mail-172.example.com and admin@mail-172.example.com . I'll provide the zimbraId, email address, and mailboxID information for user1@mail-172.example.com below so you can confirm it's reported in the logs correctly as well.
First, see what zmbackupquery shows for a particular user in regards to the backup labels they are included in.


<pre>
<pre>
[zimbra@mail-172 redolog]$ zmprov ga user1@mail-172.example.com zimbraId
$ zmbackupquery -a user1@mail-172.example.com
# name user1@mail-172.example.com
Account: user1@mail-172.example.com
zimbraId: 3c397edc-c015-4d82-ae8a-1084692f8a93
 
    Label:  incr-20150304.073543.151
    Type:    incremental
    Started: Wed, 2015/03/04 02:35:43.151 EST
    Ended:  Wed, 2015/03/04 02:35:51.981 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93


[zimbra@mail-172 redolog]$ zmprov gmi user1@mail-172.example.com
    Label:  full-20150304.012000.193
mailboxId: 6
    Type:    full
quotaUsed: 1209
    Started: Tue, 2015/03/03 20:20:00.193 EST
    Ended:  Tue, 2015/03/03 20:20:28.630 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93
</pre>
</pre>


Now, continuing with the log events. Notice the reference to the redo log file sequence, putting the account into and out of maintenance, number of blobs and index files backed up, and the statement it's finished backing up the account. Also within the section we can see the user's zimbraId [3c397edc-c015-4d82-ae8a-1084692f8a93] , email address [user1@mail-172.example.com] , and mailboxId [Mailbox 6].
Now let's review the accounts/session xml files that list all the users within those particular backup sessions. The /opt/zimbra/backup/accounts.xml is the referenced to determine what is the last know full backup session the user was in. If the /opt/zimbra/backup/accounts.xml is corrupted, unreadable, or incorrect - this can cause issues with restoring users.
 
Also notice that it also references the zimbraId for the user at the time of the backup. If you were to delete a user and then create a new account using the same email address [user1@mail-172.example.com] that new account will have a different zimbraId and therefor the backup/restore commands will not work as you might expected because of the differing zimbraId. The zmrestore command only can take an email address for the -a option, it can not use the zimbraID.


<pre>
<pre>
[cut]
[zimbra@mail-172 backup]$ pwd
2015-03-03 20:20:06,560 INFO  [FullBackupThread] [name=user1@mail-172.example.com;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/6/index/0
/opt/zimbra/backup
2015-03-03 20:20:06,560 INFO  [FullBackupThread] [name=user1@mail-172.example.com;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 LOADED
 
2015-03-03 20:20:06,560 INFO  [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - redo log file sequence is 10 at full backup for user1@mail-172.example.com
[zimbra@mail-172 backup]$ cat accounts.xml
2015-03-03 20:20:06,565 INFO  [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - Full backup started for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) mailbox 6
<?xml version="1.0" encoding="utf-8"?>
2015-03-03 20:20:06,565 INFO  [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] mailbox - Putting mailbox 6 under maintenance.
2015-03-03 20:20:06,818 INFO  [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - Number of blobs to backup for mailbox 6: 1
2015-03-03 20:20:06,818 INFO  [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] mailbox - Ending maintenance on mailbox 6.
2015-03-03 20:20:06,818 INFO  [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 AVAILABLE
2015-03-03 20:20:06,966 INFO  [BackupSetWorkerThread-BLOB] [name=user1@mail-172.example.com;mid=6;] backup - Stored 1 blob files from volume message1
2015-03-03 20:20:06,966 INFO  [BackupSetWorkerThread-BLOB] [name=user1@mail-172.example.com;mid=6;] backup - Stored 1 blob files
2015-03-03 20:20:07,014 INFO  [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - Account user1@mail-172.example.com in backup set full-20150304.012000.193: All pending file IO completed (1 out of 1)
2015-03-03 20:20:07,016 INFO  [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] mailbox - Putting mailbox 6 under maintenance.
2015-03-03 20:20:07,021 INFO  [BackupSetWorkerThread-INDEX] [name=user1@mail-172.example.com;mid=6;] backup - Stored 2 index files
2015-03-03 20:20:07,496 INFO  [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - Account user1@mail-172.example.com in backup set full-20150304.012000.193: All pending file IO completed (3 out of 3)
2015-03-03 20:20:07,497 INFO  [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] mailbox - Ending maintenance on mailbox 6.
2015-03-03 20:20:07,497 INFO  [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 AVAILABLE
2015-03-03 20:20:07,497 INFO  [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - Account user1@mail-172.example.com in backup set full-20150304.012000.193: All pending file IO completed (3 out of 3)
2015-03-03 20:20:07,497 INFO  [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - Full backup finished for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) mailbox 6
</pre>


Events for the admin user's backup:
<backupMetadata xmlns="urn:zimbraBackupMeta" version="7.2" dbVersion="103">
 
  <allAccounts>
<pre>
    <account zimbraId="af06b0b4-3160-4ea3-a55b-81163a9914a0" email="user5@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193">
2015-03-03 20:20:09,894 INFO  [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - redo log file sequence is 10 at full backup for admin@mail-172.example.com
    </account>
2015-03-03 20:20:09,914 INFO  [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - Full backup started for account admin@mail-172.example.com (7ac8de92-13fe-4d19-b9da-6872169fb5c9) mailbox 1
    <account zimbraId="75d3d364-a384-4077-8ac5-87b6960b57cd" email="user3@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193">
2015-03-03 20:20:09,915 INFO  [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] mailbox - Putting mailbox 1 under maintenance.
    </account>
2015-03-03 20:20:10,176 INFO  [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - Number of blobs to backup for mailbox 1: 73
    <account zimbraId="4343a299-5259-4826-b1c1-becbfbb9b8f0" email="galsync.qva4ekog0@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193">
2015-03-03 20:20:10,177 INFO  [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] mailbox - Ending maintenance on mailbox 1.
    </account>
2015-03-03 20:20:10,177 INFO  [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] mbxmgr - Mailbox 1 account 7ac8de92-13fe-4d19-b9da-6872169fb5c9 AVAILABLE
    <account zimbraId="89fc5986-31a3-406e-8084-0ce0b98a251f" email="user2@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193">
2015-03-03 20:20:10,343 INFO  [BackupSetWorkerThread-BLOB] [name=admin@mail-172.example.com;mid=1;] backup - Stored 73 blob files from volume message1
    </account>
2015-03-03 20:20:10,353 INFO  [BackupSetWorkerThread-BLOB] [name=admin@mail-172.example.com;mid=1;] backup - Stored 73 blob files
    <account zimbraId="42380a6e-5718-4d68-ab37-e8d5bc9e6977" email="spam.umhx3owv3s@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193">
2015-03-03 20:20:11,047 INFO  [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - Account admin@mail-172.example.com in backup set full-20150304.012000.193: All pending file IO completed (73 out of 73)
    </account>
2015-03-03 20:20:11,051 INFO  [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] mailbox - Putting mailbox 1 under maintenance.
    <account zimbraId="eab3d56e-ffd9-4412-95de-48f102d2d252" email="user4@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193">
2015-03-03 20:20:11,108 INFO  [BackupSetWorkerThread-INDEX] [name=admin@mail-172.example.com;mid=1;] backup - Stored 26 index files
    </account>
2015-03-03 20:20:12,450 INFO  [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - Account admin@mail-172.example.com in backup set full-20150304.012000.193: All pending file IO completed (99 out of 99)
    <account zimbraId="7ac8de92-13fe-4d19-b9da-6872169fb5c9" email="admin@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193">
2015-03-03 20:20:12,463 INFO  [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] mailbox - Ending maintenance on mailbox 1.
    </account>
2015-03-03 20:20:12,463 INFO  [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] mbxmgr - Mailbox 1 account 7ac8de92-13fe-4d19-b9da-6872169fb5c9 AVAILABLE
    <account zimbraId="4e898a68-1886-4a80-b3dc-ac2ab650bf8c" email="user6@mail-172.example.com" latestFullBackupLabel="full-20150304.085818.889">
2015-03-03 20:20:12,463 INFO  [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - Account admin@mail-172.example.com in backup set full-20150304.012000.193: All pending file IO completed (99 out of 99)
    </account>
2015-03-03 20:20:12,463 INFO  [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - Full backup finished for account admin@mail-172.example.com (7ac8de92-13fe-4d19-b9da-6872169fb5c9) mailbox 1
    <account zimbraId="a46dfc59-a3e9-4f09-adfb-af8efc42aa9f" email="virus-quarantine.gusiso1cd@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193">
    </account>
    <account zimbraId="3c397edc-c015-4d82-ae8a-1084692f8a93" email="user1@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193">
    </account>
    <account zimbraId="c5315c5d-97cb-4d84-bc70-374a67f13309" email="ham.jgfdb3oqjf@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193">
    </account>
  </allAccounts>
</backupMetadata>
</pre>
</pre>


And then a final line confirming completion and the number of accounts successfully backed up.
In each backup session, there will be a sessions.xml file that acts as a reference or summary file of the backup. At the top, you'll see what version of ZCS was running and what options the backup environment had set. It will then report the final status for each user for the backup attempt against the account. At the end, there's a section in regards to errors and if any were encountered.  


<pre>
<pre>
2015-03-03 20:20:12,463 INFO  [FullBackupThread] [] backup - Finished backing up 10 of 10 accounts
[zimbra@mail-172 backup]$ cd sessions/
</pre>
[zimbra@mail-172 sessions]$ cd full-20150304.012000.193
[zimbra@mail-172 full-20150304.012000.193]$ cat session.xml
<?xml version="1.0" encoding="utf-8"?>


One thing to point out is the log events were mentioning the redo log sequence as being number 10 but in the redolog/archive directory there is only a sequence 9 redo log. This is because the "seq 10" is actually the /opt/zimbra/redolog/redo.log file. When the redo.log file rolls over and is moved to the archive directory it will have seq10 in it's filename.  
<backupMetadata xmlns="urn:zimbraBackupMeta" version="7.2" dbVersion="103">
 
  <backupSet label="full-20150304.012000.193" zcsRelease="8.6.0_GA_1153 20141215151258 20141215-1518 NETWORK" startTime="1425432000193" endTime="1425432028630" minRedoSeq="10" maxRedoSeq="10" sharedBlobsZipped="true" sharedBlobsZipNameDigestChars="1" sharedBlobsDirectoryDepth="5" sharedBlobsCharsPerDirectory="2" type="full" accountsDirectoryDepth="2">
<pre>
    <desc>Full backup</desc>
[zimbra@mail-172 archive]$ pwd ; ls -la *.log
    <accounts>
/opt/zimbra/redolog/archive
      <account zimbraId="af06b0b4-3160-4ea3-a55b-81163a9914a0" email="user5@mail-172.example.com" status="COMPLETED">
-rw-r----- 1 zimbra zimbra 146723 Mar  3 20:23 redo-20150303.060015.816-seq9.log
      </account>
</pre>
      <account zimbraId="75d3d364-a384-4077-8ac5-87b6960b57cd" email="user3@mail-172.example.com" status="COMPLETED">
 
      </account>
The contents of the backup session directory. I removed the sub-directories under accounts that weren't either the admin or user1 account. Notice the zimbraId is used for the sub-directory naming scheme.
      <account zimbraId="89fc5986-31a3-406e-8084-0ce0b98a251f" email="user2@mail-172.example.com" status="COMPLETED">
 
      </account>
<pre>
      <account zimbraId="4343a299-5259-4826-b1c1-becbfbb9b8f0" email="galsync.qva4ekog0@mail-172.example.com" status="COMPLETED">
[zimbra@mail-172 full-20150304.012000.193]$ ls -R
      </account>
.:
      <account zimbraId="eab3d56e-ffd9-4412-95de-48f102d2d252" email="user4@mail-172.example.com" status="COMPLETED">
accounts  ldap  session.xml  shared_blobs  sys
      </account>
 
      <account zimbraId="42380a6e-5718-4d68-ab37-e8d5bc9e6977" email="spam.umhx3owv3s@mail-172.example.com" status="COMPLETED">
./accounts:
      </account>
3c3  423  434  75d  7ac  89f  a46  af0  c53  eab
      <account zimbraId="7ac8de92-13fe-4d19-b9da-6872169fb5c9" email="admin@mail-172.example.com" status="COMPLETED">
 
      </account>
./accounts/3c3:
      <account zimbraId="a46dfc59-a3e9-4f09-adfb-af8efc42aa9f" email="virus-quarantine.gusiso1cd@mail-172.example.com" status="COMPLETED">
97e
      </account>
 
      <account zimbraId="3c397edc-c015-4d82-ae8a-1084692f8a93" email="user1@mail-172.example.com" status="COMPLETED">
./accounts/3c3/97e:
      </account>
3c397edc-c015-4d82-ae8a-1084692f8a93
      <account zimbraId="c5315c5d-97cb-4d84-bc70-374a67f13309" email="ham.jgfdb3oqjf@mail-172.example.com" status="COMPLETED">
 
      </account>
./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93:
    </accounts>
blobs  db  index  ldap.xml  meta.xml
    <stats>
 
      <counter name="total_time" numSamples="1" sum="28437" unit="ms">
./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93/blobs:
      </counter>
blobs-1.zip
      <counter name="sysdb_time" numSamples="1" sum="1111" unit="ms">
 
      </counter>
./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93/db:
      <counter name="sysdb_bytes" numSamples="1" sum="35210" unit="bytes">
db_schema.xml  mail_item.dat  mailbox.dat  mailbox_metadata.dat  open_conversation.dat  tag.dat  tagged_item.dat
      </counter>
 
      <counter name="redologs_time" numSamples="0" sum="0" unit="ms">
./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93/index:
      </counter>
0
      <counter name="redologs_bytes" numSamples="0" sum="0" unit="bytes">
 
      </counter>
./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93/index/0:
      <counter name="accounts_time" numSamples="10" sum="4827" unit="ms">
segments.gen  segments_1
      </counter>
 
      <counter name="ldap_time" numSamples="10" sum="172" unit="ms">
[cut]
      </counter>
 
      <counter name="ldap_bytes" numSamples="10" sum="25665" unit="bytes">
./accounts/7ac:
      </counter>
8de
      <counter name="db_time" numSamples="10" sum="4787" unit="ms">
 
      </counter>
./accounts/7ac/8de:
      <counter name="db_bytes" numSamples="10" sum="78690" unit="bytes">
7ac8de92-13fe-4d19-b9da-6872169fb5c9
      </counter>
      <counter name="db_digest_map_load_time" numSamples="10" sum="2400" unit="ms">
      </counter>
      <counter name="msgs_time" numSamples="10" sum="451" unit="ms">
      </counter>
      <counter name="msg_copy_time" numSamples="74" sum="25" unit="ms">
      </counter>
      <counter name="msg_copy_bytes" numSamples="74" sum="149611" unit="bytes">
      </counter>
      <counter name="msg_link_time" numSamples="0" sum="0" unit="ms">
      </counter>
      <counter name="index_time" numSamples="48" sum="80" unit="ms">
      </counter>
      <counter name="index_bytes" numSamples="38" sum="73510" unit="bytes">
      </counter>
    </stats>
    <errors>
    </errors>
  </backupSet>
</backupMetadata>
</pre>


./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9:
An incremental session.xml file is similar to a full backups session.xml .
blobs  db  index  ldap.xml meta.xml


./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9/blobs:
<pre>
blobs-1.zip  blobs-2.zip  blobs-3.zip  blobs-4.zip
[zimbra@mail-172 sessions]$ cd incr-20150304.073543.151
[zimbra@mail-172 incr-20150304.073543.151]$ cat session.xml
<?xml version="1.0" encoding="utf-8"?>


./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9/db:
<backupMetadata xmlns="urn:zimbraBackupMeta" version="7.2" dbVersion="103">
db_schema.xml  mail_item.dat  mailbox.dat  open_conversation.dat  tag.dat  tagged_item.dat
  <backupSet label="incr-20150304.073543.151" zcsRelease="8.6.0_GA_1153 20141215151258 20141215-1518 NETWORK" startTime="1425454543151" endTime="1425454551981" minRedoSeq="10" maxRedoSeq="11" sharedBlobsZipped="true" sharedBlobsZipNameDigestChars="1" sharedBlobsDirectoryDepth="5" sharedBlobsCharsPerDirectory="2" type="incremental" accountsDirectoryDepth="2">
 
    <desc>Incremental backup</desc>
./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9/index:
    <accounts>
0
      <account zimbraId="af06b0b4-3160-4ea3-a55b-81163a9914a0" email="user5@mail-172.example.com" status="COMPLETED">
 
      </account>
./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9/index/0:
      <account zimbraId="75d3d364-a384-4077-8ac5-87b6960b57cd" email="user3@mail-172.example.com" status="COMPLETED">
_0.fdt  _0.fnm  _0.nrm  _0.tii  _1.fdt  _1.fnm  _1.nrm  _1.tii  _2.fdt  _2.fnm  _2.nrm  _2.tii  segments.gen
      </account>
_0.fdx  _0.frq  _0.prx  _0.tis  _1.fdx  _1.frq  _1.prx  _1.tis  _2.fdx  _2.frq  _2.prx  _2.tis  segments_3
      <account zimbraId="89fc5986-31a3-406e-8084-0ce0b98a251f" email="user2@mail-172.example.com" status="COMPLETED">
 
      </account>
[cut]
      <account zimbraId="4343a299-5259-4826-b1c1-becbfbb9b8f0" email="galsync.qva4ekog0@mail-172.example.com" status="COMPLETED">
 
      </account>
./ldap:
      <account zimbraId="eab3d56e-ffd9-4412-95de-48f102d2d252" email="user4@mail-172.example.com" status="COMPLETED">
ldap-config.bak.gz  ldap.bak.gz
      </account>
 
      <account zimbraId="42380a6e-5718-4d68-ab37-e8d5bc9e6977" email="spam.umhx3owv3s@mail-172.example.com" status="COMPLETED">
./shared_blobs:
      </account>
 
      <account zimbraId="7ac8de92-13fe-4d19-b9da-6872169fb5c9" email="admin@mail-172.example.com" status="COMPLETED">
./sys:
      </account>
config.dat  current_volumes.dat  db_schema.xml  localconfig.xml  ssh  ssl  volume.dat
      <account zimbraId="a46dfc59-a3e9-4f09-adfb-af8efc42aa9f" email="virus-quarantine.gusiso1cd@mail-172.example.com" status="COMPLETED">
 
      </account>
./sys/ssh:
      <account zimbraId="3c397edc-c015-4d82-ae8a-1084692f8a93" email="user1@mail-172.example.com" status="COMPLETED">
authorized_keys  zimbra_identity  zimbra_identity.pub
      </account>
 
      <account zimbraId="c5315c5d-97cb-4d84-bc70-374a67f13309" email="ham.jgfdb3oqjf@mail-172.example.com" status="COMPLETED">
./sys/ssl:
      </account>
zimbra
    </accounts>
 
    <stats>
./sys/ssl/zimbra:
      <counter name="total_time" numSamples="1" sum="8830" unit="ms">
ca  commercial  jetty.pkcs12  server
      </counter>
 
      <counter name="sysdb_time" numSamples="1" sum="563" unit="ms">
./sys/ssl/zimbra/ca:
      </counter>
ca.key  ca.pem  ca.srl  ca.srl.old  index.txt  index.txt.attr  index.txt.old  newcerts  zmssl.cnf
      <counter name="sysdb_bytes" numSamples="1" sum="35210" unit="bytes">
 
      </counter>
./sys/ssl/zimbra/ca/newcerts:
      <counter name="redologs_time" numSamples="1" sum="310" unit="ms">
1424523586.pem  1424523591.pem  1424523597.pem  1424523603.pem
      </counter>
 
      <counter name="redologs_bytes" numSamples="1" sum="126307" unit="bytes">
./sys/ssl/zimbra/commercial:
      </counter>
 
      <counter name="accounts_time" numSamples="10" sum="196" unit="ms">
./sys/ssl/zimbra/server:
      </counter>
server.crt  server.csr  server.key
      <counter name="ldap_time" numSamples="10" sum="129" unit="ms">
</pre>
      </counter>
 
      <counter name="ldap_bytes" numSamples="10" sum="25665" unit="bytes">
====Incremental Backup And Log Events Generated====
      </counter>
 
      <counter name="db_time" numSamples="0" sum="0" unit="ms">
Example of manually starting an incremental backup using the Standard zimbraBackupMode and doing a query to confirm it's completion and status.
      </counter>
 
      <counter name="db_bytes" numSamples="0" sum="0" unit="bytes">
<pre>
      </counter>
[zimbra@mail-172 archive]$ zmprov gs `zmhostname` zimbraBackupMode
      <counter name="db_digest_map_load_time" numSamples="0" sum="0" unit="ms">
# name mail-172.example.com
      </counter>
zimbraBackupMode: Standard
      <counter name="msgs_time" numSamples="0" sum="0" unit="ms">
 
      </counter>
[zimbra@mail-172 archive]$ ls
      <counter name="msg_copy_time" numSamples="0" sum="0" unit="ms">
redo-20150303.060015.816-seq9.log  redo-20150304.070025.476-seq10.log
      </counter>
      <counter name="msg_copy_bytes" numSamples="0" sum="0" unit="bytes">
      </counter>
      <counter name="msg_link_time" numSamples="0" sum="0" unit="ms">
      </counter>
      <counter name="index_time" numSamples="0" sum="0" unit="ms">
      </counter>
      <counter name="index_bytes" numSamples="0" sum="0" unit="bytes">
      </counter>
    </stats>
    <errors>
    </errors>
  </backupSet>
</backupMetadata>
</pre>
 
===User And Log Events Generated===
 
Please beware that prior backups were deleted and new backups were made on this ZCS test server. So the backup label names might not match prior examples.
 
For this example, I'm going to replicate a common situation that system administrators find themselves when trying to restore a user and they fail to include the proper restore options to have the restore stop prior to replaying a message that was deleted by a user. Proper examples of doing the restore will be shown afterwards.


[zimbra@mail-172 archive]$ zmbackup -i
Information on the account we'll restore:
incr-20150304.073543.151


[zimbra@mail-172 archive]$ zmbackupquery -lb incr-20150304.073543.151 -v
<pre>
Label:  incr-20150304.073543.151
[zimbra@mail-172 backup]$ zmprov gmi user1@mail-172.example.com
Type:    incremental
mailboxId: 6
Status:  completed
quotaUsed: 1209
Started: Wed, 2015/03/04 02:35:43.151 EST
Ended:  Wed, 2015/03/04 02:35:51.981 EST
Redo log sequence range: 10 .. 11
Number of accounts: 10 out of 10 completed
Accounts:
  admin@mail-172.example.com: completed
  galsync.qva4ekog0@mail-172.example.com: completed
  ham.jgfdb3oqjf@mail-172.example.com: completed
  spam.umhx3owv3s@mail-172.example.com: completed
  user1@mail-172.example.com: completed
  user2@mail-172.example.com: completed
  user3@mail-172.example.com: completed
  user4@mail-172.example.com: completed
  user5@mail-172.example.com: completed
  virus-quarantine.gusiso1cd@mail-172.example.com: completed


Total space: 14020MB
[zimbra@mail-172 backup]$ zmprov ga user1@mail-172.example.com zimbraId
Free space: 6032MB
# name user1@mail-172.example.com
zimbraId: 3c397edc-c015-4d82-ae8a-1084692f8a93
</pre>
</pre>


Notice how the log events are soap requests and differ from the full backup above and lack the INFO [*Backup*] formatting. Important to note this if your attempting to grep for backup/restore activity. Just like with the full, it starts with an authentication/acl request and states it's starting, how many accounts found, AsyncFileCopier starting, and the label name for this backup session.
Example of error you'll receive if you attempt to use the zimbraId rather than the email address for the zmrestore command. This is important to note if your trying to restore an account that was deleted and then later created again and the various backups and accounts.xml file has different zimbraId's associated with the same email address.
 
<pre>
[zimbra@mail-172 backup]$ zmrestore -a 3c397edc-c015-4d82-ae8a-1084692f8a93 -ca -pre restore1_
Error occurred: invalid request: invalid account email address: 3c397edc-c015-4d82-ae8a-1084692f8a93
</pre>


From /opt/zimbra/log/mailbox.log
Showing that we have one message in the Inbox for the user we'll be restoring and also the message id of that email :


<pre>
<pre>
2015-03-04 02:35:42,356 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] soap - AuthRequest elapsed=94
[zimbra@mail-172 backup]$ zmmailbox -z -m user1@mail-172.example.com gaf
2015-03-04 02:35:42,481 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup request started
        Id  View      Unread  Msg Count  Path
2015-03-04 02:35:42,917 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Found 10 accounts on server mail-172.example.com
----------  ----  ----------  ----------  ----------
2015-03-04 02:35:43,254 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] io - AsyncFileCopier is starting
        1  unkn          0          0  /
2015-03-04 02:35:43,255 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup started for backup set; label: incr-20150304.073543.151
        16  docu          0          0  /Briefcase
        10  appo          0           0 /Calendar
        14  mess          0          0  /Chats
        7  cont          0          0  /Contacts
        6  mess          0          0  /Drafts
        13  cont          0           0 /Emailed Contacts
        2  mess          1          1 /Inbox
        4  mess          0          0 /Junk
        5  mess          0           0 /Sent
        15  task          0           0 /Tasks
        3 unkn          0           0 /Trash
 
[zimbra@mail-172 backup]$ zmmailbox -z -m user1@mail-172.example.com s -t mess in:Inbox
num: 1, more: false
 
    Id  Type  From                  Subject                                            Date
  ----  ----  -------------------- -------------------------------------------------- --------------
1. 257  mess  admin                 Test1                                              02/23/15 09:03
</pre>
</pre>


The log events are again trimmed below just for the user1 and admin account for the user backup section of the log file. Notice the absence of any statements of the accounts being put into maintenance mode.
Now to delete the email from the account:


<pre>
<pre>
2015-03-04 02:35:44,118 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/6/index/0
[zimbra@mail-172 backup]$ zmmailbox -z -m user1@mail-172.example.com dm 257
2015-03-04 02:35:44,119 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 LOADED
 
2015-03-04 02:35:44,120 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup is started for account user1@mail-172.example.com
And to confirm it's no longer there:
2015-03-04 02:35:44,120 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Saving account information from LDAP
2015-03-04 02:35:44,127 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup has ended
2015-03-04 02:35:44,127 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Account user1@mail-172.example.com in backup set incr-20150304.073543.151: All pending file IO completed (0 out of 0)
[cut]
2015-03-04 02:35:44,212 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup is started for account admin@mail-172.example.com
2015-03-04 02:35:44,212 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Saving account information from LDAP
2015-03-04 02:35:44,215 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup has ended
2015-03-04 02:35:44,217 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Account admin@mail-172.example.com in backup set incr-20150304.073543.151: All pending file IO completed (0 out of 0)
</pre>


The incremental will then end with log events related to copying the redologs into the backup directory and then purging the redologs that can be deleted. This example states redo log sequence 9 can be deleted. We'll confirm it is later.
[zimbra@mail-172 backup]$ zmmailbox -z -m user1@mail-172.example.com s -t mess in:Inbox
 
num: 0, more: false
<pre>
2015-03-04 02:35:44,618 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] redolog - waiting for FileLogWriter.FsyncThread-1425452599560 to finish.
2015-03-04 02:35:44,628 INFO  [FileLogWriter.FsyncThread-1425452599560] [] redolog - fsync thread exiting
2015-03-04 02:35:44,628 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] redolog - FileLogWriter.FsyncThread-1425452599560 finished
2015-03-04 02:35:44,757 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] redolog - Redo log rollover took 540ms
2015-03-04 02:35:44,758 INFO  [FileLogWriter.FsyncThread-1425454544757] [] redolog - Starting fsync thread with interval 10
2015-03-04 02:35:44,776 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Found 3 redo logs to backup
2015-03-04 02:35:44,861 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Last backed-up redo log sequence = 9
2015-03-04 02:35:44,943 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Marking redo log /opt/zimbra/redolog/archive/redo-20150303.060015.816-seq9.log for deletion
2015-03-04 02:35:44,943 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Copying redo log /opt/zimbra/redolog/archive/redo-20150304.070025.476-seq10.log to /opt/zimbra/backup/tmp/incr-20150304.073543.151/redologs/redo-20150304.070025.476-seq10.log
2015-03-04 02:35:45,053 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Retaining recent redo log /opt/zimbra/redolog/archive/redo-20150304.070025.476-seq10.log
2015-03-04 02:35:45,053 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Copying redo log /opt/zimbra/redolog/archive/redo-20150304.073544.755-seq11.log to /opt/zimbra/backup/tmp/incr-20150304.073543.151/redologs/redo-20150304.073544.755-seq11.log
2015-03-04 02:35:45,067 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Retaining recent redo log /opt/zimbra/redolog/archive/redo-20150304.073544.755-seq11.log
</pre>
</pre>


The last limes confirm the backup finished successfully.
Below is an example of a delete operation being logged in the redo.log and how to query events for a particular user with the zmredodump command. Notice the DeleteItem events at the end.


<pre>
<pre>
2015-03-04 02:35:51,935 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup set incr-20150304.073543.151: All pending file IO completed (0 out of 0)
[zimbra@mail-172 backup]$ zmredodump -m 6 --show-blob /opt/zimbra/redolog/redo.log
2015-03-04 02:35:51,981 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] io - AsyncFileCopier is shut down
VERIFYING: /opt/zimbra/redolog/redo.log
2015-03-04 02:35:51,981 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup finished for backup set; label: incr-20150304.073543.151
HEADER
2015-03-04 02:35:51,982 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Deleting redo log /opt/zimbra/redolog/archive/redo-20150303.060015.816-seq9.log
------
2015-03-04 02:35:52,909 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup request finished
sequence: 14
2015-03-04 02:35:52,909 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] soap - BackupRequest elapsed=10451
open:    1
filesize: 512
serverId: 4407a03f-894b-4b1a-817a-684fc34c4f60
created:  Wed, 2015/03/04 03:58:10.876 EST (1425459490876)
first op: Wed, 2015/03/04 03:58:18.744 EST (1425459498744)
last op:  Wed, 2015/03/04 03:58:18.744 EST (1425459498744) (not up to date)
version1.42
------
[00001601 - 0000162e: 46 bytes; tstamp: 2015/03/04 04:00:26.677 EST] txn 1425452599.101 [PurgeOldMessages] ver=1.42, tstamp=1425459626677, change=1508, mailbox=6
[0000162f - 00001660: 50 bytes; tstamp: 2015/03/04 04:00:26.683 EST] txn 1425452599.101 [CommitTxn] ver=1.42, tstamp=1425459626683, mailbox=6, txnType=PurgeOldMessages
[00002691 - 000026be: 46 bytes; tstamp: 2015/03/04 04:10:26.816 EST] txn 1425452599.113 [PurgeOldMessages] ver=1.42, tstamp=1425460226816, change=1509, mailbox=6
[000026bf - 000026f0: 50 bytes; tstamp: 2015/03/04 04:10:26.832 EST] txn 1425452599.113 [CommitTxn] ver=1.42, tstamp=1425460226832, mailbox=6, txnType=PurgeOldMessages
[00002ab1 - 00002ade: 46 bytes; tstamp: 2015/03/04 04:21:27.376 EST] txn 1425452599.124 [PurgeOldMessages] ver=1.42, tstamp=1425460887376, change=1510, mailbox=6
[00002adf - 00002b10: 50 bytes; tstamp: 2015/03/04 04:21:27.381 EST] txn 1425452599.124 [CommitTxn] ver=1.42, tstamp=1425460887381, mailbox=6, txnType=PurgeOldMessages
[00002ed1 - 00002efe: 46 bytes; tstamp: 2015/03/04 04:32:27.610 EST] txn 1425452599.135 [PurgeOldMessages] ver=1.42, tstamp=1425461547610, change=1511, mailbox=6
[00002eff - 00002f30: 50 bytes; tstamp: 2015/03/04 04:32:27.617 EST] txn 1425452599.135 [CommitTxn] ver=1.42, tstamp=1425461547617, mailbox=6, txnType=PurgeOldMessages
[000032f1 - 0000331e: 46 bytes; tstamp: 2015/03/04 04:43:27.828 EST] txn 1425452599.146 [PurgeOldMessages] ver=1.42, tstamp=1425462207828, change=1512, mailbox=6
[0000331f - 00003350: 50 bytes; tstamp: 2015/03/04 04:43:27.839 EST] txn 1425452599.146 [CommitTxn] ver=1.42, tstamp=1425462207839, mailbox=6, txnType=PurgeOldMessages
[000034d1 - 0000350c: 60 bytes; tstamp: 2015/03/04 04:48:18.119 EST] txn 1425452599.151 [DeleteItem] ver=1.42, tstamp=1425462498119, change=1513, mailbox=6, ids=[257], type=message
[0000350d - 0000353e: 50 bytes; tstamp: 2015/03/04 04:48:18.306 EST] txn 1425452599.151 [CommitTxn] ver=1.42, tstamp=1425462498306, mailbox=6, txnType=DeleteItem
</pre>
</pre>


For an incremental, the account sub-directories will have ldap data saved also for each user but blobs and indexes aren't. That data resides in the redologs from the system under the redologs directory.
Listing of the backups that are currently associated with the account we'll be restoring:


<pre>
<pre>
[zimbra@mail-172 incr-20150304.073543.151]$ ls -R
[zimbra@mail-172 backup]$ zmbackupquery -a user1@mail-172.example.com
.:
Account: user1@mail-172.example.com
accounts  ldap  redologs  session.xml  sys


./accounts:
    Label:  incr-20150304.085810.220
3c3  423  434  75d  7ac  89f  a46  af0  c53  eab
    Type:    incremental
    Started: Wed, 2015/03/04 03:58:10.220 EST
    Ended:  Wed, 2015/03/04 03:58:16.393 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93


./accounts/3c3:
    Label:  full-20150304.082015.535
97e
    Type:    full
    Started: Wed, 2015/03/04 03:20:15.535 EST
    Ended:  Wed, 2015/03/04 03:20:39.109 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93
</pre>


./accounts/3c3/97e:
Run a zmrestore with no other options other than the account flag and the email address of the account we are restoring. '''Note''' - this example is to show why this often times isn't what you'll do in real circumstances and will require other zmrestore options to achieve what you want.
3c397edc-c015-4d82-ae8a-1084692f8a93


./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93:
<pre>
ldap.xml
[zimbra@mail-172 backup]$ zmrestore -a user1@mail-172.example.com
</pre>


[cut]
From the /opt/zimbra/log/mailbox.log we have the following log events. We see the restore request kick off and the full backup label that it'll reference as the starting point. It also lists the zimbraId rather than the email address for the restore initially.


./accounts/7ac:
<pre>
8de
2015-03-04 04:53:02,221 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - AuthRequest elapsed=60
2015-03-04 04:53:02,519 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is starting
2015-03-04 04:53:02,565 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Using full backup full-20150304.082015.535 as starting point for restoring account 3c397edc-c015-4d82-ae8a-1084692f8a93
</pre>


./accounts/7ac/8de:
And then we'll see the reference to the email address and the mailboxId associated with the account/zimbraId .
7ac8de92-13fe-4d19-b9da-6872169fb5c9
<pre>
2015-03-04 04:53:02,993 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - waiting for FileLogWriter.FsyncThread-1425459490899 to finish.
2015-03-04 04:53:02,997 INFO  [FileLogWriter.FsyncThread-1425459490899] [] redolog - fsync thread exiting
2015-03-04 04:53:02,997 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - FileLogWriter.FsyncThread-1425459490899 finished
2015-03-04 04:53:03,028 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - Redo log rollover took 463ms
2015-03-04 04:53:03,028 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore started for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6
</pre>


./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9:
Since we are basically over-writing the account, the account is placed into maintenance mode first and then existing data is deleted. Only after the account has been deleted will the account be restored using the backup data. Since we didn't use the -ca and -pre options for zmrestore, this account will reuse it's existing zimbraId, mailboxId, and of course - email address.
ldap.xml


[cut]
<pre>
2015-03-04 04:53:03,042 INFO  [FileLogWriter.FsyncThread-1425462783028] [] redolog - Starting fsync thread with interval 10
2015-03-04 04:53:03,096 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - Putting mailbox 6 under maintenance.
2015-03-04 04:53:03,097 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - clearing contents of mailbox 6, group 6
2015-03-04 04:53:03,209 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restoring to mailbox id 6 for existing account 3c397edc-c015-4d82-ae8a-1084692f8a93.
2015-03-04 04:53:03,256 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loading schema with version 7.2
2015-03-04 04:53:03,287 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Renaming volume_id column to locator in source table mail_item
2015-03-04 04:53:03,584 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Deleting directory /opt/zimbra/store/0/6/msg
2015-03-04 04:53:03,599 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 1 blobs during restore
2015-03-04 04:53:03,622 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Deleting directory /opt/zimbra/index/0/6/index
</pre>


./ldap:
At this point, the data for the account was restored to the state of how it was when the full backup was ran.  
ldap-config.bak.gz  ldap.bak.gz


./redologs:
<pre>
redo-20150304.070025.476-seq10.log redo-20150304.073544.755-seq11.log
2015-03-04 04:53:03,664 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 2 index files during restore
2015-03-04 04:53:03,665 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 AVAILABLE
2015-03-04 04:53:03,670 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Account user1@mail-172.example.com: All pending file IO completed (2 out of 2)
</pre>


./sys:
Now we see the restore start playing redologs that were made after the full backup. This section is the redologs from incr-20150304.085810.220 backup session, which were : redo-20150304.070025.476-seq10.log redo-20150304.073544.755-seq11.log redo-20150304.074024.058-seq12.log redo-20150304.085810.879-seq13.log . Located in, /opt/zimbra/backup/sessions/incr-20150304.085810.220/redologs .
config.dat current_volumes.dat db_schema.xml localconfig.xml  ssh  ssl  volume.dat


./sys/ssh:
<pre>
authorized_keys zimbra_identity zimbra_identity.pub
2015-03-04 04:53:03,671 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Run redo operations since last full backup
2015-03-04 04:53:03,733 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/6/index/0
2015-03-04 04:53:03,733 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 LOADED
2015-03-04 04:53:03,733 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
2015-03-04 04:53:03,733 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] cache - initializing folder and tag caches for mailbox 6
2015-03-04 04:53:03,746 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
2015-03-04 04:53:03,752 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
2015-03-04 04:53:03,757 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
</pre>


./sys/ssl:
Notice it will reference the sequence number of the redo log when it is plays those in /opt/zimbra/redolog/archive/ . Currently, the server has the following in that directory : redo-20150304.085810.879-seq13.log  redo-20150304.095303.026-seq14.log  redo-20150304.102039.541-seq15.log .
zimbra


./sys/ssl/zimbra:
<pre>
ca commercial jetty.pkcs12 server
2015-03-04 04:53:03,763 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Run redo operations from archived redo logs starting at sequence 13
 
2015-03-04 04:53:03,764 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
./sys/ssl/zimbra/ca:
2015-03-04 04:53:03,774 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
ca.key  ca.pem  ca.srl  ca.srl.old  index.txt  index.txt.attr  index.txt.old  newcerts  zmssl.cnf
2015-03-04 04:53:03,780 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
2015-03-04 04:53:03,787 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
2015-03-04 04:53:03,795 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
</pre>
 
And while it replays redo log sequence 13, we see that it replays the delete operation for message id 257.


./sys/ssl/zimbra/ca/newcerts:
<pre>
1424523586.pem  1424523591.pem  1424523597.pem  1424523603.pem
2015-03-04 04:53:03,824 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailop - Deleting Message (id=257).
</pre>


./sys/ssl/zimbra/commercial:
Final log events showing restore is finished.


./sys/ssl/zimbra/server:
<pre>
server.crt  server.csr  server.key
2015-03-04 04:53:03,832 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore finished for user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox 6 to mailbox 6
2015-03-04 04:53:03,833 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is shut down
2015-03-04 04:53:03,833 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - RestoreRequest elapsed=1535
</pre>
</pre>


Notice that redo log sequence 9 was indeed deleted from the redolog/archive directory.
Confirm that the message is still missing from the account:


<pre>
<pre>
[zimbra@mail-172 archive]$ pwd ; ls
[zimbra@mail-172 backup]$ zmmailbox -z -m user1@mail-172.example.com s -t mess in:Inbox
/opt/zimbra/redolog/archive
num: 0, more: false
redo-20150304.070025.476-seq10.log  redo-20150304.073544.755-seq11.log  redo-20150304.074024.058-seq12.log
</pre>
</pre>


If you had created new users after your last full backup, you would see that there was also a full backup started as well. For example :
Confirm that the account is still using the same zimbraId, mailboxId, and email address :


<pre>
<pre>
[zimbra@mail-172 log]$ zmbackup -i
[zimbra@mail-172 backup]$ zmprov gmi user1@mail-172.example.com
incr-20150304.085810.220
mailboxId: 6
full-20150304.085818.889
quotaUsed: 0


[zimbra@mail-172 log]$ zmbackupquery -a user6@mail-172.example.com
[zimbra@mail-172 backup]$ zmprov ga user1@mail-172.example.com zimbraId
Account: user6@mail-172.example.com
# name user1@mail-172.example.com
 
zimbraId: 3c397edc-c015-4d82-ae8a-1084692f8a93
    Label:  full-20150304.085818.889
    Type:    full
    Started: Wed, 2015/03/04 03:58:18.889 EST
    Ended:  Wed, 2015/03/04 03:58:25.997 EST
    Acct ID: 4e898a68-1886-4a80-b3dc-ac2ab650bf8c
 
[zimbra@mail-172 log]$ zmbackupquery -lb full-20150304.085818.889 -v
Label:  full-20150304.085818.889
Type:    full
Status:  completed
Started: Wed, 2015/03/04 03:58:18.889 EST
Ended:  Wed, 2015/03/04 03:58:25.997 EST
Redo log sequence range: 14 .. 14
Number of accounts: 1 out of 1 completed
Accounts:
  user6@mail-172.example.com: completed
 
Total space: 14020MB
Free space: 6038MB
</pre>
</pre>


====Contents Of accounts.xml And sessions.xml From The Full And Incremental Backup Examples====
====Using The -restoreToTime Option To Restore Prior To A Message Being Deleted====


First, see what zmbackupquery shows for a particular user in regards to the backup labels they are included in.
First, let's identify the exact time the delete operation took place. From our previous examples above I already know what redo log sequence file it was logged in.


<pre>
<pre>
$ zmbackupquery -a user1@mail-172.example.com
[zimbra@mail-172 backup]$ zmredodump -m 6 --show-blob /opt/zimbra/redolog/archive/redo-20150304.095303.026-seq14.log | grep DeleteItem
Account: user1@mail-172.example.com
[000034d1 - 0000350c: 60 bytes; tstamp: 2015/03/04 04:48:18.119 EST] txn 1425452599.151 [DeleteItem] ver=1.42, tstamp=1425462498119, change=1513, mailbox=6, ids=[257], type=message
[0000350d - 0000353e: 50 bytes; tstamp: 2015/03/04 04:48:18.306 EST] txn 1425452599.151 [CommitTxn] ver=1.42, tstamp=1425462498306, mailbox=6, txnType=DeleteItem
</pre>


    Label:  incr-20150304.073543.151
The first line above shows me the delete request was done at "2015/03/04 04:48:18.119" for mailbox=6, ids=[257], type=message . Ids is the message id, that were already used to delete the message in the above example and was shown in the zmmailbox search command.
    Type:    incremental
    Started: Wed, 2015/03/04 02:35:43.151 EST
    Ended:  Wed, 2015/03/04 02:35:51.981 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93
 
    Label:  full-20150304.012000.193
    Type:    full
    Started: Tue, 2015/03/03 20:20:00.193 EST
    Ended:  Tue, 2015/03/03 20:20:28.630 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93
</pre>


Now let's review the accounts/session xml files that list all the users within those particular backup sessions. The /opt/zimbra/backup/accounts.xml is the referenced to determine what is the last know full backup session the user was in. If the /opt/zimbra/backup/accounts.xml is corrupted, unreadable, or incorrect - this can cause issues with restoring users.


Also notice that it also references the zimbraId for the user at the time of the backup. If you were to delete a user and then create a new account using the same email address [user1@mail-172.example.com] that new account will have a different zimbraId and therefor the backup/restore commands will not work as you might expected because of the differing zimbraId. The zmrestore command only can take an email address for the -a option, it can not use the zimbraID.
The example uses the time above but subtracted one from it - 04:48:18.118 vs 04:48:18.119 .


<pre>
<pre>
[zimbra@mail-172 backup]$ pwd
[zimbra@mail-172 backup]$ zmrestore -a user1@mail-172.example.com -restoreToTime "2015/03/04 04:48:18.118"
/opt/zimbra/backup
using restore to time of 2015/03/04 04:48:18
</pre>


[zimbra@mail-172 backup]$ cat accounts.xml
And to now confirm the message is back in the account:
<?xml version="1.0" encoding="utf-8"?>


<backupMetadata xmlns="urn:zimbraBackupMeta" version="7.2" dbVersion="103">
<pre>
  <allAccounts>
[zimbra@mail-172 backup]$ zmmailbox -z -m user1@mail-172.example.com s -t mess in:Inbox
    <account zimbraId="af06b0b4-3160-4ea3-a55b-81163a9914a0" email="user5@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193">
num: 1, more: false
    </account>
 
    <account zimbraId="75d3d364-a384-4077-8ac5-87b6960b57cd" email="user3@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193">
    Id  Type  From                  Subject                                            Date
    </account>
  ----  ----  --------------------  -------------------------------------------------- --------------
    <account zimbraId="4343a299-5259-4826-b1c1-becbfbb9b8f0" email="galsync.qva4ekog0@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193">
1. 257  mess   admin                Test1                                              02/23/15 09:03
    </account>
    <account zimbraId="89fc5986-31a3-406e-8084-0ce0b98a251f" email="user2@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193">
    </account>
    <account zimbraId="42380a6e-5718-4d68-ab37-e8d5bc9e6977" email="spam.umhx3owv3s@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193">
    </account>
    <account zimbraId="eab3d56e-ffd9-4412-95de-48f102d2d252" email="user4@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193">
    </account>
    <account zimbraId="7ac8de92-13fe-4d19-b9da-6872169fb5c9" email="admin@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193">
    </account>
    <account zimbraId="4e898a68-1886-4a80-b3dc-ac2ab650bf8c" email="user6@mail-172.example.com" latestFullBackupLabel="full-20150304.085818.889">
    </account>
    <account zimbraId="a46dfc59-a3e9-4f09-adfb-af8efc42aa9f" email="virus-quarantine.gusiso1cd@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193">
    </account>
    <account zimbraId="3c397edc-c015-4d82-ae8a-1084692f8a93" email="user1@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193">
    </account>
    <account zimbraId="c5315c5d-97cb-4d84-bc70-374a67f13309" email="ham.jgfdb3oqjf@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193">
    </account>
   </allAccounts>
</backupMetadata>
</pre>
</pre>


In each backup session, there will be a sessions.xml file that acts as a reference or summary file of the backup. At the top, you'll see what version of ZCS was running and what options the backup environment had set. It will then report the final status for each user for the backup attempt against the account. At the end, there's a section in regards to errors and if any were encountered.
Note that this will also mean all other data or changes after 04:48:18.118 are lost also in this restored account. Normally in circumstances of trying to recovery data that a user is requesting because they delete messages and now need them recovered you would use the -ca and -pre options for zmrestore so you could then recover them and import the messages back into the existing account they have. Without the -ca and -pre , your over writing the existing account on the system.
 
The log events in /opt/zimbra/log/mailbox.log for this request were:


<pre>
<pre>
[zimbra@mail-172 backup]$ cd sessions/
2015-03-04 05:20:39,194 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is starting
[zimbra@mail-172 sessions]$ cd full-20150304.012000.193
2015-03-04 05:20:39,238 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Using full backup full-20150304.082015.535 as starting point for restoring account 3c397edc-c015-4d82-ae8a-1084692f8a93
[zimbra@mail-172 full-20150304.012000.193]$ cat session.xml
2015-03-04 05:20:39,526 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - waiting for FileLogWriter.FsyncThread-1425462783028 to finish.
<?xml version="1.0" encoding="utf-8"?>
2015-03-04 05:20:39,533 INFO  [FileLogWriter.FsyncThread-1425462783028] [] redolog - fsync thread exiting
 
2015-03-04 05:20:39,533 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - FileLogWriter.FsyncThread-1425462783028 finished
<backupMetadata xmlns="urn:zimbraBackupMeta" version="7.2" dbVersion="103">
2015-03-04 05:20:39,543 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - Redo log rollover took 305ms
  <backupSet label="full-20150304.012000.193" zcsRelease="8.6.0_GA_1153 20141215151258 20141215-1518 NETWORK" startTime="1425432000193" endTime="1425432028630" minRedoSeq="10" maxRedoSeq="10" sharedBlobsZipped="true" sharedBlobsZipNameDigestChars="1" sharedBlobsDirectoryDepth="5" sharedBlobsCharsPerDirectory="2" type="full" accountsDirectoryDepth="2">
2015-03-04 05:20:39,543 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore started for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6
    <desc>Full backup</desc>
2015-03-04 05:20:39,550 INFO  [FileLogWriter.FsyncThread-1425464439542] [] redolog - Starting fsync thread with interval 10
    <accounts>
2015-03-04 05:20:39,593 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - Putting mailbox 6 under maintenance.
      <account zimbraId="af06b0b4-3160-4ea3-a55b-81163a9914a0" email="user5@mail-172.example.com" status="COMPLETED">
2015-03-04 05:20:39,593 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - clearing contents of mailbox 6, group 6
      </account>
2015-03-04 05:20:39,614 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restoring to mailbox id 6 for existing account 3c397edc-c015-4d82-ae8a-1084692f8a93.
      <account zimbraId="75d3d364-a384-4077-8ac5-87b6960b57cd" email="user3@mail-172.example.com" status="COMPLETED">
2015-03-04 05:20:39,652 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loading schema with version 7.2
      </account>
2015-03-04 05:20:39,660 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Renaming volume_id column to locator in source table mail_item
      <account zimbraId="89fc5986-31a3-406e-8084-0ce0b98a251f" email="user2@mail-172.example.com" status="COMPLETED">
2015-03-04 05:20:39,692 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Deleting directory /opt/zimbra/store/0/6/msg
      </account>
2015-03-04 05:20:39,700 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 1 blobs during restore
      <account zimbraId="4343a299-5259-4826-b1c1-becbfbb9b8f0" email="galsync.qva4ekog0@mail-172.example.com" status="COMPLETED">
2015-03-04 05:20:39,703 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Deleting directory /opt/zimbra/index/0/6/index
      </account>
2015-03-04 05:20:39,713 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 2 index files during restore
      <account zimbraId="eab3d56e-ffd9-4412-95de-48f102d2d252" email="user4@mail-172.example.com" status="COMPLETED">
2015-03-04 05:20:39,715 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 AVAILABLE
      </account>
2015-03-04 05:20:39,717 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Account user1@mail-172.example.com: All pending file IO completed (2 out of 2)
      <account zimbraId="42380a6e-5718-4d68-ab37-e8d5bc9e6977" email="spam.umhx3owv3s@mail-172.example.com" status="COMPLETED">
2015-03-04 05:20:39,717 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Run redo operations since last full backup
      </account>
2015-03-04 05:20:39,732 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/6/index/0
      <account zimbraId="7ac8de92-13fe-4d19-b9da-6872169fb5c9" email="admin@mail-172.example.com" status="COMPLETED">
2015-03-04 05:20:39,732 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 LOADED
      </account>
2015-03-04 05:20:39,732 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
      <account zimbraId="a46dfc59-a3e9-4f09-adfb-af8efc42aa9f" email="virus-quarantine.gusiso1cd@mail-172.example.com" status="COMPLETED">
2015-03-04 05:20:39,732 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] cache - initializing folder and tag caches for mailbox 6
      </account>
2015-03-04 05:20:39,743 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
      <account zimbraId="3c397edc-c015-4d82-ae8a-1084692f8a93" email="user1@mail-172.example.com" status="COMPLETED">
2015-03-04 05:20:39,749 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
      </account>
2015-03-04 05:20:39,754 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
      <account zimbraId="c5315c5d-97cb-4d84-bc70-374a67f13309" email="ham.jgfdb3oqjf@mail-172.example.com" status="COMPLETED">
2015-03-04 05:20:39,760 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Run redo operations from archived redo logs starting at sequence 13
      </account>
2015-03-04 05:20:39,762 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
    </accounts>
2015-03-04 05:20:39,770 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
    <stats>
2015-03-04 05:20:39,774 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
      <counter name="total_time" numSamples="1" sum="28437" unit="ms">
2015-03-04 05:20:39,780 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
      </counter>
2015-03-04 05:20:39,786 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
      <counter name="sysdb_time" numSamples="1" sum="1111" unit="ms">
2015-03-04 05:20:39,789 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore finished for user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox 6 to mailbox 6
      </counter>
2015-03-04 05:20:39,791 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is shut down
      <counter name="sysdb_bytes" numSamples="1" sum="35210" unit="bytes">
2015-03-04 05:20:39,791 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - RestoreRequest elapsed=646
      </counter>
</pre>
      <counter name="redologs_time" numSamples="0" sum="0" unit="ms">
 
      </counter>
Notice that this restore, unlike the first example, does '''NOT''' have the following line :
      <counter name="redologs_bytes" numSamples="0" sum="0" unit="bytes">
<pre>
      </counter>
INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailop - Deleting Message (id=257).
      <counter name="accounts_time" numSamples="10" sum="4827" unit="ms">
</pre>
      </counter>
 
      <counter name="ldap_time" numSamples="10" sum="172" unit="ms">
====Blobs In Backup And Blobs In Production====
      </counter>
 
      <counter name="ldap_bytes" numSamples="10" sum="25665" unit="bytes">
This might be a good time to take a look at the blobs and how they are saved/stored in the backups and how they compare to the store volume. The example below is from our user and message details from above.
      </counter>
 
      <counter name="db_time" numSamples="10" sum="4787" unit="ms">
<pre>
      </counter>
[zimbra@mail-172 blobs]$ pwd
      <counter name="db_bytes" numSamples="10" sum="78690" unit="bytes">
/opt/zimbra/backup/sessions/full-20150304.082015.535/accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93/blobs
      </counter>
      <counter name="db_digest_map_load_time" numSamples="10" sum="2400" unit="ms">
      </counter>
      <counter name="msgs_time" numSamples="10" sum="451" unit="ms">
      </counter>
      <counter name="msg_copy_time" numSamples="74" sum="25" unit="ms">
      </counter>
      <counter name="msg_copy_bytes" numSamples="74" sum="149611" unit="bytes">
      </counter>
      <counter name="msg_link_time" numSamples="0" sum="0" unit="ms">
      </counter>
      <counter name="index_time" numSamples="48" sum="80" unit="ms">
      </counter>
      <counter name="index_bytes" numSamples="38" sum="73510" unit="bytes">
      </counter>
    </stats>
    <errors>
    </errors>
  </backupSet>
</backupMetadata>
</pre>


An incremental session.xml file is similar to a full backups session.xml .
[zimbra@mail-172 blobs]$ unzip -Z blobs-1.zip
Archive:  blobs-1.zip
Zip file size: 1439 bytes, number of entries: 1
-rw----    4.5 fat    1209 b- stor 15-Mar-04 03:20 1/0/sha256_mBS9YKqyNpLknKCDWY5WNEbTgotX,UNrXMA5ga+s6LA=_257-2.msg1
1 file, 1209 bytes uncompressed, 1209 bytes compressed:  0.0%


<pre>
[zimbra@mail-172 blobs]$ ls -la /opt/zimbra/store/0/6/msg/0/257-2.msg
[zimbra@mail-172 sessions]$ cd incr-20150304.073543.151
-rw-r----- 1 zimbra zimbra 1209 Mar  4 05:20 /opt/zimbra/store/0/6/msg/0/257-2.msg
[zimbra@mail-172 incr-20150304.073543.151]$ cat session.xml
 
<?xml version="1.0" encoding="utf-8"?>
[zimbra@mail-172 blobs]$ unzip blobs-1.zip
Archive:  blobs-1.zip
extracting: 1/0/sha256_mBS9YKqyNpLknKCDWY5WNEbTgotX,UNrXMA5ga+s6LA=_257-2.msg1
 
[zimbra@mail-172 blobs]$ ls
1  blobs-1.zip
 
[zimbra@mail-172 blobs]$ ls -la 1/0/sha256_mBS9YKqyNpLknKCDWY5WNEbTgotX\,UNrXMA5ga+s6LA\=_257-2.msg1
-rw-r----- 1 zimbra zimbra 1209 Mar  4 03:20 1/0/sha256_mBS9YKqyNpLknKCDWY5WNEbTgotX,UNrXMA5ga+s6LA=_257-2.msg1


<backupMetadata xmlns="urn:zimbraBackupMeta" version="7.2" dbVersion="103">
[zimbra@mail-172 blobs]$ head 1/0/sha256_mBS9YKqyNpLknKCDWY5WNEbTgotX\,UNrXMA5ga+s6LA\=_257-2.msg1
  <backupSet label="incr-20150304.073543.151" zcsRelease="8.6.0_GA_1153 20141215151258 20141215-1518 NETWORK" startTime="1425454543151" endTime="1425454551981" minRedoSeq="10" maxRedoSeq="11" sharedBlobsZipped="true" sharedBlobsZipNameDigestChars="1" sharedBlobsDirectoryDepth="5" sharedBlobsCharsPerDirectory="2" type="incremental" accountsDirectoryDepth="2">
Return-Path: zimbra@mail-172.example.com
    <desc>Incremental backup</desc>
Received: from mail-172.example.com (LHLO mail-172.example.com)
    <accounts>
(192.168.1.172) by mail-172.example.com with LMTP; Mon, 23 Feb 2015
      <account zimbraId="af06b0b4-3160-4ea3-a55b-81163a9914a0" email="user5@mail-172.example.com" status="COMPLETED">
09:02:58 -0500 (EST)
      </account>
Received: from localhost (localhost.localdomain [127.0.0.1])
      <account zimbraId="75d3d364-a384-4077-8ac5-87b6960b57cd" email="user3@mail-172.example.com" status="COMPLETED">
        by mail-172.example.com (Postfix) with ESMTP id 4F8AE8433D
      </account>
        for <user1@mail-172.example.com>; Mon, 23 Feb 2015 09:02:58 -0500 (EST)
      <account zimbraId="89fc5986-31a3-406e-8084-0ce0b98a251f" email="user2@mail-172.example.com" status="COMPLETED">
X-Virus-Scanned: amavisd-new at mail-172.example.com
      </account>
X-Spam-Flag: NO
      <account zimbraId="4343a299-5259-4826-b1c1-becbfbb9b8f0" email="galsync.qva4ekog0@mail-172.example.com" status="COMPLETED">
X-Spam-Score: -1.9
      </account>
 
      <account zimbraId="eab3d56e-ffd9-4412-95de-48f102d2d252" email="user4@mail-172.example.com" status="COMPLETED">
[zimbra@mail-172 blobs]$ head /opt/zimbra/store/0/6/msg/0/257-2.msg
      </account>
Return-Path: zimbra@mail-172.example.com
      <account zimbraId="42380a6e-5718-4d68-ab37-e8d5bc9e6977" email="spam.umhx3owv3s@mail-172.example.com" status="COMPLETED">
Received: from mail-172.example.com (LHLO mail-172.example.com)
      </account>
(192.168.1.172) by mail-172.example.com with LMTP; Mon, 23 Feb 2015
      <account zimbraId="7ac8de92-13fe-4d19-b9da-6872169fb5c9" email="admin@mail-172.example.com" status="COMPLETED">
09:02:58 -0500 (EST)
      </account>
Received: from localhost (localhost.localdomain [127.0.0.1])
      <account zimbraId="a46dfc59-a3e9-4f09-adfb-af8efc42aa9f" email="virus-quarantine.gusiso1cd@mail-172.example.com" status="COMPLETED">
        by mail-172.example.com (Postfix) with ESMTP id 4F8AE8433D
      </account>
        for <user1@mail-172.example.com>; Mon, 23 Feb 2015 09:02:58 -0500 (EST)
      <account zimbraId="3c397edc-c015-4d82-ae8a-1084692f8a93" email="user1@mail-172.example.com" status="COMPLETED">
X-Virus-Scanned: amavisd-new at mail-172.example.com
      </account>
X-Spam-Flag: NO
      <account zimbraId="c5315c5d-97cb-4d84-bc70-374a67f13309" email="ham.jgfdb3oqjf@mail-172.example.com" status="COMPLETED">
X-Spam-Score: -1.9
      </account>
</pre>
    </accounts>
 
    <stats>
====Knowing When The -br [To Incremental] Or -rf [Only Full] Option Is Required====
      <counter name="total_time" numSamples="1" sum="8830" unit="ms">
 
      </counter>
=====Pre-Steps=====
      <counter name="sysdb_time" numSamples="1" sum="563" unit="ms">
 
      </counter>
Let's query current backups and then create a new full and new incremental where we also delete a message between the full and the incremental.
      <counter name="sysdb_bytes" numSamples="1" sum="35210" unit="bytes">
 
      </counter>
<pre>
      <counter name="redologs_time" numSamples="1" sum="310" unit="ms">
[zimbra@mail-172 ~]$ zmbackupquery -a user1@`zmhostname`
      </counter>
Account: user1@mail-172.example.com
      <counter name="redologs_bytes" numSamples="1" sum="126307" unit="bytes">
 
      </counter>
     Label:  incr-20150305.060013.901
      <counter name="accounts_time" numSamples="10" sum="196" unit="ms">
     Type:    incremental
      </counter>
     Started: Thu, 2015/03/05 01:00:13.901 EST
      <counter name="ldap_time" numSamples="10" sum="129" unit="ms">
    Ended:   Thu, 2015/03/05 01:00:26.273 EST
      </counter>
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93
      <counter name="ldap_bytes" numSamples="10" sum="25665" unit="bytes">
      </counter>
      <counter name="db_time" numSamples="0" sum="0" unit="ms">
      </counter>
      <counter name="db_bytes" numSamples="0" sum="0" unit="bytes">
      </counter>
      <counter name="db_digest_map_load_time" numSamples="0" sum="0" unit="ms">
      </counter>
      <counter name="msgs_time" numSamples="0" sum="0" unit="ms">
      </counter>
      <counter name="msg_copy_time" numSamples="0" sum="0" unit="ms">
      </counter>
      <counter name="msg_copy_bytes" numSamples="0" sum="0" unit="bytes">
      </counter>
      <counter name="msg_link_time" numSamples="0" sum="0" unit="ms">
      </counter>
      <counter name="index_time" numSamples="0" sum="0" unit="ms">
      </counter>
      <counter name="index_bytes" numSamples="0" sum="0" unit="bytes">
      </counter>
     </stats>
     <errors>
     </errors>
   </backupSet>
</backupMetadata>
</pre>


====User And Log Events Generated====
    Label:  incr-20150304.085810.220
    Type:    incremental
    Started: Wed, 2015/03/04 03:58:10.220 EST
    Ended:  Wed, 2015/03/04 03:58:16.393 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93


Please beware that prior backups were deleted and new backups were made on this ZCS test server. So the backup label names might not match prior examples.
    Label:  full-20150304.082015.535
    Type:    full
    Started: Wed, 2015/03/04 03:20:15.535 EST
    Ended:  Wed, 2015/03/04 03:20:39.109 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93


For this example, I'm going to replicate a common situation that system administrators find themselves when trying to restore a user and they fail to include the proper restore options to have the restore stop prior to replaying a message that was deleted by a user. Proper examples of doing the restore will be shown afterwards.
[zimbra@mail-172 ~]$ zmbackup -f -a all
full-20150306.043202.011


Information on the account we'll restore:
[zimbra@mail-172 ~]$ zmmailbox -z -m user1@mail-172.example.com s -t mess in:Inbox
num: 1, more: false


<pre>
    Id  Type  From                  Subject                                            Date
[zimbra@mail-172 backup]$ zmprov gmi user1@mail-172.example.com
  ----  ----  --------------------  --------------------------------------------------  --------------
mailboxId: 6
1. 257  mess  admin                Test1                                              02/23/15 09:03
quotaUsed: 1209


[zimbra@mail-172 backup]$ zmprov ga user1@mail-172.example.com zimbraId
[zimbra@mail-172 ~]$ zmmailbox -z -m user1@mail-172.example.com dm 257
# name user1@mail-172.example.com
zimbraId: 3c397edc-c015-4d82-ae8a-1084692f8a93
</pre>


Example of error you'll receive if you attempt to use the zimbraId rather than the email address for the zmrestore command. This is important to note if your trying to restore an account that was deleted and then later created again and the various backups and accounts.xml file has different zimbraId's associated with the same email address.
[zimbra@mail-172 ~]$ zmbackup -i
incr-20150306.043321.683


<pre>
[zimbra@mail-172 ~]$ zmbackupquery -a user1@`zmhostname`
[zimbra@mail-172 backup]$ zmrestore -a 3c397edc-c015-4d82-ae8a-1084692f8a93 -ca -pre restore1_
Account: user1@mail-172.example.com
Error occurred: invalid request: invalid account email address: 3c397edc-c015-4d82-ae8a-1084692f8a93
</pre>


Showing that we have one message in the Inbox for the user we'll be restoring and also the message id of that email :
    Label:   incr-20150306.043321.683
    Type:    incremental
    Started: Thu, 2015/03/05 23:33:21.683 EST
    Ended:  Thu, 2015/03/05 23:33:27.495 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93


<pre>
    Label:  full-20150306.043202.011
[zimbra@mail-172 backup]$ zmmailbox -z -m user1@mail-172.example.com gaf
    Type:    full
        Id  View      Unread  Msg Count  Path
    Started: Thu, 2015/03/05 23:32:02.011 EST
----------  ----  ----------  ----------  ----------
    Ended:  Thu, 2015/03/05 23:32:13.316 EST
        1  unkn          0          0  /
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93
        16  docu          0          0  /Briefcase
        10  appo          0          0  /Calendar
        14  mess          0          0  /Chats
        7  cont          0          0  /Contacts
        6  mess          0          0  /Drafts
        13 cont          0          0  /Emailed Contacts
        2  mess          1          1  /Inbox
        4  mess          0          0  /Junk
        5  mess          0          0  /Sent
        15  task          0          0  /Tasks
        3  unkn          0          0  /Trash


[zimbra@mail-172 backup]$ zmmailbox -z -m user1@mail-172.example.com s -t mess in:Inbox
    Label:  incr-20150305.060013.901
num: 1, more: false
    Type:    incremental
    Started: Thu, 2015/03/05 01:00:13.901 EST
    Ended:   Thu, 2015/03/05 01:00:26.273 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93


    Id  Type  From                  Subject                                            Date
    Label:  incr-20150304.085810.220
  ----  ----  --------------------  --------------------------------------------------  --------------
    Type:    incremental
1. 257  mess   admin                Test1                                              02/23/15 09:03
    Started: Wed, 2015/03/04 03:58:10.220 EST
    Ended:   Wed, 2015/03/04 03:58:16.393 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93
 
    Label:   full-20150304.082015.535
    Type:    full
    Started: Wed, 2015/03/04 03:20:15.535 EST
    Ended:   Wed, 2015/03/04 03:20:39.109 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93
</pre>
</pre>


Now to delete the email from the account:
=====First Example - Options -lb, -ca, -pre restore_=====


<pre>
For the following examples, we'll include the -ca [create account] and the -pre [add prefix to new account name] . This first example uses the oldest full backup [full-20150304.082015.535 vs. full-20150306.043202.011] the account has with the -lb option. The message we just deleted, existed at the time of the full backup. We'll '''NOT''' be including either the -rf or -br option in this first example.
[zimbra@mail-172 backup]$ zmmailbox -z -m user1@mail-172.example.com dm 257


And to confirm it's no longer there:
Then we'll review the mailbox.log events. Note the following:
 
* That the log only mentions the parent account, there isn't a reference to the email address with the prefixed name.
[zimbra@mail-172 backup]$ zmmailbox -z -m user1@mail-172.example.com s -t mess in:Inbox
** There is a RFE to include the new account name in the log, [https://bugzilla.zimbra.com/show_bug.cgi?id=52641 Bug 52641 - RFE: include -ca -pre new account in mailbox.log log event]
num: 0, more: false
* Though the log will state the new mailboxId and zimbraId for the user, though:
</pre>
** "user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6" vs "Restoring to mailbox id 12 for new account 016c9f7c-e2e2-4f8f-96d0-e14a4327821d"
 
* Since we failed to include the -rf option, the restore continues past the full backup and goes through all the other backups that exist, hence we see:
Below is an example of a delete operation being logged in the redo.log and how to query events for a particular user with the zmredodump command. Notice the DeleteItem events at the end.
** "backup - Run redo operations since last full backup"
* One of the redologs in our incremental had the delete operation for the message in question and we see evidence of that delete in the log:
** "mailop - Deleting Message (id=257)."
* Since we didn't include the -br option, the restore also continues past the backups and replays the redologs in /opt/zimbra/redolog and it's subdirectory, archive. Notice the reference to sequence 17.
** "backup - Run redo operations from archived redo logs starting at sequence 17"
** /opt/zimbra/redolog/archive/ has : redo-20150306.043322.070-seq17.log  redo-20150306.043611.912-seq18.log
* Towards the end, we'll get a confirmation that the restore has finished with some reference information again - mailboxId's .
** "Restore finished for user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox 6 to mailbox 12"


<pre>
<pre>
[zimbra@mail-172 backup]$ zmredodump -m 6 --show-blob /opt/zimbra/redolog/redo.log
[zimbra@mail-172 ~]$ cat /dev/null > /opt/zimbra/log/mailbox.log
VERIFYING: /opt/zimbra/redolog/redo.log
 
HEADER
[zimbra@mail-172 ~]$ zmrestore -a user1@mail-172.example.com -lb full-20150304.082015.535 -ca -pre restore_
------
 
sequence: 14
[zimbra@mail-172 ~]$ cat /opt/zimbra/log/mailbox.log
open:     1
2015-03-05 23:35:40,815 INFO  [MailboxPurge] [name=spam.umhx3owv3s@mail-172.example.com;mid=3;] purge - Purging messages.
filesize: 512
2015-03-05 23:36:11,619 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - AuthRequest elapsed=3
serverId: 4407a03f-894b-4b1a-817a-684fc34c4f60
2015-03-05 23:36:11,784 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is starting
createdWed, 2015/03/04 03:58:10.876 EST (1425459490876)
2015-03-05 23:36:11,905 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - waiting for FileLogWriter.FsyncThread-1425616402072 to finish.
first op: Wed, 2015/03/04 03:58:18.744 EST (1425459498744)
2015-03-05 23:36:11,908 INFO  [FileLogWriter.FsyncThread-1425616402072] [] redolog - fsync thread exiting
last op: Wed, 2015/03/04 03:58:18.744 EST (1425459498744) (not up to date)
2015-03-05 23:36:11,908 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - FileLogWriter.FsyncThread-1425616402072 finished
version: 1.42
2015-03-05 23:36:11,913 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - Redo log rollover took 128ms
------
2015-03-05 23:36:11,913 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore started for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6
[00001601 - 0000162e: 46 bytes; tstamp: 2015/03/04 04:00:26.677 EST] txn 1425452599.101 [PurgeOldMessages] ver=1.42, tstamp=1425459626677, change=1508, mailbox=6
2015-03-05 23:36:11,914 INFO  [FileLogWriter.FsyncThread-1425616571913] [] redolog - Starting fsync thread with interval 10
[0000162f - 00001660: 50 bytes; tstamp: 2015/03/04 04:00:26.683 EST] txn 1425452599.101 [CommitTxn] ver=1.42, tstamp=1425459626683, mailbox=6, txnType=PurgeOldMessages
2015-03-05 23:36:13,006 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - Creating database mboxgroup12
[00002691 - 000026be: 46 bytes; tstamp: 2015/03/04 04:10:26.816 EST] txn 1425452599.113 [PurgeOldMessages] ver=1.42, tstamp=1425460226816, change=1509, mailbox=6
2015-03-05 23:36:14,150 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restoring to mailbox id 12 for new account 016c9f7c-e2e2-4f8f-96d0-e14a4327821d.
[000026bf - 000026f0: 50 bytes; tstamp: 2015/03/04 04:10:26.832 EST] txn 1425452599.113 [CommitTxn] ver=1.42, tstamp=1425460226832, mailbox=6, txnType=PurgeOldMessages
2015-03-05 23:36:14,237 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loading schema with version 7.2
[00002ab1 - 00002ade: 46 bytes; tstamp: 2015/03/04 04:21:27.376 EST] txn 1425452599.124 [PurgeOldMessages] ver=1.42, tstamp=1425460887376, change=1510, mailbox=6
2015-03-05 23:36:14,276 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Renaming volume_id column to locator in source table mail_item
[00002adf - 00002b10: 50 bytes; tstamp: 2015/03/04 04:21:27.381 EST] txn 1425452599.124 [CommitTxn] ver=1.42, tstamp=1425460887381, mailbox=6, txnType=PurgeOldMessages
2015-03-05 23:36:14,315 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 1 blobs during restore
[00002ed1 - 00002efe: 46 bytes; tstamp: 2015/03/04 04:32:27.610 EST] txn 1425452599.135 [PurgeOldMessages] ver=1.42, tstamp=1425461547610, change=1511, mailbox=6
2015-03-05 23:36:14,316 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Deleting directory /opt/zimbra/index/0/12/index
[00002eff - 00002f30: 50 bytes; tstamp: 2015/03/04 04:32:27.617 EST] txn 1425452599.135 [CommitTxn] ver=1.42, tstamp=1425461547617, mailbox=6, txnType=PurgeOldMessages
2015-03-05 23:36:14,317 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 2 index files during restore
[000032f1 - 0000331e: 46 bytes; tstamp: 2015/03/04 04:43:27.828 EST] txn 1425452599.146 [PurgeOldMessages] ver=1.42, tstamp=1425462207828, change=1512, mailbox=6
2015-03-05 23:36:14,319 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/12/index/0
[0000331f - 00003350: 50 bytes; tstamp: 2015/03/04 04:43:27.839 EST] txn 1425452599.146 [CommitTxn] ver=1.42, tstamp=1425462207839, mailbox=6, txnType=PurgeOldMessages
2015-03-05 23:36:14,319 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox 12 account 016c9f7c-e2e2-4f8f-96d0-e14a4327821d LOADED
[000034d1 - 0000350c: 60 bytes; tstamp: 2015/03/04 04:48:18.119 EST] txn 1425452599.151 [DeleteItem] ver=1.42, tstamp=1425462498119, change=1513, mailbox=6, ids=[257], type=message
2015-03-05 23:36:14,319 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] cache - initializing folder and tag caches for mailbox 12
[0000350d - 0000353e: 50 bytes; tstamp: 2015/03/04 04:48:18.306 EST] txn 1425452599.151 [CommitTxn] ver=1.42, tstamp=1425462498306, mailbox=6, txnType=DeleteItem
2015-03-05 23:36:14,367 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Account user1@mail-172.example.com: All pending file IO completed (2 out of 2)
</pre>
2015-03-05 23:36:14,367 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Run redo operations since last full backup
2015-03-05 23:36:14,400 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.


Listing of the backups that are currently associated with the account we'll be restoring:
[multiple repeats of this message]


<pre>
2015-03-05 23:36:14,428 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
[zimbra@mail-172 backup]$ zmbackupquery -a user1@mail-172.example.com
2015-03-05 23:36:14,465 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailop - Deleting Message (id=257).
Account: user1@mail-172.example.com
2015-03-05 23:36:14,474 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
 
[multiple repeats of this message]


    Label:   incr-20150304.085810.220
2015-03-05 23:36:15,152 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
    Type:   incremental
2015-03-05 23:36:15,172 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Run redo operations from archived redo logs starting at sequence 17
    Started: Wed, 2015/03/04 03:58:10.220 EST
2015-03-05 23:36:15,172 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore finished for user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox 6 to mailbox 12
    Ended:   Wed, 2015/03/04 03:58:16.393 EST
2015-03-05 23:36:15,174 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is shut down
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93
2015-03-05 23:36:15,174 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - RestoreRequest elapsed=3481


    Label:  full-20150304.082015.535
[zimbra@mail-172 ~]$ ls /opt/zimbra/redolog/archive/
    Type:    full
redo-20150306.043322.070-seq17.log  redo-20150306.043611.912-seq18.log
    Started: Wed, 2015/03/04 03:20:15.535 EST
    Ended:  Wed, 2015/03/04 03:20:39.109 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93
</pre>
</pre>


Run a zmrestore with no other options other than the account flag and the email address of the account we are restoring. '''Note''' - this example is to show why this often times isn't what you'll do in real circumstances and will require other zmrestore options to achieve what you want.
=====Second Example - Options -lb , -ca , -pre restore2_ , -br=====
 
<pre>
[zimbra@mail-172 backup]$ zmrestore -a user1@mail-172.example.com
</pre>


From the /opt/zimbra/log/mailbox.log we have the following log events. We see the restore request kick off and the full backup label that it'll reference as the starting point. It also lists the zimbraId rather than the email address for the restore initially.
For the second example, we'll again use the oldest full backup [full-20150304.082015.535 vs. full-20150306.043202.011] the account has with the -lb option. The message we had deleted, existed at the time of the full backup but did NOT in the last incremental backup. Actually, the delete operation is within the redologs of the last incremental. For the second example, we'll use the -br option rather than the -rf . We'll also be adjusting the -pre option to be restore2_ vs. restore_ in the first example.


<pre>
Then we'll review the mailbox.log events. Note the following:
2015-03-04 04:53:02,221 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - AuthRequest elapsed=60
* Though the log will state the new mailboxId and zimbraId for the user, though:
2015-03-04 04:53:02,519 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is starting
** "user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6" vs "Restoring to mailbox id 13 for new account 4c99a395-2c39-46ec-8c1b-285d97d899df."
2015-03-04 04:53:02,565 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Using full backup full-20150304.082015.535 as starting point for restoring account 3c397edc-c015-4d82-ae8a-1084692f8a93
** Notice the mailboxId and zimbraId for restore2_ is different than our first example when we used restore_ for the -pre option.
</pre>
* Since we failed to include the -rf option, the restore continues past the full backup and goes through all the other backups that exist, hence we see:
** "backup - Run redo operations since last full backup"
* One of the redologs in our incremental had the delete operation for the message in question and we see evidence of that delete in the log:
** "mailop - Deleting Message (id=257)."
* Since we '''DID''' include the -br option, the restore '''STOPS''' at our last backup session. It therefor '''DOESN'T''' replay the redologs in /opt/zimbra/redolog and it's subdirectory, archive and we do '''NOT''' have the following log events like we did in our first example.
** "backup - Run redo operations from archived redo logs starting at sequence 17"
* Towards the end, we'll get a confirmation that the restore has finished with some reference information again - mailboxId's .
** "Restore finished for user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox 6 to mailbox 13"


And then we'll see the reference to the email address and the mailboxId associated with the account/zimbraId .
<pre>
<pre>
2015-03-04 04:53:02,993 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - waiting for FileLogWriter.FsyncThread-1425459490899 to finish.
[zimbra@mail-172 ~]$ cat /dev/null > /opt/zimbra/log/mailbox.log
2015-03-04 04:53:02,997 INFO  [FileLogWriter.FsyncThread-1425459490899] [] redolog - fsync thread exiting
2015-03-04 04:53:02,997 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - FileLogWriter.FsyncThread-1425459490899 finished
2015-03-04 04:53:03,028 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - Redo log rollover took 463ms
2015-03-04 04:53:03,028 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore started for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6
</pre>


Since we are basically over-writing the account, the account is placed into maintenance mode first and then existing data is deleted. Only after the account has been deleted will the account be restored using the backup data. Since we didn't use the -ca and -pre options for zmrestore, this account will reuse it's existing zimbraId, mailboxId, and of course - email address.
[zimbra@mail-172 ~]$ zmrestore -a user1@mail-172.example.com -lb full-20150304.082015.535 -br -ca -pre restore2_


<pre>
[zimbra@mail-172 ~]$ cat /opt/zimbra/log/mailbox.log
2015-03-04 04:53:03,042 INFO  [FileLogWriter.FsyncThread-1425462783028] [] redolog - Starting fsync thread with interval 10
2015-03-05 23:39:30,176 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - AuthRequest elapsed=2
2015-03-04 04:53:03,096 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - Putting mailbox 6 under maintenance.
2015-03-05 23:39:30,263 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is starting
2015-03-04 04:53:03,097 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - clearing contents of mailbox 6, group 6
2015-03-05 23:39:30,352 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - waiting for FileLogWriter.FsyncThread-1425616571913 to finish.
2015-03-04 04:53:03,209 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restoring to mailbox id 6 for existing account 3c397edc-c015-4d82-ae8a-1084692f8a93.
2015-03-05 23:39:30,360 INFO  [FileLogWriter.FsyncThread-1425616571913] [] redolog - fsync thread exiting
2015-03-04 04:53:03,256 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loading schema with version 7.2
2015-03-05 23:39:30,360 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - FileLogWriter.FsyncThread-1425616571913 finished
2015-03-04 04:53:03,287 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Renaming volume_id column to locator in source table mail_item
2015-03-05 23:39:30,371 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - Redo log rollover took 107ms
2015-03-04 04:53:03,584 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Deleting directory /opt/zimbra/store/0/6/msg
2015-03-05 23:39:30,371 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore started for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6
2015-03-04 04:53:03,599 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 1 blobs during restore
2015-03-05 23:39:30,377 INFO  [FileLogWriter.FsyncThread-1425616770369] [] redolog - Starting fsync thread with interval 10
2015-03-04 04:53:03,622 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Deleting directory /opt/zimbra/index/0/6/index
2015-03-05 23:39:30,381 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - Creating database mboxgroup13
</pre>
2015-03-05 23:39:30,992 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restoring to mailbox id 13 for new account 4c99a395-2c39-46ec-8c1b-285d97d899df.
2015-03-05 23:39:30,993 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loading schema with version 7.2
2015-03-05 23:39:30,997 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Renaming volume_id column to locator in source table mail_item
2015-03-05 23:39:31,022 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 1 blobs during restore
2015-03-05 23:39:31,023 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Deleting directory /opt/zimbra/index/0/13/index
2015-03-05 23:39:31,023 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 2 index files during restore
2015-03-05 23:39:31,024 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/13/index/0
2015-03-05 23:39:31,024 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox 13 account 4c99a395-2c39-46ec-8c1b-285d97d899df LOADED
2015-03-05 23:39:31,024 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] cache - initializing folder and tag caches for mailbox 13
2015-03-05 23:39:31,050 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Account user1@mail-172.example.com: All pending file IO completed (2 out of 2)
2015-03-05 23:39:31,050 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Run redo operations since last full backup
2015-03-05 23:39:31,053 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.


At this point, the data for the account was restored to the state of how it was when the full backup was ran.
[multiple repeats of this message]


<pre>
2015-03-05 23:39:31,070 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
2015-03-04 04:53:03,664 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 2 index files during restore
2015-03-05 23:39:31,075 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailop - Deleting Message (id=257).
2015-03-04 04:53:03,665 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 AVAILABLE
2015-03-05 23:39:31,079 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
2015-03-04 04:53:03,670 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Account user1@mail-172.example.com: All pending file IO completed (2 out of 2)
</pre>


Now we see the restore start playing redologs that were made after the full backup. This section is the redologs from incr-20150304.085810.220 backup session, which were : redo-20150304.070025.476-seq10.log redo-20150304.073544.755-seq11.log  redo-20150304.074024.058-seq12.log  redo-20150304.085810.879-seq13.log . Located in, /opt/zimbra/backup/sessions/incr-20150304.085810.220/redologs .
  [multiple repeats of this message]


<pre>
2015-03-05 23:39:31,829 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
2015-03-04 04:53:03,671 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Run redo operations since last full backup
2015-03-05 23:39:31,832 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore finished for user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox 6 to mailbox 13
2015-03-04 04:53:03,733 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/6/index/0
2015-03-05 23:39:31,833 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is shut down
2015-03-04 04:53:03,733 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 LOADED
2015-03-05 23:39:31,834 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - RestoreRequest elapsed=1579
2015-03-04 04:53:03,733 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
2015-03-05 23:39:40,932 INFO  [MailboxPurge] [name=user5@mail-172.example.com;mid=7;] purge - Purging messages.
2015-03-04 04:53:03,733 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] cache - initializing folder and tag caches for mailbox 6
[zimbra@mail-172 ~]$
2015-03-04 04:53:03,746 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
2015-03-04 04:53:03,752 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
2015-03-04 04:53:03,757 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
</pre>
</pre>


Notice it will reference the sequence number of the redo log when it is plays those in /opt/zimbra/redolog/archive/ . Currently, the server has the following in that directory : redo-20150304.085810.879-seq13.log  redo-20150304.095303.026-seq14.log  redo-20150304.102039.541-seq15.log .
=====Third Example - Options -lb , -ca , -pre restore3_ , -rf=====


<pre>
For the third example, we'll again use the oldest full backup [full-20150304.082015.535 vs. full-20150306.043202.011] the account has with the -lb option. The message we had deleted, existed at the time of the full backup but did NOT in the last incremental backup. For the third example, we'll use the -rf option rather than the -br . We'll also be adjusting the -pre option to be restore3_ vs. restore_ or restore2_ in the other examples.
2015-03-04 04:53:03,763 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Run redo operations from archived redo logs starting at sequence 13
2015-03-04 04:53:03,764 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
2015-03-04 04:53:03,774 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
2015-03-04 04:53:03,780 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
2015-03-04 04:53:03,787 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
2015-03-04 04:53:03,795 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
</pre>


And while it replays redo log sequence 13, we see that it replays the delete operation for message id 257.
Then we'll review the mailbox.log events. Note the following:
* Though the log will state the new mailboxId and zimbraId for the user, though:
** "user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6" vs "Restoring to mailbox id 14 for new account d72a6d8d-8117-4a90-b8cf-f972f4af056d."
** Notice the mailboxId and zimbraId for restore3_ is different than our other examples.
* Since we '''INCLUDED''' the -rf option, the restore '''STOPS''' once the full backup is restored. Hence, we '''DON'T''' see the following line like we did in the other examples:
** "backup - Run redo operations since last full backup"
* And because it '''DIDN'T''' replay any of the redologs, there is '''NO''' reference to the following line either in the log:
** "mailop - Deleting Message (id=257)."
* '''NOR''' is there a reference to the following line either in the log:
** "backup - Run redo operations from archived redo logs starting at sequence 17"
* Towards the end, we'll get a confirmation that the restore has finished with some reference information again - mailboxId's .
** "Restore finished for user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox 6 to mailbox 14"


<pre>
<pre>
2015-03-04 04:53:03,824 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailop - Deleting Message (id=257).
[zimbra@mail-172 ~]$ cat /dev/null > /opt/zimbra/log/mailbox.log
</pre>
[zimbra@mail-172 ~]$ zmrestore -a user1@mail-172.example.com -lb full-20150304.082015.535 -rf -ca -pre restore3_
 
[zimbra@mail-172 ~]$ cat /opt/zimbra/log/mailbox.log
Final log events showing restore is finished.
2015-03-05 23:42:50,233 INFO  [Timer-Zimbra] [] SessionCache - Removed 1 idle sessions (SOAP). 1 active sessions remain.
 
2015-03-05 23:42:58,194 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - AuthRequest elapsed=3
<pre>
2015-03-05 23:42:58,282 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is starting
2015-03-04 04:53:03,832 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore finished for user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox 6 to mailbox 6
2015-03-05 23:42:58,475 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - waiting for FileLogWriter.FsyncThread-1425616770369 to finish.
2015-03-04 04:53:03,833 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is shut down
2015-03-05 23:42:58,477 INFO  [FileLogWriter.FsyncThread-1425616770369] [] redolog - fsync thread exiting
2015-03-04 04:53:03,833 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - RestoreRequest elapsed=1535
2015-03-05 23:42:58,477 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - FileLogWriter.FsyncThread-1425616770369 finished
</pre>
2015-03-05 23:42:58,485 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - Redo log rollover took 193ms
2015-03-05 23:42:58,485 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore started for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6
2015-03-05 23:42:58,495 INFO  [FileLogWriter.FsyncThread-1425616978485] [] redolog - Starting fsync thread with interval 10
2015-03-05 23:42:58,515 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - Creating database mboxgroup14
2015-03-05 23:43:06,041 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restoring to mailbox id 14 for new account d72a6d8d-8117-4a90-b8cf-f972f4af056d.
2015-03-05 23:43:06,042 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loading schema with version 7.2
2015-03-05 23:43:06,048 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Renaming volume_id column to locator in source table mail_item
2015-03-05 23:43:06,092 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 1 blobs during restore
2015-03-05 23:43:06,093 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Deleting directory /opt/zimbra/index/0/14/index
2015-03-05 23:43:06,093 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 2 index files during restore
2015-03-05 23:43:06,095 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/14/index/0
2015-03-05 23:43:06,095 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox 14 account d72a6d8d-8117-4a90-b8cf-f972f4af056d LOADED
2015-03-05 23:43:06,095 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] cache - initializing folder and tag caches for mailbox 14
2015-03-05 23:43:06,264 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Account user1@mail-172.example.com: All pending file IO completed (2 out of 2)
2015-03-05 23:43:06,265 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore finished for user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox 6 to mailbox 14
2015-03-05 23:43:06,266 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is shut down
2015-03-05 23:43:06,266 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - RestoreRequest elapsed=7999
[zimbra@mail-172 ~]$
</pre>


Confirm that the message is still missing from the account:
Below is to confirm that indeed, it was only the restore with the -rf that did in fact have the message that was deleted after it.


<pre>
<pre>
[zimbra@mail-172 backup]$ zmmailbox -z -m user1@mail-172.example.com s -t mess in:Inbox
[zimbra@mail-172 ~]$ zmmailbox -z -m restore_user1@mail-172.example.com s -t mess in:Inbox
num: 0, more: false
num: 0, more: false
</pre>


Confirm that the account is still using the same zimbraId, mailboxId, and email address :
[zimbra@mail-172 ~]$ zmmailbox -z -m restore2_user1@mail-172.example.com s -t mess in:Inbox
num: 0, more: false


<pre>
[zimbra@mail-172 ~]$ zmmailbox -z -m restore3_user1@mail-172.example.com s -t mess in:Inbox
[zimbra@mail-172 backup]$ zmprov gmi user1@mail-172.example.com
num: 1, more: false
mailboxId: 6
quotaUsed: 0


[zimbra@mail-172 backup]$ zmprov ga user1@mail-172.example.com zimbraId
    Id  Type  From                  Subject                                            Date
# name user1@mail-172.example.com
  ----  ----  --------------------  --------------------------------------------------  --------------
zimbraId: 3c397edc-c015-4d82-ae8a-1084692f8a93
1.  257  mess  admin                Test1                                              02/23/15 09:03
</pre>
</pre>


=====Using The -restoreToTime Option To Restore Prior To A Message Being Deleted=====
=====Fourth Example - To Confirm -lb Label Does Matter=====


First, let's identify the exact time the delete operation took place. From our previous examples above I already know what redo log sequence file it was logged in.
This example is to show that the -lb does indeed matter. Earlier examples used the oldest full we had that did include the deleted message in the account. We'll create a new full backup of the parent account, user1@ , where the message is absent and then do two restores using the differing full backup session labels with the proper option of -rf since we are wanting the restore to stop once the full session data is restored.


<pre>
<pre>
[zimbra@mail-172 backup]$ zmredodump -m 6 --show-blob /opt/zimbra/redolog/archive/redo-20150304.095303.026-seq14.log | grep DeleteItem
[zimbra@mail-172 ~]$ zmmailbox -z -m user1@mail-172.example.com s -t mess in:Inbox
[000034d1 - 0000350c: 60 bytes; tstamp: 2015/03/04 04:48:18.119 EST] txn 1425452599.151 [DeleteItem] ver=1.42, tstamp=1425462498119, change=1513, mailbox=6, ids=[257], type=message
num: 0, more: false
[0000350d - 0000353e: 50 bytes; tstamp: 2015/03/04 04:48:18.306 EST] txn 1425452599.151 [CommitTxn] ver=1.42, tstamp=1425462498306, mailbox=6, txnType=DeleteItem
</pre>


The first line above shows me the delete request was done at "2015/03/04 04:48:18.119" for mailbox=6, ids=[257], type=message . Ids is the message id, that were already used to delete the message in the above example and was shown in the zmmailbox search command.
[zimbra@mail-172 ~]$ zmbackup -f -a all
full-20150306.055845.429


[zimbra@mail-172 ~]$ zmbackupquery -a user1@`zmhostname`
Account: user1@mail-172.example.com


The example uses the time above but subtracted one from it - 04:48:18.118 vs 04:48:18.119 .
    Label:  full-20150306.055845.429
    Type:    full
    Started: Fri, 2015/03/06 00:58:45.429 EST
    Ended:  Fri, 2015/03/06 00:58:52.357 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93


<pre>
    Label:  incr-20150306.043321.683
[zimbra@mail-172 backup]$ zmrestore -a user1@mail-172.example.com -restoreToTime "2015/03/04 04:48:18.118"
    Type:    incremental
using restore to time of 2015/03/04 04:48:18
    Started: Thu, 2015/03/05 23:33:21.683 EST
</pre>
    Ended:  Thu, 2015/03/05 23:33:27.495 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93


And to now confirm the message is back in the account:
    Label:   full-20150306.043202.011
    Type:    full
    Started: Thu, 2015/03/05 23:32:02.011 EST
    Ended:  Thu, 2015/03/05 23:32:13.316 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93


<pre>
    Label:  incr-20150305.060013.901
[zimbra@mail-172 backup]$ zmmailbox -z -m user1@mail-172.example.com s -t mess in:Inbox
    Type:    incremental
num: 1, more: false
    Started: Thu, 2015/03/05 01:00:13.901 EST
    Ended:   Thu, 2015/03/05 01:00:26.273 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93


    Id  Type   From                  Subject                                            Date
    Label:   incr-20150304.085810.220
   ----  ----  --------------------  --------------------------------------------------  --------------
    Type:   incremental
1. 257  mess   admin                Test1                                              02/23/15 09:03
    Started: Wed, 2015/03/04 03:58:10.220 EST
</pre>
    Ended:   Wed, 2015/03/04 03:58:16.393 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93


Note that this will also mean all other data or changes after 04:48:18.118 are lost also in this restored account. Normally in circumstances of trying to recovery data that a user is requesting because they delete messages and now need them recovered you would use the -ca and -pre options for zmrestore so you could then recover them and import the messages back into the existing account they have. Without the -ca and -pre , your over writing the existing account on the system.
    Label:  full-20150304.082015.535
    Type:    full
    Started: Wed, 2015/03/04 03:20:15.535 EST
    Ended:  Wed, 2015/03/04 03:20:39.109 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93


The log events in /opt/zimbra/log/mailbox.log for this request were:
[zimbra@mail-172 ~]$ zmrestore -a user1@mail-172.example.com -lb full-20150306.055845.429 -rf -ca -pre restore4_
 
[zimbra@mail-172 ~]$ zmrestore -a user1@mail-172.example.com -lb full-20150304.082015.535 -rf -ca -pre restore5_
 
[zimbra@mail-172 ~]$ zmmailbox -z -m restore4_user1@mail-172.example.com s -t mess in:Inbox
num: 0, more: false
 
[zimbra@mail-172 ~]$ zmmailbox -z -m restore5_user1@mail-172.example.com s -t mess in:Inbox
num: 1, more: false
 
    Id  Type  From                  Subject                                            Date
  ----  ----  --------------------  --------------------------------------------------  --------------
1.  257  mess  admin                Test1                                              02/23/15 09:03
 
[zimbra@mail-172 ~]$
</pre>
 
====What Is Different Between The Parent Account And The Restored Accounts Where We Used The -ca And -pre Options [LDAP]====
 
Below is a difference between the zmprov ga output between the parent account [user1@] and our last restore example [restore5_user1@]. Note, in our example here the parent account was created with default values and there was no customization's or changes to the defaults.  


<pre>
<pre>
2015-03-04 05:20:39,194 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is starting
[zimbra@mail-172 ~]$ zmprov ga user1@`zmhostname` > /tmp/user1
2015-03-04 05:20:39,238 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Using full backup full-20150304.082015.535 as starting point for restoring account 3c397edc-c015-4d82-ae8a-1084692f8a93
 
2015-03-04 05:20:39,526 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - waiting for FileLogWriter.FsyncThread-1425462783028 to finish.
[zimbra@mail-172 ~]$ zmprov ga restore5_user1@`zmhostname` > /tmp/restore5
2015-03-04 05:20:39,533 INFO  [FileLogWriter.FsyncThread-1425462783028] [] redolog - fsync thread exiting
 
2015-03-04 05:20:39,533 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - FileLogWriter.FsyncThread-1425462783028 finished
[zimbra@mail-172 ~]$ diff /tmp/user1 /tmp/restore5
2015-03-04 05:20:39,543 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - Redo log rollover took 305ms
1c1
2015-03-04 05:20:39,543 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore started for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6
< # name user1@mail-172.example.com
2015-03-04 05:20:39,550 INFO  [FileLogWriter.FsyncThread-1425464439542] [] redolog - Starting fsync thread with interval 10
---
2015-03-04 05:20:39,593 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - Putting mailbox 6 under maintenance.
> # name restore5_user1@mail-172.example.com
2015-03-04 05:20:39,593 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - clearing contents of mailbox 6, group 6
3c3
2015-03-04 05:20:39,614 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restoring to mailbox id 6 for existing account 3c397edc-c015-4d82-ae8a-1084692f8a93.
< mail: user1@mail-172.example.com
2015-03-04 05:20:39,652 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loading schema with version 7.2
---
2015-03-04 05:20:39,660 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Renaming volume_id column to locator in source table mail_item
> mail: restore5_user1@mail-172.example.com
2015-03-04 05:20:39,692 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Deleting directory /opt/zimbra/store/0/6/msg
8c8
2015-03-04 05:20:39,700 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 1 blobs during restore
< uid: user1
2015-03-04 05:20:39,703 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Deleting directory /opt/zimbra/index/0/6/index
---
2015-03-04 05:20:39,713 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 2 index files during restore
> uid: restore5_user1
2015-03-04 05:20:39,715 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 AVAILABLE
20,22c20
2015-03-04 05:20:39,717 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Account user1@mail-172.example.com: All pending file IO completed (2 out of 2)
< zimbraAuthTokens: 661297364|1425702763611|8.6.0_GA_1153
2015-03-04 05:20:39,717 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Run redo operations since last full backup
< zimbraAuthTokens: 414162390|1425702787760|8.6.0_GA_1153
2015-03-04 05:20:39,732 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/6/index/0
< zimbraAuthTokens: 2046929787|1425707910566|8.6.0_GA_1153
2015-03-04 05:20:39,732 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 LOADED
---
2015-03-04 05:20:39,732 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
> zimbraAuthTokens: 1030982216|1425708106344|8.6.0_GA_1153
2015-03-04 05:20:39,732 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] cache - initializing folder and tag caches for mailbox 6
159c157
2015-03-04 05:20:39,743 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
< zimbraId: 3c397edc-c015-4d82-ae8a-1084692f8a93
2015-03-04 05:20:39,749 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
---
2015-03-04 05:20:39,754 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
> zimbraId: e5c38a4b-6bea-45bb-9e14-dd3f39159a61
2015-03-04 05:20:39,760 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Run redo operations from archived redo logs starting at sequence 13
170c168
2015-03-04 05:20:39,762 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
< zimbraMailDeliveryAddress: user1@mail-172.example.com
2015-03-04 05:20:39,770 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
---
2015-03-04 05:20:39,774 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
> zimbraMailDeliveryAddress: restore5_user1@mail-172.example.com
2015-03-04 05:20:39,780 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
2015-03-04 05:20:39,786 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
2015-03-04 05:20:39,789 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore finished for user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox 6 to mailbox 6
2015-03-04 05:20:39,791 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is shut down
2015-03-04 05:20:39,791 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - RestoreRequest elapsed=646
</pre>
</pre>


Notice that this restore, unlike the first example, does '''NOT''' have the following line :
====Restore encountered redo log sequence error: Found gap in redo log sequence====
 
Example error:
<pre>
<pre>
INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailop - Deleting Message (id=257).
Error occurred: system failure: Restore encountered redo log sequence error:  
Found gap in redo log sequence; missing 3899 through 3895; To avoid future restore problems,
discard all existing backups and take a full backup of all accounts; If this error occurred
during restore, try the --ignoreRedoErrors option
</pre>
</pre>


=====Blobs In Backup And Blobs In Production=====
This error actually gives the two basic options you have when you encounter this. Depending on your circumstances, you might first do the restore using the --ignoreRedoErrors options and recovery what data you can prior to discarding the 'bad' backup sessions in question and attempting a new full backup of the users.


This might be a good time to take a look at the blobs and how they are saved/stored in the backups and how they compare to the store volume. The example below is from our user and message details from above.
 
=====How-To Example - Extended Look At RedoErrors With Log Events=====
 
First, let's replicate a redolog failure situation, which can be done by moving the redologs from the /opt/zimbra/redolog/archive/ to a temporary directory.


<pre>
<pre>
[zimbra@mail-172 blobs]$ pwd
[zimbra@mail-172 redolog]$ cd archive/
/opt/zimbra/backup/sessions/full-20150304.082015.535/accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93/blobs


[zimbra@mail-172 blobs]$ unzip -Z blobs-1.zip
[zimbra@mail-172 archive]$ ls
Archive: blobs-1.zip
redo-20150306.060015.331-seq21.log redo-20150306.060109.166-seq22.log
Zip file size: 1439 bytes, number of entries: 1
-rw----    4.5 fat    1209 b- stor 15-Mar-04 03:20 1/0/sha256_mBS9YKqyNpLknKCDWY5WNEbTgotX,UNrXMA5ga+s6LA=_257-2.msg1
1 file, 1209 bytes uncompressed, 1209 bytes compressed:  0.0%


[zimbra@mail-172 blobs]$ ls -la /opt/zimbra/store/0/6/msg/0/257-2.msg
[zimbra@mail-172 archive]$ mkdir /tmp/redolog-archive
-rw-r----- 1 zimbra zimbra 1209 Mar  4 05:20 /opt/zimbra/store/0/6/msg/0/257-2.msg


[zimbra@mail-172 blobs]$ unzip blobs-1.zip
[zimbra@mail-172 archive]$ mv * /tmp/redolog-archive/
Archive:  blobs-1.zip
extracting: 1/0/sha256_mBS9YKqyNpLknKCDWY5WNEbTgotX,UNrXMA5ga+s6LA=_257-2.msg1


[zimbra@mail-172 blobs]$ ls
[zimbra@mail-172 archive]$ cat /dev/null > /opt/zimbra/log/mailbox.log
1  blobs-1.zip
</pre>


[zimbra@mail-172 blobs]$ ls -la 1/0/sha256_mBS9YKqyNpLknKCDWY5WNEbTgotX\,UNrXMA5ga+s6LA\=_257-2.msg1
Now that we've moved the redolog sequence files out of the redolog/archive directory path, we can attempt our zmrestore command. Note that the incremental backup label for this attempt is called incr-20150306.170810.793 . Though it fails because of the missing redologs, the backup is still created.
-rw-r----- 1 zimbra zimbra 1209 Mar  4 03:20 1/0/sha256_mBS9YKqyNpLknKCDWY5WNEbTgotX,UNrXMA5ga+s6LA=_257-2.msg1


[zimbra@mail-172 blobs]$ head 1/0/sha256_mBS9YKqyNpLknKCDWY5WNEbTgotX\,UNrXMA5ga+s6LA\=_257-2.msg1
<pre>
Return-Path: zimbra@mail-172.example.com
[zimbra@mail-172 archive]$ zmbackup -i
Received: from mail-172.example.com (LHLO mail-172.example.com)
Error occurred: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option
(192.168.1.172) by mail-172.example.com with LMTP; Mon, 23 Feb 2015
09:02:58 -0500 (EST)
Received: from localhost (localhost.localdomain [127.0.0.1])
        by mail-172.example.com (Postfix) with ESMTP id 4F8AE8433D
        for <user1@mail-172.example.com>; Mon, 23 Feb 2015 09:02:58 -0500 (EST)
X-Virus-Scanned: amavisd-new at mail-172.example.com
X-Spam-Flag: NO
X-Spam-Score: -1.9


[zimbra@mail-172 blobs]$ head /opt/zimbra/store/0/6/msg/0/257-2.msg
[zimbra@mail-172 archive]$ cat /opt/zimbra/log/mailbox.log
Return-Path: zimbra@mail-172.example.com
2015-03-06 12:08:06,442 INFO  [MailboxPurge] [name=ham.jgfdb3oqjf@mail-172.example.com;mid=4;] purge - Purging messages.
Received: from mail-172.example.com (LHLO mail-172.example.com)
2015-03-06 12:08:10,350 INFO  [qtp509886383-484:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] soap - AuthRequest elapsed=4
  (192.168.1.172) by mail-172.example.com with LMTP; Mon, 23 Feb 2015
2015-03-06 12:08:10,413 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup request started
  09:02:58 -0500 (EST)
2015-03-06 12:08:10,579 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Found 16 accounts on server mail-172.example.com
Received: from localhost (localhost.localdomain [127.0.0.1])
2015-03-06 12:08:10,797 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] io - AsyncFileCopier is starting
        by mail-172.example.com (Postfix) with ESMTP id 4F8AE8433D
2015-03-06 12:08:10,798 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup started for backup set; label: incr-20150306.170810.793
        for <user1@mail-172.example.com>; Mon, 23 Feb 2015 09:02:58 -0500 (EST)
[cut]
X-Virus-Scanned: amavisd-new at mail-172.example.com
2015-03-06 12:08:11,277 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup is started for account user1@mail-172.example.com
X-Spam-Flag: NO
2015-03-06 12:08:11,277 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Saving account information from LDAP
X-Spam-Score: -1.9
2015-03-06 12:08:11,281 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup has ended
</pre>
2015-03-06 12:08:11,281 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Account user1@mail-172.example.com in backup set incr-20150306.170810.793: All pending file IO completed (0 out of 0)
 
[cut]
=====Knowing When The -br [To Incremental] Or -rf [Only Full] Option Is Required=====
2015-03-06 12:08:11,310 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup is started for account admin@mail-172.example.com
 
2015-03-06 12:08:11,310 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Saving account information from LDAP
======Pre-Steps======
2015-03-06 12:08:11,312 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup has ended
 
2015-03-06 12:08:11,312 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Account admin@mail-172.example.com in backup set incr-20150306.170810.793: All pending file IO completed (0 out of 0)
Let's query current backups and then create a new full and new incremental where we also delete a message between the full and the incremental.
2015-03-06 12:08:11,711 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] redolog - waiting for FileLogWriter.FsyncThread-1425621669168 to finish.
 
2015-03-06 12:08:11,712 INFO [FileLogWriter.FsyncThread-1425621669168] [] redolog - fsync thread exiting
<pre>
2015-03-06 12:08:11,712 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] redolog - FileLogWriter.FsyncThread-1425621669168 finished
[zimbra@mail-172 ~]$ zmbackupquery -a user1@`zmhostname`
2015-03-06 12:08:11,799 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] redolog - Redo log rollover took 487ms
Account: user1@mail-172.example.com
2015-03-06 12:08:11,799 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Found 1 redo logs to backup
 
2015-03-06 12:08:11,800 INFO  [FileLogWriter.FsyncThread-1425661691796] [] redolog - Starting fsync thread with interval 10
    Label:  incr-20150305.060013.901
2015-03-06 12:08:11,864 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Last backed-up redo log sequence = 21
    Type:    incremental
2015-03-06 12:08:11,864 ERROR [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option
    Started: Thu, 2015/03/05 01:00:13.901 EST
2015-03-06 12:08:11,973 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Copying redo log /opt/zimbra/redolog/archive/redo-20150306.170811.794-seq23.log to /opt/zimbra/backup/tmp/incr-20150306.170810.793/redologs/redo-20150306.170811.794-seq23.log
    Ended:   Thu, 2015/03/05 01:00:26.273 EST
2015-03-06 12:08:12,100 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Retaining recent redo log /opt/zimbra/redolog/archive/redo-20150306.170811.794-seq23.log
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93
2015-03-06 12:08:19,375 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup set incr-20150306.170810.793: All pending file IO completed (0 out of 0)
 
2015-03-06 12:08:19,378 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] io - AsyncFileCopier is shut down
    Label:  incr-20150304.085810.220
2015-03-06 12:08:19,378 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup finished for backup set; label: incr-20150306.170810.793
    Type:   incremental
2015-03-06 12:08:20,565 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] SoapEngine - handler exception
    Started: Wed, 2015/03/04 03:58:10.220 EST
com.zimbra.cs.backup.BackupServiceException: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option
    Ended:   Wed, 2015/03/04 03:58:16.393 EST
ExceptionId:qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest:1425661691864:14586f805b4c1b0e
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93
Code:backup.REDOLOG_OUT_OF_SEQUENCE
 
        at com.zimbra.cs.backup.BackupServiceException.REDOLOG_OUT_OF_SEQUENCE(BackupServiceException.java:91)
    Label:   full-20150304.082015.535
        at com.zimbra.cs.backup.util.Utils.splitRedoLogsAtSeq(Utils.java:302)
    Type:   full
        at com.zimbra.cs.backup.FileBackupTarget$FileBackupSet.backupRedoLogs(FileBackupTarget.java:1360)
    Started: Wed, 2015/03/04 03:20:15.535 EST
        at com.zimbra.cs.backup.BackupSet.startIncrementalBackup(BackupSet.java:817)
    Ended:   Wed, 2015/03/04 03:20:39.109 EST
        at com.zimbra.cs.backup.FileBackupTarget$FileBackupSet.startIncrementalBackup(FileBackupTarget.java:1067)
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93
        at com.zimbra.cs.backup.BackupManager.backupIncremental(BackupManager.java:336)
 
        at com.zimbra.cs.service.backup.Backup.handleNetworkRequest(Backup.java:153)
[zimbra@mail-172 ~]$ zmbackup -f -a all
        at com.zimbra.cs.service.NetworkDocumentHandler.handle(NetworkDocumentHandler.java:23)
full-20150306.043202.011
        at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEngine.java:569)
 
        at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:432)
[zimbra@mail-172 ~]$ zmmailbox -z -m user1@mail-172.example.com s -t mess in:Inbox
        at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:266)
num: 1, more: false
        at com.zimbra.soap.SoapServlet.doWork(SoapServlet.java:303)
 
        at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:213)
    Id  Type  From                  Subject                                            Date
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
  ----  ----   --------------------  -------------------------------------------------- --------------
        at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:209)
1.  257  mess  admin                 Test1                                              02/23/15 09:03
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
 
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738)
[zimbra@mail-172 ~]$ zmmailbox -z -m user1@mail-172.example.com dm 257
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651)
 
        at com.zimbra.cs.servlet.RequestStringFilter.doFilter(RequestStringFilter.java:54)
[zimbra@mail-172 ~]$ zmbackup -i
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
incr-20150306.043321.683
        at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:59)
 
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
[zimbra@mail-172 ~]$ zmbackupquery -a user1@`zmhostname`
        at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
Account: user1@mail-172.example.com
        at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:351)
 
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
    Label:  incr-20150306.043321.683
        at com.zimbra.cs.servlet.ETagHeaderFilter.doFilter(ETagHeaderFilter.java:47)
    Type:   incremental
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
    Started: Thu, 2015/03/05 23:33:21.683 EST
        at com.zimbra.cs.servlet.ContextPathBasedThreadPoolBalancerFilter.doFilter(ContextPathBasedThreadPoolBalancerFilter.java:107)
    Ended:   Thu, 2015/03/05 23:33:27.495 EST
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93
        at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(ZimbraQoSFilter.java:107)
 
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
    Label:   full-20150306.043202.011
        at com.zimbra.cs.servlet.ZimbraInvalidLoginFilter.doFilter(ZimbraInvalidLoginFilter.java:117)
    Type:   full
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
    Started: Thu, 2015/03/05 23:32:02.011 EST
        at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:457)
    Ended:   Thu, 2015/03/05 23:32:13.316 EST
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:326)
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:299)
 
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
    Label:   incr-20150305.060013.901
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549)
    Type:   incremental
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    Started: Thu, 2015/03/05 01:00:13.901 EST
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:544)
    Ended:   Thu, 2015/03/05 01:00:26.273 EST
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
 
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
    Label:   incr-20150304.085810.220
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
    Type:   incremental
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
    Started: Wed, 2015/03/04 03:58:10.220 EST
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    Ended:   Wed, 2015/03/04 03:58:16.393 EST
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:199)
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
 
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
    Label:   full-20150304.082015.535
        at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:309)
    Type:   full
        at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:81)
    Started: Wed, 2015/03/04 03:20:15.535 EST
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
    Ended:   Wed, 2015/03/04 03:20:39.109 EST
        at org.eclipse.jetty.server.Server.handle(Server.java:462)
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
        at java.lang.Thread.run(Thread.java:745)
2015-03-06 12:08:20,566 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] soap - BackupRequest elapsed=10154
</pre>
</pre>


======First Example - Options -lb, -ca, -pre restore_======
To confirm the backup label was made and also to see how the zmbackupquery reports it, we can do the following:


For the following examples, we'll include the -ca [create account] and the -pre [add prefix to new account name] . This first example uses the oldest full backup [full-20150304.082015.535 vs. full-20150306.043202.011] the account has with the -lb option. The message we just deleted, existed at the time of the full backup. We'll '''NOT''' be including either the -rf or -br option in this first example.
<pre>
 
[zimbra@mail-172 archive]$ zmbackupquery -lb incr-20150306.170810.793 -v
Then we'll review the mailbox.log events. Note the following:
Label:  incr-20150306.170810.793
* That the log only mentions the parent account, there isn't a reference to the email address with the prefixed name.
Type:    incremental
** There is a RFE to include the new account name in the log, [https://bugzilla.zimbra.com/show_bug.cgi?id=52641 Bug 52641 - RFE: include -ca -pre new account in mailbox.log log event]
Status:  completed (with errors)
* Though the log will state the new mailboxId and zimbraId for the user, though:
Started: Fri, 2015/03/06 12:08:10.793 EST
** "user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6" vs "Restoring to mailbox id 12 for new account 016c9f7c-e2e2-4f8f-96d0-e14a4327821d"
Ended:  Fri, 2015/03/06 12:08:19.378 EST
* Since we failed to include the -rf option, the restore continues past the full backup and goes through all the other backups that exist, hence we see:
Redo log sequence range: 23 .. 23
** "backup - Run redo operations since last full backup"
Number of accounts: 14 out of 14 completed
* One of the redologs in our incremental had the delete operation for the message in question and we see evidence of that delete in the log:
Number of errors: 1
** "mailop - Deleting Message (id=257)."
Accounts:
* Since we didn't include the -br option, the restore also continues past the backups and replays the redologs in /opt/zimbra/redolog and it's subdirectory, archive. Notice the reference to sequence 17.
  admin@mail-172.example.com: completed
** "backup - Run redo operations from archived redo logs starting at sequence 17"
  galsync.qva4ekog0@mail-172.example.com: completed
** /opt/zimbra/redolog/archive/ has : redo-20150306.043322.070-seq17.log  redo-20150306.043611.912-seq18.log
  ham.jgfdb3oqjf@mail-172.example.com: completed
* Towards the end, we'll get a confirmation that the restore has finished with some reference information again - mailboxId's .
  restore2_user1@mail-172.example.com: completed
** "Restore finished for user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox 6 to mailbox 12"
  restore3_user1@mail-172.example.com: completed
 
  restore_user1@mail-172.example.com: completed
<pre>
  spam.umhx3owv3s@mail-172.example.com: completed
[zimbra@mail-172 ~]$ cat /dev/null > /opt/zimbra/log/mailbox.log
  user1@mail-172.example.com: completed
 
  user2@mail-172.example.com: completed
[zimbra@mail-172 ~]$ zmrestore -a user1@mail-172.example.com -lb full-20150304.082015.535 -ca -pre restore_
  user3@mail-172.example.com: completed
 
  user4@mail-172.example.com: completed
[zimbra@mail-172 ~]$ cat /opt/zimbra/log/mailbox.log
  user5@mail-172.example.com: completed
2015-03-05 23:35:40,815 INFO  [MailboxPurge] [name=spam.umhx3owv3s@mail-172.example.com;mid=3;] purge - Purging messages.
  user6@mail-172.example.com: completed
2015-03-05 23:36:11,619 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - AuthRequest elapsed=3
  virus-quarantine.gusiso1cd@mail-172.example.com: completed
2015-03-05 23:36:11,784 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is starting
Error: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option
2015-03-05 23:36:11,905 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - waiting for FileLogWriter.FsyncThread-1425616402072 to finish.
com.zimbra.cs.backup.BackupServiceException: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option
2015-03-05 23:36:11,908 INFO  [FileLogWriter.FsyncThread-1425616402072] [] redolog - fsync thread exiting
ExceptionId:qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest:1425661691864:14586f805b4c1b0e
2015-03-05 23:36:11,908 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - FileLogWriter.FsyncThread-1425616402072 finished
Code:backup.REDOLOG_OUT_OF_SEQUENCE
2015-03-05 23:36:11,913 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - Redo log rollover took 128ms
        at com.zimbra.cs.backup.BackupServiceException.REDOLOG_OUT_OF_SEQUENCE(BackupServiceException.java:91)
2015-03-05 23:36:11,913 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore started for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6
        at com.zimbra.cs.backup.util.Utils.splitRedoLogsAtSeq(Utils.java:302)
2015-03-05 23:36:11,914 INFO  [FileLogWriter.FsyncThread-1425616571913] [] redolog - Starting fsync thread with interval 10
        at com.zimbra.cs.backup.FileBackupTarget$FileBackupSet.backupRedoLogs(FileBackupTarget.java:1360)
2015-03-05 23:36:13,006 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - Creating database mboxgroup12
        at com.zimbra.cs.backup.BackupSet.startIncrementalBackup(BackupSet.java:817)
2015-03-05 23:36:14,150 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restoring to mailbox id 12 for new account 016c9f7c-e2e2-4f8f-96d0-e14a4327821d.
        at com.zimbra.cs.backup.FileBackupTarget$FileBackupSet.startIncrementalBackup(FileBackupTarget.java:1067)
2015-03-05 23:36:14,237 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loading schema with version 7.2
        at com.zimbra.cs.backup.BackupManager.backupIncremental(BackupManager.java:336)
2015-03-05 23:36:14,276 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Renaming volume_id column to locator in source table mail_item
        at com.zimbra.cs.service.backup.Backup.handleNetworkRequest(Backup.java:153)
2015-03-05 23:36:14,315 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 1 blobs during restore
        at com.zimbra.cs.service.NetworkDocumentHandler.handle(NetworkDocumentHandler.java:23)
2015-03-05 23:36:14,316 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Deleting directory /opt/zimbra/index/0/12/index
        at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEngine.java:569)
2015-03-05 23:36:14,317 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 2 index files during restore
        at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:432)
2015-03-05 23:36:14,319 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/12/index/0
        at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:266)
2015-03-05 23:36:14,319 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox 12 account 016c9f7c-e2e2-4f8f-96d0-e14a4327821d LOADED
        at com.zimbra.soap.SoapServlet.doWork(SoapServlet.java:303)
2015-03-05 23:36:14,319 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] cache - initializing folder and tag caches for mailbox 12
        at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:213)
2015-03-05 23:36:14,367 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Account user1@mail-172.example.com: All pending file IO completed (2 out of 2)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
2015-03-05 23:36:14,367 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Run redo operations since last full backup
        at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:209)
2015-03-05 23:36:14,400 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
 
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738)
[multiple repeats of this message]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651)
 
        at com.zimbra.cs.servlet.RequestStringFilter.doFilter(RequestStringFilter.java:54)
2015-03-05 23:36:14,428 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
2015-03-05 23:36:14,465 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailop - Deleting Message (id=257).
        at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:59)
2015-03-05 23:36:14,474 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
 
        at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
[multiple repeats of this message]
        at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:351)
 
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
2015-03-05 23:36:15,152 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
        at com.zimbra.cs.servlet.ETagHeaderFilter.doFilter(ETagHeaderFilter.java:47)
2015-03-05 23:36:15,172 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Run redo operations from archived redo logs starting at sequence 17
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
2015-03-05 23:36:15,172 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore finished for user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox 6 to mailbox 12
        at com.zimbra.cs.servlet.ContextPathBasedThreadPoolBalancerFilter.doFilter(ContextPathBasedThreadPoolBalancerFilter.java:107)
2015-03-05 23:36:15,174 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is shut down
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
2015-03-05 23:36:15,174 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - RestoreRequest elapsed=3481
        at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(ZimbraQoSFilter.java:107)
 
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
[zimbra@mail-172 ~]$ ls /opt/zimbra/redolog/archive/
        at com.zimbra.cs.servlet.ZimbraInvalidLoginFilter.doFilter(ZimbraInvalidLoginFilter.java:117)
redo-20150306.043322.070-seq17.log  redo-20150306.043611.912-seq18.log
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
</pre>
        at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:457)
 
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:326)
======Second Example - Options -lb , -ca , -pre restore2_ , -br======
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:299)
 
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
For the second example, we'll again use the oldest full backup [full-20150304.082015.535 vs. full-20150306.043202.011] the account has with the -lb option. The message we had deleted, existed at the time of the full backup but did NOT in the last incremental backup. Actually, the delete operation is within the redologs of the last incremental. For the second example, we'll use the -br option rather than the -rf . We'll also be adjusting the -pre option to be restore2_ vs. restore_ in the first example.
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549)
 
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
Then we'll review the mailbox.log events. Note the following:
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:544)
* Though the log will state the new mailboxId and zimbraId for the user, though:
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
** "user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6" vs "Restoring to mailbox id 13 for new account 4c99a395-2c39-46ec-8c1b-285d97d899df."
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
** Notice the mailboxId and zimbraId for restore2_ is different than our first example when we used restore_ for the -pre option.
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
* Since we failed to include the -rf option, the restore continues past the full backup and goes through all the other backups that exist, hence we see:
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
** "backup - Run redo operations since last full backup"
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
* One of the redologs in our incremental had the delete operation for the message in question and we see evidence of that delete in the log:
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
** "mailop - Deleting Message (id=257)."
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:199)
* Since we '''DID''' include the -br option, the restore '''STOPS''' at our last backup session. It therefor '''DOESN'T''' replay the redologs in /opt/zimbra/redolog and it's subdirectory, archive and we do '''NOT''' have the following log events like we did in our first example.
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
** "backup - Run redo operations from archived redo logs starting at sequence 17"
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
* Towards the end, we'll get a confirmation that the restore has finished with some reference information again - mailboxId's .
        at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:309)
** "Restore finished for user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox 6 to mailbox 13"
        at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:81)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:462)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
        at java.lang.Thread.run(Thread.java:745)
Total space: 14020MB
  Free space: 6031MB
[zimbra@mail-172 archive]$
</pre>
 
Let's now try doing a restore of an account.


<pre>
<pre>
[zimbra@mail-172 ~]$ cat /dev/null > /opt/zimbra/log/mailbox.log
[zimbra@mail-172 archive]$ cat /dev/null > /opt/zimbra/log/mailbox.log


[zimbra@mail-172 ~]$ zmrestore -a user1@mail-172.example.com -lb full-20150304.082015.535 -br -ca -pre restore2_
[zimbra@mail-172 archive]$ zmrestore -a user1@`zmhostname` -ca -pre restore6_
 
Error occurred: system failure: Restore encountered redo log sequence error: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option
[zimbra@mail-172 ~]$ cat /opt/zimbra/log/mailbox.log
2015-03-05 23:39:30,176 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - AuthRequest elapsed=2
2015-03-05 23:39:30,263 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is starting
2015-03-05 23:39:30,352 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - waiting for FileLogWriter.FsyncThread-1425616571913 to finish.
2015-03-05 23:39:30,360 INFO  [FileLogWriter.FsyncThread-1425616571913] [] redolog - fsync thread exiting
2015-03-05 23:39:30,360 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - FileLogWriter.FsyncThread-1425616571913 finished
2015-03-05 23:39:30,371 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - Redo log rollover took 107ms
2015-03-05 23:39:30,371 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore started for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6
2015-03-05 23:39:30,377 INFO  [FileLogWriter.FsyncThread-1425616770369] [] redolog - Starting fsync thread with interval 10
2015-03-05 23:39:30,381 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - Creating database mboxgroup13
2015-03-05 23:39:30,992 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restoring to mailbox id 13 for new account 4c99a395-2c39-46ec-8c1b-285d97d899df.
2015-03-05 23:39:30,993 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loading schema with version 7.2
2015-03-05 23:39:30,997 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Renaming volume_id column to locator in source table mail_item
2015-03-05 23:39:31,022 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 1 blobs during restore
2015-03-05 23:39:31,023 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Deleting directory /opt/zimbra/index/0/13/index
2015-03-05 23:39:31,023 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 2 index files during restore
2015-03-05 23:39:31,024 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/13/index/0
2015-03-05 23:39:31,024 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox 13 account 4c99a395-2c39-46ec-8c1b-285d97d899df LOADED
2015-03-05 23:39:31,024 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] cache - initializing folder and tag caches for mailbox 13
2015-03-05 23:39:31,050 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Account user1@mail-172.example.com: All pending file IO completed (2 out of 2)
2015-03-05 23:39:31,050 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Run redo operations since last full backup
2015-03-05 23:39:31,053 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
 
[multiple repeats of this message]
 
2015-03-05 23:39:31,070 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
2015-03-05 23:39:31,075 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailop - Deleting Message (id=257).
2015-03-05 23:39:31,079 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
 
[multiple repeats of this message]
 
2015-03-05 23:39:31,829 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
2015-03-05 23:39:31,832 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore finished for user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox 6 to mailbox 13
2015-03-05 23:39:31,833 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is shut down
2015-03-05 23:39:31,834 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - RestoreRequest elapsed=1579
2015-03-05 23:39:40,932 INFO  [MailboxPurge] [name=user5@mail-172.example.com;mid=7;] purge - Purging messages.
[zimbra@mail-172 ~]$
</pre>
 
======Third Example - Options -lb , -ca , -pre restore3_ , -rf======
 
For the third example, we'll again use the oldest full backup [full-20150304.082015.535 vs. full-20150306.043202.011] the account has with the -lb option. The message we had deleted, existed at the time of the full backup but did NOT in the last incremental backup. For the third example, we'll use the -rf option rather than the -br . We'll also be adjusting the -pre option to be restore3_ vs. restore_ or restore2_ in the other examples.
 
Then we'll review the mailbox.log events. Note the following:
* Though the log will state the new mailboxId and zimbraId for the user, though:
** "user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6" vs "Restoring to mailbox id 14 for new account d72a6d8d-8117-4a90-b8cf-f972f4af056d."
** Notice the mailboxId and zimbraId for restore3_ is different than our other examples.
* Since we '''INCLUDED''' the -rf option, the restore '''STOPS''' once the full backup is restored. Hence, we '''DON'T''' see the following line like we did in the other examples:
** "backup - Run redo operations since last full backup"
* And because it '''DIDN'T''' replay any of the redologs, there is '''NO''' reference to the following line either in the log:
** "mailop - Deleting Message (id=257)."
* '''NOR''' is there a reference to the following line either in the log:
** "backup - Run redo operations from archived redo logs starting at sequence 17"
* Towards the end, we'll get a confirmation that the restore has finished with some reference information again - mailboxId's .
** "Restore finished for user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox 6 to mailbox 14"
 
<pre>
[zimbra@mail-172 ~]$ cat /dev/null > /opt/zimbra/log/mailbox.log
[zimbra@mail-172 ~]$ zmrestore -a user1@mail-172.example.com -lb full-20150304.082015.535 -rf -ca -pre restore3_
[zimbra@mail-172 ~]$ cat /opt/zimbra/log/mailbox.log
2015-03-05 23:42:50,233 INFO  [Timer-Zimbra] [] SessionCache - Removed 1 idle sessions (SOAP). 1 active sessions remain.
2015-03-05 23:42:58,194 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - AuthRequest elapsed=3
2015-03-05 23:42:58,282 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is starting
2015-03-05 23:42:58,475 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - waiting for FileLogWriter.FsyncThread-1425616770369 to finish.
2015-03-05 23:42:58,477 INFO  [FileLogWriter.FsyncThread-1425616770369] [] redolog - fsync thread exiting
2015-03-05 23:42:58,477 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - FileLogWriter.FsyncThread-1425616770369 finished
2015-03-05 23:42:58,485 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - Redo log rollover took 193ms
2015-03-05 23:42:58,485 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore started for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6
2015-03-05 23:42:58,495 INFO  [FileLogWriter.FsyncThread-1425616978485] [] redolog - Starting fsync thread with interval 10
2015-03-05 23:42:58,515 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - Creating database mboxgroup14
2015-03-05 23:43:06,041 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restoring to mailbox id 14 for new account d72a6d8d-8117-4a90-b8cf-f972f4af056d.
2015-03-05 23:43:06,042 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loading schema with version 7.2
2015-03-05 23:43:06,048 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Renaming volume_id column to locator in source table mail_item
2015-03-05 23:43:06,092 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 1 blobs during restore
2015-03-05 23:43:06,093 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Deleting directory /opt/zimbra/index/0/14/index
2015-03-05 23:43:06,093 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 2 index files during restore
2015-03-05 23:43:06,095 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/14/index/0
2015-03-05 23:43:06,095 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox 14 account d72a6d8d-8117-4a90-b8cf-f972f4af056d LOADED
2015-03-05 23:43:06,095 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] cache - initializing folder and tag caches for mailbox 14
2015-03-05 23:43:06,264 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Account user1@mail-172.example.com: All pending file IO completed (2 out of 2)
2015-03-05 23:43:06,265 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore finished for user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox 6 to mailbox 14
2015-03-05 23:43:06,266 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is shut down
2015-03-05 23:43:06,266 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - RestoreRequest elapsed=7999
[zimbra@mail-172 ~]$
</pre>
 
Below is to confirm that indeed, it was only the restore with the -rf that did in fact have the message that was deleted after it.
 
<pre>
[zimbra@mail-172 ~]$ zmmailbox -z -m restore_user1@mail-172.example.com s -t mess in:Inbox
num: 0, more: false
 
[zimbra@mail-172 ~]$ zmmailbox -z -m restore2_user1@mail-172.example.com s -t mess in:Inbox
num: 0, more: false
 
[zimbra@mail-172 ~]$ zmmailbox -z -m restore3_user1@mail-172.example.com s -t mess in:Inbox
num: 1, more: false
 
    Id  Type  From                  Subject                                            Date
  ----  ----  --------------------  --------------------------------------------------  --------------
1.  257  mess  admin                Test1                                              02/23/15 09:03
</pre>
 
======Fourth Example - To Confirm -lb Label Does Matter======
 
This example is to show that the -lb does indeed matter. Earlier examples used the oldest full we had that did include the deleted message in the account. We'll create a new full backup of the parent account, user1@ , where the message is absent and then do two restores using the differing full backup session labels with the proper option of -rf since we are wanting the restore to stop once the full session data is restored.
 
<pre>
[zimbra@mail-172 ~]$ zmmailbox -z -m user1@mail-172.example.com s -t mess in:Inbox
num: 0, more: false
 
[zimbra@mail-172 ~]$ zmbackup -f -a all
full-20150306.055845.429
 
[zimbra@mail-172 ~]$ zmbackupquery -a user1@`zmhostname`
Account: user1@mail-172.example.com
 
    Label:  full-20150306.055845.429
    Type:    full
    Started: Fri, 2015/03/06 00:58:45.429 EST
    Ended:  Fri, 2015/03/06 00:58:52.357 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93
 
    Label:  incr-20150306.043321.683
    Type:    incremental
    Started: Thu, 2015/03/05 23:33:21.683 EST
    Ended:  Thu, 2015/03/05 23:33:27.495 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93
 
    Label:  full-20150306.043202.011
    Type:    full
    Started: Thu, 2015/03/05 23:32:02.011 EST
    Ended:  Thu, 2015/03/05 23:32:13.316 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93
 
    Label:  incr-20150305.060013.901
    Type:    incremental
    Started: Thu, 2015/03/05 01:00:13.901 EST
    Ended:  Thu, 2015/03/05 01:00:26.273 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93
 
    Label:  incr-20150304.085810.220
    Type:    incremental
    Started: Wed, 2015/03/04 03:58:10.220 EST
    Ended:  Wed, 2015/03/04 03:58:16.393 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93
 
    Label:  full-20150304.082015.535
    Type:    full
    Started: Wed, 2015/03/04 03:20:15.535 EST
    Ended:  Wed, 2015/03/04 03:20:39.109 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93
 
[zimbra@mail-172 ~]$ zmrestore -a user1@mail-172.example.com -lb full-20150306.055845.429 -rf -ca -pre restore4_
 
[zimbra@mail-172 ~]$ zmrestore -a user1@mail-172.example.com -lb full-20150304.082015.535 -rf -ca -pre restore5_
 
[zimbra@mail-172 ~]$ zmmailbox -z -m restore4_user1@mail-172.example.com s -t mess in:Inbox
num: 0, more: false
 
[zimbra@mail-172 ~]$ zmmailbox -z -m restore5_user1@mail-172.example.com s -t mess in:Inbox
num: 1, more: false
 
    Id  Type  From                  Subject                                            Date
  ----  ----  --------------------  --------------------------------------------------  --------------
1.  257  mess  admin                Test1                                              02/23/15 09:03
 
[zimbra@mail-172 ~]$
</pre>
 
=====What Is Different Between The Parent Account And The Restored Accounts Where We Used The -ca And -pre Options [LDAP]=====
 
Below is a difference between the zmprov ga output between the parent account [user1@] and our last restore example [restore5_user1@]. Note, in our example here the parent account was created with default values and there was no customization's or changes to the defaults.
 
<pre>
[zimbra@mail-172 ~]$ zmprov ga user1@`zmhostname` > /tmp/user1
 
[zimbra@mail-172 ~]$ zmprov ga restore5_user1@`zmhostname` > /tmp/restore5
 
[zimbra@mail-172 ~]$ diff /tmp/user1 /tmp/restore5
1c1
< # name user1@mail-172.example.com
---
> # name restore5_user1@mail-172.example.com
3c3
< mail: user1@mail-172.example.com
---
> mail: restore5_user1@mail-172.example.com
8c8
< uid: user1
---
> uid: restore5_user1
20,22c20
< zimbraAuthTokens: 661297364|1425702763611|8.6.0_GA_1153
< zimbraAuthTokens: 414162390|1425702787760|8.6.0_GA_1153
< zimbraAuthTokens: 2046929787|1425707910566|8.6.0_GA_1153
---
> zimbraAuthTokens: 1030982216|1425708106344|8.6.0_GA_1153
159c157
< zimbraId: 3c397edc-c015-4d82-ae8a-1084692f8a93
---
> zimbraId: e5c38a4b-6bea-45bb-9e14-dd3f39159a61
170c168
< zimbraMailDeliveryAddress: user1@mail-172.example.com
---
> zimbraMailDeliveryAddress: restore5_user1@mail-172.example.com
</pre>
 
=====Restore encountered redo log sequence error: Found gap in redo log sequence=====
 
Example error:
<pre>
Error occurred: system failure: Restore encountered redo log sequence error:  
Found gap in redo log sequence; missing 3899 through 3895; To avoid future restore problems,  
discard all existing backups and take a full backup of all accounts; If this error occurred  
during restore, try the --ignoreRedoErrors option
</pre>
 
This error actually gives the two basic options you have when you encounter this. Depending on your circumstances, you might first do the restore using the --ignoreRedoErrors options and recovery what data you can prior to discarding the 'bad' backup sessions in question and attempting a new full backup of the users.


[zimbra@mail-172 archive]$ cat /opt/zimbra/log/mailbox.log
2015-03-06 12:12:31,884 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - RestoreRequest elapsed=38
2015-03-06 12:13:06,650 INFO  [MailboxPurge] [name=user3@mail-172.example.com;mid=9;] purge - Purging messages.
2015-03-06 12:13:45,329 INFO  [qtp509886383-484:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - AuthRequest elapsed=2
2015-03-06 12:13:45,474 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is starting
2015-03-06 12:13:45,512 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Using full backup full-20150306.055845.429 as starting point for restoring account 3c397edc-c015-4d82-ae8a-1084692f8a93
2015-03-06 12:13:45,662 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - waiting for FileLogWriter.FsyncThread-1425661691796 to finish.
2015-03-06 12:13:45,669 INFO  [FileLogWriter.FsyncThread-1425661691796] [] redolog - fsync thread exiting
2015-03-06 12:13:45,669 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - FileLogWriter.FsyncThread-1425661691796 finished
2015-03-06 12:13:45,678 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - Redo log rollover took 166ms
2015-03-06 12:13:45,678 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore started for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6
2015-03-06 12:13:45,700 INFO  [FileLogWriter.FsyncThread-1425662025678] [] redolog - Starting fsync thread with interval 10
2015-03-06 12:13:45,981 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - Creating database mboxgroup17
2015-03-06 12:13:46,695 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restoring to mailbox id 17 for new account f7f4c1a8-15d0-41a1-bd11-6e05359ea3e8.
2015-03-06 12:13:46,745 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loading schema with version 7.2
2015-03-06 12:13:46,785 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Renaming volume_id column to locator in source table mail_item
2015-03-06 12:13:46,820 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 0 blobs during restore
2015-03-06 12:13:46,822 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Deleting directory /opt/zimbra/index/0/17/index
2015-03-06 12:13:46,882 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 2 index files during restore
2015-03-06 12:13:46,887 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/17/index/0
2015-03-06 12:13:46,887 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox 17 account f7f4c1a8-15d0-41a1-bd11-6e05359ea3e8 LOADED
2015-03-06 12:13:46,887 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] cache - initializing folder and tag caches for mailbox 17
2015-03-06 12:13:46,912 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Account user1@mail-172.example.com: All pending file IO completed (2 out of 2)
2015-03-06 12:13:46,912 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Run redo operations since last full backup
2015-03-06 12:13:47,047 ERROR [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option
2015-03-06 12:13:47,047 WARN  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Redo log sequence error: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option
com.zimbra.cs.backup.BackupServiceException: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option
ExceptionId:qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest:1425662027047:14586f805b4c1b0e
Code:backup.REDOLOG_OUT_OF_SEQUENCE
        at com.zimbra.cs.backup.BackupServiceException.REDOLOG_OUT_OF_SEQUENCE(BackupServiceException.java:91)
        at com.zimbra.cs.backup.util.Utils.splitRedoLogsAtSeq(Utils.java:302)
        at com.zimbra.cs.backup.RestoreAccountSession.checkRedoLogSequenceContiguity(RestoreAccountSession.java:534)
        at com.zimbra.cs.backup.RestoreAccountSession.endRestore(RestoreAccountSession.java:350)
        at com.zimbra.cs.backup.FileBackupTarget$RestoreAcctSession.endRestore(FileBackupTarget.java:2524)
        at com.zimbra.cs.backup.FileBackupTarget.restore(FileBackupTarget.java:490)
        at com.zimbra.cs.backup.BackupManager.restore(BackupManager.java:826)
        at com.zimbra.cs.service.backup.Restore.handle(Restore.java:137)
        at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEngine.java:569)
        at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:432)
        at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:266)
        at com.zimbra.soap.SoapServlet.doWork(SoapServlet.java:303)
        at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:213)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
        at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:209)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651)
        at com.zimbra.cs.servlet.RequestStringFilter.doFilter(RequestStringFilter.java:54)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:59)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
        at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:351)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ETagHeaderFilter.doFilter(ETagHeaderFilter.java:47)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ContextPathBasedThreadPoolBalancerFilter.doFilter(ContextPathBasedThreadPoolBalancerFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(ZimbraQoSFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ZimbraInvalidLoginFilter.doFilter(ZimbraInvalidLoginFilter.java:117)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:457)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:326)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:299)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:544)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:199)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:309)
        at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:81)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:462)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
        at java.lang.Thread.run(Thread.java:745)
2015-03-06 12:13:47,047 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Cleaning up data from failed restore (mid=17)
2015-03-06 12:13:47,063 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - Putting mailbox 17 under maintenance.
2015-03-06 12:13:47,084 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - clearing contents of mailbox 17, group 17
2015-03-06 12:13:47,331 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox for account f7f4c1a8-15d0-41a1-bd11-6e05359ea3e8 DELETED
2015-03-06 12:13:47,375 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Cleaning up account created during failed restore (account=f7f4c1a8-15d0-41a1-bd11-6e05359ea3e8)
2015-03-06 12:13:47,752 WARN  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Error occurred during restore account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93)
com.zimbra.common.service.ServiceException: system failure: Restore encountered redo log sequence error: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option
ExceptionId:qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest:1425662027047:14586f805b4c1b0e
Code:service.FAILURE
        at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:260)
        at com.zimbra.cs.backup.RestoreAccountSession.checkRedoLogSequenceContiguity(RestoreAccountSession.java:545)
        at com.zimbra.cs.backup.RestoreAccountSession.endRestore(RestoreAccountSession.java:350)
        at com.zimbra.cs.backup.FileBackupTarget$RestoreAcctSession.endRestore(FileBackupTarget.java:2524)
        at com.zimbra.cs.backup.FileBackupTarget.restore(FileBackupTarget.java:490)
        at com.zimbra.cs.backup.BackupManager.restore(BackupManager.java:826)
        at com.zimbra.cs.service.backup.Restore.handle(Restore.java:137)
        at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEngine.java:569)
        at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:432)
        at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:266)
        at com.zimbra.soap.SoapServlet.doWork(SoapServlet.java:303)
        at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:213)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
        at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:209)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651)
        at com.zimbra.cs.servlet.RequestStringFilter.doFilter(RequestStringFilter.java:54)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:59)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
        at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:351)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ETagHeaderFilter.doFilter(ETagHeaderFilter.java:47)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ContextPathBasedThreadPoolBalancerFilter.doFilter(ContextPathBasedThreadPoolBalancerFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(ZimbraQoSFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ZimbraInvalidLoginFilter.doFilter(ZimbraInvalidLoginFilter.java:117)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:457)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:326)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:299)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:544)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:199)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:309)
        at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:81)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:462)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
        at java.lang.Thread.run(Thread.java:745)
Caused by: com.zimbra.cs.backup.BackupServiceException: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option
ExceptionId:qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest:1425662027047:14586f805b4c1b0e
Code:backup.REDOLOG_OUT_OF_SEQUENCE
        at com.zimbra.cs.backup.BackupServiceException.REDOLOG_OUT_OF_SEQUENCE(BackupServiceException.java:91)
        at com.zimbra.cs.backup.util.Utils.splitRedoLogsAtSeq(Utils.java:302)
        at com.zimbra.cs.backup.RestoreAccountSession.checkRedoLogSequenceContiguity(RestoreAccountSession.java:534)
        ... 56 more
2015-03-06 12:13:47,755 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is shut down
2015-03-06 12:13:47,755 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] SoapEngine - handler exception
com.zimbra.common.service.ServiceException: system failure: Restore encountered redo log sequence error: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option
ExceptionId:qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest:1425662027047:14586f805b4c1b0e
Code:service.FAILURE
        at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:260)
        at com.zimbra.cs.backup.RestoreAccountSession.checkRedoLogSequenceContiguity(RestoreAccountSession.java:545)
        at com.zimbra.cs.backup.RestoreAccountSession.endRestore(RestoreAccountSession.java:350)
        at com.zimbra.cs.backup.FileBackupTarget$RestoreAcctSession.endRestore(FileBackupTarget.java:2524)
        at com.zimbra.cs.backup.FileBackupTarget.restore(FileBackupTarget.java:490)
        at com.zimbra.cs.backup.BackupManager.restore(BackupManager.java:826)
        at com.zimbra.cs.service.backup.Restore.handle(Restore.java:137)
        at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEngine.java:569)
        at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:432)
        at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:266)
        at com.zimbra.soap.SoapServlet.doWork(SoapServlet.java:303)
        at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:213)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
        at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:209)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651)
        at com.zimbra.cs.servlet.RequestStringFilter.doFilter(RequestStringFilter.java:54)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:59)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
        at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:351)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ETagHeaderFilter.doFilter(ETagHeaderFilter.java:47)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ContextPathBasedThreadPoolBalancerFilter.doFilter(ContextPathBasedThreadPoolBalancerFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(ZimbraQoSFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ZimbraInvalidLoginFilter.doFilter(ZimbraInvalidLoginFilter.java:117)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:457)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:326)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:299)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:544)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:199)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:309)
        at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:81)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:462)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
        at java.lang.Thread.run(Thread.java:745)
Caused by: com.zimbra.cs.backup.BackupServiceException: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option
ExceptionId:qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest:1425662027047:14586f805b4c1b0e
Code:backup.REDOLOG_OUT_OF_SEQUENCE
        at com.zimbra.cs.backup.BackupServiceException.REDOLOG_OUT_OF_SEQUENCE(BackupServiceException.java:91)
        at com.zimbra.cs.backup.util.Utils.splitRedoLogsAtSeq(Utils.java:302)
        at com.zimbra.cs.backup.RestoreAccountSession.checkRedoLogSequenceContiguity(RestoreAccountSession.java:534)
        ... 56 more
2015-03-06 12:13:47,755 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - RestoreRequest elapsed=2358


======How-To Example - Extended Look At RedoErrors With Log Events======
[zimbra@mail-172 archive]$ zmprov ga restore6_user1@`zmhostname` | head
ERROR: account.NO_SUCH_ACCOUNT (no such account: restore6_user1@mail-172.example.com)
</pre>


First, let's replicate a redolog failure situation, which can be done by moving the redologs from the /opt/zimbra/redolog/archive/ to a temporary directory.
Let's now see what happens if we move the missing redolog seq files back into /opt/zimbra/redolog/archive and then if we move/copy them into the backup sessions redolog directory. With restore attempts being done after each move/copy.


<pre>
<pre>
[zimbra@mail-172 redolog]$ cd archive/
[zimbra@mail-172 archive]$ mv /tmp/redolog-archive/* ./
 
[zimbra@mail-172 archive]$ pwd
/opt/zimbra/redolog/archive


[zimbra@mail-172 archive]$ ls
[zimbra@mail-172 archive]$ ls
redo-20150306.060015.331-seq21.log  redo-20150306.060109.166-seq22.log
redo-20150306.060015.331-seq21.log  redo-20150306.060109.166-seq22.log  redo-20150306.170811.794-seq23.log  redo-20150306.171345.676-seq24.log


[zimbra@mail-172 archive]$ mkdir /tmp/redolog-archive
[zimbra@mail-172 archive]$ zmrestore -a user1@`zmhostname` -ca -pre restore6_
Error occurred: system failure: Restore encountered redo log sequence error: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option


[zimbra@mail-172 archive]$ mv * /tmp/redolog-archive/
[zimbra@mail-172 archive]$ pwd
/opt/zimbra/redolog/archive
 
[zimbra@mail-172 archive]$ cp *seq21.log *seq22.log /opt/zimbra/backup/sessions/incr-20150306.170810.793/redologs/
 
[zimbra@mail-172 archive]$ zmrestore -a user1@`zmhostname` -ca -pre restore6_


[zimbra@mail-172 archive]$ cat /dev/null > /opt/zimbra/log/mailbox.log
[zimbra@mail-172 archive]$ zmprov ga restore6_user1@`zmhostname` | head -2
# name restore6_user1@mail-172.example.com
cn: user1
</pre>
</pre>


Now that we've moved the redolog sequence files out of the redolog/archive directory path, we can attempt our zmrestore command. Note that the incremental backup label for this attempt is called incr-20150306.170810.793 . Though it fails because of the missing redologs, the backup is still created.
====Redirected Restore With No Deletes To Then Import Data From A Specific Time Range Into A Subdirectory Of Users Archive Account====
 
This example would be the steps one might do if you discovered that the archiving service wasn't working as expected for a given date range and you wanted to populate the users archive account with the missing data. Normally, this would involve many users or all of your users - hence, why the issue about licenses is addressed.
 
Note, I use the following variables below but you should be replacing them to suit your needs.
 
* $USER = users production email
* recovery_$USER = the new email address that the restore created [-ca -pre recovery_]
* $DOMAIN = the domain for the user
* $USER-archive@$DOMAIN.archive = the production users archive email address
 
Determine all the necessary variables for your restore command.
 
* zmrestore -a $USER
** -ca -pre recovery_
** -lb determine what full backup label to use
** determine if you need to state the incremental backup label [-restoreToIncrLabel] and/or a specific time to stop the restore at [-restoreToTime]
** determine if you need to use -rf or -br
** --skipDeletes
 
For example:


<pre>
<pre>
[zimbra@mail-172 archive]$ zmbackup -i
$ zmrestore -a user1@`zmhostname` -ca -pre recovery_ -lb full-20150321.050015.999 -restoreToTime 20150323000000 -br --skipDeletes
Error occurred: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option
 
using restore to time of 2015/03/23 00:00:00
</pre>
 
Set the restored account to not consume a normal license [notice recovery_$USER vs $USER]:
 
<pre>$ zmprov ma recovery_$USER zimbraIsSystemResource TRUE</pre>
 
Get what the archive accounts address is for the main account:
 
<pre>$ zmprov ga $USER amavisArchiveQuarantineTo</pre>


[zimbra@mail-172 archive]$ cat /opt/zimbra/log/mailbox.log
Set the restored account to not consume an archive license [notice recovery_$USER vs $USER]:
2015-03-06 12:08:06,442 INFO  [MailboxPurge] [name=ham.jgfdb3oqjf@mail-172.example.com;mid=4;] purge - Purging messages.
 
2015-03-06 12:08:10,350 INFO  [qtp509886383-484:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] soap - AuthRequest elapsed=4
<pre>$ zmprov ma recovery_$USER amavisArchiveQuarantineTo ""</pre>
2015-03-06 12:08:10,413 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup request started
 
2015-03-06 12:08:10,579 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Found 16 accounts on server mail-172.example.com
Create a directory to dump all the export data in. You might want to have this shared across your mailstores [that hold archive accounts] so you can then import them locally to the mailstore the archive account resides on.
2015-03-06 12:08:10,797 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] io - AsyncFileCopier is starting
 
2015-03-06 12:08:10,798 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup started for backup set; label: incr-20150306.170810.793
<pre>
[cut]
$ mkdir /tmp/export
2015-03-06 12:08:11,277 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup is started for account user1@mail-172.example.com
 
2015-03-06 12:08:11,277 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Saving account information from LDAP
$ cd /tmp/export
2015-03-06 12:08:11,281 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup has ended
</pre>
2015-03-06 12:08:11,281 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Account user1@mail-172.example.com in backup set incr-20150306.170810.793: All pending file IO completed (0 out of 0)
 
[cut]
Example exports out all data between a given date , adjust the account and date string below. Also, the filename that is being saved.
2015-03-06 12:08:11,310 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup is started for account admin@mail-172.example.com
 
2015-03-06 12:08:11,310 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Saving account information from LDAP
<pre>
2015-03-06 12:08:11,312 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup has ended
$ zmmailbox -z -m recovery_$USER@$DOMAIN gru '//?fmt=tgz&query=under:/ after:"4/8/15" AND before:"4/11/15"' > /tmp/export/$USER.tgz
2015-03-06 12:08:11,312 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Account admin@mail-172.example.com in backup set incr-20150306.170810.793: All pending file IO completed (0 out of 0)
</pre>
2015-03-06 12:08:11,711 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] redolog - waiting for FileLogWriter.FsyncThread-1425621669168 to finish.
 
2015-03-06 12:08:11,712 INFO  [FileLogWriter.FsyncThread-1425621669168] [] redolog - fsync thread exiting
Example of what is dumped
2015-03-06 12:08:11,712 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] redolog - FileLogWriter.FsyncThread-1425621669168 finished
 
2015-03-06 12:08:11,799 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] redolog - Redo log rollover took 487ms
<pre>
2015-03-06 12:08:11,799 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Found 1 redo logs to backup
$ pwd
2015-03-06 12:08:11,800 INFO  [FileLogWriter.FsyncThread-1425661691796] [] redolog - Starting fsync thread with interval 10
/tmp/export
2015-03-06 12:08:11,864 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Last backed-up redo log sequence = 21
 
2015-03-06 12:08:11,864 ERROR [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option
$ ls
2015-03-06 12:08:11,973 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Copying redo log /opt/zimbra/redolog/archive/redo-20150306.170811.794-seq23.log to /opt/zimbra/backup/tmp/incr-20150306.170810.793/redologs/redo-20150306.170811.794-seq23.log
admin.tgz
2015-03-06 12:08:12,100 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Retaining recent redo log /opt/zimbra/redolog/archive/redo-20150306.170811.794-seq23.log
 
2015-03-06 12:08:19,375 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup set incr-20150306.170810.793: All pending file IO completed (0 out of 0)
$ tar zxvf $USER.tgz
2015-03-06 12:08:19,378 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] io - AsyncFileCopier is shut down
Inbox/0000000776-ZCS Backup Report_ SUCCESS.eml.meta
2015-03-06 12:08:19,378 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup finished for backup set; label: incr-20150306.170810.793
Inbox/0000000776-ZCS Backup Report_ SUCCESS.eml
2015-03-06 12:08:20,565 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] SoapEngine - handler exception
Inbox/0000000777-Daily mail report for 2015-04-09.eml.meta
com.zimbra.cs.backup.BackupServiceException: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option
Inbox/0000000777-Daily mail report for 2015-04-09.eml
ExceptionId:qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest:1425661691864:14586f805b4c1b0e
Inbox/0000000778-ZCS Backup Report_ SUCCESS.eml.meta
Code:backup.REDOLOG_OUT_OF_SEQUENCE
Inbox/0000000778-ZCS Backup Report_ SUCCESS.eml
        at com.zimbra.cs.backup.BackupServiceException.REDOLOG_OUT_OF_SEQUENCE(BackupServiceException.java:91)
Inbox/0000000779-Daily mail report for 2015-04-10.eml.meta
        at com.zimbra.cs.backup.util.Utils.splitRedoLogsAtSeq(Utils.java:302)
Inbox/0000000779-Daily mail report for 2015-04-10.eml
        at com.zimbra.cs.backup.FileBackupTarget$FileBackupSet.backupRedoLogs(FileBackupTarget.java:1360)
</pre>
        at com.zimbra.cs.backup.BackupSet.startIncrementalBackup(BackupSet.java:817)
 
        at com.zimbra.cs.backup.FileBackupTarget$FileBackupSet.startIncrementalBackup(FileBackupTarget.java:1067)
Now to import the data into the users Archive account. Below example will dump the imported data into a specfic folder that will be created [I'm using Imported]. Adjust the account and filename below.
        at com.zimbra.cs.backup.BackupManager.backupIncremental(BackupManager.java:336)
 
        at com.zimbra.cs.service.backup.Backup.handleNetworkRequest(Backup.java:153)
<pre>
        at com.zimbra.cs.service.NetworkDocumentHandler.handle(NetworkDocumentHandler.java:23)
$ zmmailbox -z -m $USER-archive@$DOMAIN.archive pru "//?fmt=tgz&subfolder=Imported" /tmp/export/$USER.tgz
        at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEngine.java:569)
 
        at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:432)
$ zmmailbox -z -m $USER-archive@$DOMAIN.archive gaf
        at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:266)
 
        at com.zimbra.soap.SoapServlet.doWork(SoapServlet.java:303)
        Id  View      Unread  Msg Count Path
        at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:213)
---------- ----  ----------  ---------- ----------
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
        1  unkn          0          0  /
        at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:209)
        16  docu          0           0 /Briefcase
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        10  appo          0          0  /Calendar
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738)
        14  mess          0          0  /Chats
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651)
        7  cont          0          0  /Contacts
        at com.zimbra.cs.servlet.RequestStringFilter.doFilter(RequestStringFilter.java:54)
        6 mess          0           0 /Drafts
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        13  cont          0          0  /Emailed Contacts
        at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:59)
      789  unkn          0           0 /Imported
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
      790  mess          4          4 /Imported/Inbox
        at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
        2  mess        415        420  /Inbox
        at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:351)
        4  mess          0           0 /Junk
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        5  mess          0          0  /Sent
        at com.zimbra.cs.servlet.ETagHeaderFilter.doFilter(ETagHeaderFilter.java:47)
        15  task          0          0  /Tasks
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        3  unkn          0           0 /Trash
        at com.zimbra.cs.servlet.ContextPathBasedThreadPoolBalancerFilter.doFilter(ContextPathBasedThreadPoolBalancerFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(ZimbraQoSFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ZimbraInvalidLoginFilter.doFilter(ZimbraInvalidLoginFilter.java:117)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:457)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:326)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:299)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:544)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:199)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:309)
        at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:81)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:462)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
        at java.lang.Thread.run(Thread.java:745)
2015-03-06 12:08:20,566 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] soap - BackupRequest elapsed=10154
</pre>
 
To confirm the backup label was made and also to see how the zmbackupquery reports it, we can do the following:
 
<pre>
[zimbra@mail-172 archive]$ zmbackupquery -lb incr-20150306.170810.793 -v
Label:  incr-20150306.170810.793
Type:    incremental
Status:  completed (with errors)
Started: Fri, 2015/03/06 12:08:10.793 EST
Ended:  Fri, 2015/03/06 12:08:19.378 EST
Redo log sequence range: 23 .. 23
Number of accounts: 14 out of 14 completed
Number of errors: 1
Accounts:
  admin@mail-172.example.com: completed
  galsync.qva4ekog0@mail-172.example.com: completed
  ham.jgfdb3oqjf@mail-172.example.com: completed
  restore2_user1@mail-172.example.com: completed
  restore3_user1@mail-172.example.com: completed
  restore_user1@mail-172.example.com: completed
  spam.umhx3owv3s@mail-172.example.com: completed
  user1@mail-172.example.com: completed
  user2@mail-172.example.com: completed
  user3@mail-172.example.com: completed
  user4@mail-172.example.com: completed
  user5@mail-172.example.com: completed
  user6@mail-172.example.com: completed
  virus-quarantine.gusiso1cd@mail-172.example.com: completed
Error: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option
com.zimbra.cs.backup.BackupServiceException: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option
ExceptionId:qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest:1425661691864:14586f805b4c1b0e
Code:backup.REDOLOG_OUT_OF_SEQUENCE
        at com.zimbra.cs.backup.BackupServiceException.REDOLOG_OUT_OF_SEQUENCE(BackupServiceException.java:91)
        at com.zimbra.cs.backup.util.Utils.splitRedoLogsAtSeq(Utils.java:302)
        at com.zimbra.cs.backup.FileBackupTarget$FileBackupSet.backupRedoLogs(FileBackupTarget.java:1360)
        at com.zimbra.cs.backup.BackupSet.startIncrementalBackup(BackupSet.java:817)
        at com.zimbra.cs.backup.FileBackupTarget$FileBackupSet.startIncrementalBackup(FileBackupTarget.java:1067)
        at com.zimbra.cs.backup.BackupManager.backupIncremental(BackupManager.java:336)
        at com.zimbra.cs.service.backup.Backup.handleNetworkRequest(Backup.java:153)
        at com.zimbra.cs.service.NetworkDocumentHandler.handle(NetworkDocumentHandler.java:23)
        at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEngine.java:569)
        at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:432)
        at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:266)
        at com.zimbra.soap.SoapServlet.doWork(SoapServlet.java:303)
        at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:213)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
        at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:209)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651)
        at com.zimbra.cs.servlet.RequestStringFilter.doFilter(RequestStringFilter.java:54)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:59)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
        at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:351)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ETagHeaderFilter.doFilter(ETagHeaderFilter.java:47)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ContextPathBasedThreadPoolBalancerFilter.doFilter(ContextPathBasedThreadPoolBalancerFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(ZimbraQoSFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ZimbraInvalidLoginFilter.doFilter(ZimbraInvalidLoginFilter.java:117)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:457)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:326)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:299)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:544)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:199)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:309)
        at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:81)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:462)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
        at java.lang.Thread.run(Thread.java:745)
Total space: 14020MB
Free space: 6031MB
[zimbra@mail-172 archive]$
</pre>
 
Let's now try doing a restore of an account.
 
<pre>
[zimbra@mail-172 archive]$ cat /dev/null > /opt/zimbra/log/mailbox.log
 
[zimbra@mail-172 archive]$ zmrestore -a user1@`zmhostname` -ca -pre restore6_
Error occurred: system failure: Restore encountered redo log sequence error: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option
 
[zimbra@mail-172 archive]$ cat /opt/zimbra/log/mailbox.log
2015-03-06 12:12:31,884 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - RestoreRequest elapsed=38
2015-03-06 12:13:06,650 INFO  [MailboxPurge] [name=user3@mail-172.example.com;mid=9;] purge - Purging messages.
2015-03-06 12:13:45,329 INFO  [qtp509886383-484:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - AuthRequest elapsed=2
2015-03-06 12:13:45,474 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is starting
2015-03-06 12:13:45,512 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Using full backup full-20150306.055845.429 as starting point for restoring account 3c397edc-c015-4d82-ae8a-1084692f8a93
2015-03-06 12:13:45,662 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - waiting for FileLogWriter.FsyncThread-1425661691796 to finish.
2015-03-06 12:13:45,669 INFO  [FileLogWriter.FsyncThread-1425661691796] [] redolog - fsync thread exiting
2015-03-06 12:13:45,669 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - FileLogWriter.FsyncThread-1425661691796 finished
2015-03-06 12:13:45,678 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - Redo log rollover took 166ms
2015-03-06 12:13:45,678 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore started for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6
2015-03-06 12:13:45,700 INFO  [FileLogWriter.FsyncThread-1425662025678] [] redolog - Starting fsync thread with interval 10
2015-03-06 12:13:45,981 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - Creating database mboxgroup17
2015-03-06 12:13:46,695 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restoring to mailbox id 17 for new account f7f4c1a8-15d0-41a1-bd11-6e05359ea3e8.
2015-03-06 12:13:46,745 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loading schema with version 7.2
2015-03-06 12:13:46,785 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Renaming volume_id column to locator in source table mail_item
2015-03-06 12:13:46,820 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 0 blobs during restore
2015-03-06 12:13:46,822 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Deleting directory /opt/zimbra/index/0/17/index
2015-03-06 12:13:46,882 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 2 index files during restore
2015-03-06 12:13:46,887 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/17/index/0
2015-03-06 12:13:46,887 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox 17 account f7f4c1a8-15d0-41a1-bd11-6e05359ea3e8 LOADED
2015-03-06 12:13:46,887 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] cache - initializing folder and tag caches for mailbox 17
2015-03-06 12:13:46,912 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Account user1@mail-172.example.com: All pending file IO completed (2 out of 2)
2015-03-06 12:13:46,912 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Run redo operations since last full backup
2015-03-06 12:13:47,047 ERROR [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option
2015-03-06 12:13:47,047 WARN  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Redo log sequence error: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option
com.zimbra.cs.backup.BackupServiceException: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option
ExceptionId:qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest:1425662027047:14586f805b4c1b0e
Code:backup.REDOLOG_OUT_OF_SEQUENCE
        at com.zimbra.cs.backup.BackupServiceException.REDOLOG_OUT_OF_SEQUENCE(BackupServiceException.java:91)
        at com.zimbra.cs.backup.util.Utils.splitRedoLogsAtSeq(Utils.java:302)
        at com.zimbra.cs.backup.RestoreAccountSession.checkRedoLogSequenceContiguity(RestoreAccountSession.java:534)
        at com.zimbra.cs.backup.RestoreAccountSession.endRestore(RestoreAccountSession.java:350)
        at com.zimbra.cs.backup.FileBackupTarget$RestoreAcctSession.endRestore(FileBackupTarget.java:2524)
        at com.zimbra.cs.backup.FileBackupTarget.restore(FileBackupTarget.java:490)
        at com.zimbra.cs.backup.BackupManager.restore(BackupManager.java:826)
        at com.zimbra.cs.service.backup.Restore.handle(Restore.java:137)
        at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEngine.java:569)
        at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:432)
        at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:266)
        at com.zimbra.soap.SoapServlet.doWork(SoapServlet.java:303)
        at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:213)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
        at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:209)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651)
        at com.zimbra.cs.servlet.RequestStringFilter.doFilter(RequestStringFilter.java:54)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:59)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
        at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:351)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ETagHeaderFilter.doFilter(ETagHeaderFilter.java:47)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ContextPathBasedThreadPoolBalancerFilter.doFilter(ContextPathBasedThreadPoolBalancerFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(ZimbraQoSFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ZimbraInvalidLoginFilter.doFilter(ZimbraInvalidLoginFilter.java:117)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:457)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:326)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:299)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:544)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:199)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:309)
        at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:81)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:462)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
        at java.lang.Thread.run(Thread.java:745)
2015-03-06 12:13:47,047 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Cleaning up data from failed restore (mid=17)
2015-03-06 12:13:47,063 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - Putting mailbox 17 under maintenance.
2015-03-06 12:13:47,084 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - clearing contents of mailbox 17, group 17
2015-03-06 12:13:47,331 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox for account f7f4c1a8-15d0-41a1-bd11-6e05359ea3e8 DELETED
2015-03-06 12:13:47,375 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Cleaning up account created during failed restore (account=f7f4c1a8-15d0-41a1-bd11-6e05359ea3e8)
2015-03-06 12:13:47,752 WARN  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Error occurred during restore account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93)
com.zimbra.common.service.ServiceException: system failure: Restore encountered redo log sequence error: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option
ExceptionId:qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest:1425662027047:14586f805b4c1b0e
Code:service.FAILURE
        at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:260)
        at com.zimbra.cs.backup.RestoreAccountSession.checkRedoLogSequenceContiguity(RestoreAccountSession.java:545)
        at com.zimbra.cs.backup.RestoreAccountSession.endRestore(RestoreAccountSession.java:350)
        at com.zimbra.cs.backup.FileBackupTarget$RestoreAcctSession.endRestore(FileBackupTarget.java:2524)
        at com.zimbra.cs.backup.FileBackupTarget.restore(FileBackupTarget.java:490)
        at com.zimbra.cs.backup.BackupManager.restore(BackupManager.java:826)
        at com.zimbra.cs.service.backup.Restore.handle(Restore.java:137)
        at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEngine.java:569)
        at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:432)
        at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:266)
        at com.zimbra.soap.SoapServlet.doWork(SoapServlet.java:303)
        at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:213)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
        at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:209)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651)
        at com.zimbra.cs.servlet.RequestStringFilter.doFilter(RequestStringFilter.java:54)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:59)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
        at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:351)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ETagHeaderFilter.doFilter(ETagHeaderFilter.java:47)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ContextPathBasedThreadPoolBalancerFilter.doFilter(ContextPathBasedThreadPoolBalancerFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(ZimbraQoSFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ZimbraInvalidLoginFilter.doFilter(ZimbraInvalidLoginFilter.java:117)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:457)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:326)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:299)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:544)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:199)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:309)
        at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:81)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:462)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
        at java.lang.Thread.run(Thread.java:745)
Caused by: com.zimbra.cs.backup.BackupServiceException: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option
ExceptionId:qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest:1425662027047:14586f805b4c1b0e
Code:backup.REDOLOG_OUT_OF_SEQUENCE
        at com.zimbra.cs.backup.BackupServiceException.REDOLOG_OUT_OF_SEQUENCE(BackupServiceException.java:91)
        at com.zimbra.cs.backup.util.Utils.splitRedoLogsAtSeq(Utils.java:302)
        at com.zimbra.cs.backup.RestoreAccountSession.checkRedoLogSequenceContiguity(RestoreAccountSession.java:534)
        ... 56 more
2015-03-06 12:13:47,755 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is shut down
2015-03-06 12:13:47,755 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] SoapEngine - handler exception
com.zimbra.common.service.ServiceException: system failure: Restore encountered redo log sequence error: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option
ExceptionId:qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest:1425662027047:14586f805b4c1b0e
Code:service.FAILURE
        at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:260)
        at com.zimbra.cs.backup.RestoreAccountSession.checkRedoLogSequenceContiguity(RestoreAccountSession.java:545)
        at com.zimbra.cs.backup.RestoreAccountSession.endRestore(RestoreAccountSession.java:350)
        at com.zimbra.cs.backup.FileBackupTarget$RestoreAcctSession.endRestore(FileBackupTarget.java:2524)
        at com.zimbra.cs.backup.FileBackupTarget.restore(FileBackupTarget.java:490)
        at com.zimbra.cs.backup.BackupManager.restore(BackupManager.java:826)
        at com.zimbra.cs.service.backup.Restore.handle(Restore.java:137)
        at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEngine.java:569)
        at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:432)
        at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:266)
        at com.zimbra.soap.SoapServlet.doWork(SoapServlet.java:303)
        at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:213)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
        at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:209)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651)
        at com.zimbra.cs.servlet.RequestStringFilter.doFilter(RequestStringFilter.java:54)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:59)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
        at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:351)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ETagHeaderFilter.doFilter(ETagHeaderFilter.java:47)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ContextPathBasedThreadPoolBalancerFilter.doFilter(ContextPathBasedThreadPoolBalancerFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(ZimbraQoSFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ZimbraInvalidLoginFilter.doFilter(ZimbraInvalidLoginFilter.java:117)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:457)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:326)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:299)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:544)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:199)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:309)
        at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:81)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:462)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
        at java.lang.Thread.run(Thread.java:745)
Caused by: com.zimbra.cs.backup.BackupServiceException: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option
ExceptionId:qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest:1425662027047:14586f805b4c1b0e
Code:backup.REDOLOG_OUT_OF_SEQUENCE
        at com.zimbra.cs.backup.BackupServiceException.REDOLOG_OUT_OF_SEQUENCE(BackupServiceException.java:91)
        at com.zimbra.cs.backup.util.Utils.splitRedoLogsAtSeq(Utils.java:302)
        at com.zimbra.cs.backup.RestoreAccountSession.checkRedoLogSequenceContiguity(RestoreAccountSession.java:534)
        ... 56 more
2015-03-06 12:13:47,755 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - RestoreRequest elapsed=2358
 
[zimbra@mail-172 archive]$ zmprov ga restore6_user1@`zmhostname` | head
ERROR: account.NO_SUCH_ACCOUNT (no such account: restore6_user1@mail-172.example.com)
</pre>
</pre>


Let's now see what happens if we move the missing redolog seq files back into /opt/zimbra/redolog/archive and then if we move/copy them into the backup sessions redolog directory. With restore attempts being done after each move/copy.
Here's a (customer-supplied) script that does the above in batch. It was used to restore archives from a date range for 29k users: https://github.com/morganlljones/zimbra_backup2archive
 
<pre>
[zimbra@mail-172 archive]$ mv /tmp/redolog-archive/* ./
 
[zimbra@mail-172 archive]$ pwd
/opt/zimbra/redolog/archive
 
[zimbra@mail-172 archive]$ ls
redo-20150306.060015.331-seq21.log  redo-20150306.060109.166-seq22.log  redo-20150306.170811.794-seq23.log  redo-20150306.171345.676-seq24.log
 
[zimbra@mail-172 archive]$ zmrestore -a user1@`zmhostname` -ca -pre restore6_
Error occurred: system failure: Restore encountered redo log sequence error: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option
 
[zimbra@mail-172 archive]$ pwd
/opt/zimbra/redolog/archive
 
[zimbra@mail-172 archive]$ cp *seq21.log *seq22.log /opt/zimbra/backup/sessions/incr-20150306.170810.793/redologs/
 
[zimbra@mail-172 archive]$ zmrestore -a user1@`zmhostname` -ca -pre restore6_
 
[zimbra@mail-172 archive]$ zmprov ga restore6_user1@`zmhostname` | head -2
# name restore6_user1@mail-172.example.com
cn: user1
</pre>


== Quiz ==
== Quiz ==
Line 2,592: Line 2,371:
{{Article_Footer|Zimbra Collaboration Suite 8.6|01/22/2015}}
{{Article_Footer|Zimbra Collaboration Suite 8.6|01/22/2015}}
[[Category:Troubleshooting_Course_Content_Rough_Drafts]]
[[Category:Troubleshooting_Course_Content_Rough_Drafts]]
[[Category: Author:Ajcody]]

Latest revision as of 20:34, 7 June 2016


Bug & RFE's I Made While Researching This Draft Write Up [Remove For Course]

Introduction

  • Will you please provide a brief paragraph introduction/overview about recovering missing data in Zimbra.
  • For example: what are some scenarios wherein you need to recover missing data.
  • What is the general method for recovery?
  • How/Why is the backup environment relevant/important.
  • Remember that this won't be obvious for all audience members.

User Issues

Questions To Ask & Address Prior To Doing A User Restore

  • Can data be restored from the user's dumpster [if enabled] ?
  • Is there a copy of the data in a third party client ? [POP email client that downloaded it, for example]
  • Does the data maybe exist in another user's account ? [For example, the senders account or others in the To/CC]
  • Does the data reside in a redolog/s ?
  • Finally, restore from backups ?
    • Do I need to overwrite the existing account with the restore or should I restore to a new account name?
      • Using the option -ca [create account] and -pre restore_ [prefix to the new account name, restore_user@ ]
    • What are your retention policies and how it might determine the backup set you need?

User Restores

  • Admin Console
    • Restore/Backup
    • Redirected restore : -ca -pre restored_
    • View Mail > TGZ export/import
  • Crossmailboxsearch
  • Importing Restored Data Into Parent Account

Restoring A User To A Specific Full Backup, Incremental Backup, Or Point In Time

One of the more common support issues around restoring a user account is the complaint that the restore didn't roll back the account to the desired state. Almost always, this is because the proper options were not passed to the zmrestore command for the restore to stop at the desired time frame. If you don't give the proper options to zmrestore, it will restore the account all the way to the most recent redolog operations on the server. For example, it replays the delete operations that caused the user to request to restore in the first place.

You MUST use the -lb [full backup label name] option when your trying to restore anything that ISN'T meant to include the latest information for the mailbox. The -lb argument should specify a full backup that took place prior to the time of the backup you wish to restore.

Find Out What Backup Session Labels You Need First

To find out what backups are associated with a particular account, you would do the following :

zmbackupquery -a user@domain

You'll want to note what is the first full that occurs before the point in time you want to restore. And then the incremental that follows right after your point in time.

Backup labels (-lb) for fulls for a particular user can be found by doing :

zmbackupquery -a user@domain --type full

Backup labels (-restoreToIncrLabel) for incrementals can be found by doing :

zmbackupquery -a user@domain --type incremental
Command Syntax Example For Restores On The CLI

Example to restore ONLY the data in the full backup session and stop :

zmrestore -a USER@DOMAIN.com -lb full-20080726.050017.306 -rf -ca -pre restore_

Example to restore only the data in the full backup session and all the incremental data after the full to a specific incremental label and stop :

zmrestore -a USER@DOMAIN.com -restoreToIncrLabel incr-20080731.060007.644 -lb full-20080726.050017.306 -br -ca -pre restore_

Example to restore to a specific time and stop :

zmrestore -a USER@DOMAIN.com -restoreToTime 20080801011800 -lb full-20080726.050017.306 -br -ca -pre restore_


Important Options You Might Want Or Need To Include

--ignoreRedoErrors : If you attempt a restore and you see an error about problems related to playing the redolog, you'll want to run the restore command again and include this option.


--skipDeletes : Please see http://bugzilla.zimbra.com/show_bug.cgi?id=31824#c5 for details on this.


-t /path/to/backup_dir : If you are restoring from another backup directory besides your current default path.


Variables that are asking for TIME rather than LABELS should follow this syntax (from zmrestore --help):

Specify date/time in one of these formats:

    2008/08/06 09:55:50
    2008/08/06 09:55:50 572
    2008/08/06 09:55:50.572
    2008/08/06-09:55:50-572
    2008/08/06-09:55:50
    20080806.095550.572
    20080806.095550
    20080806095550572
    20080806095550

Specify year, month, date, hour, minute, second, and optionally millisecond.
Month/date/hour/minute/second are 0-padded to 2 digits, millisecond to 3 digits.
Hour must be specified in 24-hour format, and time is in local time zone.

Trouble-shooting Backup/Restore Issues And Other General Questions

Backup And Restore Compatibility Between ZCS Versions

  • You can use backups from older versions of ZCS for account restores.
  • You can NOT use backups from old versions of ZCS for system restores though. Disaster Recovery restores have to be done with the same version of ZCS as the backups were done with.

References:

Trouble-shooting

Ref:

Basic Backup Information To Submit To Support

Disk Space Usage Issues
Trend Data

If there is concerns about disks/partitions getting full, this command would be helpful for trending data on your server. Send support the resulting df.tar file . Note - adjust the tail command if you want more than 20 day's worth of trending data, the -n 20 option.

[zimbra@zcs806 tmp]$ /tmp

[zimbra@zcs806 tmp]$ tar cvf /tmp/df.tar `find /opt/zimbra/zmstat -name df.cs\* | sort | tail -n 20`
tar: Removing leading `/' from member names
/opt/zimbra/zmstat/2014-03-10/df.csv.gz
/opt/zimbra/zmstat/2014-03-11/df.csv.gz
 [cut - Ajcody]
/opt/zimbra/zmstat/2014-03-27/df.csv.gz
/opt/zimbra/zmstat/2014-03-28/df.csv.gz
/opt/zimbra/zmstat/df.csv

[zimbra@zcs806 tmp]$ ls -lah /tmp/df.tar
-rw-r----- 1 zimbra zimbra 80K Mar 29 06:44 /tmp/df.tar

[zimbra@zcs806 tmp]$ tar tvf /tmp/df.tar
-rw-r----- zimbra/zimbra  2566 2014-03-11 00:00 opt/zimbra/zmstat/2014-03-10/df.csv.gz
-rw-r----- zimbra/zimbra  2553 2014-03-12 00:00 opt/zimbra/zmstat/2014-03-11/df.csv.gz
 [cut - Ajcody]
-rw-r----- zimbra/zimbra  2513 2014-03-28 00:00 opt/zimbra/zmstat/2014-03-27/df.csv.gz
-rw-r----- zimbra/zimbra  2531 2014-03-29 00:00 opt/zimbra/zmstat/2014-03-28/df.csv.gz
-rw-r----- zimbra/zimbra  8013 2014-03-29 06:40 opt/zimbra/zmstat/df.csv
Directory Sizes In /opt/zimbra

Please see the following and provide the output to support. Note, even though this method is faster than doing a du it still can take awhile.

* Ajcody-Server-Misc-Topics#Faster_Way_To_Get_Directory_Size_On_Filesytem_-_find_vs_du
Adjusting The Disk Alert Threshold

Note - zmlocalconfig smtp_notify must return yes if you want to receive the notifications.

If you just need to adjust the disk alert threshold, then see the following:

See current values:

 zmlocalconfig | grep zmdisklog

Example adjustment:

 su - zimbra
 zmlocalconfig -e zmdisklog_critical_threshold=98
 zmlocalconfig -e zmdisklog_warn_threshold=95
 zmstatctl

To exclude a partition from the checks [example of two being excluded]:

 su - zimbra
 zmlocalconfig -e zmstat_df_excludes="/mount/point:/mount/point2"
 zmstatctl

Note, depending on your version of ZCS, you might be hitting a bug where you'll keep getting emails until a logrotate happens.

Some things to do to confirm and share with support or in bug. As zimbra

su - zimbra

ls -la /var/log/zimbra.log

df -h
 /dev/mapper/vg_rhel664-lv_root
                      5.5G  3.5G  1.7G  68% /
 tmpfs                 939M     0  939M   0% /dev/shm
 /dev/sda1             485M   79M  381M  18% /boot
 /dev/sdb1              30G  6.2G   23G  22% /opt

date

zmlocalconfig | grep zmdisklog
 zmdisklog_critical_threshold = 80
 zmdisklog_warn_threshold = 85  

zmlocalconfig -e zmdisklog_critical_threshold=95

zmlocalconfig -e zmdisklog_warn_threshold=90

zmlocalconfig | grep zmdisklog
 zmdisklog_critical_threshold = 95
 zmdisklog_warn_threshold = 90

zmstatctl restart

date

ps -eaf | grep zmstat-df

ls -la /var/log/zimbra.log

date ; grep "Disk warning" /var/log/zimbra* ; zmmailbox -z -m admin@`zmhostname` s -l 100 -t message "Subject: Disk and after:yesterday"

##Note - Emails by default go out every 10 minutes - for example:

[zimbra@zcs803 ~]$ date ; grep "Disk warning" /var/log/zimbra* ; zmmailbox -z -m admin@`zmhostname` s -l 100 -t message "Subject: Disk and after:yesterday"
Thu May 22 09:40:08 PDT 2014
/var/log/zimbra.log:May 22 08:30:00 zcs803 zimbramon[18826]: 18826:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82%
/var/log/zimbra.log:May 22 08:40:00 zcs803 zimbramon[22970]: 22970:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82%
/var/log/zimbra.log:May 22 08:50:00 zcs803 zimbramon[22970]: 22970:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82%
/var/log/zimbra.log:May 22 09:00:00 zcs803 zimbramon[22970]: 22970:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82%
  ## Note - I had readjusted the variable to not warn during this time segment ##
/var/log/zimbra.log:May 22 09:20:00 zcs803 zimbramon[8322]: 8322:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82%
/var/log/zimbra.log:May 22 09:30:00 zcs803 zimbramon[8322]: 8322:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82%
/var/log/zimbra.log:May 22 09:40:00 zcs803 zimbramon[8322]: 8322:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82%
num: 7, more: false

     Id  Type   From                  Subject                                             Date
   ----  ----   --------------------  --------------------------------------------------  --------------
1.  328  mess   admin                 Disk / at 82% on zcs803.DOMAIN.com:           05/22/14 09:40
2.  327  mess   admin                 Disk / at 82% on zcs803.DOMAIN.com:           05/22/14 09:30
3.  326  mess   admin                 Disk / at 82% on zcs803.DOMAIN.com:           05/22/14 09:20
  ## Note - I had readjusted the variable to not warn during this time segment ##
4.  325  mess   admin                 Disk / at 82% on zcs803.DOMAIN.com:           05/22/14 09:00
5.  324  mess   admin                 Disk / at 82% on zcs803.DOMAIN.com:           05/22/14 08:50
6.  323  mess   admin                 Disk / at 82% on zcs803.DOMAIN.com:           05/22/14 08:40
7.  320  mess   admin                 Disk / at 82% on zcs803.DOMAIN.com:           05/22/14 08:31

Continue to monitor your zmmailbox search results for an hour.

The Basic Information Support Needs

as root:

  • cat /etc/fstab
    • Shows us what is mounted upon boot
  • cat /proc/mounts
    • Shows us what is currently mounted and its status - you can see if a mount is read-only here.
  • df -hT
    • Lists current mounts using human-readable size information and also notes the filesystem type.

as zimbra:

  • zmprov -l gs `zmhostname` | egrep 'Back|Redo'
    • Will show us a number of variables related to backup and redologs. Also tell us if your using auto-group or the default method.
  • du -sh /opt/zimbra/redolog
    • Will might notice your redolog logs aren't rolling over, causing a possible issue.
  • ls -latr /opt/zimbra/backup
    • This is the default backup target, please adjust this path here and below if you are using a different zimbraBackupTarget value.
    • zmprov gs `zmhostname` zimbraBackupTarget
    • We'll be able to confirm permissions are right.
  • ls -latr /opt/zimbra/backup/tmp
    • This will show us if you have failed backup jobs and confirm tmp is being cleaned appropriately after the backup is done.
  • ls -latr /opt/zimbra/backup/sessions
    • This will show us what backup sessions are available and confirm permissions are correct.
    • Adjust path if your zimbraBackupTarget value is not the default path.
  • Some directory sizes in the backup directory:
    • Default path first
      • du -sh `find /opt/zimbra/backup -maxdepth 2 -type d`
    • If your using a different backup target, check that directory also. Replace /opt/zimbra/backup above with your backup path.
  • zmbackupquery
    • This should match what's in the sessions directory and it will also tell us if status of each backup and how many accts were done.
  • crontab -l | grep -i back
    • This will show use when backups are support to run and with what options they are running with.
  • zmlocalconfig | grep -i back
    • This is useful to see a number of backup options not exposed in the crontab, things related to the zip options.
  • zmvolume -l
    • This is useful to see how many volumes are being used, if HSM is being used, and if compression is being done at the volume level.
Additional Log Files Support Might Need

And send the following logs:

  • /var/log/messages
    • Filesystem issues often times are noted here and also in syslog. This might explain an interruption in the backup process. Server restarts, filesystem going full, filesystem going read-only, etc.
  • /var/log/syslog
  • /opt/zimbra/log/mailbox.log
    • The backup activity is logged here.
    • And any other mailbox.log file that would cover the event

Some Specific Example Restore/Backup Issues

NO_SUCH_ACCOUNT_BACKUP

Example errors:

Error code: backup.NO_SUCH_ACCOUNT_BACKUP 
Message: no such backup for account: Missing full backup earlier than restore-to time for account 3c397edc-c015-4d82-ae8a-1084692f8a93 
Details:soap:Sender 

Error occurred: no such backup for account: 3c397edc-c015-4d82-ae8a-1084692f8a93

Message: no such backup for account: Missing full backup earlier than restore-to time for account 3c397edc-c015-4d82-ae8a-1084692f8a93 

3c397edc-c015-4d82-ae8a-1084692f8a93 is an example of the zimbraId of a user [ user1@mail-172.example.com ] .

Confirm the current email address is mapped to the appropriate zimbraId. You'll want to search what is in ldap and the mailstores database for the exiting email address and then also search the reference files in your backups. Often times, this happens when an account was deleted and then created again. Account meaning the email address. When this is does, the new account will have a new zimbraId.

$ zmprov -l ga user1@mail-172.example.com zimbraId
# name user1@mail-172.example.com
zimbraId: 3c397edc-c015-4d82-ae8a-1084692f8a93

$ mysql -e 'SELECT * FROM zimbra.mailbox WHERE comment LIKE "user1@mail-172.example.com"\G'

$ mysql -e 'SELECT * FROM zimbra.mailbox WHERE account_id="3c397edc-c015-4d82-ae8a-1084692f8a93"\G'

$ grep -C1 3c397edc-c015-4d82-ae8a-1084692f8a93 /opt/zimbra/backup/accounts.xml

   [using the default path; /opt/zimbra/backup/accounts.xml]

$ egrep "user1@mail-172.example.com|3c397edc-c015-4d82-ae8a-1084692f8a93" /opt/zimbra/backup/accounts.xml

$ egrep "user1@mail-172.example.com|3c397edc-c015-4d82-ae8a-1084692f8a93" /opt/zimbra/backup/sessions/*/session.xml
exception during auth

Example error:

LDAP backup failed: system failure: exception during auth 
 {RemoteManager: zimbramail.example.com->zimbra@zimbramail.example.com:22}
 com.zimbra.common.service.ServiceException: system failure: LDAP backup failed: 
 system failure: exception during auth {RemoteManager: zimbramail.example.com->
 zimbra@zimbramail.example.com:22}

This indicates the ssh keys aren't configured correctly in the environment. You should be able to do the following to resolve the issue:

su - zimbra
zmsshkeygen 
zmupdateauthkeys

Note - You should also confirm that you do indeed have the ssh port available. The default is port 22. If your unable to ssh into the server using port 22, then this might be the issue. If you configured ssh to listen on a different port on this server, you'll need to adjust the following variable - zmprov gcf zimbraRemoteManagementPort . You might also have the ssh port blocked - confirm your firewall, iptables, etc is not blocking the ssh port.

LICENSE_ERROR_Message: AccountsLimits exceeded

Example error:

LICENSE_ERROR_Message: AccountsLimits exceeded

You should check your license and confirm it's valid and active:

su - zimbra
zmlicense -p

And confirm how many licenses are used: First, flush the cache on all servers for the license information. Then do the query via soap.

zmprov fc -a license
zmsoap -z GetLicenseRequest

Four lines that are generally of interest are shown below. The first two state how many one is licensed for, the bottom two show how many are currently used :

 <attr name="AccountsLimit">1000</attr>
 <attr name="ArchivingAccountsLimit">1000</attr>

 <attr name="TotalAccounts">3</attr>
 <attr name="ArchivingAccounts">3</attr>

Hands On How-To And Reviewing Of Log Events

Full Backup And Log Events Generated

Example of manually starting a full backup using the Standard zimbraBackupMode and doing a query to confirm it's completion and status.

[zimbra@mail-172 archive]$ zmprov gs `zmhostname` zimbraBackupMode
# name mail-172.example.com
zimbraBackupMode: Standard

[zimbra@mail-172 archive]$ pwd ; ls *.log
/opt/zimbra/redolog/archive
redo-20150303.060015.816-seq9.log

[zimbra@mail-172 archive]$ zmbackup -f -a all
full-20150304.012000.193

[zimbra@mail-172 archive]$ zmbackupquery -lb full-20150304.012000.193 -v
Label:   full-20150304.012000.193
Type:    full
Status:  completed
Started: Tue, 2015/03/03 20:20:00.193 EST
Ended:   Tue, 2015/03/03 20:20:28.630 EST
Redo log sequence range: 10 .. 10
Number of accounts: 10 out of 10 completed
Accounts:
  admin@mail-172.example.com: completed
  galsync.qva4ekog0@mail-172.example.com: completed
  ham.jgfdb3oqjf@mail-172.example.com: completed
  spam.umhx3owv3s@mail-172.example.com: completed
  user1@mail-172.example.com: completed
  user2@mail-172.example.com: completed
  user3@mail-172.example.com: completed
  user4@mail-172.example.com: completed
  user5@mail-172.example.com: completed
  virus-quarantine.gusiso1cd@mail-172.example.com: completed

Total space: 14020MB
 Free space: 6017MB

Example of what log events are generated for a full backup in /opt/zimbra/log/mailbox.log . First, you'll see the authentication/acl request and then statements about the backup request.

2015-03-03 20:19:58,499 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [] AuthProvider - Adding auth provider: zimbra com.zimbra.cs.service.ZimbraAuthProvider
2015-03-03 20:19:59,458 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] soap - AuthRequest elapsed=834
2015-03-03 20:19:59,751 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] account - Initialized access manager: com.zimbra.cs.account.accesscontrol.ACLAccessManager
2015-03-03 20:19:59,770 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup request started
2015-03-03 20:20:00,193 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Found 10 accounts on server mail-172.example.com
2015-03-03 20:20:00,464 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup request finished
2015-03-03 20:20:00,464 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] soap - BackupRequest elapsed=760

Your log events will now reference Backup in them. AsyncFileCopier will start and it will give the backup session label for this backup job.

2015-03-03 20:20:01,092 INFO  [FullBackupThread] [] io - AsyncFileCopier is starting
2015-03-03 20:20:01,187 INFO  [FullBackupThread] [] backup - Full backup started for backup set; label: full-20150304.012000.193

It proceeds to backing up your accounts. I'll list the backup log events for just two accounts on this system - user1@mail-172.example.com and admin@mail-172.example.com . I'll provide the zimbraId, email address, and mailboxID information for user1@mail-172.example.com below so you can confirm it's reported in the logs correctly as well.

[zimbra@mail-172 redolog]$ zmprov ga user1@mail-172.example.com zimbraId
# name user1@mail-172.example.com
zimbraId: 3c397edc-c015-4d82-ae8a-1084692f8a93

[zimbra@mail-172 redolog]$ zmprov gmi user1@mail-172.example.com
mailboxId: 6
quotaUsed: 1209

Now, continuing with the log events. Notice the reference to the redo log file sequence, putting the account into and out of maintenance, number of blobs and index files backed up, and the statement it's finished backing up the account. Also within the section we can see the user's zimbraId [3c397edc-c015-4d82-ae8a-1084692f8a93] , email address [user1@mail-172.example.com] , and mailboxId [Mailbox 6].

[cut]
2015-03-03 20:20:06,560 INFO  [FullBackupThread] [name=user1@mail-172.example.com;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/6/index/0
2015-03-03 20:20:06,560 INFO  [FullBackupThread] [name=user1@mail-172.example.com;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 LOADED
2015-03-03 20:20:06,560 INFO  [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - redo log file sequence is 10 at full backup for user1@mail-172.example.com
2015-03-03 20:20:06,565 INFO  [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - Full backup started for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) mailbox 6
2015-03-03 20:20:06,565 INFO  [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] mailbox - Putting mailbox 6 under maintenance.
2015-03-03 20:20:06,818 INFO  [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - Number of blobs to backup for mailbox 6: 1
2015-03-03 20:20:06,818 INFO  [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] mailbox - Ending maintenance on mailbox 6.
2015-03-03 20:20:06,818 INFO  [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 AVAILABLE
2015-03-03 20:20:06,966 INFO  [BackupSetWorkerThread-BLOB] [name=user1@mail-172.example.com;mid=6;] backup - Stored 1 blob files from volume message1
2015-03-03 20:20:06,966 INFO  [BackupSetWorkerThread-BLOB] [name=user1@mail-172.example.com;mid=6;] backup - Stored 1 blob files
2015-03-03 20:20:07,014 INFO  [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - Account user1@mail-172.example.com in backup set full-20150304.012000.193: All pending file IO completed (1 out of 1)
2015-03-03 20:20:07,016 INFO  [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] mailbox - Putting mailbox 6 under maintenance.
2015-03-03 20:20:07,021 INFO  [BackupSetWorkerThread-INDEX] [name=user1@mail-172.example.com;mid=6;] backup - Stored 2 index files
2015-03-03 20:20:07,496 INFO  [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - Account user1@mail-172.example.com in backup set full-20150304.012000.193: All pending file IO completed (3 out of 3)
2015-03-03 20:20:07,497 INFO  [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] mailbox - Ending maintenance on mailbox 6.
2015-03-03 20:20:07,497 INFO  [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 AVAILABLE
2015-03-03 20:20:07,497 INFO  [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - Account user1@mail-172.example.com in backup set full-20150304.012000.193: All pending file IO completed (3 out of 3)
2015-03-03 20:20:07,497 INFO  [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - Full backup finished for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) mailbox 6

Events for the admin user's backup:

2015-03-03 20:20:09,894 INFO  [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - redo log file sequence is 10 at full backup for admin@mail-172.example.com
2015-03-03 20:20:09,914 INFO  [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - Full backup started for account admin@mail-172.example.com (7ac8de92-13fe-4d19-b9da-6872169fb5c9) mailbox 1
2015-03-03 20:20:09,915 INFO  [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] mailbox - Putting mailbox 1 under maintenance.
2015-03-03 20:20:10,176 INFO  [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - Number of blobs to backup for mailbox 1: 73
2015-03-03 20:20:10,177 INFO  [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] mailbox - Ending maintenance on mailbox 1.
2015-03-03 20:20:10,177 INFO  [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] mbxmgr - Mailbox 1 account 7ac8de92-13fe-4d19-b9da-6872169fb5c9 AVAILABLE
2015-03-03 20:20:10,343 INFO  [BackupSetWorkerThread-BLOB] [name=admin@mail-172.example.com;mid=1;] backup - Stored 73 blob files from volume message1
2015-03-03 20:20:10,353 INFO  [BackupSetWorkerThread-BLOB] [name=admin@mail-172.example.com;mid=1;] backup - Stored 73 blob files
2015-03-03 20:20:11,047 INFO  [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - Account admin@mail-172.example.com in backup set full-20150304.012000.193: All pending file IO completed (73 out of 73)
2015-03-03 20:20:11,051 INFO  [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] mailbox - Putting mailbox 1 under maintenance.
2015-03-03 20:20:11,108 INFO  [BackupSetWorkerThread-INDEX] [name=admin@mail-172.example.com;mid=1;] backup - Stored 26 index files
2015-03-03 20:20:12,450 INFO  [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - Account admin@mail-172.example.com in backup set full-20150304.012000.193: All pending file IO completed (99 out of 99)
2015-03-03 20:20:12,463 INFO  [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] mailbox - Ending maintenance on mailbox 1.
2015-03-03 20:20:12,463 INFO  [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] mbxmgr - Mailbox 1 account 7ac8de92-13fe-4d19-b9da-6872169fb5c9 AVAILABLE
2015-03-03 20:20:12,463 INFO  [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - Account admin@mail-172.example.com in backup set full-20150304.012000.193: All pending file IO completed (99 out of 99)
2015-03-03 20:20:12,463 INFO  [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - Full backup finished for account admin@mail-172.example.com (7ac8de92-13fe-4d19-b9da-6872169fb5c9) mailbox 1

And then a final line confirming completion and the number of accounts successfully backed up.

2015-03-03 20:20:12,463 INFO  [FullBackupThread] [] backup - Finished backing up 10 of 10 accounts

One thing to point out is the log events were mentioning the redo log sequence as being number 10 but in the redolog/archive directory there is only a sequence 9 redo log. This is because the "seq 10" is actually the /opt/zimbra/redolog/redo.log file. When the redo.log file rolls over and is moved to the archive directory it will have seq10 in it's filename.

[zimbra@mail-172 archive]$ pwd ; ls -la *.log
/opt/zimbra/redolog/archive
-rw-r----- 1 zimbra zimbra 146723 Mar  3 20:23 redo-20150303.060015.816-seq9.log

The contents of the backup session directory. I removed the sub-directories under accounts that weren't either the admin or user1 account. Notice the zimbraId is used for the sub-directory naming scheme.

[zimbra@mail-172 full-20150304.012000.193]$ ls -R
.:
accounts  ldap  session.xml  shared_blobs  sys

./accounts:
3c3  423  434  75d  7ac  89f  a46  af0  c53  eab

./accounts/3c3:
97e

./accounts/3c3/97e:
3c397edc-c015-4d82-ae8a-1084692f8a93

./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93:
blobs  db  index  ldap.xml  meta.xml

./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93/blobs:
blobs-1.zip

./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93/db:
db_schema.xml  mail_item.dat  mailbox.dat  mailbox_metadata.dat  open_conversation.dat  tag.dat  tagged_item.dat

./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93/index:
0

./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93/index/0:
segments.gen  segments_1

[cut]

./accounts/7ac:
8de

./accounts/7ac/8de:
7ac8de92-13fe-4d19-b9da-6872169fb5c9

./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9:
blobs  db  index  ldap.xml  meta.xml

./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9/blobs:
blobs-1.zip  blobs-2.zip  blobs-3.zip  blobs-4.zip

./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9/db:
db_schema.xml  mail_item.dat  mailbox.dat  open_conversation.dat  tag.dat  tagged_item.dat

./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9/index:
0

./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9/index/0:
_0.fdt  _0.fnm  _0.nrm  _0.tii  _1.fdt  _1.fnm  _1.nrm  _1.tii  _2.fdt  _2.fnm  _2.nrm  _2.tii  segments.gen
_0.fdx  _0.frq  _0.prx  _0.tis  _1.fdx  _1.frq  _1.prx  _1.tis  _2.fdx  _2.frq  _2.prx  _2.tis  segments_3

[cut]

./ldap:
ldap-config.bak.gz  ldap.bak.gz

./shared_blobs:

./sys:
config.dat  current_volumes.dat  db_schema.xml  localconfig.xml  ssh  ssl  volume.dat

./sys/ssh:
authorized_keys  zimbra_identity  zimbra_identity.pub

./sys/ssl:
zimbra

./sys/ssl/zimbra:
ca  commercial  jetty.pkcs12  server

./sys/ssl/zimbra/ca:
ca.key  ca.pem  ca.srl  ca.srl.old  index.txt  index.txt.attr  index.txt.old  newcerts  zmssl.cnf

./sys/ssl/zimbra/ca/newcerts:
1424523586.pem  1424523591.pem  1424523597.pem  1424523603.pem

./sys/ssl/zimbra/commercial:

./sys/ssl/zimbra/server:
server.crt  server.csr  server.key

Incremental Backup And Log Events Generated

Example of manually starting an incremental backup using the Standard zimbraBackupMode and doing a query to confirm it's completion and status.

[zimbra@mail-172 archive]$ zmprov gs `zmhostname` zimbraBackupMode
# name mail-172.example.com
zimbraBackupMode: Standard

[zimbra@mail-172 archive]$ ls
redo-20150303.060015.816-seq9.log  redo-20150304.070025.476-seq10.log

[zimbra@mail-172 archive]$ zmbackup -i
incr-20150304.073543.151

[zimbra@mail-172 archive]$ zmbackupquery -lb incr-20150304.073543.151 -v
Label:   incr-20150304.073543.151
Type:    incremental
Status:  completed
Started: Wed, 2015/03/04 02:35:43.151 EST
Ended:   Wed, 2015/03/04 02:35:51.981 EST
Redo log sequence range: 10 .. 11
Number of accounts: 10 out of 10 completed
Accounts:
  admin@mail-172.example.com: completed
  galsync.qva4ekog0@mail-172.example.com: completed
  ham.jgfdb3oqjf@mail-172.example.com: completed
  spam.umhx3owv3s@mail-172.example.com: completed
  user1@mail-172.example.com: completed
  user2@mail-172.example.com: completed
  user3@mail-172.example.com: completed
  user4@mail-172.example.com: completed
  user5@mail-172.example.com: completed
  virus-quarantine.gusiso1cd@mail-172.example.com: completed

Total space: 14020MB
 Free space: 6032MB

Notice how the log events are soap requests and differ from the full backup above and lack the INFO [*Backup*] formatting. Important to note this if your attempting to grep for backup/restore activity. Just like with the full, it starts with an authentication/acl request and states it's starting, how many accounts found, AsyncFileCopier starting, and the label name for this backup session.

From /opt/zimbra/log/mailbox.log

2015-03-04 02:35:42,356 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] soap - AuthRequest elapsed=94
2015-03-04 02:35:42,481 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup request started
2015-03-04 02:35:42,917 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Found 10 accounts on server mail-172.example.com
2015-03-04 02:35:43,254 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] io - AsyncFileCopier is starting
2015-03-04 02:35:43,255 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup started for backup set; label: incr-20150304.073543.151

The log events are again trimmed below just for the user1 and admin account for the user backup section of the log file. Notice the absence of any statements of the accounts being put into maintenance mode.

2015-03-04 02:35:44,118 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/6/index/0
2015-03-04 02:35:44,119 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 LOADED
2015-03-04 02:35:44,120 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup is started for account user1@mail-172.example.com
2015-03-04 02:35:44,120 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Saving account information from LDAP
2015-03-04 02:35:44,127 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup has ended
2015-03-04 02:35:44,127 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Account user1@mail-172.example.com in backup set incr-20150304.073543.151: All pending file IO completed (0 out of 0)
[cut]
2015-03-04 02:35:44,212 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup is started for account admin@mail-172.example.com
2015-03-04 02:35:44,212 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Saving account information from LDAP
2015-03-04 02:35:44,215 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup has ended
2015-03-04 02:35:44,217 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Account admin@mail-172.example.com in backup set incr-20150304.073543.151: All pending file IO completed (0 out of 0)

The incremental will then end with log events related to copying the redologs into the backup directory and then purging the redologs that can be deleted. This example states redo log sequence 9 can be deleted. We'll confirm it is later.

2015-03-04 02:35:44,618 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] redolog - waiting for FileLogWriter.FsyncThread-1425452599560 to finish.
2015-03-04 02:35:44,628 INFO  [FileLogWriter.FsyncThread-1425452599560] [] redolog - fsync thread exiting
2015-03-04 02:35:44,628 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] redolog - FileLogWriter.FsyncThread-1425452599560 finished
2015-03-04 02:35:44,757 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] redolog - Redo log rollover took 540ms
2015-03-04 02:35:44,758 INFO  [FileLogWriter.FsyncThread-1425454544757] [] redolog - Starting fsync thread with interval 10
2015-03-04 02:35:44,776 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Found 3 redo logs to backup
2015-03-04 02:35:44,861 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Last backed-up redo log sequence = 9
2015-03-04 02:35:44,943 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Marking redo log /opt/zimbra/redolog/archive/redo-20150303.060015.816-seq9.log for deletion
2015-03-04 02:35:44,943 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Copying redo log /opt/zimbra/redolog/archive/redo-20150304.070025.476-seq10.log to /opt/zimbra/backup/tmp/incr-20150304.073543.151/redologs/redo-20150304.070025.476-seq10.log
2015-03-04 02:35:45,053 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Retaining recent redo log /opt/zimbra/redolog/archive/redo-20150304.070025.476-seq10.log
2015-03-04 02:35:45,053 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Copying redo log /opt/zimbra/redolog/archive/redo-20150304.073544.755-seq11.log to /opt/zimbra/backup/tmp/incr-20150304.073543.151/redologs/redo-20150304.073544.755-seq11.log
2015-03-04 02:35:45,067 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Retaining recent redo log /opt/zimbra/redolog/archive/redo-20150304.073544.755-seq11.log

The last limes confirm the backup finished successfully.

2015-03-04 02:35:51,935 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup set incr-20150304.073543.151: All pending file IO completed (0 out of 0)
2015-03-04 02:35:51,981 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] io - AsyncFileCopier is shut down
2015-03-04 02:35:51,981 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup finished for backup set; label: incr-20150304.073543.151
2015-03-04 02:35:51,982 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Deleting redo log /opt/zimbra/redolog/archive/redo-20150303.060015.816-seq9.log
2015-03-04 02:35:52,909 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup request finished
2015-03-04 02:35:52,909 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] soap - BackupRequest elapsed=10451

For an incremental, the account sub-directories will have ldap data saved also for each user but blobs and indexes aren't. That data resides in the redologs from the system under the redologs directory.

[zimbra@mail-172 incr-20150304.073543.151]$ ls -R
.:
accounts  ldap  redologs  session.xml  sys

./accounts:
3c3  423  434  75d  7ac  89f  a46  af0  c53  eab

./accounts/3c3:
97e

./accounts/3c3/97e:
3c397edc-c015-4d82-ae8a-1084692f8a93

./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93:
ldap.xml

[cut]

./accounts/7ac:
8de

./accounts/7ac/8de:
7ac8de92-13fe-4d19-b9da-6872169fb5c9

./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9:
ldap.xml

[cut]

./ldap:
ldap-config.bak.gz  ldap.bak.gz

./redologs:
redo-20150304.070025.476-seq10.log  redo-20150304.073544.755-seq11.log

./sys:
config.dat  current_volumes.dat  db_schema.xml  localconfig.xml  ssh  ssl  volume.dat

./sys/ssh:
authorized_keys  zimbra_identity  zimbra_identity.pub

./sys/ssl:
zimbra

./sys/ssl/zimbra:
ca  commercial  jetty.pkcs12  server

./sys/ssl/zimbra/ca:
ca.key  ca.pem  ca.srl  ca.srl.old  index.txt  index.txt.attr  index.txt.old  newcerts  zmssl.cnf

./sys/ssl/zimbra/ca/newcerts:
1424523586.pem  1424523591.pem  1424523597.pem  1424523603.pem

./sys/ssl/zimbra/commercial:

./sys/ssl/zimbra/server:
server.crt  server.csr  server.key

Notice that redo log sequence 9 was indeed deleted from the redolog/archive directory.

[zimbra@mail-172 archive]$ pwd ; ls
/opt/zimbra/redolog/archive
redo-20150304.070025.476-seq10.log  redo-20150304.073544.755-seq11.log  redo-20150304.074024.058-seq12.log

If you had created new users after your last full backup, you would see that there was also a full backup started as well. For example :

[zimbra@mail-172 log]$ zmbackup -i
incr-20150304.085810.220
full-20150304.085818.889

[zimbra@mail-172 log]$ zmbackupquery -a user6@mail-172.example.com
Account: user6@mail-172.example.com

    Label:   full-20150304.085818.889
    Type:    full
    Started: Wed, 2015/03/04 03:58:18.889 EST
    Ended:   Wed, 2015/03/04 03:58:25.997 EST
    Acct ID: 4e898a68-1886-4a80-b3dc-ac2ab650bf8c

[zimbra@mail-172 log]$ zmbackupquery -lb full-20150304.085818.889 -v
Label:   full-20150304.085818.889
Type:    full
Status:  completed
Started: Wed, 2015/03/04 03:58:18.889 EST
Ended:   Wed, 2015/03/04 03:58:25.997 EST
Redo log sequence range: 14 .. 14
Number of accounts: 1 out of 1 completed
Accounts:
  user6@mail-172.example.com: completed

Total space: 14020MB
 Free space: 6038MB

Contents Of accounts.xml And sessions.xml From The Full And Incremental Backup Examples

First, see what zmbackupquery shows for a particular user in regards to the backup labels they are included in.

$ zmbackupquery -a user1@mail-172.example.com
Account: user1@mail-172.example.com

    Label:   incr-20150304.073543.151
    Type:    incremental
    Started: Wed, 2015/03/04 02:35:43.151 EST
    Ended:   Wed, 2015/03/04 02:35:51.981 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93

    Label:   full-20150304.012000.193
    Type:    full
    Started: Tue, 2015/03/03 20:20:00.193 EST
    Ended:   Tue, 2015/03/03 20:20:28.630 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93

Now let's review the accounts/session xml files that list all the users within those particular backup sessions. The /opt/zimbra/backup/accounts.xml is the referenced to determine what is the last know full backup session the user was in. If the /opt/zimbra/backup/accounts.xml is corrupted, unreadable, or incorrect - this can cause issues with restoring users.

Also notice that it also references the zimbraId for the user at the time of the backup. If you were to delete a user and then create a new account using the same email address [user1@mail-172.example.com] that new account will have a different zimbraId and therefor the backup/restore commands will not work as you might expected because of the differing zimbraId. The zmrestore command only can take an email address for the -a option, it can not use the zimbraID.

[zimbra@mail-172 backup]$ pwd
/opt/zimbra/backup

[zimbra@mail-172 backup]$ cat accounts.xml
<?xml version="1.0" encoding="utf-8"?>

<backupMetadata xmlns="urn:zimbraBackupMeta" version="7.2" dbVersion="103">
  <allAccounts>
    <account zimbraId="af06b0b4-3160-4ea3-a55b-81163a9914a0" email="user5@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193">
    </account>
    <account zimbraId="75d3d364-a384-4077-8ac5-87b6960b57cd" email="user3@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193">
    </account>
    <account zimbraId="4343a299-5259-4826-b1c1-becbfbb9b8f0" email="galsync.qva4ekog0@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193">
    </account>
    <account zimbraId="89fc5986-31a3-406e-8084-0ce0b98a251f" email="user2@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193">
    </account>
    <account zimbraId="42380a6e-5718-4d68-ab37-e8d5bc9e6977" email="spam.umhx3owv3s@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193">
    </account>
    <account zimbraId="eab3d56e-ffd9-4412-95de-48f102d2d252" email="user4@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193">
    </account>
    <account zimbraId="7ac8de92-13fe-4d19-b9da-6872169fb5c9" email="admin@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193">
    </account>
    <account zimbraId="4e898a68-1886-4a80-b3dc-ac2ab650bf8c" email="user6@mail-172.example.com" latestFullBackupLabel="full-20150304.085818.889">
    </account>
    <account zimbraId="a46dfc59-a3e9-4f09-adfb-af8efc42aa9f" email="virus-quarantine.gusiso1cd@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193">
    </account>
    <account zimbraId="3c397edc-c015-4d82-ae8a-1084692f8a93" email="user1@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193">
    </account>
    <account zimbraId="c5315c5d-97cb-4d84-bc70-374a67f13309" email="ham.jgfdb3oqjf@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193">
    </account>
  </allAccounts>
</backupMetadata>

In each backup session, there will be a sessions.xml file that acts as a reference or summary file of the backup. At the top, you'll see what version of ZCS was running and what options the backup environment had set. It will then report the final status for each user for the backup attempt against the account. At the end, there's a section in regards to errors and if any were encountered.

[zimbra@mail-172 backup]$ cd sessions/
[zimbra@mail-172 sessions]$ cd full-20150304.012000.193
[zimbra@mail-172 full-20150304.012000.193]$ cat session.xml
<?xml version="1.0" encoding="utf-8"?>

<backupMetadata xmlns="urn:zimbraBackupMeta" version="7.2" dbVersion="103">
  <backupSet label="full-20150304.012000.193" zcsRelease="8.6.0_GA_1153 20141215151258 20141215-1518 NETWORK" startTime="1425432000193" endTime="1425432028630" minRedoSeq="10" maxRedoSeq="10" sharedBlobsZipped="true" sharedBlobsZipNameDigestChars="1" sharedBlobsDirectoryDepth="5" sharedBlobsCharsPerDirectory="2" type="full" accountsDirectoryDepth="2">
    <desc>Full backup</desc>
    <accounts>
      <account zimbraId="af06b0b4-3160-4ea3-a55b-81163a9914a0" email="user5@mail-172.example.com" status="COMPLETED">
      </account>
      <account zimbraId="75d3d364-a384-4077-8ac5-87b6960b57cd" email="user3@mail-172.example.com" status="COMPLETED">
      </account>
      <account zimbraId="89fc5986-31a3-406e-8084-0ce0b98a251f" email="user2@mail-172.example.com" status="COMPLETED">
      </account>
      <account zimbraId="4343a299-5259-4826-b1c1-becbfbb9b8f0" email="galsync.qva4ekog0@mail-172.example.com" status="COMPLETED">
      </account>
      <account zimbraId="eab3d56e-ffd9-4412-95de-48f102d2d252" email="user4@mail-172.example.com" status="COMPLETED">
      </account>
      <account zimbraId="42380a6e-5718-4d68-ab37-e8d5bc9e6977" email="spam.umhx3owv3s@mail-172.example.com" status="COMPLETED">
      </account>
      <account zimbraId="7ac8de92-13fe-4d19-b9da-6872169fb5c9" email="admin@mail-172.example.com" status="COMPLETED">
      </account>
      <account zimbraId="a46dfc59-a3e9-4f09-adfb-af8efc42aa9f" email="virus-quarantine.gusiso1cd@mail-172.example.com" status="COMPLETED">
      </account>
      <account zimbraId="3c397edc-c015-4d82-ae8a-1084692f8a93" email="user1@mail-172.example.com" status="COMPLETED">
      </account>
      <account zimbraId="c5315c5d-97cb-4d84-bc70-374a67f13309" email="ham.jgfdb3oqjf@mail-172.example.com" status="COMPLETED">
      </account>
    </accounts>
    <stats>
      <counter name="total_time" numSamples="1" sum="28437" unit="ms">
      </counter>
      <counter name="sysdb_time" numSamples="1" sum="1111" unit="ms">
      </counter>
      <counter name="sysdb_bytes" numSamples="1" sum="35210" unit="bytes">
      </counter>
      <counter name="redologs_time" numSamples="0" sum="0" unit="ms">
      </counter>
      <counter name="redologs_bytes" numSamples="0" sum="0" unit="bytes">
      </counter>
      <counter name="accounts_time" numSamples="10" sum="4827" unit="ms">
      </counter>
      <counter name="ldap_time" numSamples="10" sum="172" unit="ms">
      </counter>
      <counter name="ldap_bytes" numSamples="10" sum="25665" unit="bytes">
      </counter>
      <counter name="db_time" numSamples="10" sum="4787" unit="ms">
      </counter>
      <counter name="db_bytes" numSamples="10" sum="78690" unit="bytes">
      </counter>
      <counter name="db_digest_map_load_time" numSamples="10" sum="2400" unit="ms">
      </counter>
      <counter name="msgs_time" numSamples="10" sum="451" unit="ms">
      </counter>
      <counter name="msg_copy_time" numSamples="74" sum="25" unit="ms">
      </counter>
      <counter name="msg_copy_bytes" numSamples="74" sum="149611" unit="bytes">
      </counter>
      <counter name="msg_link_time" numSamples="0" sum="0" unit="ms">
      </counter>
      <counter name="index_time" numSamples="48" sum="80" unit="ms">
      </counter>
      <counter name="index_bytes" numSamples="38" sum="73510" unit="bytes">
      </counter>
    </stats>
    <errors>
    </errors>
  </backupSet>
</backupMetadata>

An incremental session.xml file is similar to a full backups session.xml .

[zimbra@mail-172 sessions]$ cd incr-20150304.073543.151
[zimbra@mail-172 incr-20150304.073543.151]$ cat session.xml
<?xml version="1.0" encoding="utf-8"?>

<backupMetadata xmlns="urn:zimbraBackupMeta" version="7.2" dbVersion="103">
  <backupSet label="incr-20150304.073543.151" zcsRelease="8.6.0_GA_1153 20141215151258 20141215-1518 NETWORK" startTime="1425454543151" endTime="1425454551981" minRedoSeq="10" maxRedoSeq="11" sharedBlobsZipped="true" sharedBlobsZipNameDigestChars="1" sharedBlobsDirectoryDepth="5" sharedBlobsCharsPerDirectory="2" type="incremental" accountsDirectoryDepth="2">
    <desc>Incremental backup</desc>
    <accounts>
      <account zimbraId="af06b0b4-3160-4ea3-a55b-81163a9914a0" email="user5@mail-172.example.com" status="COMPLETED">
      </account>
      <account zimbraId="75d3d364-a384-4077-8ac5-87b6960b57cd" email="user3@mail-172.example.com" status="COMPLETED">
      </account>
      <account zimbraId="89fc5986-31a3-406e-8084-0ce0b98a251f" email="user2@mail-172.example.com" status="COMPLETED">
      </account>
      <account zimbraId="4343a299-5259-4826-b1c1-becbfbb9b8f0" email="galsync.qva4ekog0@mail-172.example.com" status="COMPLETED">
      </account>
      <account zimbraId="eab3d56e-ffd9-4412-95de-48f102d2d252" email="user4@mail-172.example.com" status="COMPLETED">
      </account>
      <account zimbraId="42380a6e-5718-4d68-ab37-e8d5bc9e6977" email="spam.umhx3owv3s@mail-172.example.com" status="COMPLETED">
      </account>
      <account zimbraId="7ac8de92-13fe-4d19-b9da-6872169fb5c9" email="admin@mail-172.example.com" status="COMPLETED">
      </account>
      <account zimbraId="a46dfc59-a3e9-4f09-adfb-af8efc42aa9f" email="virus-quarantine.gusiso1cd@mail-172.example.com" status="COMPLETED">
      </account>
      <account zimbraId="3c397edc-c015-4d82-ae8a-1084692f8a93" email="user1@mail-172.example.com" status="COMPLETED">
      </account>
      <account zimbraId="c5315c5d-97cb-4d84-bc70-374a67f13309" email="ham.jgfdb3oqjf@mail-172.example.com" status="COMPLETED">
      </account>
    </accounts>
    <stats>
      <counter name="total_time" numSamples="1" sum="8830" unit="ms">
      </counter>
      <counter name="sysdb_time" numSamples="1" sum="563" unit="ms">
      </counter>
      <counter name="sysdb_bytes" numSamples="1" sum="35210" unit="bytes">
      </counter>
      <counter name="redologs_time" numSamples="1" sum="310" unit="ms">
      </counter>
      <counter name="redologs_bytes" numSamples="1" sum="126307" unit="bytes">
      </counter>
      <counter name="accounts_time" numSamples="10" sum="196" unit="ms">
      </counter>
      <counter name="ldap_time" numSamples="10" sum="129" unit="ms">
      </counter>
      <counter name="ldap_bytes" numSamples="10" sum="25665" unit="bytes">
      </counter>
      <counter name="db_time" numSamples="0" sum="0" unit="ms">
      </counter>
      <counter name="db_bytes" numSamples="0" sum="0" unit="bytes">
      </counter>
      <counter name="db_digest_map_load_time" numSamples="0" sum="0" unit="ms">
      </counter>
      <counter name="msgs_time" numSamples="0" sum="0" unit="ms">
      </counter>
      <counter name="msg_copy_time" numSamples="0" sum="0" unit="ms">
      </counter>
      <counter name="msg_copy_bytes" numSamples="0" sum="0" unit="bytes">
      </counter>
      <counter name="msg_link_time" numSamples="0" sum="0" unit="ms">
      </counter>
      <counter name="index_time" numSamples="0" sum="0" unit="ms">
      </counter>
      <counter name="index_bytes" numSamples="0" sum="0" unit="bytes">
      </counter>
    </stats>
    <errors>
    </errors>
  </backupSet>
</backupMetadata>

User And Log Events Generated

Please beware that prior backups were deleted and new backups were made on this ZCS test server. So the backup label names might not match prior examples.

For this example, I'm going to replicate a common situation that system administrators find themselves when trying to restore a user and they fail to include the proper restore options to have the restore stop prior to replaying a message that was deleted by a user. Proper examples of doing the restore will be shown afterwards.

Information on the account we'll restore:

[zimbra@mail-172 backup]$ zmprov gmi user1@mail-172.example.com
mailboxId: 6
quotaUsed: 1209

[zimbra@mail-172 backup]$ zmprov ga user1@mail-172.example.com zimbraId
# name user1@mail-172.example.com
zimbraId: 3c397edc-c015-4d82-ae8a-1084692f8a93

Example of error you'll receive if you attempt to use the zimbraId rather than the email address for the zmrestore command. This is important to note if your trying to restore an account that was deleted and then later created again and the various backups and accounts.xml file has different zimbraId's associated with the same email address.

[zimbra@mail-172 backup]$ zmrestore -a 3c397edc-c015-4d82-ae8a-1084692f8a93 -ca -pre restore1_
Error occurred: invalid request: invalid account email address: 3c397edc-c015-4d82-ae8a-1084692f8a93

Showing that we have one message in the Inbox for the user we'll be restoring and also the message id of that email :

[zimbra@mail-172 backup]$ zmmailbox -z -m user1@mail-172.example.com gaf
        Id  View      Unread   Msg Count  Path
----------  ----  ----------  ----------  ----------
         1  unkn           0           0  /
        16  docu           0           0  /Briefcase
        10  appo           0           0  /Calendar
        14  mess           0           0  /Chats
         7  cont           0           0  /Contacts
         6  mess           0           0  /Drafts
        13  cont           0           0  /Emailed Contacts
         2  mess           1           1  /Inbox
         4  mess           0           0  /Junk
         5  mess           0           0  /Sent
        15  task           0           0  /Tasks
         3  unkn           0           0  /Trash

[zimbra@mail-172 backup]$ zmmailbox -z -m user1@mail-172.example.com s -t mess in:Inbox
num: 1, more: false

     Id  Type   From                  Subject                                             Date
   ----  ----   --------------------  --------------------------------------------------  --------------
1.  257  mess   admin                 Test1                                               02/23/15 09:03

Now to delete the email from the account:

[zimbra@mail-172 backup]$ zmmailbox -z -m user1@mail-172.example.com dm 257

And to confirm it's no longer there:

[zimbra@mail-172 backup]$ zmmailbox -z -m user1@mail-172.example.com s -t mess in:Inbox
num: 0, more: false

Below is an example of a delete operation being logged in the redo.log and how to query events for a particular user with the zmredodump command. Notice the DeleteItem events at the end.

[zimbra@mail-172 backup]$ zmredodump -m 6 --show-blob /opt/zimbra/redolog/redo.log
VERIFYING: /opt/zimbra/redolog/redo.log
HEADER
------
sequence: 14
open:     1
filesize: 512
serverId: 4407a03f-894b-4b1a-817a-684fc34c4f60
created:  Wed, 2015/03/04 03:58:10.876 EST (1425459490876)
first op: Wed, 2015/03/04 03:58:18.744 EST (1425459498744)
last op:  Wed, 2015/03/04 03:58:18.744 EST (1425459498744) (not up to date)
version:  1.42
------
[00001601 - 0000162e: 46 bytes; tstamp: 2015/03/04 04:00:26.677 EST] txn 1425452599.101 [PurgeOldMessages] ver=1.42, tstamp=1425459626677, change=1508, mailbox=6
[0000162f - 00001660: 50 bytes; tstamp: 2015/03/04 04:00:26.683 EST] txn 1425452599.101 [CommitTxn] ver=1.42, tstamp=1425459626683, mailbox=6, txnType=PurgeOldMessages
[00002691 - 000026be: 46 bytes; tstamp: 2015/03/04 04:10:26.816 EST] txn 1425452599.113 [PurgeOldMessages] ver=1.42, tstamp=1425460226816, change=1509, mailbox=6
[000026bf - 000026f0: 50 bytes; tstamp: 2015/03/04 04:10:26.832 EST] txn 1425452599.113 [CommitTxn] ver=1.42, tstamp=1425460226832, mailbox=6, txnType=PurgeOldMessages
[00002ab1 - 00002ade: 46 bytes; tstamp: 2015/03/04 04:21:27.376 EST] txn 1425452599.124 [PurgeOldMessages] ver=1.42, tstamp=1425460887376, change=1510, mailbox=6
[00002adf - 00002b10: 50 bytes; tstamp: 2015/03/04 04:21:27.381 EST] txn 1425452599.124 [CommitTxn] ver=1.42, tstamp=1425460887381, mailbox=6, txnType=PurgeOldMessages
[00002ed1 - 00002efe: 46 bytes; tstamp: 2015/03/04 04:32:27.610 EST] txn 1425452599.135 [PurgeOldMessages] ver=1.42, tstamp=1425461547610, change=1511, mailbox=6
[00002eff - 00002f30: 50 bytes; tstamp: 2015/03/04 04:32:27.617 EST] txn 1425452599.135 [CommitTxn] ver=1.42, tstamp=1425461547617, mailbox=6, txnType=PurgeOldMessages
[000032f1 - 0000331e: 46 bytes; tstamp: 2015/03/04 04:43:27.828 EST] txn 1425452599.146 [PurgeOldMessages] ver=1.42, tstamp=1425462207828, change=1512, mailbox=6
[0000331f - 00003350: 50 bytes; tstamp: 2015/03/04 04:43:27.839 EST] txn 1425452599.146 [CommitTxn] ver=1.42, tstamp=1425462207839, mailbox=6, txnType=PurgeOldMessages
[000034d1 - 0000350c: 60 bytes; tstamp: 2015/03/04 04:48:18.119 EST] txn 1425452599.151 [DeleteItem] ver=1.42, tstamp=1425462498119, change=1513, mailbox=6, ids=[257], type=message
[0000350d - 0000353e: 50 bytes; tstamp: 2015/03/04 04:48:18.306 EST] txn 1425452599.151 [CommitTxn] ver=1.42, tstamp=1425462498306, mailbox=6, txnType=DeleteItem

Listing of the backups that are currently associated with the account we'll be restoring:

[zimbra@mail-172 backup]$ zmbackupquery -a user1@mail-172.example.com
Account: user1@mail-172.example.com

    Label:   incr-20150304.085810.220
    Type:    incremental
    Started: Wed, 2015/03/04 03:58:10.220 EST
    Ended:   Wed, 2015/03/04 03:58:16.393 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93

    Label:   full-20150304.082015.535
    Type:    full
    Started: Wed, 2015/03/04 03:20:15.535 EST
    Ended:   Wed, 2015/03/04 03:20:39.109 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93

Run a zmrestore with no other options other than the account flag and the email address of the account we are restoring. Note - this example is to show why this often times isn't what you'll do in real circumstances and will require other zmrestore options to achieve what you want.

[zimbra@mail-172 backup]$ zmrestore -a user1@mail-172.example.com

From the /opt/zimbra/log/mailbox.log we have the following log events. We see the restore request kick off and the full backup label that it'll reference as the starting point. It also lists the zimbraId rather than the email address for the restore initially.

2015-03-04 04:53:02,221 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - AuthRequest elapsed=60
2015-03-04 04:53:02,519 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is starting
2015-03-04 04:53:02,565 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Using full backup full-20150304.082015.535 as starting point for restoring account 3c397edc-c015-4d82-ae8a-1084692f8a93

And then we'll see the reference to the email address and the mailboxId associated with the account/zimbraId .

2015-03-04 04:53:02,993 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - waiting for FileLogWriter.FsyncThread-1425459490899 to finish.
2015-03-04 04:53:02,997 INFO  [FileLogWriter.FsyncThread-1425459490899] [] redolog - fsync thread exiting
2015-03-04 04:53:02,997 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - FileLogWriter.FsyncThread-1425459490899 finished
2015-03-04 04:53:03,028 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - Redo log rollover took 463ms
2015-03-04 04:53:03,028 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore started for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6

Since we are basically over-writing the account, the account is placed into maintenance mode first and then existing data is deleted. Only after the account has been deleted will the account be restored using the backup data. Since we didn't use the -ca and -pre options for zmrestore, this account will reuse it's existing zimbraId, mailboxId, and of course - email address.

2015-03-04 04:53:03,042 INFO  [FileLogWriter.FsyncThread-1425462783028] [] redolog - Starting fsync thread with interval 10
2015-03-04 04:53:03,096 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - Putting mailbox 6 under maintenance.
2015-03-04 04:53:03,097 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - clearing contents of mailbox 6, group 6
2015-03-04 04:53:03,209 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restoring to mailbox id 6 for existing account 3c397edc-c015-4d82-ae8a-1084692f8a93.
2015-03-04 04:53:03,256 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loading schema with version 7.2
2015-03-04 04:53:03,287 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Renaming volume_id column to locator in source table mail_item
2015-03-04 04:53:03,584 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Deleting directory /opt/zimbra/store/0/6/msg
2015-03-04 04:53:03,599 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 1 blobs during restore
2015-03-04 04:53:03,622 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Deleting directory /opt/zimbra/index/0/6/index

At this point, the data for the account was restored to the state of how it was when the full backup was ran.

2015-03-04 04:53:03,664 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 2 index files during restore
2015-03-04 04:53:03,665 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 AVAILABLE
2015-03-04 04:53:03,670 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Account user1@mail-172.example.com: All pending file IO completed (2 out of 2)

Now we see the restore start playing redologs that were made after the full backup. This section is the redologs from incr-20150304.085810.220 backup session, which were : redo-20150304.070025.476-seq10.log redo-20150304.073544.755-seq11.log redo-20150304.074024.058-seq12.log redo-20150304.085810.879-seq13.log . Located in, /opt/zimbra/backup/sessions/incr-20150304.085810.220/redologs .

2015-03-04 04:53:03,671 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Run redo operations since last full backup
2015-03-04 04:53:03,733 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/6/index/0
2015-03-04 04:53:03,733 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 LOADED
2015-03-04 04:53:03,733 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
2015-03-04 04:53:03,733 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] cache - initializing folder and tag caches for mailbox 6
2015-03-04 04:53:03,746 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
2015-03-04 04:53:03,752 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
2015-03-04 04:53:03,757 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.

Notice it will reference the sequence number of the redo log when it is plays those in /opt/zimbra/redolog/archive/ . Currently, the server has the following in that directory : redo-20150304.085810.879-seq13.log redo-20150304.095303.026-seq14.log redo-20150304.102039.541-seq15.log .

2015-03-04 04:53:03,763 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Run redo operations from archived redo logs starting at sequence 13
2015-03-04 04:53:03,764 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
2015-03-04 04:53:03,774 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
2015-03-04 04:53:03,780 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
2015-03-04 04:53:03,787 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
2015-03-04 04:53:03,795 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.

And while it replays redo log sequence 13, we see that it replays the delete operation for message id 257.

2015-03-04 04:53:03,824 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailop - Deleting Message (id=257).

Final log events showing restore is finished.

2015-03-04 04:53:03,832 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore finished for user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox 6 to mailbox 6
2015-03-04 04:53:03,833 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is shut down
2015-03-04 04:53:03,833 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - RestoreRequest elapsed=1535

Confirm that the message is still missing from the account:

[zimbra@mail-172 backup]$ zmmailbox -z -m user1@mail-172.example.com s -t mess in:Inbox
num: 0, more: false

Confirm that the account is still using the same zimbraId, mailboxId, and email address :

[zimbra@mail-172 backup]$ zmprov gmi user1@mail-172.example.com
mailboxId: 6
quotaUsed: 0

[zimbra@mail-172 backup]$ zmprov ga user1@mail-172.example.com zimbraId
# name user1@mail-172.example.com
zimbraId: 3c397edc-c015-4d82-ae8a-1084692f8a93

Using The -restoreToTime Option To Restore Prior To A Message Being Deleted

First, let's identify the exact time the delete operation took place. From our previous examples above I already know what redo log sequence file it was logged in.

[zimbra@mail-172 backup]$ zmredodump -m 6 --show-blob /opt/zimbra/redolog/archive/redo-20150304.095303.026-seq14.log | grep DeleteItem
[000034d1 - 0000350c: 60 bytes; tstamp: 2015/03/04 04:48:18.119 EST] txn 1425452599.151 [DeleteItem] ver=1.42, tstamp=1425462498119, change=1513, mailbox=6, ids=[257], type=message
[0000350d - 0000353e: 50 bytes; tstamp: 2015/03/04 04:48:18.306 EST] txn 1425452599.151 [CommitTxn] ver=1.42, tstamp=1425462498306, mailbox=6, txnType=DeleteItem

The first line above shows me the delete request was done at "2015/03/04 04:48:18.119" for mailbox=6, ids=[257], type=message . Ids is the message id, that were already used to delete the message in the above example and was shown in the zmmailbox search command.


The example uses the time above but subtracted one from it - 04:48:18.118 vs 04:48:18.119 .

[zimbra@mail-172 backup]$ zmrestore -a user1@mail-172.example.com -restoreToTime "2015/03/04 04:48:18.118"
using restore to time of 2015/03/04 04:48:18

And to now confirm the message is back in the account:

[zimbra@mail-172 backup]$ zmmailbox -z -m user1@mail-172.example.com s -t mess in:Inbox
num: 1, more: false

     Id  Type   From                  Subject                                             Date
   ----  ----   --------------------  --------------------------------------------------  --------------
1.  257  mess   admin                 Test1                                               02/23/15 09:03

Note that this will also mean all other data or changes after 04:48:18.118 are lost also in this restored account. Normally in circumstances of trying to recovery data that a user is requesting because they delete messages and now need them recovered you would use the -ca and -pre options for zmrestore so you could then recover them and import the messages back into the existing account they have. Without the -ca and -pre , your over writing the existing account on the system.

The log events in /opt/zimbra/log/mailbox.log for this request were:

2015-03-04 05:20:39,194 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is starting
2015-03-04 05:20:39,238 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Using full backup full-20150304.082015.535 as starting point for restoring account 3c397edc-c015-4d82-ae8a-1084692f8a93
2015-03-04 05:20:39,526 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - waiting for FileLogWriter.FsyncThread-1425462783028 to finish.
2015-03-04 05:20:39,533 INFO  [FileLogWriter.FsyncThread-1425462783028] [] redolog - fsync thread exiting
2015-03-04 05:20:39,533 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - FileLogWriter.FsyncThread-1425462783028 finished
2015-03-04 05:20:39,543 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - Redo log rollover took 305ms
2015-03-04 05:20:39,543 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore started for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6
2015-03-04 05:20:39,550 INFO  [FileLogWriter.FsyncThread-1425464439542] [] redolog - Starting fsync thread with interval 10
2015-03-04 05:20:39,593 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - Putting mailbox 6 under maintenance.
2015-03-04 05:20:39,593 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - clearing contents of mailbox 6, group 6
2015-03-04 05:20:39,614 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restoring to mailbox id 6 for existing account 3c397edc-c015-4d82-ae8a-1084692f8a93.
2015-03-04 05:20:39,652 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loading schema with version 7.2
2015-03-04 05:20:39,660 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Renaming volume_id column to locator in source table mail_item
2015-03-04 05:20:39,692 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Deleting directory /opt/zimbra/store/0/6/msg
2015-03-04 05:20:39,700 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 1 blobs during restore
2015-03-04 05:20:39,703 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Deleting directory /opt/zimbra/index/0/6/index
2015-03-04 05:20:39,713 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 2 index files during restore
2015-03-04 05:20:39,715 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 AVAILABLE
2015-03-04 05:20:39,717 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Account user1@mail-172.example.com: All pending file IO completed (2 out of 2)
2015-03-04 05:20:39,717 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Run redo operations since last full backup
2015-03-04 05:20:39,732 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/6/index/0
2015-03-04 05:20:39,732 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 LOADED
2015-03-04 05:20:39,732 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
2015-03-04 05:20:39,732 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] cache - initializing folder and tag caches for mailbox 6
2015-03-04 05:20:39,743 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
2015-03-04 05:20:39,749 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
2015-03-04 05:20:39,754 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
2015-03-04 05:20:39,760 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Run redo operations from archived redo logs starting at sequence 13
2015-03-04 05:20:39,762 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
2015-03-04 05:20:39,770 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
2015-03-04 05:20:39,774 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
2015-03-04 05:20:39,780 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
2015-03-04 05:20:39,786 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
2015-03-04 05:20:39,789 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore finished for user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox 6 to mailbox 6
2015-03-04 05:20:39,791 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is shut down
2015-03-04 05:20:39,791 INFO  [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - RestoreRequest elapsed=646

Notice that this restore, unlike the first example, does NOT have the following line :

 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailop - Deleting Message (id=257).

Blobs In Backup And Blobs In Production

This might be a good time to take a look at the blobs and how they are saved/stored in the backups and how they compare to the store volume. The example below is from our user and message details from above.

[zimbra@mail-172 blobs]$ pwd
/opt/zimbra/backup/sessions/full-20150304.082015.535/accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93/blobs

[zimbra@mail-172 blobs]$ unzip -Z blobs-1.zip
Archive:  blobs-1.zip
Zip file size: 1439 bytes, number of entries: 1
-rw----     4.5 fat     1209 b- stor 15-Mar-04 03:20 1/0/sha256_mBS9YKqyNpLknKCDWY5WNEbTgotX,UNrXMA5ga+s6LA=_257-2.msg1
1 file, 1209 bytes uncompressed, 1209 bytes compressed:  0.0%

[zimbra@mail-172 blobs]$ ls -la /opt/zimbra/store/0/6/msg/0/257-2.msg
-rw-r----- 1 zimbra zimbra 1209 Mar  4 05:20 /opt/zimbra/store/0/6/msg/0/257-2.msg

[zimbra@mail-172 blobs]$ unzip blobs-1.zip
Archive:  blobs-1.zip
 extracting: 1/0/sha256_mBS9YKqyNpLknKCDWY5WNEbTgotX,UNrXMA5ga+s6LA=_257-2.msg1

[zimbra@mail-172 blobs]$ ls
1  blobs-1.zip

[zimbra@mail-172 blobs]$ ls -la 1/0/sha256_mBS9YKqyNpLknKCDWY5WNEbTgotX\,UNrXMA5ga+s6LA\=_257-2.msg1
-rw-r----- 1 zimbra zimbra 1209 Mar  4 03:20 1/0/sha256_mBS9YKqyNpLknKCDWY5WNEbTgotX,UNrXMA5ga+s6LA=_257-2.msg1

[zimbra@mail-172 blobs]$ head 1/0/sha256_mBS9YKqyNpLknKCDWY5WNEbTgotX\,UNrXMA5ga+s6LA\=_257-2.msg1
Return-Path: zimbra@mail-172.example.com
Received: from mail-172.example.com (LHLO mail-172.example.com)
 (192.168.1.172) by mail-172.example.com with LMTP; Mon, 23 Feb 2015
 09:02:58 -0500 (EST)
Received: from localhost (localhost.localdomain [127.0.0.1])
        by mail-172.example.com (Postfix) with ESMTP id 4F8AE8433D
        for <user1@mail-172.example.com>; Mon, 23 Feb 2015 09:02:58 -0500 (EST)
X-Virus-Scanned: amavisd-new at mail-172.example.com
X-Spam-Flag: NO
X-Spam-Score: -1.9

[zimbra@mail-172 blobs]$ head /opt/zimbra/store/0/6/msg/0/257-2.msg
Return-Path: zimbra@mail-172.example.com
Received: from mail-172.example.com (LHLO mail-172.example.com)
 (192.168.1.172) by mail-172.example.com with LMTP; Mon, 23 Feb 2015
 09:02:58 -0500 (EST)
Received: from localhost (localhost.localdomain [127.0.0.1])
        by mail-172.example.com (Postfix) with ESMTP id 4F8AE8433D
        for <user1@mail-172.example.com>; Mon, 23 Feb 2015 09:02:58 -0500 (EST)
X-Virus-Scanned: amavisd-new at mail-172.example.com
X-Spam-Flag: NO
X-Spam-Score: -1.9

Knowing When The -br [To Incremental] Or -rf [Only Full] Option Is Required

Pre-Steps

Let's query current backups and then create a new full and new incremental where we also delete a message between the full and the incremental.

[zimbra@mail-172 ~]$ zmbackupquery -a user1@`zmhostname`
Account: user1@mail-172.example.com

    Label:   incr-20150305.060013.901
    Type:    incremental
    Started: Thu, 2015/03/05 01:00:13.901 EST
    Ended:   Thu, 2015/03/05 01:00:26.273 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93

    Label:   incr-20150304.085810.220
    Type:    incremental
    Started: Wed, 2015/03/04 03:58:10.220 EST
    Ended:   Wed, 2015/03/04 03:58:16.393 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93

    Label:   full-20150304.082015.535
    Type:    full
    Started: Wed, 2015/03/04 03:20:15.535 EST
    Ended:   Wed, 2015/03/04 03:20:39.109 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93

[zimbra@mail-172 ~]$ zmbackup -f -a all
full-20150306.043202.011

[zimbra@mail-172 ~]$ zmmailbox -z -m user1@mail-172.example.com s -t mess in:Inbox
num: 1, more: false

     Id  Type   From                  Subject                                             Date
   ----  ----   --------------------  --------------------------------------------------  --------------
1.  257  mess   admin                 Test1                                               02/23/15 09:03

[zimbra@mail-172 ~]$ zmmailbox -z -m user1@mail-172.example.com dm 257

[zimbra@mail-172 ~]$ zmbackup -i
incr-20150306.043321.683

[zimbra@mail-172 ~]$ zmbackupquery -a user1@`zmhostname`
Account: user1@mail-172.example.com

    Label:   incr-20150306.043321.683
    Type:    incremental
    Started: Thu, 2015/03/05 23:33:21.683 EST
    Ended:   Thu, 2015/03/05 23:33:27.495 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93

    Label:   full-20150306.043202.011
    Type:    full
    Started: Thu, 2015/03/05 23:32:02.011 EST
    Ended:   Thu, 2015/03/05 23:32:13.316 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93

    Label:   incr-20150305.060013.901
    Type:    incremental
    Started: Thu, 2015/03/05 01:00:13.901 EST
    Ended:   Thu, 2015/03/05 01:00:26.273 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93

    Label:   incr-20150304.085810.220
    Type:    incremental
    Started: Wed, 2015/03/04 03:58:10.220 EST
    Ended:   Wed, 2015/03/04 03:58:16.393 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93

    Label:   full-20150304.082015.535
    Type:    full
    Started: Wed, 2015/03/04 03:20:15.535 EST
    Ended:   Wed, 2015/03/04 03:20:39.109 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93
First Example - Options -lb, -ca, -pre restore_

For the following examples, we'll include the -ca [create account] and the -pre [add prefix to new account name] . This first example uses the oldest full backup [full-20150304.082015.535 vs. full-20150306.043202.011] the account has with the -lb option. The message we just deleted, existed at the time of the full backup. We'll NOT be including either the -rf or -br option in this first example.

Then we'll review the mailbox.log events. Note the following:

  • That the log only mentions the parent account, there isn't a reference to the email address with the prefixed name.
  • Though the log will state the new mailboxId and zimbraId for the user, though:
    • "user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6" vs "Restoring to mailbox id 12 for new account 016c9f7c-e2e2-4f8f-96d0-e14a4327821d"
  • Since we failed to include the -rf option, the restore continues past the full backup and goes through all the other backups that exist, hence we see:
    • "backup - Run redo operations since last full backup"
  • One of the redologs in our incremental had the delete operation for the message in question and we see evidence of that delete in the log:
    • "mailop - Deleting Message (id=257)."
  • Since we didn't include the -br option, the restore also continues past the backups and replays the redologs in /opt/zimbra/redolog and it's subdirectory, archive. Notice the reference to sequence 17.
    • "backup - Run redo operations from archived redo logs starting at sequence 17"
    • /opt/zimbra/redolog/archive/ has : redo-20150306.043322.070-seq17.log redo-20150306.043611.912-seq18.log
  • Towards the end, we'll get a confirmation that the restore has finished with some reference information again - mailboxId's .
    • "Restore finished for user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox 6 to mailbox 12"
[zimbra@mail-172 ~]$ cat /dev/null > /opt/zimbra/log/mailbox.log

[zimbra@mail-172 ~]$ zmrestore -a user1@mail-172.example.com -lb full-20150304.082015.535 -ca -pre restore_

[zimbra@mail-172 ~]$ cat /opt/zimbra/log/mailbox.log
2015-03-05 23:35:40,815 INFO  [MailboxPurge] [name=spam.umhx3owv3s@mail-172.example.com;mid=3;] purge - Purging messages.
2015-03-05 23:36:11,619 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - AuthRequest elapsed=3
2015-03-05 23:36:11,784 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is starting
2015-03-05 23:36:11,905 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - waiting for FileLogWriter.FsyncThread-1425616402072 to finish.
2015-03-05 23:36:11,908 INFO  [FileLogWriter.FsyncThread-1425616402072] [] redolog - fsync thread exiting
2015-03-05 23:36:11,908 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - FileLogWriter.FsyncThread-1425616402072 finished
2015-03-05 23:36:11,913 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - Redo log rollover took 128ms
2015-03-05 23:36:11,913 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore started for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6
2015-03-05 23:36:11,914 INFO  [FileLogWriter.FsyncThread-1425616571913] [] redolog - Starting fsync thread with interval 10
2015-03-05 23:36:13,006 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - Creating database mboxgroup12
2015-03-05 23:36:14,150 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restoring to mailbox id 12 for new account 016c9f7c-e2e2-4f8f-96d0-e14a4327821d.
2015-03-05 23:36:14,237 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loading schema with version 7.2
2015-03-05 23:36:14,276 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Renaming volume_id column to locator in source table mail_item
2015-03-05 23:36:14,315 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 1 blobs during restore
2015-03-05 23:36:14,316 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Deleting directory /opt/zimbra/index/0/12/index
2015-03-05 23:36:14,317 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 2 index files during restore
2015-03-05 23:36:14,319 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/12/index/0
2015-03-05 23:36:14,319 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox 12 account 016c9f7c-e2e2-4f8f-96d0-e14a4327821d LOADED
2015-03-05 23:36:14,319 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] cache - initializing folder and tag caches for mailbox 12
2015-03-05 23:36:14,367 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Account user1@mail-172.example.com: All pending file IO completed (2 out of 2)
2015-03-05 23:36:14,367 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Run redo operations since last full backup
2015-03-05 23:36:14,400 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.

 [multiple repeats of this message]

2015-03-05 23:36:14,428 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
2015-03-05 23:36:14,465 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailop - Deleting Message (id=257).
2015-03-05 23:36:14,474 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.

 [multiple repeats of this message]

2015-03-05 23:36:15,152 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
2015-03-05 23:36:15,172 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Run redo operations from archived redo logs starting at sequence 17
2015-03-05 23:36:15,172 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore finished for user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox 6 to mailbox 12
2015-03-05 23:36:15,174 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is shut down
2015-03-05 23:36:15,174 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - RestoreRequest elapsed=3481

[zimbra@mail-172 ~]$ ls /opt/zimbra/redolog/archive/
redo-20150306.043322.070-seq17.log  redo-20150306.043611.912-seq18.log
Second Example - Options -lb , -ca , -pre restore2_ , -br

For the second example, we'll again use the oldest full backup [full-20150304.082015.535 vs. full-20150306.043202.011] the account has with the -lb option. The message we had deleted, existed at the time of the full backup but did NOT in the last incremental backup. Actually, the delete operation is within the redologs of the last incremental. For the second example, we'll use the -br option rather than the -rf . We'll also be adjusting the -pre option to be restore2_ vs. restore_ in the first example.

Then we'll review the mailbox.log events. Note the following:

  • Though the log will state the new mailboxId and zimbraId for the user, though:
    • "user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6" vs "Restoring to mailbox id 13 for new account 4c99a395-2c39-46ec-8c1b-285d97d899df."
    • Notice the mailboxId and zimbraId for restore2_ is different than our first example when we used restore_ for the -pre option.
  • Since we failed to include the -rf option, the restore continues past the full backup and goes through all the other backups that exist, hence we see:
    • "backup - Run redo operations since last full backup"
  • One of the redologs in our incremental had the delete operation for the message in question and we see evidence of that delete in the log:
    • "mailop - Deleting Message (id=257)."
  • Since we DID include the -br option, the restore STOPS at our last backup session. It therefor DOESN'T replay the redologs in /opt/zimbra/redolog and it's subdirectory, archive and we do NOT have the following log events like we did in our first example.
    • "backup - Run redo operations from archived redo logs starting at sequence 17"
  • Towards the end, we'll get a confirmation that the restore has finished with some reference information again - mailboxId's .
    • "Restore finished for user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox 6 to mailbox 13"
[zimbra@mail-172 ~]$ cat /dev/null > /opt/zimbra/log/mailbox.log

[zimbra@mail-172 ~]$ zmrestore -a user1@mail-172.example.com -lb full-20150304.082015.535 -br -ca -pre restore2_

[zimbra@mail-172 ~]$ cat /opt/zimbra/log/mailbox.log
2015-03-05 23:39:30,176 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - AuthRequest elapsed=2
2015-03-05 23:39:30,263 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is starting
2015-03-05 23:39:30,352 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - waiting for FileLogWriter.FsyncThread-1425616571913 to finish.
2015-03-05 23:39:30,360 INFO  [FileLogWriter.FsyncThread-1425616571913] [] redolog - fsync thread exiting
2015-03-05 23:39:30,360 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - FileLogWriter.FsyncThread-1425616571913 finished
2015-03-05 23:39:30,371 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - Redo log rollover took 107ms
2015-03-05 23:39:30,371 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore started for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6
2015-03-05 23:39:30,377 INFO  [FileLogWriter.FsyncThread-1425616770369] [] redolog - Starting fsync thread with interval 10
2015-03-05 23:39:30,381 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - Creating database mboxgroup13
2015-03-05 23:39:30,992 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restoring to mailbox id 13 for new account 4c99a395-2c39-46ec-8c1b-285d97d899df.
2015-03-05 23:39:30,993 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loading schema with version 7.2
2015-03-05 23:39:30,997 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Renaming volume_id column to locator in source table mail_item
2015-03-05 23:39:31,022 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 1 blobs during restore
2015-03-05 23:39:31,023 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Deleting directory /opt/zimbra/index/0/13/index
2015-03-05 23:39:31,023 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 2 index files during restore
2015-03-05 23:39:31,024 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/13/index/0
2015-03-05 23:39:31,024 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox 13 account 4c99a395-2c39-46ec-8c1b-285d97d899df LOADED
2015-03-05 23:39:31,024 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] cache - initializing folder and tag caches for mailbox 13
2015-03-05 23:39:31,050 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Account user1@mail-172.example.com: All pending file IO completed (2 out of 2)
2015-03-05 23:39:31,050 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Run redo operations since last full backup
2015-03-05 23:39:31,053 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.

 [multiple repeats of this message]

2015-03-05 23:39:31,070 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
2015-03-05 23:39:31,075 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailop - Deleting Message (id=257).
2015-03-05 23:39:31,079 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.

 [multiple repeats of this message]

2015-03-05 23:39:31,829 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
2015-03-05 23:39:31,832 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore finished for user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox 6 to mailbox 13
2015-03-05 23:39:31,833 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is shut down
2015-03-05 23:39:31,834 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - RestoreRequest elapsed=1579
2015-03-05 23:39:40,932 INFO  [MailboxPurge] [name=user5@mail-172.example.com;mid=7;] purge - Purging messages.
[zimbra@mail-172 ~]$
Third Example - Options -lb , -ca , -pre restore3_ , -rf

For the third example, we'll again use the oldest full backup [full-20150304.082015.535 vs. full-20150306.043202.011] the account has with the -lb option. The message we had deleted, existed at the time of the full backup but did NOT in the last incremental backup. For the third example, we'll use the -rf option rather than the -br . We'll also be adjusting the -pre option to be restore3_ vs. restore_ or restore2_ in the other examples.

Then we'll review the mailbox.log events. Note the following:

  • Though the log will state the new mailboxId and zimbraId for the user, though:
    • "user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6" vs "Restoring to mailbox id 14 for new account d72a6d8d-8117-4a90-b8cf-f972f4af056d."
    • Notice the mailboxId and zimbraId for restore3_ is different than our other examples.
  • Since we INCLUDED the -rf option, the restore STOPS once the full backup is restored. Hence, we DON'T see the following line like we did in the other examples:
    • "backup - Run redo operations since last full backup"
  • And because it DIDN'T replay any of the redologs, there is NO reference to the following line either in the log:
    • "mailop - Deleting Message (id=257)."
  • NOR is there a reference to the following line either in the log:
    • "backup - Run redo operations from archived redo logs starting at sequence 17"
  • Towards the end, we'll get a confirmation that the restore has finished with some reference information again - mailboxId's .
    • "Restore finished for user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox 6 to mailbox 14"
[zimbra@mail-172 ~]$ cat /dev/null > /opt/zimbra/log/mailbox.log
[zimbra@mail-172 ~]$ zmrestore -a user1@mail-172.example.com -lb full-20150304.082015.535 -rf -ca -pre restore3_
[zimbra@mail-172 ~]$ cat /opt/zimbra/log/mailbox.log
2015-03-05 23:42:50,233 INFO  [Timer-Zimbra] [] SessionCache - Removed 1 idle sessions (SOAP). 1 active sessions remain.
2015-03-05 23:42:58,194 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - AuthRequest elapsed=3
2015-03-05 23:42:58,282 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is starting
2015-03-05 23:42:58,475 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - waiting for FileLogWriter.FsyncThread-1425616770369 to finish.
2015-03-05 23:42:58,477 INFO  [FileLogWriter.FsyncThread-1425616770369] [] redolog - fsync thread exiting
2015-03-05 23:42:58,477 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - FileLogWriter.FsyncThread-1425616770369 finished
2015-03-05 23:42:58,485 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - Redo log rollover took 193ms
2015-03-05 23:42:58,485 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore started for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6
2015-03-05 23:42:58,495 INFO  [FileLogWriter.FsyncThread-1425616978485] [] redolog - Starting fsync thread with interval 10
2015-03-05 23:42:58,515 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - Creating database mboxgroup14
2015-03-05 23:43:06,041 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restoring to mailbox id 14 for new account d72a6d8d-8117-4a90-b8cf-f972f4af056d.
2015-03-05 23:43:06,042 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loading schema with version 7.2
2015-03-05 23:43:06,048 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Renaming volume_id column to locator in source table mail_item
2015-03-05 23:43:06,092 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 1 blobs during restore
2015-03-05 23:43:06,093 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Deleting directory /opt/zimbra/index/0/14/index
2015-03-05 23:43:06,093 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 2 index files during restore
2015-03-05 23:43:06,095 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/14/index/0
2015-03-05 23:43:06,095 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox 14 account d72a6d8d-8117-4a90-b8cf-f972f4af056d LOADED
2015-03-05 23:43:06,095 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] cache - initializing folder and tag caches for mailbox 14
2015-03-05 23:43:06,264 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Account user1@mail-172.example.com: All pending file IO completed (2 out of 2)
2015-03-05 23:43:06,265 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore finished for user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox 6 to mailbox 14
2015-03-05 23:43:06,266 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is shut down
2015-03-05 23:43:06,266 INFO  [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - RestoreRequest elapsed=7999
[zimbra@mail-172 ~]$

Below is to confirm that indeed, it was only the restore with the -rf that did in fact have the message that was deleted after it.

[zimbra@mail-172 ~]$ zmmailbox -z -m restore_user1@mail-172.example.com s -t mess in:Inbox
num: 0, more: false

[zimbra@mail-172 ~]$ zmmailbox -z -m restore2_user1@mail-172.example.com s -t mess in:Inbox
num: 0, more: false

[zimbra@mail-172 ~]$ zmmailbox -z -m restore3_user1@mail-172.example.com s -t mess in:Inbox
num: 1, more: false

     Id  Type   From                  Subject                                             Date
   ----  ----   --------------------  --------------------------------------------------  --------------
1.  257  mess   admin                 Test1                                               02/23/15 09:03
Fourth Example - To Confirm -lb Label Does Matter

This example is to show that the -lb does indeed matter. Earlier examples used the oldest full we had that did include the deleted message in the account. We'll create a new full backup of the parent account, user1@ , where the message is absent and then do two restores using the differing full backup session labels with the proper option of -rf since we are wanting the restore to stop once the full session data is restored.

[zimbra@mail-172 ~]$ zmmailbox -z -m user1@mail-172.example.com s -t mess in:Inbox
num: 0, more: false

[zimbra@mail-172 ~]$ zmbackup -f -a all
full-20150306.055845.429

[zimbra@mail-172 ~]$ zmbackupquery -a user1@`zmhostname`
Account: user1@mail-172.example.com

    Label:   full-20150306.055845.429
    Type:    full
    Started: Fri, 2015/03/06 00:58:45.429 EST
    Ended:   Fri, 2015/03/06 00:58:52.357 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93

    Label:   incr-20150306.043321.683
    Type:    incremental
    Started: Thu, 2015/03/05 23:33:21.683 EST
    Ended:   Thu, 2015/03/05 23:33:27.495 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93

    Label:   full-20150306.043202.011
    Type:    full
    Started: Thu, 2015/03/05 23:32:02.011 EST
    Ended:   Thu, 2015/03/05 23:32:13.316 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93

    Label:   incr-20150305.060013.901
    Type:    incremental
    Started: Thu, 2015/03/05 01:00:13.901 EST
    Ended:   Thu, 2015/03/05 01:00:26.273 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93

    Label:   incr-20150304.085810.220
    Type:    incremental
    Started: Wed, 2015/03/04 03:58:10.220 EST
    Ended:   Wed, 2015/03/04 03:58:16.393 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93

    Label:   full-20150304.082015.535
    Type:    full
    Started: Wed, 2015/03/04 03:20:15.535 EST
    Ended:   Wed, 2015/03/04 03:20:39.109 EST
    Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93

[zimbra@mail-172 ~]$ zmrestore -a user1@mail-172.example.com -lb full-20150306.055845.429 -rf -ca -pre restore4_

[zimbra@mail-172 ~]$ zmrestore -a user1@mail-172.example.com -lb full-20150304.082015.535 -rf -ca -pre restore5_

[zimbra@mail-172 ~]$ zmmailbox -z -m restore4_user1@mail-172.example.com s -t mess in:Inbox
num: 0, more: false

[zimbra@mail-172 ~]$ zmmailbox -z -m restore5_user1@mail-172.example.com s -t mess in:Inbox
num: 1, more: false

     Id  Type   From                  Subject                                             Date
   ----  ----   --------------------  --------------------------------------------------  --------------
1.  257  mess   admin                 Test1                                               02/23/15 09:03

[zimbra@mail-172 ~]$

What Is Different Between The Parent Account And The Restored Accounts Where We Used The -ca And -pre Options [LDAP]

Below is a difference between the zmprov ga output between the parent account [user1@] and our last restore example [restore5_user1@]. Note, in our example here the parent account was created with default values and there was no customization's or changes to the defaults.

[zimbra@mail-172 ~]$ zmprov ga user1@`zmhostname` > /tmp/user1

[zimbra@mail-172 ~]$ zmprov ga restore5_user1@`zmhostname` > /tmp/restore5

[zimbra@mail-172 ~]$ diff /tmp/user1 /tmp/restore5
1c1
< # name user1@mail-172.example.com
---
> # name restore5_user1@mail-172.example.com
3c3
< mail: user1@mail-172.example.com
---
> mail: restore5_user1@mail-172.example.com
8c8
< uid: user1
---
> uid: restore5_user1
20,22c20
< zimbraAuthTokens: 661297364|1425702763611|8.6.0_GA_1153
< zimbraAuthTokens: 414162390|1425702787760|8.6.0_GA_1153
< zimbraAuthTokens: 2046929787|1425707910566|8.6.0_GA_1153
---
> zimbraAuthTokens: 1030982216|1425708106344|8.6.0_GA_1153
159c157
< zimbraId: 3c397edc-c015-4d82-ae8a-1084692f8a93
---
> zimbraId: e5c38a4b-6bea-45bb-9e14-dd3f39159a61
170c168
< zimbraMailDeliveryAddress: user1@mail-172.example.com
---
> zimbraMailDeliveryAddress: restore5_user1@mail-172.example.com

Restore encountered redo log sequence error: Found gap in redo log sequence

Example error:

Error occurred: system failure: Restore encountered redo log sequence error: 
 Found gap in redo log sequence; missing 3899 through 3895; To avoid future restore problems, 
 discard all existing backups and take a full backup of all accounts; If this error occurred 
 during restore, try the --ignoreRedoErrors option

This error actually gives the two basic options you have when you encounter this. Depending on your circumstances, you might first do the restore using the --ignoreRedoErrors options and recovery what data you can prior to discarding the 'bad' backup sessions in question and attempting a new full backup of the users.


How-To Example - Extended Look At RedoErrors With Log Events

First, let's replicate a redolog failure situation, which can be done by moving the redologs from the /opt/zimbra/redolog/archive/ to a temporary directory.

[zimbra@mail-172 redolog]$ cd archive/

[zimbra@mail-172 archive]$ ls
redo-20150306.060015.331-seq21.log  redo-20150306.060109.166-seq22.log

[zimbra@mail-172 archive]$ mkdir /tmp/redolog-archive

[zimbra@mail-172 archive]$ mv * /tmp/redolog-archive/

[zimbra@mail-172 archive]$ cat /dev/null > /opt/zimbra/log/mailbox.log

Now that we've moved the redolog sequence files out of the redolog/archive directory path, we can attempt our zmrestore command. Note that the incremental backup label for this attempt is called incr-20150306.170810.793 . Though it fails because of the missing redologs, the backup is still created.

[zimbra@mail-172 archive]$ zmbackup -i
Error occurred: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option

[zimbra@mail-172 archive]$ cat /opt/zimbra/log/mailbox.log
2015-03-06 12:08:06,442 INFO  [MailboxPurge] [name=ham.jgfdb3oqjf@mail-172.example.com;mid=4;] purge - Purging messages.
2015-03-06 12:08:10,350 INFO  [qtp509886383-484:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] soap - AuthRequest elapsed=4
2015-03-06 12:08:10,413 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup request started
2015-03-06 12:08:10,579 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Found 16 accounts on server mail-172.example.com
2015-03-06 12:08:10,797 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] io - AsyncFileCopier is starting
2015-03-06 12:08:10,798 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup started for backup set; label: incr-20150306.170810.793
[cut]
2015-03-06 12:08:11,277 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup is started for account user1@mail-172.example.com
2015-03-06 12:08:11,277 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Saving account information from LDAP
2015-03-06 12:08:11,281 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup has ended
2015-03-06 12:08:11,281 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Account user1@mail-172.example.com in backup set incr-20150306.170810.793: All pending file IO completed (0 out of 0)
[cut]
2015-03-06 12:08:11,310 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup is started for account admin@mail-172.example.com
2015-03-06 12:08:11,310 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Saving account information from LDAP
2015-03-06 12:08:11,312 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup has ended
2015-03-06 12:08:11,312 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Account admin@mail-172.example.com in backup set incr-20150306.170810.793: All pending file IO completed (0 out of 0)
2015-03-06 12:08:11,711 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] redolog - waiting for FileLogWriter.FsyncThread-1425621669168 to finish.
2015-03-06 12:08:11,712 INFO  [FileLogWriter.FsyncThread-1425621669168] [] redolog - fsync thread exiting
2015-03-06 12:08:11,712 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] redolog - FileLogWriter.FsyncThread-1425621669168 finished
2015-03-06 12:08:11,799 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] redolog - Redo log rollover took 487ms
2015-03-06 12:08:11,799 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Found 1 redo logs to backup
2015-03-06 12:08:11,800 INFO  [FileLogWriter.FsyncThread-1425661691796] [] redolog - Starting fsync thread with interval 10
2015-03-06 12:08:11,864 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Last backed-up redo log sequence = 21
2015-03-06 12:08:11,864 ERROR [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option
2015-03-06 12:08:11,973 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Copying redo log /opt/zimbra/redolog/archive/redo-20150306.170811.794-seq23.log to /opt/zimbra/backup/tmp/incr-20150306.170810.793/redologs/redo-20150306.170811.794-seq23.log
2015-03-06 12:08:12,100 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Retaining recent redo log /opt/zimbra/redolog/archive/redo-20150306.170811.794-seq23.log
2015-03-06 12:08:19,375 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup set incr-20150306.170810.793: All pending file IO completed (0 out of 0)
2015-03-06 12:08:19,378 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] io - AsyncFileCopier is shut down
2015-03-06 12:08:19,378 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup finished for backup set; label: incr-20150306.170810.793
2015-03-06 12:08:20,565 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] SoapEngine - handler exception
com.zimbra.cs.backup.BackupServiceException: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option
ExceptionId:qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest:1425661691864:14586f805b4c1b0e
Code:backup.REDOLOG_OUT_OF_SEQUENCE
        at com.zimbra.cs.backup.BackupServiceException.REDOLOG_OUT_OF_SEQUENCE(BackupServiceException.java:91)
        at com.zimbra.cs.backup.util.Utils.splitRedoLogsAtSeq(Utils.java:302)
        at com.zimbra.cs.backup.FileBackupTarget$FileBackupSet.backupRedoLogs(FileBackupTarget.java:1360)
        at com.zimbra.cs.backup.BackupSet.startIncrementalBackup(BackupSet.java:817)
        at com.zimbra.cs.backup.FileBackupTarget$FileBackupSet.startIncrementalBackup(FileBackupTarget.java:1067)
        at com.zimbra.cs.backup.BackupManager.backupIncremental(BackupManager.java:336)
        at com.zimbra.cs.service.backup.Backup.handleNetworkRequest(Backup.java:153)
        at com.zimbra.cs.service.NetworkDocumentHandler.handle(NetworkDocumentHandler.java:23)
        at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEngine.java:569)
        at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:432)
        at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:266)
        at com.zimbra.soap.SoapServlet.doWork(SoapServlet.java:303)
        at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:213)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
        at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:209)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651)
        at com.zimbra.cs.servlet.RequestStringFilter.doFilter(RequestStringFilter.java:54)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:59)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
        at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:351)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ETagHeaderFilter.doFilter(ETagHeaderFilter.java:47)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ContextPathBasedThreadPoolBalancerFilter.doFilter(ContextPathBasedThreadPoolBalancerFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(ZimbraQoSFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ZimbraInvalidLoginFilter.doFilter(ZimbraInvalidLoginFilter.java:117)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:457)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:326)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:299)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:544)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:199)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:309)
        at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:81)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:462)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
        at java.lang.Thread.run(Thread.java:745)
2015-03-06 12:08:20,566 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] soap - BackupRequest elapsed=10154

To confirm the backup label was made and also to see how the zmbackupquery reports it, we can do the following:

[zimbra@mail-172 archive]$ zmbackupquery -lb incr-20150306.170810.793 -v
Label:   incr-20150306.170810.793
Type:    incremental
Status:  completed (with errors)
Started: Fri, 2015/03/06 12:08:10.793 EST
Ended:   Fri, 2015/03/06 12:08:19.378 EST
Redo log sequence range: 23 .. 23
Number of accounts: 14 out of 14 completed
Number of errors: 1
Accounts:
  admin@mail-172.example.com: completed
  galsync.qva4ekog0@mail-172.example.com: completed
  ham.jgfdb3oqjf@mail-172.example.com: completed
  restore2_user1@mail-172.example.com: completed
  restore3_user1@mail-172.example.com: completed
  restore_user1@mail-172.example.com: completed
  spam.umhx3owv3s@mail-172.example.com: completed
  user1@mail-172.example.com: completed
  user2@mail-172.example.com: completed
  user3@mail-172.example.com: completed
  user4@mail-172.example.com: completed
  user5@mail-172.example.com: completed
  user6@mail-172.example.com: completed
  virus-quarantine.gusiso1cd@mail-172.example.com: completed
Error: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option
com.zimbra.cs.backup.BackupServiceException: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option
ExceptionId:qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest:1425661691864:14586f805b4c1b0e
Code:backup.REDOLOG_OUT_OF_SEQUENCE
        at com.zimbra.cs.backup.BackupServiceException.REDOLOG_OUT_OF_SEQUENCE(BackupServiceException.java:91)
        at com.zimbra.cs.backup.util.Utils.splitRedoLogsAtSeq(Utils.java:302)
        at com.zimbra.cs.backup.FileBackupTarget$FileBackupSet.backupRedoLogs(FileBackupTarget.java:1360)
        at com.zimbra.cs.backup.BackupSet.startIncrementalBackup(BackupSet.java:817)
        at com.zimbra.cs.backup.FileBackupTarget$FileBackupSet.startIncrementalBackup(FileBackupTarget.java:1067)
        at com.zimbra.cs.backup.BackupManager.backupIncremental(BackupManager.java:336)
        at com.zimbra.cs.service.backup.Backup.handleNetworkRequest(Backup.java:153)
        at com.zimbra.cs.service.NetworkDocumentHandler.handle(NetworkDocumentHandler.java:23)
        at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEngine.java:569)
        at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:432)
        at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:266)
        at com.zimbra.soap.SoapServlet.doWork(SoapServlet.java:303)
        at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:213)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
        at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:209)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651)
        at com.zimbra.cs.servlet.RequestStringFilter.doFilter(RequestStringFilter.java:54)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:59)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
        at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:351)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ETagHeaderFilter.doFilter(ETagHeaderFilter.java:47)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ContextPathBasedThreadPoolBalancerFilter.doFilter(ContextPathBasedThreadPoolBalancerFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(ZimbraQoSFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ZimbraInvalidLoginFilter.doFilter(ZimbraInvalidLoginFilter.java:117)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:457)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:326)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:299)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:544)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:199)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:309)
        at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:81)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:462)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
        at java.lang.Thread.run(Thread.java:745)
Total space: 14020MB
 Free space: 6031MB
[zimbra@mail-172 archive]$

Let's now try doing a restore of an account.

[zimbra@mail-172 archive]$ cat /dev/null > /opt/zimbra/log/mailbox.log

[zimbra@mail-172 archive]$ zmrestore -a user1@`zmhostname` -ca -pre restore6_
Error occurred: system failure: Restore encountered redo log sequence error: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option

[zimbra@mail-172 archive]$ cat /opt/zimbra/log/mailbox.log
2015-03-06 12:12:31,884 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - RestoreRequest elapsed=38
2015-03-06 12:13:06,650 INFO  [MailboxPurge] [name=user3@mail-172.example.com;mid=9;] purge - Purging messages.
2015-03-06 12:13:45,329 INFO  [qtp509886383-484:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - AuthRequest elapsed=2
2015-03-06 12:13:45,474 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is starting
2015-03-06 12:13:45,512 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Using full backup full-20150306.055845.429 as starting point for restoring account 3c397edc-c015-4d82-ae8a-1084692f8a93
2015-03-06 12:13:45,662 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - waiting for FileLogWriter.FsyncThread-1425661691796 to finish.
2015-03-06 12:13:45,669 INFO  [FileLogWriter.FsyncThread-1425661691796] [] redolog - fsync thread exiting
2015-03-06 12:13:45,669 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - FileLogWriter.FsyncThread-1425661691796 finished
2015-03-06 12:13:45,678 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - Redo log rollover took 166ms
2015-03-06 12:13:45,678 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore started for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6
2015-03-06 12:13:45,700 INFO  [FileLogWriter.FsyncThread-1425662025678] [] redolog - Starting fsync thread with interval 10
2015-03-06 12:13:45,981 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - Creating database mboxgroup17
2015-03-06 12:13:46,695 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restoring to mailbox id 17 for new account f7f4c1a8-15d0-41a1-bd11-6e05359ea3e8.
2015-03-06 12:13:46,745 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loading schema with version 7.2
2015-03-06 12:13:46,785 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Renaming volume_id column to locator in source table mail_item
2015-03-06 12:13:46,820 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 0 blobs during restore
2015-03-06 12:13:46,822 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Deleting directory /opt/zimbra/index/0/17/index
2015-03-06 12:13:46,882 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 2 index files during restore
2015-03-06 12:13:46,887 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/17/index/0
2015-03-06 12:13:46,887 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox 17 account f7f4c1a8-15d0-41a1-bd11-6e05359ea3e8 LOADED
2015-03-06 12:13:46,887 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] cache - initializing folder and tag caches for mailbox 17
2015-03-06 12:13:46,912 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Account user1@mail-172.example.com: All pending file IO completed (2 out of 2)
2015-03-06 12:13:46,912 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Run redo operations since last full backup
2015-03-06 12:13:47,047 ERROR [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option
2015-03-06 12:13:47,047 WARN  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Redo log sequence error: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option
com.zimbra.cs.backup.BackupServiceException: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option
ExceptionId:qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest:1425662027047:14586f805b4c1b0e
Code:backup.REDOLOG_OUT_OF_SEQUENCE
        at com.zimbra.cs.backup.BackupServiceException.REDOLOG_OUT_OF_SEQUENCE(BackupServiceException.java:91)
        at com.zimbra.cs.backup.util.Utils.splitRedoLogsAtSeq(Utils.java:302)
        at com.zimbra.cs.backup.RestoreAccountSession.checkRedoLogSequenceContiguity(RestoreAccountSession.java:534)
        at com.zimbra.cs.backup.RestoreAccountSession.endRestore(RestoreAccountSession.java:350)
        at com.zimbra.cs.backup.FileBackupTarget$RestoreAcctSession.endRestore(FileBackupTarget.java:2524)
        at com.zimbra.cs.backup.FileBackupTarget.restore(FileBackupTarget.java:490)
        at com.zimbra.cs.backup.BackupManager.restore(BackupManager.java:826)
        at com.zimbra.cs.service.backup.Restore.handle(Restore.java:137)
        at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEngine.java:569)
        at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:432)
        at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:266)
        at com.zimbra.soap.SoapServlet.doWork(SoapServlet.java:303)
        at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:213)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
        at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:209)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651)
        at com.zimbra.cs.servlet.RequestStringFilter.doFilter(RequestStringFilter.java:54)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:59)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
        at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:351)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ETagHeaderFilter.doFilter(ETagHeaderFilter.java:47)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ContextPathBasedThreadPoolBalancerFilter.doFilter(ContextPathBasedThreadPoolBalancerFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(ZimbraQoSFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ZimbraInvalidLoginFilter.doFilter(ZimbraInvalidLoginFilter.java:117)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:457)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:326)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:299)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:544)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:199)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:309)
        at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:81)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:462)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
        at java.lang.Thread.run(Thread.java:745)
2015-03-06 12:13:47,047 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Cleaning up data from failed restore (mid=17)
2015-03-06 12:13:47,063 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - Putting mailbox 17 under maintenance.
2015-03-06 12:13:47,084 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - clearing contents of mailbox 17, group 17
2015-03-06 12:13:47,331 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox for account f7f4c1a8-15d0-41a1-bd11-6e05359ea3e8 DELETED
2015-03-06 12:13:47,375 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Cleaning up account created during failed restore (account=f7f4c1a8-15d0-41a1-bd11-6e05359ea3e8)
2015-03-06 12:13:47,752 WARN  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Error occurred during restore account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93)
com.zimbra.common.service.ServiceException: system failure: Restore encountered redo log sequence error: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option
ExceptionId:qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest:1425662027047:14586f805b4c1b0e
Code:service.FAILURE
        at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:260)
        at com.zimbra.cs.backup.RestoreAccountSession.checkRedoLogSequenceContiguity(RestoreAccountSession.java:545)
        at com.zimbra.cs.backup.RestoreAccountSession.endRestore(RestoreAccountSession.java:350)
        at com.zimbra.cs.backup.FileBackupTarget$RestoreAcctSession.endRestore(FileBackupTarget.java:2524)
        at com.zimbra.cs.backup.FileBackupTarget.restore(FileBackupTarget.java:490)
        at com.zimbra.cs.backup.BackupManager.restore(BackupManager.java:826)
        at com.zimbra.cs.service.backup.Restore.handle(Restore.java:137)
        at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEngine.java:569)
        at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:432)
        at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:266)
        at com.zimbra.soap.SoapServlet.doWork(SoapServlet.java:303)
        at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:213)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
        at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:209)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651)
        at com.zimbra.cs.servlet.RequestStringFilter.doFilter(RequestStringFilter.java:54)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:59)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
        at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:351)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ETagHeaderFilter.doFilter(ETagHeaderFilter.java:47)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ContextPathBasedThreadPoolBalancerFilter.doFilter(ContextPathBasedThreadPoolBalancerFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(ZimbraQoSFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ZimbraInvalidLoginFilter.doFilter(ZimbraInvalidLoginFilter.java:117)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:457)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:326)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:299)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:544)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:199)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:309)
        at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:81)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:462)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
        at java.lang.Thread.run(Thread.java:745)
Caused by: com.zimbra.cs.backup.BackupServiceException: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option
ExceptionId:qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest:1425662027047:14586f805b4c1b0e
Code:backup.REDOLOG_OUT_OF_SEQUENCE
        at com.zimbra.cs.backup.BackupServiceException.REDOLOG_OUT_OF_SEQUENCE(BackupServiceException.java:91)
        at com.zimbra.cs.backup.util.Utils.splitRedoLogsAtSeq(Utils.java:302)
        at com.zimbra.cs.backup.RestoreAccountSession.checkRedoLogSequenceContiguity(RestoreAccountSession.java:534)
        ... 56 more
2015-03-06 12:13:47,755 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is shut down
2015-03-06 12:13:47,755 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] SoapEngine - handler exception
com.zimbra.common.service.ServiceException: system failure: Restore encountered redo log sequence error: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option
ExceptionId:qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest:1425662027047:14586f805b4c1b0e
Code:service.FAILURE
        at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:260)
        at com.zimbra.cs.backup.RestoreAccountSession.checkRedoLogSequenceContiguity(RestoreAccountSession.java:545)
        at com.zimbra.cs.backup.RestoreAccountSession.endRestore(RestoreAccountSession.java:350)
        at com.zimbra.cs.backup.FileBackupTarget$RestoreAcctSession.endRestore(FileBackupTarget.java:2524)
        at com.zimbra.cs.backup.FileBackupTarget.restore(FileBackupTarget.java:490)
        at com.zimbra.cs.backup.BackupManager.restore(BackupManager.java:826)
        at com.zimbra.cs.service.backup.Restore.handle(Restore.java:137)
        at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEngine.java:569)
        at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:432)
        at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:266)
        at com.zimbra.soap.SoapServlet.doWork(SoapServlet.java:303)
        at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:213)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
        at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:209)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651)
        at com.zimbra.cs.servlet.RequestStringFilter.doFilter(RequestStringFilter.java:54)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:59)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
        at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:351)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ETagHeaderFilter.doFilter(ETagHeaderFilter.java:47)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ContextPathBasedThreadPoolBalancerFilter.doFilter(ContextPathBasedThreadPoolBalancerFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(ZimbraQoSFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ZimbraInvalidLoginFilter.doFilter(ZimbraInvalidLoginFilter.java:117)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:457)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:326)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:299)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:544)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:199)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:309)
        at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:81)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:462)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
        at java.lang.Thread.run(Thread.java:745)
Caused by: com.zimbra.cs.backup.BackupServiceException: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option
ExceptionId:qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest:1425662027047:14586f805b4c1b0e
Code:backup.REDOLOG_OUT_OF_SEQUENCE
        at com.zimbra.cs.backup.BackupServiceException.REDOLOG_OUT_OF_SEQUENCE(BackupServiceException.java:91)
        at com.zimbra.cs.backup.util.Utils.splitRedoLogsAtSeq(Utils.java:302)
        at com.zimbra.cs.backup.RestoreAccountSession.checkRedoLogSequenceContiguity(RestoreAccountSession.java:534)
        ... 56 more
2015-03-06 12:13:47,755 INFO  [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - RestoreRequest elapsed=2358

[zimbra@mail-172 archive]$ zmprov ga restore6_user1@`zmhostname` | head
ERROR: account.NO_SUCH_ACCOUNT (no such account: restore6_user1@mail-172.example.com)

Let's now see what happens if we move the missing redolog seq files back into /opt/zimbra/redolog/archive and then if we move/copy them into the backup sessions redolog directory. With restore attempts being done after each move/copy.

[zimbra@mail-172 archive]$ mv /tmp/redolog-archive/* ./

[zimbra@mail-172 archive]$ pwd
/opt/zimbra/redolog/archive

[zimbra@mail-172 archive]$ ls
redo-20150306.060015.331-seq21.log  redo-20150306.060109.166-seq22.log  redo-20150306.170811.794-seq23.log  redo-20150306.171345.676-seq24.log

[zimbra@mail-172 archive]$ zmrestore -a user1@`zmhostname` -ca -pre restore6_
Error occurred: system failure: Restore encountered redo log sequence error: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option

[zimbra@mail-172 archive]$ pwd
/opt/zimbra/redolog/archive

[zimbra@mail-172 archive]$ cp *seq21.log *seq22.log /opt/zimbra/backup/sessions/incr-20150306.170810.793/redologs/

[zimbra@mail-172 archive]$ zmrestore -a user1@`zmhostname` -ca -pre restore6_

[zimbra@mail-172 archive]$ zmprov ga restore6_user1@`zmhostname` | head -2
# name restore6_user1@mail-172.example.com
cn: user1

Redirected Restore With No Deletes To Then Import Data From A Specific Time Range Into A Subdirectory Of Users Archive Account

This example would be the steps one might do if you discovered that the archiving service wasn't working as expected for a given date range and you wanted to populate the users archive account with the missing data. Normally, this would involve many users or all of your users - hence, why the issue about licenses is addressed.

Note, I use the following variables below but you should be replacing them to suit your needs.

  • $USER = users production email
  • recovery_$USER = the new email address that the restore created [-ca -pre recovery_]
  • $DOMAIN = the domain for the user
  • $USER-archive@$DOMAIN.archive = the production users archive email address

Determine all the necessary variables for your restore command.

  • zmrestore -a $USER
    • -ca -pre recovery_
    • -lb determine what full backup label to use
    • determine if you need to state the incremental backup label [-restoreToIncrLabel] and/or a specific time to stop the restore at [-restoreToTime]
    • determine if you need to use -rf or -br
    • --skipDeletes

For example:

 $ zmrestore -a user1@`zmhostname` -ca -pre recovery_ -lb full-20150321.050015.999 -restoreToTime 20150323000000 -br --skipDeletes

using restore to time of 2015/03/23 00:00:00

Set the restored account to not consume a normal license [notice recovery_$USER vs $USER]:

$ zmprov ma recovery_$USER zimbraIsSystemResource TRUE

Get what the archive accounts address is for the main account:

$ zmprov ga $USER amavisArchiveQuarantineTo

Set the restored account to not consume an archive license [notice recovery_$USER vs $USER]:

$ zmprov ma recovery_$USER amavisArchiveQuarantineTo ""

Create a directory to dump all the export data in. You might want to have this shared across your mailstores [that hold archive accounts] so you can then import them locally to the mailstore the archive account resides on.

$ mkdir /tmp/export

$ cd /tmp/export

Example exports out all data between a given date , adjust the account and date string below. Also, the filename that is being saved.

$ zmmailbox -z -m recovery_$USER@$DOMAIN gru '//?fmt=tgz&query=under:/ after:"4/8/15" AND before:"4/11/15"' > /tmp/export/$USER.tgz

Example of what is dumped

$ pwd
/tmp/export

$ ls
admin.tgz

$ tar zxvf $USER.tgz
Inbox/0000000776-ZCS Backup Report_ SUCCESS.eml.meta
Inbox/0000000776-ZCS Backup Report_ SUCCESS.eml
Inbox/0000000777-Daily mail report for 2015-04-09.eml.meta
Inbox/0000000777-Daily mail report for 2015-04-09.eml
Inbox/0000000778-ZCS Backup Report_ SUCCESS.eml.meta
Inbox/0000000778-ZCS Backup Report_ SUCCESS.eml
Inbox/0000000779-Daily mail report for 2015-04-10.eml.meta
Inbox/0000000779-Daily mail report for 2015-04-10.eml

Now to import the data into the users Archive account. Below example will dump the imported data into a specfic folder that will be created [I'm using Imported]. Adjust the account and filename below.

$ zmmailbox -z -m $USER-archive@$DOMAIN.archive pru "//?fmt=tgz&subfolder=Imported" /tmp/export/$USER.tgz

$ zmmailbox -z -m $USER-archive@$DOMAIN.archive gaf

        Id  View      Unread   Msg Count  Path
----------  ----  ----------  ----------  ----------
         1  unkn           0           0  /
        16  docu           0           0  /Briefcase
        10  appo           0           0  /Calendar
        14  mess           0           0  /Chats
         7  cont           0           0  /Contacts
         6  mess           0           0  /Drafts
        13  cont           0           0  /Emailed Contacts
       789  unkn           0           0  /Imported
       790  mess           4           4  /Imported/Inbox
         2  mess         415         420  /Inbox
         4  mess           0           0  /Junk
         5  mess           0           0  /Sent
        15  task           0           0  /Tasks
         3  unkn           0           0  /Trash

Here's a (customer-supplied) script that does the above in batch. It was used to restore archives from a date range for 29k users: https://github.com/morganlljones/zimbra_backup2archive

Quiz

  1. What are three methods you can locate which backups a user is in?
  2. To stop a restore from playing all the way up to the current time, you would need to use one of these two zmrestore variables - what are they?
  3. If you see an error about being unable to restore because of redologs, what option might you need to include?
  4. What is the primary log file on the ZCS server you'll monitor or review for restore or backup issues?
  5. Provide two methods to increase logging/debug output when attempting a restore?
  6. Can you perform a full system restore from the backups done on an older ZCS version?
  7. What are the two identify variables that zmbackup/zmrestore uses for an individual?
  8. What user variable would you use to locate the directory a users data is stored in within a backup session?
Verified Against: Zimbra Collaboration Suite 8.6 Date Created: 01/22/2015
Article ID: https://wiki.zimbra.com/index.php?title=Troubleshooting_Course_Content_Rough_Drafts-Recover_Missing_Data_-_User Date Modified: 2016-06-07



Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »

Jump to: navigation, search