Troubleshooting Course Content Rough Drafts-Recover Missing Data - User: Difference between revisions
mNo edit summary |
|||
(129 intermediate revisions by 3 users not shown) | |||
Line 1: | Line 1: | ||
__FORCETOC__ | |||
==Bug & RFE's I Made While Researching This Draft Write Up [Remove For Course]== | |||
== | * [story] Ability to search data within backup and do "item" restores or identify locations of search results | ||
** https://bugzilla.zimbra.com/show_bug.cgi?id=97551 | |||
* admin console backup label view doesn't list accounts in the all accounts tab | |||
** https://bugzilla.zimbra.com/show_bug.cgi?id=97555 | |||
* admin console restore - doesn't autocomplete / suggest account matches when filling out email address box | |||
** https://bugzilla.zimbra.com/show_bug.cgi?id=97556 | |||
* document new restore functions / options with ZCS 8+ for admin console restore | |||
** https://bugzilla.zimbra.com/show_bug.cgi?id=97558 | |||
* admin console restore - rename "Selected Servers" panel to "Restore Options" | |||
** https://bugzilla.zimbra.com/show_bug.cgi?id=97559 | |||
*admin console restore - if only one mailstore in env. then state such in second panel of restore about "server for the restored accounts" | |||
** https://bugzilla.zimbra.com/show_bug.cgi?id=97560 | |||
*admin console restore - expand restore To options - To full backup label, To incremental target | |||
** https://bugzilla.zimbra.com/show_bug.cgi?id=97561 | |||
*admin console restore - "restore to the latest backup" incorrectly described / broken | |||
** https://bugzilla.zimbra.com/show_bug.cgi?id=97562 | |||
* admin console restore - unable to restore individual accounts [sort of] | |||
** https://bugzilla.zimbra.com/show_bug.cgi?id=97564 | |||
* admin console restore - reuse GAL/Contact Picker Window for "restore individual accounts" | |||
** https://bugzilla.zimbra.com/show_bug.cgi?id=97565 | |||
* Default COS value [ zimbraPrefShowSelectionCheckbox ] for checkboxes in lists should be TRUE and improve 'select all' functionality | |||
** https://bugzilla.zimbra.com/show_bug.cgi?id=97554 | |||
* zmschedulebackup ignores zmbackup options in auto-grouped mode | |||
** https://bugzilla.zimbra.com/show_bug.cgi?id=97154 | |||
== | ==Introduction== | ||
* ''Will you please provide a brief paragraph introduction/overview about recovering missing data in Zimbra.'' | |||
* | * ''For example: what are some scenarios wherein you need to recover missing data. '' | ||
* | * ''What is the general method for recovery? '' | ||
* | * ''How/Why is the backup environment relevant/important. '' | ||
* ''Remember that this won't be obvious for all audience members.'' | |||
== | ==User Issues== | ||
===Questions To Ask & Address Prior To Doing A User Restore=== | |||
* | * Can data be restored from the user's dumpster [if enabled] ? | ||
** | * Is there a copy of the data in a third party client ? [POP email client that downloaded it, for example] | ||
* Does the data maybe exist in another user's account ? [For example, the senders account or others in the To/CC] | |||
* Does the data reside in a redolog/s ? | |||
* Finally, restore from backups ? | |||
** Do I need to overwrite the existing account with the restore or should I restore to a new account name? | |||
*** Using the option -ca [create account] and -pre restore_ [prefix to the new account name, restore_user@ ] | |||
** What are your retention policies and how it might determine the backup set you need? | |||
===User Restores=== | |||
* Admin Console | |||
: | ** Restore/Backup | ||
** Redirected restore : -ca -pre restored_ | |||
** View Mail > TGZ export/import | |||
* Crossmailboxsearch | |||
* Importing Restored Data Into Parent Account | |||
====Restoring A User To A Specific Full Backup, Incremental Backup, Or Point In Time==== | |||
One of the more common support issues around restoring a user account is the complaint that the restore didn't roll back the account to the desired state. Almost always, this is because the proper options were not passed to the zmrestore command for the restore to stop at the desired time frame. If you don't give the proper options to zmrestore, it will restore the account all the way to the most recent redolog operations on the server. For example, it replays the delete operations that caused the user to request to restore in the first place. | |||
You '''MUST''' use the -lb [full backup label name] option when your trying to restore anything that ISN'T meant to include the latest information for the mailbox. '''''The -lb argument should specify a full backup that took place prior to the time of the backup you wish to restore.''''' | |||
=====Find Out What Backup Session Labels You Need First===== | |||
To find out what backups are associated with a particular account, you would do the following : | |||
zmbackupquery -a user@domain | |||
You'll want to note what is the first full that occurs before the point in time you want to restore. And then the incremental that follows right after your point in time. | |||
Backup labels (-lb) for fulls for a particular user can be found by doing : | |||
zmbackupquery -a user@domain --type full | |||
Backup labels (-restoreToIncrLabel) for incrementals can be found by doing : | |||
zmbackupquery -a user@domain --type incremental | |||
=====Command Syntax Example For Restores On The CLI===== | |||
Example to restore ONLY the data in the full backup session and stop : | |||
zmrestore -a USER@DOMAIN.com -lb full-20080726.050017.306 -rf -ca -pre restore_ | |||
Example to restore only the data in the full backup session and all the incremental data after the full to a specific incremental label and stop : | |||
zmrestore -a USER@DOMAIN.com -restoreToIncrLabel incr-20080731.060007.644 -lb full-20080726.050017.306 -br -ca -pre restore_ | |||
Example to restore to a specific time and stop : | |||
zmrestore -a USER@DOMAIN.com -restoreToTime 20080801011800 -lb full-20080726.050017.306 -br -ca -pre restore_ | |||
'''Important Options You Might Want Or Need To Include''' | |||
'''--ignoreRedoErrors''' : If you attempt a restore and you see an error about problems related to playing the redolog, you'll want to run the restore command again and include this option. | |||
= | '''--skipDeletes''' : Please see http://bugzilla.zimbra.com/show_bug.cgi?id=31824#c5 for details on this. | ||
'''-t /path/to/backup_dir''' : If you are restoring from another backup directory besides your current default path. | |||
Variables that are asking for TIME rather than LABELS should follow this syntax (from zmrestore --help): | |||
<pre> | |||
Specify date/time in one of these formats: | |||
2008/08/06 09:55:50 | |||
2008/08/06 09:55:50 572 | |||
2008/08/06 09:55:50.572 | |||
2008/08/06-09:55:50-572 | |||
2008/08/06-09:55:50 | |||
20080806.095550.572 | |||
20080806.095550 | |||
20080806095550572 | |||
20080806095550 | |||
Specify year, month, date, hour, minute, second, and optionally millisecond. | |||
Month/date/hour/minute/second are 0-padded to 2 digits, millisecond to 3 digits. | |||
Hour must be specified in 24-hour format, and time is in local time zone. | |||
</pre> | |||
==Trouble-shooting Backup/Restore Issues And Other General Questions== | ==Trouble-shooting Backup/Restore Issues And Other General Questions== | ||
Line 163: | Line 129: | ||
===Backup And Restore Compatibility Between ZCS Versions=== | ===Backup And Restore Compatibility Between ZCS Versions=== | ||
''' | * You can use backups from older versions of ZCS for account restores. | ||
* You can '''NOT''' use backups from old versions of ZCS for system restores though. Disaster Recovery restores have to be done with the same version of ZCS as the backups were done with. | |||
References: | References: | ||
Line 250: | Line 217: | ||
</pre> | </pre> | ||
Note, depending on your version of ZCS, you might be hitting a bug where you'll keep getting emails until a logrotate happens. | |||
* Changing Zmstat-df values do not take affect until logrotate | * Changing Zmstat-df values do not take affect until logrotate | ||
Line 371: | Line 338: | ||
** And any other mailbox.log file that would cover the event | ** And any other mailbox.log file that would cover the event | ||
==== | ====Some Specific Example Restore/Backup Issues==== | ||
=====NO_SUCH_ACCOUNT_BACKUP===== | |||
Example errors: | |||
<pre> | |||
Error code: backup.NO_SUCH_ACCOUNT_BACKUP | |||
Message: no such backup for account: Missing full backup earlier than restore-to time for account 3c397edc-c015-4d82-ae8a-1084692f8a93 | |||
Details:soap:Sender | |||
Error occurred: no such backup for account: 3c397edc-c015-4d82-ae8a-1084692f8a93 | |||
Message: no such backup for account: Missing full backup earlier than restore-to time for account 3c397edc-c015-4d82-ae8a-1084692f8a93 | |||
3c397edc-c015-4d82-ae8a-1084692f8a93 is an example of the zimbraId of a user [ user1@mail-172.example.com ] . | |||
</pre> | |||
Confirm the current email address is mapped to the appropriate zimbraId. You'll want to search what is in ldap and the mailstores database for the exiting email address and then also search the reference files in your backups. Often times, this happens when an account was deleted and then created again. Account meaning the email address. When this is does, the new account will have a new zimbraId. | |||
<pre> | |||
$ zmprov -l ga user1@mail-172.example.com zimbraId | |||
# name user1@mail-172.example.com | |||
zimbraId: 3c397edc-c015-4d82-ae8a-1084692f8a93 | |||
$ mysql -e 'SELECT * FROM zimbra.mailbox WHERE comment LIKE "user1@mail-172.example.com"\G' | |||
$ mysql -e 'SELECT * FROM zimbra.mailbox WHERE account_id="3c397edc-c015-4d82-ae8a-1084692f8a93"\G' | |||
$ grep -C1 3c397edc-c015-4d82-ae8a-1084692f8a93 /opt/zimbra/backup/accounts.xml | |||
[using the default path; /opt/zimbra/backup/accounts.xml] | |||
$ egrep "user1@mail-172.example.com|3c397edc-c015-4d82-ae8a-1084692f8a93" /opt/zimbra/backup/accounts.xml | |||
$ egrep "user1@mail-172.example.com|3c397edc-c015-4d82-ae8a-1084692f8a93" /opt/zimbra/backup/sessions/*/session.xml | |||
</pre> | |||
=====exception during auth===== | |||
Example error: | |||
<pre> | |||
LDAP backup failed: system failure: exception during auth | |||
{RemoteManager: zimbramail.example.com->zimbra@zimbramail.example.com:22} | |||
com.zimbra.common.service.ServiceException: system failure: LDAP backup failed: | |||
system failure: exception during auth {RemoteManager: zimbramail.example.com-> | |||
zimbra@zimbramail.example.com:22} | |||
</pre> | |||
This indicates the ssh keys aren't configured correctly in the environment. You should be able to do the following to resolve the issue: | |||
su - zimbra | |||
zmsshkeygen | |||
zmupdateauthkeys | |||
'''Note''' - You should also confirm that you do indeed have the ssh port available. The default is port 22. If your unable to ssh into the server using port 22, then this might be the issue. If you configured ssh to listen on a different port on this server, you'll need to adjust the following variable - zmprov gcf zimbraRemoteManagementPort . You might also have the ssh port blocked - confirm your firewall, iptables, etc is not blocking the ssh port. | |||
=====LICENSE_ERROR_Message: AccountsLimits exceeded===== | |||
Example error: | |||
LICENSE_ERROR_Message: AccountsLimits exceeded | |||
You should check your license and confirm it's valid and active: | |||
su - zimbra | |||
zmlicense -p | |||
And confirm how many licenses are used: First, flush the cache on all servers for the license information. Then do the query via soap. | |||
zmprov fc -a license | |||
zmsoap -z GetLicenseRequest | |||
Four lines that are generally of interest are shown below. The first two state how many one is licensed for, the bottom two show how many are currently used : | |||
<pre> | |||
<attr name="AccountsLimit">1000</attr> | |||
<attr name="ArchivingAccountsLimit">1000</attr> | |||
<attr name="TotalAccounts">3</attr> | |||
<attr name="ArchivingAccounts">3</attr> | |||
</pre> | |||
==Hands On How-To And Reviewing Of Log Events== | |||
===Full Backup And Log Events Generated=== | |||
Example of manually starting a full backup using the Standard zimbraBackupMode and doing a query to confirm it's completion and status. | |||
<pre> | |||
[zimbra@mail-172 archive]$ zmprov gs `zmhostname` zimbraBackupMode | |||
# name mail-172.example.com | |||
zimbraBackupMode: Standard | |||
[zimbra@mail-172 archive]$ pwd ; ls *.log | |||
/opt/zimbra/redolog/archive | |||
redo-20150303.060015.816-seq9.log | |||
[zimbra@mail-172 archive]$ zmbackup -f -a all | |||
full-20150304.012000.193 | |||
[zimbra@mail-172 archive]$ zmbackupquery -lb full-20150304.012000.193 -v | |||
Label: full-20150304.012000.193 | |||
Type: full | |||
Status: completed | |||
Started: Tue, 2015/03/03 20:20:00.193 EST | |||
Ended: Tue, 2015/03/03 20:20:28.630 EST | |||
Redo log sequence range: 10 .. 10 | |||
Number of accounts: 10 out of 10 completed | |||
Accounts: | |||
admin@mail-172.example.com: completed | |||
galsync.qva4ekog0@mail-172.example.com: completed | |||
ham.jgfdb3oqjf@mail-172.example.com: completed | |||
spam.umhx3owv3s@mail-172.example.com: completed | |||
user1@mail-172.example.com: completed | |||
user2@mail-172.example.com: completed | |||
user3@mail-172.example.com: completed | |||
user4@mail-172.example.com: completed | |||
user5@mail-172.example.com: completed | |||
virus-quarantine.gusiso1cd@mail-172.example.com: completed | |||
Total space: 14020MB | |||
Free space: 6017MB | |||
</pre> | |||
Example of what log events are generated for a full backup in /opt/zimbra/log/mailbox.log . First, you'll see the authentication/acl request and then statements about the backup request. | |||
<pre> | |||
2015-03-03 20:19:58,499 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [] AuthProvider - Adding auth provider: zimbra com.zimbra.cs.service.ZimbraAuthProvider | |||
2015-03-03 20:19:59,458 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] soap - AuthRequest elapsed=834 | |||
2015-03-03 20:19:59,751 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] account - Initialized access manager: com.zimbra.cs.account.accesscontrol.ACLAccessManager | |||
2015-03-03 20:19:59,770 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup request started | |||
2015-03-03 20:20:00,193 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Found 10 accounts on server mail-172.example.com | |||
2015-03-03 20:20:00,464 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup request finished | |||
2015-03-03 20:20:00,464 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] soap - BackupRequest elapsed=760 | |||
</pre> | |||
Your log events will now reference Backup in them. AsyncFileCopier will start and it will give the backup session label for this backup job. | |||
<pre> | |||
2015-03-03 20:20:01,092 INFO [FullBackupThread] [] io - AsyncFileCopier is starting | |||
2015-03-03 20:20:01,187 INFO [FullBackupThread] [] backup - Full backup started for backup set; label: full-20150304.012000.193 | |||
</pre> | |||
It proceeds to backing up your accounts. I'll list the backup log events for just two accounts on this system - user1@mail-172.example.com and admin@mail-172.example.com . I'll provide the zimbraId, email address, and mailboxID information for user1@mail-172.example.com below so you can confirm it's reported in the logs correctly as well. | |||
<pre> | |||
[zimbra@mail-172 redolog]$ zmprov ga user1@mail-172.example.com zimbraId | |||
# name user1@mail-172.example.com | |||
zimbraId: 3c397edc-c015-4d82-ae8a-1084692f8a93 | |||
[zimbra@mail-172 redolog]$ zmprov gmi user1@mail-172.example.com | |||
mailboxId: 6 | |||
quotaUsed: 1209 | |||
</pre> | |||
Now, continuing with the log events. Notice the reference to the redo log file sequence, putting the account into and out of maintenance, number of blobs and index files backed up, and the statement it's finished backing up the account. Also within the section we can see the user's zimbraId [3c397edc-c015-4d82-ae8a-1084692f8a93] , email address [user1@mail-172.example.com] , and mailboxId [Mailbox 6]. | |||
<pre> | |||
[cut] | |||
2015-03-03 20:20:06,560 INFO [FullBackupThread] [name=user1@mail-172.example.com;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/6/index/0 | |||
2015-03-03 20:20:06,560 INFO [FullBackupThread] [name=user1@mail-172.example.com;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 LOADED | |||
2015-03-03 20:20:06,560 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - redo log file sequence is 10 at full backup for user1@mail-172.example.com | |||
2015-03-03 20:20:06,565 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - Full backup started for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) mailbox 6 | |||
2015-03-03 20:20:06,565 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] mailbox - Putting mailbox 6 under maintenance. | |||
2015-03-03 20:20:06,818 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - Number of blobs to backup for mailbox 6: 1 | |||
2015-03-03 20:20:06,818 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] mailbox - Ending maintenance on mailbox 6. | |||
2015-03-03 20:20:06,818 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 AVAILABLE | |||
2015-03-03 20:20:06,966 INFO [BackupSetWorkerThread-BLOB] [name=user1@mail-172.example.com;mid=6;] backup - Stored 1 blob files from volume message1 | |||
2015-03-03 20:20:06,966 INFO [BackupSetWorkerThread-BLOB] [name=user1@mail-172.example.com;mid=6;] backup - Stored 1 blob files | |||
2015-03-03 20:20:07,014 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - Account user1@mail-172.example.com in backup set full-20150304.012000.193: All pending file IO completed (1 out of 1) | |||
2015-03-03 20:20:07,016 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] mailbox - Putting mailbox 6 under maintenance. | |||
2015-03-03 20:20:07,021 INFO [BackupSetWorkerThread-INDEX] [name=user1@mail-172.example.com;mid=6;] backup - Stored 2 index files | |||
2015-03-03 20:20:07,496 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - Account user1@mail-172.example.com in backup set full-20150304.012000.193: All pending file IO completed (3 out of 3) | |||
2015-03-03 20:20:07,497 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] mailbox - Ending maintenance on mailbox 6. | |||
2015-03-03 20:20:07,497 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 AVAILABLE | |||
2015-03-03 20:20:07,497 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - Account user1@mail-172.example.com in backup set full-20150304.012000.193: All pending file IO completed (3 out of 3) | |||
2015-03-03 20:20:07,497 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - Full backup finished for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) mailbox 6 | |||
</pre> | |||
Events for the admin user's backup: | |||
<pre> | |||
2015-03-03 20:20:09,894 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - redo log file sequence is 10 at full backup for admin@mail-172.example.com | |||
2015-03-03 20:20:09,914 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - Full backup started for account admin@mail-172.example.com (7ac8de92-13fe-4d19-b9da-6872169fb5c9) mailbox 1 | |||
2015-03-03 20:20:09,915 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] mailbox - Putting mailbox 1 under maintenance. | |||
2015-03-03 20:20:10,176 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - Number of blobs to backup for mailbox 1: 73 | |||
2015-03-03 20:20:10,177 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] mailbox - Ending maintenance on mailbox 1. | |||
2015-03-03 20:20:10,177 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] mbxmgr - Mailbox 1 account 7ac8de92-13fe-4d19-b9da-6872169fb5c9 AVAILABLE | |||
2015-03-03 20:20:10,343 INFO [BackupSetWorkerThread-BLOB] [name=admin@mail-172.example.com;mid=1;] backup - Stored 73 blob files from volume message1 | |||
2015-03-03 20:20:10,353 INFO [BackupSetWorkerThread-BLOB] [name=admin@mail-172.example.com;mid=1;] backup - Stored 73 blob files | |||
2015-03-03 20:20:11,047 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - Account admin@mail-172.example.com in backup set full-20150304.012000.193: All pending file IO completed (73 out of 73) | |||
2015-03-03 20:20:11,051 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] mailbox - Putting mailbox 1 under maintenance. | |||
2015-03-03 20:20:11,108 INFO [BackupSetWorkerThread-INDEX] [name=admin@mail-172.example.com;mid=1;] backup - Stored 26 index files | |||
2015-03-03 20:20:12,450 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - Account admin@mail-172.example.com in backup set full-20150304.012000.193: All pending file IO completed (99 out of 99) | |||
2015-03-03 20:20:12,463 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] mailbox - Ending maintenance on mailbox 1. | |||
2015-03-03 20:20:12,463 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] mbxmgr - Mailbox 1 account 7ac8de92-13fe-4d19-b9da-6872169fb5c9 AVAILABLE | |||
2015-03-03 20:20:12,463 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - Account admin@mail-172.example.com in backup set full-20150304.012000.193: All pending file IO completed (99 out of 99) | |||
2015-03-03 20:20:12,463 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - Full backup finished for account admin@mail-172.example.com (7ac8de92-13fe-4d19-b9da-6872169fb5c9) mailbox 1 | |||
</pre> | |||
And then a final line confirming completion and the number of accounts successfully backed up. | |||
<pre> | |||
2015-03-03 20:20:12,463 INFO [FullBackupThread] [] backup - Finished backing up 10 of 10 accounts | |||
</pre> | |||
One thing to point out is the log events were mentioning the redo log sequence as being number 10 but in the redolog/archive directory there is only a sequence 9 redo log. This is because the "seq 10" is actually the /opt/zimbra/redolog/redo.log file. When the redo.log file rolls over and is moved to the archive directory it will have seq10 in it's filename. | |||
<pre> | |||
[zimbra@mail-172 archive]$ pwd ; ls -la *.log | |||
/opt/zimbra/redolog/archive | |||
-rw-r----- 1 zimbra zimbra 146723 Mar 3 20:23 redo-20150303.060015.816-seq9.log | |||
</pre> | |||
The contents of the backup session directory. I removed the sub-directories under accounts that weren't either the admin or user1 account. Notice the zimbraId is used for the sub-directory naming scheme. | |||
<pre> | |||
[zimbra@mail-172 full-20150304.012000.193]$ ls -R | |||
.: | |||
accounts ldap session.xml shared_blobs sys | |||
./accounts: | |||
3c3 423 434 75d 7ac 89f a46 af0 c53 eab | |||
./accounts/3c3: | |||
97e | |||
./accounts/3c3/97e: | |||
3c397edc-c015-4d82-ae8a-1084692f8a93 | |||
./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93: | |||
blobs db index ldap.xml meta.xml | |||
./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93/blobs: | |||
blobs-1.zip | |||
./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93/db: | |||
db_schema.xml mail_item.dat mailbox.dat mailbox_metadata.dat open_conversation.dat tag.dat tagged_item.dat | |||
./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93/index: | |||
0 | |||
./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93/index/0: | |||
segments.gen segments_1 | |||
[cut] | |||
./accounts/7ac: | |||
8de | |||
./accounts/7ac/8de: | |||
7ac8de92-13fe-4d19-b9da-6872169fb5c9 | |||
./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9: | |||
blobs db index ldap.xml meta.xml | |||
-- | ./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9/blobs: | ||
blobs-1.zip blobs-2.zip blobs-3.zip blobs-4.zip | |||
./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9/db: | |||
db_schema.xml mail_item.dat mailbox.dat open_conversation.dat tag.dat tagged_item.dat | |||
./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9/index: | |||
0 | |||
mail: | ./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9/index/0: | ||
zimbra$ ls blobs-1.zip | _0.fdt _0.fnm _0.nrm _0.tii _1.fdt _1.fnm _1.nrm _1.tii _2.fdt _2.fnm _2.nrm _2.tii segments.gen | ||
_0.fdx _0.frq _0.prx _0.tis _1.fdx _1.frq _1.prx _1.tis _2.fdx _2.frq _2.prx _2.tis segments_3 | |||
[cut] | |||
./ldap: | |||
ldap-config.bak.gz ldap.bak.gz | |||
./shared_blobs: | |||
./sys: | |||
config.dat current_volumes.dat db_schema.xml localconfig.xml ssh ssl volume.dat | |||
./sys/ssh: | |||
authorized_keys zimbra_identity zimbra_identity.pub | |||
./sys/ssl: | |||
zimbra | |||
./sys/ssl/zimbra: | |||
ca commercial jetty.pkcs12 server | |||
./sys/ssl/zimbra/ca: | |||
ca.key ca.pem ca.srl ca.srl.old index.txt index.txt.attr index.txt.old newcerts zmssl.cnf | |||
./sys/ssl/zimbra/ca/newcerts: | |||
1424523586.pem 1424523591.pem 1424523597.pem 1424523603.pem | |||
./sys/ssl/zimbra/commercial: | |||
./sys/ssl/zimbra/server: | |||
server.crt server.csr server.key | |||
</pre> | |||
===Incremental Backup And Log Events Generated=== | |||
Example of manually starting an incremental backup using the Standard zimbraBackupMode and doing a query to confirm it's completion and status. | |||
<pre> | |||
[zimbra@mail-172 archive]$ zmprov gs `zmhostname` zimbraBackupMode | |||
# name mail-172.example.com | |||
zimbraBackupMode: Standard | |||
[zimbra@mail-172 archive]$ ls | |||
redo-20150303.060015.816-seq9.log redo-20150304.070025.476-seq10.log | |||
[zimbra@mail-172 archive]$ zmbackup -i | |||
incr-20150304.073543.151 | |||
[zimbra@mail-172 archive]$ zmbackupquery -lb incr-20150304.073543.151 -v | |||
Label: incr-20150304.073543.151 | |||
Type: incremental | |||
Status: completed | |||
Started: Wed, 2015/03/04 02:35:43.151 EST | |||
Ended: Wed, 2015/03/04 02:35:51.981 EST | |||
Redo log sequence range: 10 .. 11 | |||
Number of accounts: 10 out of 10 completed | |||
Accounts: | |||
admin@mail-172.example.com: completed | |||
galsync.qva4ekog0@mail-172.example.com: completed | |||
ham.jgfdb3oqjf@mail-172.example.com: completed | |||
spam.umhx3owv3s@mail-172.example.com: completed | |||
user1@mail-172.example.com: completed | |||
user2@mail-172.example.com: completed | |||
user3@mail-172.example.com: completed | |||
user4@mail-172.example.com: completed | |||
user5@mail-172.example.com: completed | |||
virus-quarantine.gusiso1cd@mail-172.example.com: completed | |||
Total space: 14020MB | |||
Free space: 6032MB | |||
</pre> | |||
Notice how the log events are soap requests and differ from the full backup above and lack the INFO [*Backup*] formatting. Important to note this if your attempting to grep for backup/restore activity. Just like with the full, it starts with an authentication/acl request and states it's starting, how many accounts found, AsyncFileCopier starting, and the label name for this backup session. | |||
From /opt/zimbra/log/mailbox.log | |||
<pre> | |||
2015-03-04 02:35:42,356 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] soap - AuthRequest elapsed=94 | |||
2015-03-04 02:35:42,481 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup request started | |||
2015-03-04 02:35:42,917 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Found 10 accounts on server mail-172.example.com | |||
2015-03-04 02:35:43,254 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] io - AsyncFileCopier is starting | |||
2015-03-04 02:35:43,255 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup started for backup set; label: incr-20150304.073543.151 | |||
</pre> | |||
The log events are again trimmed below just for the user1 and admin account for the user backup section of the log file. Notice the absence of any statements of the accounts being put into maintenance mode. | |||
<pre> | |||
2015-03-04 02:35:44,118 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/6/index/0 | |||
2015-03-04 02:35:44,119 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 LOADED | |||
2015-03-04 02:35:44,120 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup is started for account user1@mail-172.example.com | |||
2015-03-04 02:35:44,120 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Saving account information from LDAP | |||
2015-03-04 02:35:44,127 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup has ended | |||
2015-03-04 02:35:44,127 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Account user1@mail-172.example.com in backup set incr-20150304.073543.151: All pending file IO completed (0 out of 0) | |||
[cut] | |||
2015-03-04 02:35:44,212 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup is started for account admin@mail-172.example.com | |||
2015-03-04 02:35:44,212 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Saving account information from LDAP | |||
2015-03-04 02:35:44,215 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup has ended | |||
2015-03-04 02:35:44,217 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Account admin@mail-172.example.com in backup set incr-20150304.073543.151: All pending file IO completed (0 out of 0) | |||
</pre> | |||
The incremental will then end with log events related to copying the redologs into the backup directory and then purging the redologs that can be deleted. This example states redo log sequence 9 can be deleted. We'll confirm it is later. | |||
<pre> | |||
2015-03-04 02:35:44,618 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] redolog - waiting for FileLogWriter.FsyncThread-1425452599560 to finish. | |||
2015-03-04 02:35:44,628 INFO [FileLogWriter.FsyncThread-1425452599560] [] redolog - fsync thread exiting | |||
2015-03-04 02:35:44,628 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] redolog - FileLogWriter.FsyncThread-1425452599560 finished | |||
2015-03-04 02:35:44,757 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] redolog - Redo log rollover took 540ms | |||
2015-03-04 02:35:44,758 INFO [FileLogWriter.FsyncThread-1425454544757] [] redolog - Starting fsync thread with interval 10 | |||
2015-03-04 02:35:44,776 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Found 3 redo logs to backup | |||
2015-03-04 02:35:44,861 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Last backed-up redo log sequence = 9 | |||
2015-03-04 02:35:44,943 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Marking redo log /opt/zimbra/redolog/archive/redo-20150303.060015.816-seq9.log for deletion | |||
2015-03-04 02:35:44,943 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Copying redo log /opt/zimbra/redolog/archive/redo-20150304.070025.476-seq10.log to /opt/zimbra/backup/tmp/incr-20150304.073543.151/redologs/redo-20150304.070025.476-seq10.log | |||
2015-03-04 02:35:45,053 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Retaining recent redo log /opt/zimbra/redolog/archive/redo-20150304.070025.476-seq10.log | |||
2015-03-04 02:35:45,053 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Copying redo log /opt/zimbra/redolog/archive/redo-20150304.073544.755-seq11.log to /opt/zimbra/backup/tmp/incr-20150304.073543.151/redologs/redo-20150304.073544.755-seq11.log | |||
2015-03-04 02:35:45,067 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Retaining recent redo log /opt/zimbra/redolog/archive/redo-20150304.073544.755-seq11.log | |||
</pre> | |||
The last limes confirm the backup finished successfully. | |||
<pre> | |||
2015-03-04 02:35:51,935 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup set incr-20150304.073543.151: All pending file IO completed (0 out of 0) | |||
2015-03-04 02:35:51,981 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] io - AsyncFileCopier is shut down | |||
2015-03-04 02:35:51,981 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup finished for backup set; label: incr-20150304.073543.151 | |||
2015-03-04 02:35:51,982 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Deleting redo log /opt/zimbra/redolog/archive/redo-20150303.060015.816-seq9.log | |||
2015-03-04 02:35:52,909 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup request finished | |||
2015-03-04 02:35:52,909 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] soap - BackupRequest elapsed=10451 | |||
</pre> | |||
For an incremental, the account sub-directories will have ldap data saved also for each user but blobs and indexes aren't. That data resides in the redologs from the system under the redologs directory. | |||
<pre> | |||
[zimbra@mail-172 incr-20150304.073543.151]$ ls -R | |||
.: | |||
accounts ldap redologs session.xml sys | |||
./accounts: | |||
3c3 423 434 75d 7ac 89f a46 af0 c53 eab | |||
./accounts/3c3: | |||
97e | |||
./accounts/3c3/97e: | |||
3c397edc-c015-4d82-ae8a-1084692f8a93 | |||
./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93: | |||
ldap.xml | |||
[cut] | |||
./accounts/7ac: | |||
8de | |||
./accounts/7ac/8de: | |||
7ac8de92-13fe-4d19-b9da-6872169fb5c9 | |||
./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9: | |||
ldap.xml | |||
[cut] | |||
./ldap: | |||
ldap-config.bak.gz ldap.bak.gz | |||
./redologs: | |||
redo-20150304.070025.476-seq10.log redo-20150304.073544.755-seq11.log | |||
./sys: | |||
config.dat current_volumes.dat db_schema.xml localconfig.xml ssh ssl volume.dat | |||
./sys/ssh: | |||
authorized_keys zimbra_identity zimbra_identity.pub | |||
./sys/ssl: | |||
zimbra | |||
./sys/ssl/zimbra: | |||
ca commercial jetty.pkcs12 server | |||
./sys/ssl/zimbra/ca: | |||
ca.key ca.pem ca.srl ca.srl.old index.txt index.txt.attr index.txt.old newcerts zmssl.cnf | |||
./sys/ssl/zimbra/ca/newcerts: | |||
1424523586.pem 1424523591.pem 1424523597.pem 1424523603.pem | |||
./sys/ssl/zimbra/commercial: | |||
./sys/ssl/zimbra/server: | |||
server.crt server.csr server.key | |||
</pre> | |||
Notice that redo log sequence 9 was indeed deleted from the redolog/archive directory. | |||
<pre> | |||
[zimbra@mail-172 archive]$ pwd ; ls | |||
/opt/zimbra/redolog/archive | |||
redo-20150304.070025.476-seq10.log redo-20150304.073544.755-seq11.log redo-20150304.074024.058-seq12.log | |||
</pre> | |||
If you had created new users after your last full backup, you would see that there was also a full backup started as well. For example : | |||
<pre> | |||
[zimbra@mail-172 log]$ zmbackup -i | |||
incr-20150304.085810.220 | |||
full-20150304.085818.889 | |||
[zimbra@mail-172 log]$ zmbackupquery -a user6@mail-172.example.com | |||
Account: user6@mail-172.example.com | |||
Label: full-20150304.085818.889 | |||
Type: full | |||
Started: Wed, 2015/03/04 03:58:18.889 EST | |||
Ended: Wed, 2015/03/04 03:58:25.997 EST | |||
Acct ID: 4e898a68-1886-4a80-b3dc-ac2ab650bf8c | |||
[zimbra@mail-172 log]$ zmbackupquery -lb full-20150304.085818.889 -v | |||
Label: full-20150304.085818.889 | |||
Type: full | |||
Status: completed | |||
Started: Wed, 2015/03/04 03:58:18.889 EST | |||
Ended: Wed, 2015/03/04 03:58:25.997 EST | |||
Redo log sequence range: 14 .. 14 | |||
Number of accounts: 1 out of 1 completed | |||
Accounts: | |||
user6@mail-172.example.com: completed | |||
Total space: 14020MB | |||
Free space: 6038MB | |||
</pre> | |||
===Contents Of accounts.xml And sessions.xml From The Full And Incremental Backup Examples=== | |||
First, see what zmbackupquery shows for a particular user in regards to the backup labels they are included in. | |||
<pre> | |||
$ zmbackupquery -a user1@mail-172.example.com | |||
Account: user1@mail-172.example.com | |||
Label: incr-20150304.073543.151 | |||
Type: incremental | |||
Started: Wed, 2015/03/04 02:35:43.151 EST | |||
Ended: Wed, 2015/03/04 02:35:51.981 EST | |||
Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93 | |||
Label: full-20150304.012000.193 | |||
Type: full | |||
Started: Tue, 2015/03/03 20:20:00.193 EST | |||
Ended: Tue, 2015/03/03 20:20:28.630 EST | |||
Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93 | |||
</pre> | |||
Now let's review the accounts/session xml files that list all the users within those particular backup sessions. The /opt/zimbra/backup/accounts.xml is the referenced to determine what is the last know full backup session the user was in. If the /opt/zimbra/backup/accounts.xml is corrupted, unreadable, or incorrect - this can cause issues with restoring users. | |||
Also notice that it also references the zimbraId for the user at the time of the backup. If you were to delete a user and then create a new account using the same email address [user1@mail-172.example.com] that new account will have a different zimbraId and therefor the backup/restore commands will not work as you might expected because of the differing zimbraId. The zmrestore command only can take an email address for the -a option, it can not use the zimbraID. | |||
<pre> | |||
[zimbra@mail-172 backup]$ pwd | |||
/opt/zimbra/backup | |||
[zimbra@mail-172 backup]$ cat accounts.xml | |||
<?xml version="1.0" encoding="utf-8"?> | |||
<backupMetadata xmlns="urn:zimbraBackupMeta" version="7.2" dbVersion="103"> | |||
<allAccounts> | |||
<account zimbraId="af06b0b4-3160-4ea3-a55b-81163a9914a0" email="user5@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193"> | |||
</account> | |||
<account zimbraId="75d3d364-a384-4077-8ac5-87b6960b57cd" email="user3@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193"> | |||
</account> | |||
<account zimbraId="4343a299-5259-4826-b1c1-becbfbb9b8f0" email="galsync.qva4ekog0@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193"> | |||
</account> | |||
<account zimbraId="89fc5986-31a3-406e-8084-0ce0b98a251f" email="user2@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193"> | |||
</account> | |||
<account zimbraId="42380a6e-5718-4d68-ab37-e8d5bc9e6977" email="spam.umhx3owv3s@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193"> | |||
</account> | |||
<account zimbraId="eab3d56e-ffd9-4412-95de-48f102d2d252" email="user4@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193"> | |||
</account> | |||
<account zimbraId="7ac8de92-13fe-4d19-b9da-6872169fb5c9" email="admin@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193"> | |||
</account> | |||
<account zimbraId="4e898a68-1886-4a80-b3dc-ac2ab650bf8c" email="user6@mail-172.example.com" latestFullBackupLabel="full-20150304.085818.889"> | |||
</account> | |||
<account zimbraId="a46dfc59-a3e9-4f09-adfb-af8efc42aa9f" email="virus-quarantine.gusiso1cd@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193"> | |||
</account> | |||
<account zimbraId="3c397edc-c015-4d82-ae8a-1084692f8a93" email="user1@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193"> | |||
</account> | |||
<account zimbraId="c5315c5d-97cb-4d84-bc70-374a67f13309" email="ham.jgfdb3oqjf@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193"> | |||
</account> | |||
</allAccounts> | |||
</backupMetadata> | |||
</pre> | |||
In each backup session, there will be a sessions.xml file that acts as a reference or summary file of the backup. At the top, you'll see what version of ZCS was running and what options the backup environment had set. It will then report the final status for each user for the backup attempt against the account. At the end, there's a section in regards to errors and if any were encountered. | |||
<pre> | |||
[zimbra@mail-172 backup]$ cd sessions/ | |||
[zimbra@mail-172 sessions]$ cd full-20150304.012000.193 | |||
[zimbra@mail-172 full-20150304.012000.193]$ cat session.xml | |||
<?xml version="1.0" encoding="utf-8"?> | |||
<backupMetadata xmlns="urn:zimbraBackupMeta" version="7.2" dbVersion="103"> | |||
<backupSet label="full-20150304.012000.193" zcsRelease="8.6.0_GA_1153 20141215151258 20141215-1518 NETWORK" startTime="1425432000193" endTime="1425432028630" minRedoSeq="10" maxRedoSeq="10" sharedBlobsZipped="true" sharedBlobsZipNameDigestChars="1" sharedBlobsDirectoryDepth="5" sharedBlobsCharsPerDirectory="2" type="full" accountsDirectoryDepth="2"> | |||
<desc>Full backup</desc> | |||
<accounts> | |||
<account zimbraId="af06b0b4-3160-4ea3-a55b-81163a9914a0" email="user5@mail-172.example.com" status="COMPLETED"> | |||
</account> | |||
<account zimbraId="75d3d364-a384-4077-8ac5-87b6960b57cd" email="user3@mail-172.example.com" status="COMPLETED"> | |||
</account> | |||
<account zimbraId="89fc5986-31a3-406e-8084-0ce0b98a251f" email="user2@mail-172.example.com" status="COMPLETED"> | |||
</account> | |||
<account zimbraId="4343a299-5259-4826-b1c1-becbfbb9b8f0" email="galsync.qva4ekog0@mail-172.example.com" status="COMPLETED"> | |||
</account> | |||
<account zimbraId="eab3d56e-ffd9-4412-95de-48f102d2d252" email="user4@mail-172.example.com" status="COMPLETED"> | |||
</account> | |||
<account zimbraId="42380a6e-5718-4d68-ab37-e8d5bc9e6977" email="spam.umhx3owv3s@mail-172.example.com" status="COMPLETED"> | |||
</account> | |||
<account zimbraId="7ac8de92-13fe-4d19-b9da-6872169fb5c9" email="admin@mail-172.example.com" status="COMPLETED"> | |||
</account> | |||
<account zimbraId="a46dfc59-a3e9-4f09-adfb-af8efc42aa9f" email="virus-quarantine.gusiso1cd@mail-172.example.com" status="COMPLETED"> | |||
</account> | |||
<account zimbraId="3c397edc-c015-4d82-ae8a-1084692f8a93" email="user1@mail-172.example.com" status="COMPLETED"> | |||
</account> | |||
<account zimbraId="c5315c5d-97cb-4d84-bc70-374a67f13309" email="ham.jgfdb3oqjf@mail-172.example.com" status="COMPLETED"> | |||
</account> | |||
</accounts> | |||
<stats> | |||
<counter name="total_time" numSamples="1" sum="28437" unit="ms"> | |||
</counter> | |||
<counter name="sysdb_time" numSamples="1" sum="1111" unit="ms"> | |||
</counter> | |||
<counter name="sysdb_bytes" numSamples="1" sum="35210" unit="bytes"> | |||
</counter> | |||
<counter name="redologs_time" numSamples="0" sum="0" unit="ms"> | |||
</counter> | |||
<counter name="redologs_bytes" numSamples="0" sum="0" unit="bytes"> | |||
</counter> | |||
<counter name="accounts_time" numSamples="10" sum="4827" unit="ms"> | |||
</counter> | |||
<counter name="ldap_time" numSamples="10" sum="172" unit="ms"> | |||
</counter> | |||
<counter name="ldap_bytes" numSamples="10" sum="25665" unit="bytes"> | |||
</counter> | |||
<counter name="db_time" numSamples="10" sum="4787" unit="ms"> | |||
</counter> | |||
<counter name="db_bytes" numSamples="10" sum="78690" unit="bytes"> | |||
</counter> | |||
<counter name="db_digest_map_load_time" numSamples="10" sum="2400" unit="ms"> | |||
</counter> | |||
<counter name="msgs_time" numSamples="10" sum="451" unit="ms"> | |||
</counter> | |||
<counter name="msg_copy_time" numSamples="74" sum="25" unit="ms"> | |||
</counter> | |||
<counter name="msg_copy_bytes" numSamples="74" sum="149611" unit="bytes"> | |||
</counter> | |||
<counter name="msg_link_time" numSamples="0" sum="0" unit="ms"> | |||
</counter> | |||
<counter name="index_time" numSamples="48" sum="80" unit="ms"> | |||
</counter> | |||
<counter name="index_bytes" numSamples="38" sum="73510" unit="bytes"> | |||
</counter> | |||
</stats> | |||
<errors> | |||
</errors> | |||
</backupSet> | |||
</backupMetadata> | |||
</pre> | |||
An incremental session.xml file is similar to a full backups session.xml . | |||
<pre> | |||
[zimbra@mail-172 sessions]$ cd incr-20150304.073543.151 | |||
[zimbra@mail-172 incr-20150304.073543.151]$ cat session.xml | |||
<?xml version="1.0" encoding="utf-8"?> | |||
<backupMetadata xmlns="urn:zimbraBackupMeta" version="7.2" dbVersion="103"> | |||
<backupSet label="incr-20150304.073543.151" zcsRelease="8.6.0_GA_1153 20141215151258 20141215-1518 NETWORK" startTime="1425454543151" endTime="1425454551981" minRedoSeq="10" maxRedoSeq="11" sharedBlobsZipped="true" sharedBlobsZipNameDigestChars="1" sharedBlobsDirectoryDepth="5" sharedBlobsCharsPerDirectory="2" type="incremental" accountsDirectoryDepth="2"> | |||
<desc>Incremental backup</desc> | |||
<accounts> | |||
<account zimbraId="af06b0b4-3160-4ea3-a55b-81163a9914a0" email="user5@mail-172.example.com" status="COMPLETED"> | |||
</account> | |||
<account zimbraId="75d3d364-a384-4077-8ac5-87b6960b57cd" email="user3@mail-172.example.com" status="COMPLETED"> | |||
</account> | |||
<account zimbraId="89fc5986-31a3-406e-8084-0ce0b98a251f" email="user2@mail-172.example.com" status="COMPLETED"> | |||
</account> | |||
<account zimbraId="4343a299-5259-4826-b1c1-becbfbb9b8f0" email="galsync.qva4ekog0@mail-172.example.com" status="COMPLETED"> | |||
</account> | |||
<account zimbraId="eab3d56e-ffd9-4412-95de-48f102d2d252" email="user4@mail-172.example.com" status="COMPLETED"> | |||
</account> | |||
<account zimbraId="42380a6e-5718-4d68-ab37-e8d5bc9e6977" email="spam.umhx3owv3s@mail-172.example.com" status="COMPLETED"> | |||
</account> | |||
<account zimbraId="7ac8de92-13fe-4d19-b9da-6872169fb5c9" email="admin@mail-172.example.com" status="COMPLETED"> | |||
</account> | |||
<account zimbraId="a46dfc59-a3e9-4f09-adfb-af8efc42aa9f" email="virus-quarantine.gusiso1cd@mail-172.example.com" status="COMPLETED"> | |||
</account> | |||
<account zimbraId="3c397edc-c015-4d82-ae8a-1084692f8a93" email="user1@mail-172.example.com" status="COMPLETED"> | |||
</account> | |||
<account zimbraId="c5315c5d-97cb-4d84-bc70-374a67f13309" email="ham.jgfdb3oqjf@mail-172.example.com" status="COMPLETED"> | |||
</account> | |||
</accounts> | |||
<stats> | |||
<counter name="total_time" numSamples="1" sum="8830" unit="ms"> | |||
</counter> | |||
<counter name="sysdb_time" numSamples="1" sum="563" unit="ms"> | |||
</counter> | |||
<counter name="sysdb_bytes" numSamples="1" sum="35210" unit="bytes"> | |||
</counter> | |||
<counter name="redologs_time" numSamples="1" sum="310" unit="ms"> | |||
</counter> | |||
<counter name="redologs_bytes" numSamples="1" sum="126307" unit="bytes"> | |||
</counter> | |||
<counter name="accounts_time" numSamples="10" sum="196" unit="ms"> | |||
</counter> | |||
<counter name="ldap_time" numSamples="10" sum="129" unit="ms"> | |||
</counter> | |||
<counter name="ldap_bytes" numSamples="10" sum="25665" unit="bytes"> | |||
</counter> | |||
<counter name="db_time" numSamples="0" sum="0" unit="ms"> | |||
</counter> | |||
<counter name="db_bytes" numSamples="0" sum="0" unit="bytes"> | |||
</counter> | |||
<counter name="db_digest_map_load_time" numSamples="0" sum="0" unit="ms"> | |||
</counter> | |||
<counter name="msgs_time" numSamples="0" sum="0" unit="ms"> | |||
</counter> | |||
<counter name="msg_copy_time" numSamples="0" sum="0" unit="ms"> | |||
</counter> | |||
<counter name="msg_copy_bytes" numSamples="0" sum="0" unit="bytes"> | |||
</counter> | |||
<counter name="msg_link_time" numSamples="0" sum="0" unit="ms"> | |||
</counter> | |||
<counter name="index_time" numSamples="0" sum="0" unit="ms"> | |||
</counter> | |||
<counter name="index_bytes" numSamples="0" sum="0" unit="bytes"> | |||
</counter> | |||
</stats> | |||
<errors> | |||
</errors> | |||
</backupSet> | |||
</backupMetadata> | |||
</pre> | |||
===User And Log Events Generated=== | |||
Please beware that prior backups were deleted and new backups were made on this ZCS test server. So the backup label names might not match prior examples. | |||
For this example, I'm going to replicate a common situation that system administrators find themselves when trying to restore a user and they fail to include the proper restore options to have the restore stop prior to replaying a message that was deleted by a user. Proper examples of doing the restore will be shown afterwards. | |||
Information on the account we'll restore: | |||
<pre> | |||
[zimbra@mail-172 backup]$ zmprov gmi user1@mail-172.example.com | |||
mailboxId: 6 | |||
quotaUsed: 1209 | |||
[zimbra@mail-172 backup]$ zmprov ga user1@mail-172.example.com zimbraId | |||
# name user1@mail-172.example.com | |||
zimbraId: 3c397edc-c015-4d82-ae8a-1084692f8a93 | |||
</pre> | |||
Example of error you'll receive if you attempt to use the zimbraId rather than the email address for the zmrestore command. This is important to note if your trying to restore an account that was deleted and then later created again and the various backups and accounts.xml file has different zimbraId's associated with the same email address. | |||
<pre> | |||
[zimbra@mail-172 backup]$ zmrestore -a 3c397edc-c015-4d82-ae8a-1084692f8a93 -ca -pre restore1_ | |||
Error occurred: invalid request: invalid account email address: 3c397edc-c015-4d82-ae8a-1084692f8a93 | |||
</pre> | |||
Showing that we have one message in the Inbox for the user we'll be restoring and also the message id of that email : | |||
<pre> | |||
[zimbra@mail-172 backup]$ zmmailbox -z -m user1@mail-172.example.com gaf | |||
Id View Unread Msg Count Path | |||
---------- ---- ---------- ---------- ---------- | |||
1 unkn 0 0 / | |||
16 docu 0 0 /Briefcase | |||
10 appo 0 0 /Calendar | |||
14 mess 0 0 /Chats | |||
7 cont 0 0 /Contacts | |||
6 mess 0 0 /Drafts | |||
13 cont 0 0 /Emailed Contacts | |||
2 mess 1 1 /Inbox | |||
4 mess 0 0 /Junk | |||
5 mess 0 0 /Sent | |||
15 task 0 0 /Tasks | |||
3 unkn 0 0 /Trash | |||
[zimbra@mail-172 backup]$ zmmailbox -z -m user1@mail-172.example.com s -t mess in:Inbox | |||
num: 1, more: false | |||
Id Type From Subject Date | |||
---- ---- -------------------- -------------------------------------------------- -------------- | |||
1. 257 mess admin Test1 02/23/15 09:03 | |||
</pre> | |||
Now to delete the email from the account: | |||
<pre> | |||
[zimbra@mail-172 backup]$ zmmailbox -z -m user1@mail-172.example.com dm 257 | |||
And to confirm it's no longer there: | |||
[zimbra@mail-172 backup]$ zmmailbox -z -m user1@mail-172.example.com s -t mess in:Inbox | |||
num: 0, more: false | |||
</pre> | |||
Below is an example of a delete operation being logged in the redo.log and how to query events for a particular user with the zmredodump command. Notice the DeleteItem events at the end. | |||
<pre> | |||
[zimbra@mail-172 backup]$ zmredodump -m 6 --show-blob /opt/zimbra/redolog/redo.log | |||
VERIFYING: /opt/zimbra/redolog/redo.log | |||
HEADER | |||
------ | |||
sequence: 14 | |||
open: 1 | |||
filesize: 512 | |||
serverId: 4407a03f-894b-4b1a-817a-684fc34c4f60 | |||
created: Wed, 2015/03/04 03:58:10.876 EST (1425459490876) | |||
first op: Wed, 2015/03/04 03:58:18.744 EST (1425459498744) | |||
last op: Wed, 2015/03/04 03:58:18.744 EST (1425459498744) (not up to date) | |||
version: 1.42 | |||
------ | |||
[00001601 - 0000162e: 46 bytes; tstamp: 2015/03/04 04:00:26.677 EST] txn 1425452599.101 [PurgeOldMessages] ver=1.42, tstamp=1425459626677, change=1508, mailbox=6 | |||
[0000162f - 00001660: 50 bytes; tstamp: 2015/03/04 04:00:26.683 EST] txn 1425452599.101 [CommitTxn] ver=1.42, tstamp=1425459626683, mailbox=6, txnType=PurgeOldMessages | |||
[00002691 - 000026be: 46 bytes; tstamp: 2015/03/04 04:10:26.816 EST] txn 1425452599.113 [PurgeOldMessages] ver=1.42, tstamp=1425460226816, change=1509, mailbox=6 | |||
[000026bf - 000026f0: 50 bytes; tstamp: 2015/03/04 04:10:26.832 EST] txn 1425452599.113 [CommitTxn] ver=1.42, tstamp=1425460226832, mailbox=6, txnType=PurgeOldMessages | |||
[00002ab1 - 00002ade: 46 bytes; tstamp: 2015/03/04 04:21:27.376 EST] txn 1425452599.124 [PurgeOldMessages] ver=1.42, tstamp=1425460887376, change=1510, mailbox=6 | |||
[00002adf - 00002b10: 50 bytes; tstamp: 2015/03/04 04:21:27.381 EST] txn 1425452599.124 [CommitTxn] ver=1.42, tstamp=1425460887381, mailbox=6, txnType=PurgeOldMessages | |||
[00002ed1 - 00002efe: 46 bytes; tstamp: 2015/03/04 04:32:27.610 EST] txn 1425452599.135 [PurgeOldMessages] ver=1.42, tstamp=1425461547610, change=1511, mailbox=6 | |||
[00002eff - 00002f30: 50 bytes; tstamp: 2015/03/04 04:32:27.617 EST] txn 1425452599.135 [CommitTxn] ver=1.42, tstamp=1425461547617, mailbox=6, txnType=PurgeOldMessages | |||
[000032f1 - 0000331e: 46 bytes; tstamp: 2015/03/04 04:43:27.828 EST] txn 1425452599.146 [PurgeOldMessages] ver=1.42, tstamp=1425462207828, change=1512, mailbox=6 | |||
[0000331f - 00003350: 50 bytes; tstamp: 2015/03/04 04:43:27.839 EST] txn 1425452599.146 [CommitTxn] ver=1.42, tstamp=1425462207839, mailbox=6, txnType=PurgeOldMessages | |||
[000034d1 - 0000350c: 60 bytes; tstamp: 2015/03/04 04:48:18.119 EST] txn 1425452599.151 [DeleteItem] ver=1.42, tstamp=1425462498119, change=1513, mailbox=6, ids=[257], type=message | |||
[0000350d - 0000353e: 50 bytes; tstamp: 2015/03/04 04:48:18.306 EST] txn 1425452599.151 [CommitTxn] ver=1.42, tstamp=1425462498306, mailbox=6, txnType=DeleteItem | |||
</pre> | |||
Listing of the backups that are currently associated with the account we'll be restoring: | |||
<pre> | |||
[zimbra@mail-172 backup]$ zmbackupquery -a user1@mail-172.example.com | |||
Account: user1@mail-172.example.com | |||
Label: incr-20150304.085810.220 | |||
Type: incremental | |||
Started: Wed, 2015/03/04 03:58:10.220 EST | |||
Ended: Wed, 2015/03/04 03:58:16.393 EST | |||
Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93 | |||
Label: full-20150304.082015.535 | |||
Type: full | |||
Started: Wed, 2015/03/04 03:20:15.535 EST | |||
Ended: Wed, 2015/03/04 03:20:39.109 EST | |||
Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93 | |||
</pre> | |||
Run a zmrestore with no other options other than the account flag and the email address of the account we are restoring. '''Note''' - this example is to show why this often times isn't what you'll do in real circumstances and will require other zmrestore options to achieve what you want. | |||
<pre> | |||
[zimbra@mail-172 backup]$ zmrestore -a user1@mail-172.example.com | |||
</pre> | |||
From the /opt/zimbra/log/mailbox.log we have the following log events. We see the restore request kick off and the full backup label that it'll reference as the starting point. It also lists the zimbraId rather than the email address for the restore initially. | |||
<pre> | |||
2015-03-04 04:53:02,221 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - AuthRequest elapsed=60 | |||
2015-03-04 04:53:02,519 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is starting | |||
2015-03-04 04:53:02,565 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Using full backup full-20150304.082015.535 as starting point for restoring account 3c397edc-c015-4d82-ae8a-1084692f8a93 | |||
</pre> | |||
And then we'll see the reference to the email address and the mailboxId associated with the account/zimbraId . | |||
<pre> | |||
2015-03-04 04:53:02,993 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - waiting for FileLogWriter.FsyncThread-1425459490899 to finish. | |||
2015-03-04 04:53:02,997 INFO [FileLogWriter.FsyncThread-1425459490899] [] redolog - fsync thread exiting | |||
2015-03-04 04:53:02,997 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - FileLogWriter.FsyncThread-1425459490899 finished | |||
2015-03-04 04:53:03,028 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - Redo log rollover took 463ms | |||
2015-03-04 04:53:03,028 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore started for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6 | |||
</pre> | |||
Since we are basically over-writing the account, the account is placed into maintenance mode first and then existing data is deleted. Only after the account has been deleted will the account be restored using the backup data. Since we didn't use the -ca and -pre options for zmrestore, this account will reuse it's existing zimbraId, mailboxId, and of course - email address. | |||
<pre> | |||
2015-03-04 04:53:03,042 INFO [FileLogWriter.FsyncThread-1425462783028] [] redolog - Starting fsync thread with interval 10 | |||
2015-03-04 04:53:03,096 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - Putting mailbox 6 under maintenance. | |||
2015-03-04 04:53:03,097 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - clearing contents of mailbox 6, group 6 | |||
2015-03-04 04:53:03,209 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restoring to mailbox id 6 for existing account 3c397edc-c015-4d82-ae8a-1084692f8a93. | |||
2015-03-04 04:53:03,256 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loading schema with version 7.2 | |||
2015-03-04 04:53:03,287 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Renaming volume_id column to locator in source table mail_item | |||
2015-03-04 04:53:03,584 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Deleting directory /opt/zimbra/store/0/6/msg | |||
2015-03-04 04:53:03,599 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 1 blobs during restore | |||
2015-03-04 04:53:03,622 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Deleting directory /opt/zimbra/index/0/6/index | |||
</pre> | |||
At this point, the data for the account was restored to the state of how it was when the full backup was ran. | |||
<pre> | |||
2015-03-04 04:53:03,664 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 2 index files during restore | |||
2015-03-04 04:53:03,665 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 AVAILABLE | |||
2015-03-04 04:53:03,670 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Account user1@mail-172.example.com: All pending file IO completed (2 out of 2) | |||
</pre> | |||
Now we see the restore start playing redologs that were made after the full backup. This section is the redologs from incr-20150304.085810.220 backup session, which were : redo-20150304.070025.476-seq10.log redo-20150304.073544.755-seq11.log redo-20150304.074024.058-seq12.log redo-20150304.085810.879-seq13.log . Located in, /opt/zimbra/backup/sessions/incr-20150304.085810.220/redologs . | |||
<pre> | |||
2015-03-04 04:53:03,671 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Run redo operations since last full backup | |||
2015-03-04 04:53:03,733 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/6/index/0 | |||
2015-03-04 04:53:03,733 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 LOADED | |||
2015-03-04 04:53:03,733 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. | |||
2015-03-04 04:53:03,733 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] cache - initializing folder and tag caches for mailbox 6 | |||
2015-03-04 04:53:03,746 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. | |||
2015-03-04 04:53:03,752 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. | |||
2015-03-04 04:53:03,757 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. | |||
</pre> | |||
Notice it will reference the sequence number of the redo log when it is plays those in /opt/zimbra/redolog/archive/ . Currently, the server has the following in that directory : redo-20150304.085810.879-seq13.log redo-20150304.095303.026-seq14.log redo-20150304.102039.541-seq15.log . | |||
<pre> | |||
2015-03-04 04:53:03,763 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Run redo operations from archived redo logs starting at sequence 13 | |||
2015-03-04 04:53:03,764 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. | |||
2015-03-04 04:53:03,774 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. | |||
2015-03-04 04:53:03,780 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. | |||
2015-03-04 04:53:03,787 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. | |||
2015-03-04 04:53:03,795 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. | |||
</pre> | |||
And while it replays redo log sequence 13, we see that it replays the delete operation for message id 257. | |||
<pre> | |||
2015-03-04 04:53:03,824 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailop - Deleting Message (id=257). | |||
</pre> | |||
Final log events showing restore is finished. | |||
<pre> | |||
2015-03-04 04:53:03,832 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore finished for user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox 6 to mailbox 6 | |||
2015-03-04 04:53:03,833 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is shut down | |||
2015-03-04 04:53:03,833 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - RestoreRequest elapsed=1535 | |||
</pre> | |||
Confirm that the message is still missing from the account: | |||
<pre> | |||
[zimbra@mail-172 backup]$ zmmailbox -z -m user1@mail-172.example.com s -t mess in:Inbox | |||
num: 0, more: false | |||
</pre> | |||
Confirm that the account is still using the same zimbraId, mailboxId, and email address : | |||
<pre> | |||
[zimbra@mail-172 backup]$ zmprov gmi user1@mail-172.example.com | |||
mailboxId: 6 | |||
quotaUsed: 0 | |||
[zimbra@mail-172 backup]$ zmprov ga user1@mail-172.example.com zimbraId | |||
# name user1@mail-172.example.com | |||
zimbraId: 3c397edc-c015-4d82-ae8a-1084692f8a93 | |||
</pre> | |||
====Using The -restoreToTime Option To Restore Prior To A Message Being Deleted==== | |||
First, let's identify the exact time the delete operation took place. From our previous examples above I already know what redo log sequence file it was logged in. | |||
<pre> | |||
[zimbra@mail-172 backup]$ zmredodump -m 6 --show-blob /opt/zimbra/redolog/archive/redo-20150304.095303.026-seq14.log | grep DeleteItem | |||
[000034d1 - 0000350c: 60 bytes; tstamp: 2015/03/04 04:48:18.119 EST] txn 1425452599.151 [DeleteItem] ver=1.42, tstamp=1425462498119, change=1513, mailbox=6, ids=[257], type=message | |||
[0000350d - 0000353e: 50 bytes; tstamp: 2015/03/04 04:48:18.306 EST] txn 1425452599.151 [CommitTxn] ver=1.42, tstamp=1425462498306, mailbox=6, txnType=DeleteItem | |||
</pre> | |||
The first line above shows me the delete request was done at "2015/03/04 04:48:18.119" for mailbox=6, ids=[257], type=message . Ids is the message id, that were already used to delete the message in the above example and was shown in the zmmailbox search command. | |||
The example uses the time above but subtracted one from it - 04:48:18.118 vs 04:48:18.119 . | |||
<pre> | |||
[zimbra@mail-172 backup]$ zmrestore -a user1@mail-172.example.com -restoreToTime "2015/03/04 04:48:18.118" | |||
using restore to time of 2015/03/04 04:48:18 | |||
</pre> | |||
And to now confirm the message is back in the account: | |||
<pre> | |||
[zimbra@mail-172 backup]$ zmmailbox -z -m user1@mail-172.example.com s -t mess in:Inbox | |||
num: 1, more: false | |||
Id Type From Subject Date | |||
---- ---- -------------------- -------------------------------------------------- -------------- | |||
1. 257 mess admin Test1 02/23/15 09:03 | |||
</pre> | |||
Note that this will also mean all other data or changes after 04:48:18.118 are lost also in this restored account. Normally in circumstances of trying to recovery data that a user is requesting because they delete messages and now need them recovered you would use the -ca and -pre options for zmrestore so you could then recover them and import the messages back into the existing account they have. Without the -ca and -pre , your over writing the existing account on the system. | |||
The log events in /opt/zimbra/log/mailbox.log for this request were: | |||
<pre> | |||
2015-03-04 05:20:39,194 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is starting | |||
2015-03-04 05:20:39,238 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Using full backup full-20150304.082015.535 as starting point for restoring account 3c397edc-c015-4d82-ae8a-1084692f8a93 | |||
2015-03-04 05:20:39,526 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - waiting for FileLogWriter.FsyncThread-1425462783028 to finish. | |||
2015-03-04 05:20:39,533 INFO [FileLogWriter.FsyncThread-1425462783028] [] redolog - fsync thread exiting | |||
2015-03-04 05:20:39,533 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - FileLogWriter.FsyncThread-1425462783028 finished | |||
2015-03-04 05:20:39,543 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - Redo log rollover took 305ms | |||
2015-03-04 05:20:39,543 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore started for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6 | |||
2015-03-04 05:20:39,550 INFO [FileLogWriter.FsyncThread-1425464439542] [] redolog - Starting fsync thread with interval 10 | |||
2015-03-04 05:20:39,593 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - Putting mailbox 6 under maintenance. | |||
2015-03-04 05:20:39,593 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - clearing contents of mailbox 6, group 6 | |||
2015-03-04 05:20:39,614 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restoring to mailbox id 6 for existing account 3c397edc-c015-4d82-ae8a-1084692f8a93. | |||
2015-03-04 05:20:39,652 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loading schema with version 7.2 | |||
2015-03-04 05:20:39,660 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Renaming volume_id column to locator in source table mail_item | |||
2015-03-04 05:20:39,692 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Deleting directory /opt/zimbra/store/0/6/msg | |||
2015-03-04 05:20:39,700 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 1 blobs during restore | |||
2015-03-04 05:20:39,703 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Deleting directory /opt/zimbra/index/0/6/index | |||
2015-03-04 05:20:39,713 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 2 index files during restore | |||
2015-03-04 05:20:39,715 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 AVAILABLE | |||
2015-03-04 05:20:39,717 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Account user1@mail-172.example.com: All pending file IO completed (2 out of 2) | |||
2015-03-04 05:20:39,717 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Run redo operations since last full backup | |||
2015-03-04 05:20:39,732 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/6/index/0 | |||
2015-03-04 05:20:39,732 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 LOADED | |||
2015-03-04 05:20:39,732 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. | |||
2015-03-04 05:20:39,732 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] cache - initializing folder and tag caches for mailbox 6 | |||
2015-03-04 05:20:39,743 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. | |||
2015-03-04 05:20:39,749 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. | |||
2015-03-04 05:20:39,754 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. | |||
2015-03-04 05:20:39,760 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Run redo operations from archived redo logs starting at sequence 13 | |||
2015-03-04 05:20:39,762 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. | |||
2015-03-04 05:20:39,770 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. | |||
2015-03-04 05:20:39,774 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. | |||
2015-03-04 05:20:39,780 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. | |||
2015-03-04 05:20:39,786 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. | |||
2015-03-04 05:20:39,789 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore finished for user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox 6 to mailbox 6 | |||
2015-03-04 05:20:39,791 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is shut down | |||
2015-03-04 05:20:39,791 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - RestoreRequest elapsed=646 | |||
</pre> | |||
Notice that this restore, unlike the first example, does '''NOT''' have the following line : | |||
<pre> | |||
INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailop - Deleting Message (id=257). | |||
</pre> | |||
====Blobs In Backup And Blobs In Production==== | |||
This might be a good time to take a look at the blobs and how they are saved/stored in the backups and how they compare to the store volume. The example below is from our user and message details from above. | |||
<pre> | |||
[zimbra@mail-172 blobs]$ pwd | |||
/opt/zimbra/backup/sessions/full-20150304.082015.535/accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93/blobs | |||
[zimbra@mail-172 blobs]$ unzip -Z blobs-1.zip | |||
Archive: blobs-1.zip | |||
Zip file size: 1439 bytes, number of entries: 1 | |||
-rw---- 4.5 fat 1209 b- stor 15-Mar-04 03:20 1/0/sha256_mBS9YKqyNpLknKCDWY5WNEbTgotX,UNrXMA5ga+s6LA=_257-2.msg1 | |||
1 file, 1209 bytes uncompressed, 1209 bytes compressed: 0.0% | |||
[zimbra@mail-172 blobs]$ ls -la /opt/zimbra/store/0/6/msg/0/257-2.msg | |||
-rw-r----- 1 zimbra zimbra 1209 Mar 4 05:20 /opt/zimbra/store/0/6/msg/0/257-2.msg | |||
[zimbra@mail-172 blobs]$ unzip blobs-1.zip | |||
Archive: blobs-1.zip | |||
extracting: 1/0/sha256_mBS9YKqyNpLknKCDWY5WNEbTgotX,UNrXMA5ga+s6LA=_257-2.msg1 | |||
[zimbra@mail-172 blobs]$ ls | |||
1 blobs-1.zip | |||
[zimbra@mail-172 blobs]$ ls -la 1/0/sha256_mBS9YKqyNpLknKCDWY5WNEbTgotX\,UNrXMA5ga+s6LA\=_257-2.msg1 | |||
-rw-r----- 1 zimbra zimbra 1209 Mar 4 03:20 1/0/sha256_mBS9YKqyNpLknKCDWY5WNEbTgotX,UNrXMA5ga+s6LA=_257-2.msg1 | |||
[zimbra@mail-172 blobs]$ head 1/0/sha256_mBS9YKqyNpLknKCDWY5WNEbTgotX\,UNrXMA5ga+s6LA\=_257-2.msg1 | |||
Return-Path: zimbra@mail-172.example.com | |||
Received: from mail-172.example.com (LHLO mail-172.example.com) | |||
(192.168.1.172) by mail-172.example.com with LMTP; Mon, 23 Feb 2015 | |||
09:02:58 -0500 (EST) | |||
Received: from localhost (localhost.localdomain [127.0.0.1]) | |||
by mail-172.example.com (Postfix) with ESMTP id 4F8AE8433D | |||
for <user1@mail-172.example.com>; Mon, 23 Feb 2015 09:02:58 -0500 (EST) | |||
X-Virus-Scanned: amavisd-new at mail-172.example.com | |||
X-Spam-Flag: NO | |||
X-Spam-Score: -1.9 | |||
[zimbra@mail-172 blobs]$ head /opt/zimbra/store/0/6/msg/0/257-2.msg | |||
Return-Path: zimbra@mail-172.example.com | |||
Received: from mail-172.example.com (LHLO mail-172.example.com) | |||
(192.168.1.172) by mail-172.example.com with LMTP; Mon, 23 Feb 2015 | |||
09:02:58 -0500 (EST) | |||
Received: from localhost (localhost.localdomain [127.0.0.1]) | |||
by mail-172.example.com (Postfix) with ESMTP id 4F8AE8433D | |||
for <user1@mail-172.example.com>; Mon, 23 Feb 2015 09:02:58 -0500 (EST) | |||
X-Virus-Scanned: amavisd-new at mail-172.example.com | |||
X-Spam-Flag: NO | |||
X-Spam-Score: -1.9 | |||
</pre> | |||
====Knowing When The -br [To Incremental] Or -rf [Only Full] Option Is Required==== | |||
=====Pre-Steps===== | |||
Let's query current backups and then create a new full and new incremental where we also delete a message between the full and the incremental. | |||
<pre> | |||
[zimbra@mail-172 ~]$ zmbackupquery -a user1@`zmhostname` | |||
Account: user1@mail-172.example.com | |||
Label: incr-20150305.060013.901 | |||
Type: incremental | |||
Started: Thu, 2015/03/05 01:00:13.901 EST | |||
Ended: Thu, 2015/03/05 01:00:26.273 EST | |||
Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93 | |||
Label: incr-20150304.085810.220 | |||
Type: incremental | |||
Started: Wed, 2015/03/04 03:58:10.220 EST | |||
Ended: Wed, 2015/03/04 03:58:16.393 EST | |||
Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93 | |||
Label: full-20150304.082015.535 | |||
Type: full | |||
Started: Wed, 2015/03/04 03:20:15.535 EST | |||
Ended: Wed, 2015/03/04 03:20:39.109 EST | |||
Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93 | |||
[zimbra@mail-172 ~]$ zmbackup -f -a all | |||
full-20150306.043202.011 | |||
[zimbra@mail-172 ~]$ zmmailbox -z -m user1@mail-172.example.com s -t mess in:Inbox | |||
num: 1, more: false | |||
Id Type From Subject Date | |||
---- ---- -------------------- -------------------------------------------------- -------------- | |||
1. 257 mess admin Test1 02/23/15 09:03 | |||
[zimbra@mail-172 ~]$ zmmailbox -z -m user1@mail-172.example.com dm 257 | |||
[zimbra@mail-172 ~]$ zmbackup -i | |||
incr-20150306.043321.683 | |||
[zimbra@mail-172 ~]$ zmbackupquery -a user1@`zmhostname` | |||
Account: user1@mail-172.example.com | |||
Label: incr-20150306.043321.683 | |||
Type: incremental | |||
Started: Thu, 2015/03/05 23:33:21.683 EST | |||
Ended: Thu, 2015/03/05 23:33:27.495 EST | |||
Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93 | |||
Label: full-20150306.043202.011 | |||
Type: full | |||
Started: Thu, 2015/03/05 23:32:02.011 EST | |||
Ended: Thu, 2015/03/05 23:32:13.316 EST | |||
Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93 | |||
Label: incr-20150305.060013.901 | |||
Type: incremental | |||
Started: Thu, 2015/03/05 01:00:13.901 EST | |||
Ended: Thu, 2015/03/05 01:00:26.273 EST | |||
Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93 | |||
Label: incr-20150304.085810.220 | |||
Type: incremental | |||
Started: Wed, 2015/03/04 03:58:10.220 EST | |||
Ended: Wed, 2015/03/04 03:58:16.393 EST | |||
Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93 | |||
Label: full-20150304.082015.535 | |||
Type: full | |||
Started: Wed, 2015/03/04 03:20:15.535 EST | |||
Ended: Wed, 2015/03/04 03:20:39.109 EST | |||
Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93 | |||
</pre> | |||
=====First Example - Options -lb, -ca, -pre restore_===== | |||
For the following examples, we'll include the -ca [create account] and the -pre [add prefix to new account name] . This first example uses the oldest full backup [full-20150304.082015.535 vs. full-20150306.043202.011] the account has with the -lb option. The message we just deleted, existed at the time of the full backup. We'll '''NOT''' be including either the -rf or -br option in this first example. | |||
Then we'll review the mailbox.log events. Note the following: | |||
* That the log only mentions the parent account, there isn't a reference to the email address with the prefixed name. | |||
** There is a RFE to include the new account name in the log, [https://bugzilla.zimbra.com/show_bug.cgi?id=52641 Bug 52641 - RFE: include -ca -pre new account in mailbox.log log event] | |||
* Though the log will state the new mailboxId and zimbraId for the user, though: | |||
** "user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6" vs "Restoring to mailbox id 12 for new account 016c9f7c-e2e2-4f8f-96d0-e14a4327821d" | |||
* Since we failed to include the -rf option, the restore continues past the full backup and goes through all the other backups that exist, hence we see: | |||
** "backup - Run redo operations since last full backup" | |||
* One of the redologs in our incremental had the delete operation for the message in question and we see evidence of that delete in the log: | |||
** "mailop - Deleting Message (id=257)." | |||
* Since we didn't include the -br option, the restore also continues past the backups and replays the redologs in /opt/zimbra/redolog and it's subdirectory, archive. Notice the reference to sequence 17. | |||
** "backup - Run redo operations from archived redo logs starting at sequence 17" | |||
** /opt/zimbra/redolog/archive/ has : redo-20150306.043322.070-seq17.log redo-20150306.043611.912-seq18.log | |||
* Towards the end, we'll get a confirmation that the restore has finished with some reference information again - mailboxId's . | |||
** "Restore finished for user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox 6 to mailbox 12" | |||
<pre> | |||
[zimbra@mail-172 ~]$ cat /dev/null > /opt/zimbra/log/mailbox.log | |||
[zimbra@mail-172 ~]$ zmrestore -a user1@mail-172.example.com -lb full-20150304.082015.535 -ca -pre restore_ | |||
[zimbra@mail-172 ~]$ cat /opt/zimbra/log/mailbox.log | |||
2015-03-05 23:35:40,815 INFO [MailboxPurge] [name=spam.umhx3owv3s@mail-172.example.com;mid=3;] purge - Purging messages. | |||
2015-03-05 23:36:11,619 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - AuthRequest elapsed=3 | |||
2015-03-05 23:36:11,784 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is starting | |||
2015-03-05 23:36:11,905 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - waiting for FileLogWriter.FsyncThread-1425616402072 to finish. | |||
2015-03-05 23:36:11,908 INFO [FileLogWriter.FsyncThread-1425616402072] [] redolog - fsync thread exiting | |||
2015-03-05 23:36:11,908 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - FileLogWriter.FsyncThread-1425616402072 finished | |||
2015-03-05 23:36:11,913 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - Redo log rollover took 128ms | |||
2015-03-05 23:36:11,913 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore started for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6 | |||
2015-03-05 23:36:11,914 INFO [FileLogWriter.FsyncThread-1425616571913] [] redolog - Starting fsync thread with interval 10 | |||
2015-03-05 23:36:13,006 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - Creating database mboxgroup12 | |||
2015-03-05 23:36:14,150 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restoring to mailbox id 12 for new account 016c9f7c-e2e2-4f8f-96d0-e14a4327821d. | |||
2015-03-05 23:36:14,237 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loading schema with version 7.2 | |||
2015-03-05 23:36:14,276 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Renaming volume_id column to locator in source table mail_item | |||
2015-03-05 23:36:14,315 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 1 blobs during restore | |||
2015-03-05 23:36:14,316 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Deleting directory /opt/zimbra/index/0/12/index | |||
2015-03-05 23:36:14,317 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 2 index files during restore | |||
2015-03-05 23:36:14,319 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/12/index/0 | |||
2015-03-05 23:36:14,319 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox 12 account 016c9f7c-e2e2-4f8f-96d0-e14a4327821d LOADED | |||
2015-03-05 23:36:14,319 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] cache - initializing folder and tag caches for mailbox 12 | |||
2015-03-05 23:36:14,367 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Account user1@mail-172.example.com: All pending file IO completed (2 out of 2) | |||
2015-03-05 23:36:14,367 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Run redo operations since last full backup | |||
2015-03-05 23:36:14,400 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. | |||
[multiple repeats of this message] | |||
2015-03-05 23:36:14,428 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. | |||
2015-03-05 23:36:14,465 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailop - Deleting Message (id=257). | |||
2015-03-05 23:36:14,474 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. | |||
[multiple repeats of this message] | |||
2015-03-05 23:36:15,152 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. | |||
2015-03-05 23:36:15,172 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Run redo operations from archived redo logs starting at sequence 17 | |||
2015-03-05 23:36:15,172 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore finished for user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox 6 to mailbox 12 | |||
2015-03-05 23:36:15,174 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is shut down | |||
2015-03-05 23:36:15,174 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - RestoreRequest elapsed=3481 | |||
[zimbra@mail-172 ~]$ ls /opt/zimbra/redolog/archive/ | |||
redo-20150306.043322.070-seq17.log redo-20150306.043611.912-seq18.log | |||
</pre> | |||
=====Second Example - Options -lb , -ca , -pre restore2_ , -br===== | |||
For the second example, we'll again use the oldest full backup [full-20150304.082015.535 vs. full-20150306.043202.011] the account has with the -lb option. The message we had deleted, existed at the time of the full backup but did NOT in the last incremental backup. Actually, the delete operation is within the redologs of the last incremental. For the second example, we'll use the -br option rather than the -rf . We'll also be adjusting the -pre option to be restore2_ vs. restore_ in the first example. | |||
Then we'll review the mailbox.log events. Note the following: | |||
* Though the log will state the new mailboxId and zimbraId for the user, though: | |||
** "user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6" vs "Restoring to mailbox id 13 for new account 4c99a395-2c39-46ec-8c1b-285d97d899df." | |||
** Notice the mailboxId and zimbraId for restore2_ is different than our first example when we used restore_ for the -pre option. | |||
* Since we failed to include the -rf option, the restore continues past the full backup and goes through all the other backups that exist, hence we see: | |||
** "backup - Run redo operations since last full backup" | |||
* One of the redologs in our incremental had the delete operation for the message in question and we see evidence of that delete in the log: | |||
** "mailop - Deleting Message (id=257)." | |||
* Since we '''DID''' include the -br option, the restore '''STOPS''' at our last backup session. It therefor '''DOESN'T''' replay the redologs in /opt/zimbra/redolog and it's subdirectory, archive and we do '''NOT''' have the following log events like we did in our first example. | |||
** "backup - Run redo operations from archived redo logs starting at sequence 17" | |||
* Towards the end, we'll get a confirmation that the restore has finished with some reference information again - mailboxId's . | |||
** "Restore finished for user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox 6 to mailbox 13" | |||
<pre> | |||
[zimbra@mail-172 ~]$ cat /dev/null > /opt/zimbra/log/mailbox.log | |||
[zimbra@mail-172 ~]$ zmrestore -a user1@mail-172.example.com -lb full-20150304.082015.535 -br -ca -pre restore2_ | |||
[zimbra@mail-172 ~]$ cat /opt/zimbra/log/mailbox.log | |||
2015-03-05 23:39:30,176 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - AuthRequest elapsed=2 | |||
2015-03-05 23:39:30,263 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is starting | |||
2015-03-05 23:39:30,352 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - waiting for FileLogWriter.FsyncThread-1425616571913 to finish. | |||
2015-03-05 23:39:30,360 INFO [FileLogWriter.FsyncThread-1425616571913] [] redolog - fsync thread exiting | |||
2015-03-05 23:39:30,360 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - FileLogWriter.FsyncThread-1425616571913 finished | |||
2015-03-05 23:39:30,371 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - Redo log rollover took 107ms | |||
2015-03-05 23:39:30,371 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore started for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6 | |||
2015-03-05 23:39:30,377 INFO [FileLogWriter.FsyncThread-1425616770369] [] redolog - Starting fsync thread with interval 10 | |||
2015-03-05 23:39:30,381 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - Creating database mboxgroup13 | |||
2015-03-05 23:39:30,992 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restoring to mailbox id 13 for new account 4c99a395-2c39-46ec-8c1b-285d97d899df. | |||
2015-03-05 23:39:30,993 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loading schema with version 7.2 | |||
2015-03-05 23:39:30,997 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Renaming volume_id column to locator in source table mail_item | |||
2015-03-05 23:39:31,022 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 1 blobs during restore | |||
2015-03-05 23:39:31,023 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Deleting directory /opt/zimbra/index/0/13/index | |||
2015-03-05 23:39:31,023 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 2 index files during restore | |||
2015-03-05 23:39:31,024 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/13/index/0 | |||
2015-03-05 23:39:31,024 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox 13 account 4c99a395-2c39-46ec-8c1b-285d97d899df LOADED | |||
2015-03-05 23:39:31,024 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] cache - initializing folder and tag caches for mailbox 13 | |||
2015-03-05 23:39:31,050 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Account user1@mail-172.example.com: All pending file IO completed (2 out of 2) | |||
2015-03-05 23:39:31,050 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Run redo operations since last full backup | |||
2015-03-05 23:39:31,053 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. | |||
[multiple repeats of this message] | |||
2015-03-05 23:39:31,070 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. | |||
2015-03-05 23:39:31,075 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailop - Deleting Message (id=257). | |||
2015-03-05 23:39:31,079 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. | |||
[multiple repeats of this message] | |||
2015-03-05 23:39:31,829 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. | |||
2015-03-05 23:39:31,832 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore finished for user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox 6 to mailbox 13 | |||
2015-03-05 23:39:31,833 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is shut down | |||
2015-03-05 23:39:31,834 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - RestoreRequest elapsed=1579 | |||
2015-03-05 23:39:40,932 INFO [MailboxPurge] [name=user5@mail-172.example.com;mid=7;] purge - Purging messages. | |||
[zimbra@mail-172 ~]$ | |||
</pre> | |||
=====Third Example - Options -lb , -ca , -pre restore3_ , -rf===== | |||
For the third example, we'll again use the oldest full backup [full-20150304.082015.535 vs. full-20150306.043202.011] the account has with the -lb option. The message we had deleted, existed at the time of the full backup but did NOT in the last incremental backup. For the third example, we'll use the -rf option rather than the -br . We'll also be adjusting the -pre option to be restore3_ vs. restore_ or restore2_ in the other examples. | |||
Then we'll review the mailbox.log events. Note the following: | |||
* Though the log will state the new mailboxId and zimbraId for the user, though: | |||
** "user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6" vs "Restoring to mailbox id 14 for new account d72a6d8d-8117-4a90-b8cf-f972f4af056d." | |||
** Notice the mailboxId and zimbraId for restore3_ is different than our other examples. | |||
* Since we '''INCLUDED''' the -rf option, the restore '''STOPS''' once the full backup is restored. Hence, we '''DON'T''' see the following line like we did in the other examples: | |||
** "backup - Run redo operations since last full backup" | |||
* And because it '''DIDN'T''' replay any of the redologs, there is '''NO''' reference to the following line either in the log: | |||
** "mailop - Deleting Message (id=257)." | |||
* '''NOR''' is there a reference to the following line either in the log: | |||
** "backup - Run redo operations from archived redo logs starting at sequence 17" | |||
* Towards the end, we'll get a confirmation that the restore has finished with some reference information again - mailboxId's . | |||
** "Restore finished for user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox 6 to mailbox 14" | |||
<pre> | |||
[zimbra@mail-172 ~]$ cat /dev/null > /opt/zimbra/log/mailbox.log | |||
[zimbra@mail-172 ~]$ zmrestore -a user1@mail-172.example.com -lb full-20150304.082015.535 -rf -ca -pre restore3_ | |||
[zimbra@mail-172 ~]$ cat /opt/zimbra/log/mailbox.log | |||
2015-03-05 23:42:50,233 INFO [Timer-Zimbra] [] SessionCache - Removed 1 idle sessions (SOAP). 1 active sessions remain. | |||
2015-03-05 23:42:58,194 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - AuthRequest elapsed=3 | |||
2015-03-05 23:42:58,282 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is starting | |||
2015-03-05 23:42:58,475 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - waiting for FileLogWriter.FsyncThread-1425616770369 to finish. | |||
2015-03-05 23:42:58,477 INFO [FileLogWriter.FsyncThread-1425616770369] [] redolog - fsync thread exiting | |||
2015-03-05 23:42:58,477 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - FileLogWriter.FsyncThread-1425616770369 finished | |||
2015-03-05 23:42:58,485 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - Redo log rollover took 193ms | |||
2015-03-05 23:42:58,485 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore started for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6 | |||
2015-03-05 23:42:58,495 INFO [FileLogWriter.FsyncThread-1425616978485] [] redolog - Starting fsync thread with interval 10 | |||
2015-03-05 23:42:58,515 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - Creating database mboxgroup14 | |||
2015-03-05 23:43:06,041 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restoring to mailbox id 14 for new account d72a6d8d-8117-4a90-b8cf-f972f4af056d. | |||
2015-03-05 23:43:06,042 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loading schema with version 7.2 | |||
2015-03-05 23:43:06,048 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Renaming volume_id column to locator in source table mail_item | |||
2015-03-05 23:43:06,092 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 1 blobs during restore | |||
2015-03-05 23:43:06,093 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Deleting directory /opt/zimbra/index/0/14/index | |||
2015-03-05 23:43:06,093 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 2 index files during restore | |||
2015-03-05 23:43:06,095 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/14/index/0 | |||
2015-03-05 23:43:06,095 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox 14 account d72a6d8d-8117-4a90-b8cf-f972f4af056d LOADED | |||
2015-03-05 23:43:06,095 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] cache - initializing folder and tag caches for mailbox 14 | |||
2015-03-05 23:43:06,264 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Account user1@mail-172.example.com: All pending file IO completed (2 out of 2) | |||
2015-03-05 23:43:06,265 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore finished for user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox 6 to mailbox 14 | |||
2015-03-05 23:43:06,266 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is shut down | |||
2015-03-05 23:43:06,266 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - RestoreRequest elapsed=7999 | |||
[zimbra@mail-172 ~]$ | |||
</pre> | |||
Below is to confirm that indeed, it was only the restore with the -rf that did in fact have the message that was deleted after it. | |||
<pre> | |||
[zimbra@mail-172 ~]$ zmmailbox -z -m restore_user1@mail-172.example.com s -t mess in:Inbox | |||
num: 0, more: false | |||
[zimbra@mail-172 ~]$ zmmailbox -z -m restore2_user1@mail-172.example.com s -t mess in:Inbox | |||
num: 0, more: false | |||
[zimbra@mail-172 ~]$ zmmailbox -z -m restore3_user1@mail-172.example.com s -t mess in:Inbox | |||
num: 1, more: false | |||
Id Type From Subject Date | |||
---- ---- -------------------- -------------------------------------------------- -------------- | |||
1. 257 mess admin Test1 02/23/15 09:03 | |||
</pre> | |||
=====Fourth Example - To Confirm -lb Label Does Matter===== | |||
This example is to show that the -lb does indeed matter. Earlier examples used the oldest full we had that did include the deleted message in the account. We'll create a new full backup of the parent account, user1@ , where the message is absent and then do two restores using the differing full backup session labels with the proper option of -rf since we are wanting the restore to stop once the full session data is restored. | |||
<pre> | |||
[zimbra@mail-172 ~]$ zmmailbox -z -m user1@mail-172.example.com s -t mess in:Inbox | |||
num: 0, more: false | |||
[zimbra@mail-172 ~]$ zmbackup -f -a all | |||
full-20150306.055845.429 | |||
[zimbra@mail-172 ~]$ zmbackupquery -a user1@`zmhostname` | |||
Account: user1@mail-172.example.com | |||
Label: full-20150306.055845.429 | |||
Type: full | |||
Started: Fri, 2015/03/06 00:58:45.429 EST | |||
Ended: Fri, 2015/03/06 00:58:52.357 EST | |||
Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93 | |||
Label: incr-20150306.043321.683 | |||
Type: incremental | |||
Started: Thu, 2015/03/05 23:33:21.683 EST | |||
Ended: Thu, 2015/03/05 23:33:27.495 EST | |||
Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93 | |||
Label: full-20150306.043202.011 | |||
Type: full | |||
Started: Thu, 2015/03/05 23:32:02.011 EST | |||
Ended: Thu, 2015/03/05 23:32:13.316 EST | |||
Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93 | |||
Label: incr-20150305.060013.901 | |||
Type: incremental | |||
Started: Thu, 2015/03/05 01:00:13.901 EST | |||
Ended: Thu, 2015/03/05 01:00:26.273 EST | |||
Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93 | |||
Label: incr-20150304.085810.220 | |||
Type: incremental | |||
Started: Wed, 2015/03/04 03:58:10.220 EST | |||
Ended: Wed, 2015/03/04 03:58:16.393 EST | |||
Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93 | |||
Label: full-20150304.082015.535 | |||
Type: full | |||
Started: Wed, 2015/03/04 03:20:15.535 EST | |||
Ended: Wed, 2015/03/04 03:20:39.109 EST | |||
Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93 | |||
[zimbra@mail-172 ~]$ zmrestore -a user1@mail-172.example.com -lb full-20150306.055845.429 -rf -ca -pre restore4_ | |||
[zimbra@mail-172 ~]$ zmrestore -a user1@mail-172.example.com -lb full-20150304.082015.535 -rf -ca -pre restore5_ | |||
[zimbra@mail-172 ~]$ zmmailbox -z -m restore4_user1@mail-172.example.com s -t mess in:Inbox | |||
num: 0, more: false | |||
[zimbra@mail-172 ~]$ zmmailbox -z -m restore5_user1@mail-172.example.com s -t mess in:Inbox | |||
num: 1, more: false | |||
Id Type From Subject Date | |||
---- ---- -------------------- -------------------------------------------------- -------------- | |||
1. 257 mess admin Test1 02/23/15 09:03 | |||
[zimbra@mail-172 ~]$ | |||
</pre> | |||
====What Is Different Between The Parent Account And The Restored Accounts Where We Used The -ca And -pre Options [LDAP]==== | |||
Below is a difference between the zmprov ga output between the parent account [user1@] and our last restore example [restore5_user1@]. Note, in our example here the parent account was created with default values and there was no customization's or changes to the defaults. | |||
<pre> | |||
[zimbra@mail-172 ~]$ zmprov ga user1@`zmhostname` > /tmp/user1 | |||
[zimbra@mail-172 ~]$ zmprov ga restore5_user1@`zmhostname` > /tmp/restore5 | |||
[zimbra@mail-172 ~]$ diff /tmp/user1 /tmp/restore5 | |||
1c1 | |||
< # name user1@mail-172.example.com | |||
--- | |||
> # name restore5_user1@mail-172.example.com | |||
3c3 | |||
< mail: user1@mail-172.example.com | |||
--- | |||
> mail: restore5_user1@mail-172.example.com | |||
8c8 | |||
< uid: user1 | |||
--- | |||
> uid: restore5_user1 | |||
20,22c20 | |||
< zimbraAuthTokens: 661297364|1425702763611|8.6.0_GA_1153 | |||
< zimbraAuthTokens: 414162390|1425702787760|8.6.0_GA_1153 | |||
< zimbraAuthTokens: 2046929787|1425707910566|8.6.0_GA_1153 | |||
--- | |||
> zimbraAuthTokens: 1030982216|1425708106344|8.6.0_GA_1153 | |||
159c157 | |||
< zimbraId: 3c397edc-c015-4d82-ae8a-1084692f8a93 | |||
--- | |||
> zimbraId: e5c38a4b-6bea-45bb-9e14-dd3f39159a61 | |||
170c168 | |||
< zimbraMailDeliveryAddress: user1@mail-172.example.com | |||
--- | |||
> zimbraMailDeliveryAddress: restore5_user1@mail-172.example.com | |||
</pre> | |||
====Restore encountered redo log sequence error: Found gap in redo log sequence==== | |||
Example error: | |||
<pre> | |||
Error occurred: system failure: Restore encountered redo log sequence error: | |||
Found gap in redo log sequence; missing 3899 through 3895; To avoid future restore problems, | |||
discard all existing backups and take a full backup of all accounts; If this error occurred | |||
during restore, try the --ignoreRedoErrors option | |||
</pre> | |||
This error actually gives the two basic options you have when you encounter this. Depending on your circumstances, you might first do the restore using the --ignoreRedoErrors options and recovery what data you can prior to discarding the 'bad' backup sessions in question and attempting a new full backup of the users. | |||
=====How-To Example - Extended Look At RedoErrors With Log Events===== | |||
First, let's replicate a redolog failure situation, which can be done by moving the redologs from the /opt/zimbra/redolog/archive/ to a temporary directory. | |||
<pre> | |||
[zimbra@mail-172 redolog]$ cd archive/ | |||
[zimbra@mail-172 archive]$ ls | |||
redo-20150306.060015.331-seq21.log redo-20150306.060109.166-seq22.log | |||
[zimbra@mail-172 archive]$ mkdir /tmp/redolog-archive | |||
[zimbra@mail-172 archive]$ mv * /tmp/redolog-archive/ | |||
[zimbra@mail-172 archive]$ cat /dev/null > /opt/zimbra/log/mailbox.log | |||
</pre> | |||
Now that we've moved the redolog sequence files out of the redolog/archive directory path, we can attempt our zmrestore command. Note that the incremental backup label for this attempt is called incr-20150306.170810.793 . Though it fails because of the missing redologs, the backup is still created. | |||
<pre> | |||
[zimbra@mail-172 archive]$ zmbackup -i | |||
Error occurred: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option | |||
[zimbra@mail-172 archive]$ cat /opt/zimbra/log/mailbox.log | |||
2015-03-06 12:08:06,442 INFO [MailboxPurge] [name=ham.jgfdb3oqjf@mail-172.example.com;mid=4;] purge - Purging messages. | |||
2015-03-06 12:08:10,350 INFO [qtp509886383-484:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] soap - AuthRequest elapsed=4 | |||
2015-03-06 12:08:10,413 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup request started | |||
2015-03-06 12:08:10,579 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Found 16 accounts on server mail-172.example.com | |||
2015-03-06 12:08:10,797 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] io - AsyncFileCopier is starting | |||
2015-03-06 12:08:10,798 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup started for backup set; label: incr-20150306.170810.793 | |||
[cut] | |||
2015-03-06 12:08:11,277 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup is started for account user1@mail-172.example.com | |||
2015-03-06 12:08:11,277 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Saving account information from LDAP | |||
2015-03-06 12:08:11,281 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup has ended | |||
2015-03-06 12:08:11,281 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Account user1@mail-172.example.com in backup set incr-20150306.170810.793: All pending file IO completed (0 out of 0) | |||
[cut] | |||
2015-03-06 12:08:11,310 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup is started for account admin@mail-172.example.com | |||
2015-03-06 12:08:11,310 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Saving account information from LDAP | |||
2015-03-06 12:08:11,312 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup has ended | |||
2015-03-06 12:08:11,312 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Account admin@mail-172.example.com in backup set incr-20150306.170810.793: All pending file IO completed (0 out of 0) | |||
2015-03-06 12:08:11,711 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] redolog - waiting for FileLogWriter.FsyncThread-1425621669168 to finish. | |||
2015-03-06 12:08:11,712 INFO [FileLogWriter.FsyncThread-1425621669168] [] redolog - fsync thread exiting | |||
2015-03-06 12:08:11,712 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] redolog - FileLogWriter.FsyncThread-1425621669168 finished | |||
2015-03-06 12:08:11,799 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] redolog - Redo log rollover took 487ms | |||
2015-03-06 12:08:11,799 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Found 1 redo logs to backup | |||
2015-03-06 12:08:11,800 INFO [FileLogWriter.FsyncThread-1425661691796] [] redolog - Starting fsync thread with interval 10 | |||
2015-03-06 12:08:11,864 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Last backed-up redo log sequence = 21 | |||
2015-03-06 12:08:11,864 ERROR [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option | |||
2015-03-06 12:08:11,973 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Copying redo log /opt/zimbra/redolog/archive/redo-20150306.170811.794-seq23.log to /opt/zimbra/backup/tmp/incr-20150306.170810.793/redologs/redo-20150306.170811.794-seq23.log | |||
2015-03-06 12:08:12,100 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Retaining recent redo log /opt/zimbra/redolog/archive/redo-20150306.170811.794-seq23.log | |||
2015-03-06 12:08:19,375 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup set incr-20150306.170810.793: All pending file IO completed (0 out of 0) | |||
2015-03-06 12:08:19,378 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] io - AsyncFileCopier is shut down | |||
2015-03-06 12:08:19,378 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup finished for backup set; label: incr-20150306.170810.793 | |||
2015-03-06 12:08:20,565 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] SoapEngine - handler exception | |||
com.zimbra.cs.backup.BackupServiceException: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option | |||
ExceptionId:qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest:1425661691864:14586f805b4c1b0e | |||
Code:backup.REDOLOG_OUT_OF_SEQUENCE | |||
at com.zimbra.cs.backup.BackupServiceException.REDOLOG_OUT_OF_SEQUENCE(BackupServiceException.java:91) | |||
at com.zimbra.cs.backup.util.Utils.splitRedoLogsAtSeq(Utils.java:302) | |||
at com.zimbra.cs.backup.FileBackupTarget$FileBackupSet.backupRedoLogs(FileBackupTarget.java:1360) | |||
at com.zimbra.cs.backup.BackupSet.startIncrementalBackup(BackupSet.java:817) | |||
at com.zimbra.cs.backup.FileBackupTarget$FileBackupSet.startIncrementalBackup(FileBackupTarget.java:1067) | |||
at com.zimbra.cs.backup.BackupManager.backupIncremental(BackupManager.java:336) | |||
at com.zimbra.cs.service.backup.Backup.handleNetworkRequest(Backup.java:153) | |||
at com.zimbra.cs.service.NetworkDocumentHandler.handle(NetworkDocumentHandler.java:23) | |||
at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEngine.java:569) | |||
at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:432) | |||
at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:266) | |||
at com.zimbra.soap.SoapServlet.doWork(SoapServlet.java:303) | |||
at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:213) | |||
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) | |||
at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:209) | |||
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) | |||
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651) | |||
at com.zimbra.cs.servlet.RequestStringFilter.doFilter(RequestStringFilter.java:54) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) | |||
at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:59) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) | |||
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) | |||
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:351) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) | |||
at com.zimbra.cs.servlet.ETagHeaderFilter.doFilter(ETagHeaderFilter.java:47) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) | |||
at com.zimbra.cs.servlet.ContextPathBasedThreadPoolBalancerFilter.doFilter(ContextPathBasedThreadPoolBalancerFilter.java:107) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) | |||
at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(ZimbraQoSFilter.java:107) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) | |||
at com.zimbra.cs.servlet.ZimbraInvalidLoginFilter.doFilter(ZimbraInvalidLoginFilter.java:117) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) | |||
at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:457) | |||
at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:326) | |||
at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:299) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) | |||
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549) | |||
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) | |||
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:544) | |||
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221) | |||
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111) | |||
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478) | |||
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183) | |||
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045) | |||
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) | |||
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:199) | |||
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109) | |||
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) | |||
at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:309) | |||
at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:81) | |||
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) | |||
at org.eclipse.jetty.server.Server.handle(Server.java:462) | |||
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279) | |||
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232) | |||
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534) | |||
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607) | |||
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536) | |||
at java.lang.Thread.run(Thread.java:745) | |||
2015-03-06 12:08:20,566 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] soap - BackupRequest elapsed=10154 | |||
</pre> | |||
To confirm the backup label was made and also to see how the zmbackupquery reports it, we can do the following: | |||
<pre> | |||
[zimbra@mail-172 archive]$ zmbackupquery -lb incr-20150306.170810.793 -v | |||
Label: incr-20150306.170810.793 | |||
Type: incremental | |||
Status: completed (with errors) | |||
Started: Fri, 2015/03/06 12:08:10.793 EST | |||
Ended: Fri, 2015/03/06 12:08:19.378 EST | |||
Redo log sequence range: 23 .. 23 | |||
Number of accounts: 14 out of 14 completed | |||
Number of errors: 1 | |||
Accounts: | |||
admin@mail-172.example.com: completed | |||
galsync.qva4ekog0@mail-172.example.com: completed | |||
ham.jgfdb3oqjf@mail-172.example.com: completed | |||
restore2_user1@mail-172.example.com: completed | |||
restore3_user1@mail-172.example.com: completed | |||
restore_user1@mail-172.example.com: completed | |||
spam.umhx3owv3s@mail-172.example.com: completed | |||
user1@mail-172.example.com: completed | |||
user2@mail-172.example.com: completed | |||
user3@mail-172.example.com: completed | |||
user4@mail-172.example.com: completed | |||
user5@mail-172.example.com: completed | |||
user6@mail-172.example.com: completed | |||
virus-quarantine.gusiso1cd@mail-172.example.com: completed | |||
Error: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option | |||
com.zimbra.cs.backup.BackupServiceException: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option | |||
ExceptionId:qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest:1425661691864:14586f805b4c1b0e | |||
Code:backup.REDOLOG_OUT_OF_SEQUENCE | |||
at com.zimbra.cs.backup.BackupServiceException.REDOLOG_OUT_OF_SEQUENCE(BackupServiceException.java:91) | |||
at com.zimbra.cs.backup.util.Utils.splitRedoLogsAtSeq(Utils.java:302) | |||
at com.zimbra.cs.backup.FileBackupTarget$FileBackupSet.backupRedoLogs(FileBackupTarget.java:1360) | |||
at com.zimbra.cs.backup.BackupSet.startIncrementalBackup(BackupSet.java:817) | |||
at com.zimbra.cs.backup.FileBackupTarget$FileBackupSet.startIncrementalBackup(FileBackupTarget.java:1067) | |||
at com.zimbra.cs.backup.BackupManager.backupIncremental(BackupManager.java:336) | |||
at com.zimbra.cs.service.backup.Backup.handleNetworkRequest(Backup.java:153) | |||
at com.zimbra.cs.service.NetworkDocumentHandler.handle(NetworkDocumentHandler.java:23) | |||
at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEngine.java:569) | |||
at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:432) | |||
at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:266) | |||
at com.zimbra.soap.SoapServlet.doWork(SoapServlet.java:303) | |||
at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:213) | |||
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) | |||
at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:209) | |||
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) | |||
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651) | |||
at com.zimbra.cs.servlet.RequestStringFilter.doFilter(RequestStringFilter.java:54) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) | |||
at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:59) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) | |||
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) | |||
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:351) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) | |||
at com.zimbra.cs.servlet.ETagHeaderFilter.doFilter(ETagHeaderFilter.java:47) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) | |||
at com.zimbra.cs.servlet.ContextPathBasedThreadPoolBalancerFilter.doFilter(ContextPathBasedThreadPoolBalancerFilter.java:107) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) | |||
at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(ZimbraQoSFilter.java:107) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) | |||
at com.zimbra.cs.servlet.ZimbraInvalidLoginFilter.doFilter(ZimbraInvalidLoginFilter.java:117) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) | |||
at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:457) | |||
at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:326) | |||
at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:299) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) | |||
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549) | |||
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) | |||
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:544) | |||
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221) | |||
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111) | |||
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478) | |||
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183) | |||
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045) | |||
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) | |||
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:199) | |||
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109) | |||
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) | |||
at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:309) | |||
at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:81) | |||
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) | |||
at org.eclipse.jetty.server.Server.handle(Server.java:462) | |||
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279) | |||
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232) | |||
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534) | |||
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607) | |||
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536) | |||
at java.lang.Thread.run(Thread.java:745) | |||
Total space: 14020MB | |||
Free space: 6031MB | |||
[zimbra@mail-172 archive]$ | |||
</pre> | |||
Let's now try doing a restore of an account. | |||
<pre> | |||
[zimbra@mail-172 archive]$ cat /dev/null > /opt/zimbra/log/mailbox.log | |||
[zimbra@mail-172 archive]$ zmrestore -a user1@`zmhostname` -ca -pre restore6_ | |||
Error occurred: system failure: Restore encountered redo log sequence error: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option | |||
[zimbra@mail-172 archive]$ cat /opt/zimbra/log/mailbox.log | |||
2015-03-06 12:12:31,884 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - RestoreRequest elapsed=38 | |||
2015-03-06 12:13:06,650 INFO [MailboxPurge] [name=user3@mail-172.example.com;mid=9;] purge - Purging messages. | |||
2015-03-06 12:13:45,329 INFO [qtp509886383-484:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - AuthRequest elapsed=2 | |||
2015-03-06 12:13:45,474 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is starting | |||
2015-03-06 12:13:45,512 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Using full backup full-20150306.055845.429 as starting point for restoring account 3c397edc-c015-4d82-ae8a-1084692f8a93 | |||
2015-03-06 12:13:45,662 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - waiting for FileLogWriter.FsyncThread-1425661691796 to finish. | |||
2015-03-06 12:13:45,669 INFO [FileLogWriter.FsyncThread-1425661691796] [] redolog - fsync thread exiting | |||
2015-03-06 12:13:45,669 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - FileLogWriter.FsyncThread-1425661691796 finished | |||
2015-03-06 12:13:45,678 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - Redo log rollover took 166ms | |||
2015-03-06 12:13:45,678 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore started for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6 | |||
2015-03-06 12:13:45,700 INFO [FileLogWriter.FsyncThread-1425662025678] [] redolog - Starting fsync thread with interval 10 | |||
2015-03-06 12:13:45,981 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - Creating database mboxgroup17 | |||
2015-03-06 12:13:46,695 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restoring to mailbox id 17 for new account f7f4c1a8-15d0-41a1-bd11-6e05359ea3e8. | |||
2015-03-06 12:13:46,745 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loading schema with version 7.2 | |||
2015-03-06 12:13:46,785 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Renaming volume_id column to locator in source table mail_item | |||
2015-03-06 12:13:46,820 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 0 blobs during restore | |||
2015-03-06 12:13:46,822 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Deleting directory /opt/zimbra/index/0/17/index | |||
2015-03-06 12:13:46,882 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 2 index files during restore | |||
2015-03-06 12:13:46,887 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/17/index/0 | |||
2015-03-06 12:13:46,887 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox 17 account f7f4c1a8-15d0-41a1-bd11-6e05359ea3e8 LOADED | |||
2015-03-06 12:13:46,887 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] cache - initializing folder and tag caches for mailbox 17 | |||
2015-03-06 12:13:46,912 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Account user1@mail-172.example.com: All pending file IO completed (2 out of 2) | |||
2015-03-06 12:13:46,912 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Run redo operations since last full backup | |||
2015-03-06 12:13:47,047 ERROR [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option | |||
2015-03-06 12:13:47,047 WARN [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Redo log sequence error: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option | |||
com.zimbra.cs.backup.BackupServiceException: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option | |||
ExceptionId:qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest:1425662027047:14586f805b4c1b0e | |||
Code:backup.REDOLOG_OUT_OF_SEQUENCE | |||
at com.zimbra.cs.backup.BackupServiceException.REDOLOG_OUT_OF_SEQUENCE(BackupServiceException.java:91) | |||
at com.zimbra.cs.backup.util.Utils.splitRedoLogsAtSeq(Utils.java:302) | |||
at com.zimbra.cs.backup.RestoreAccountSession.checkRedoLogSequenceContiguity(RestoreAccountSession.java:534) | |||
at com.zimbra.cs.backup.RestoreAccountSession.endRestore(RestoreAccountSession.java:350) | |||
at com.zimbra.cs.backup.FileBackupTarget$RestoreAcctSession.endRestore(FileBackupTarget.java:2524) | |||
at com.zimbra.cs.backup.FileBackupTarget.restore(FileBackupTarget.java:490) | |||
at com.zimbra.cs.backup.BackupManager.restore(BackupManager.java:826) | |||
at com.zimbra.cs.service.backup.Restore.handle(Restore.java:137) | |||
at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEngine.java:569) | |||
at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:432) | |||
at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:266) | |||
at com.zimbra.soap.SoapServlet.doWork(SoapServlet.java:303) | |||
at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:213) | |||
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) | |||
at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:209) | |||
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) | |||
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651) | |||
at com.zimbra.cs.servlet.RequestStringFilter.doFilter(RequestStringFilter.java:54) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) | |||
at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:59) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) | |||
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) | |||
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:351) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) | |||
at com.zimbra.cs.servlet.ETagHeaderFilter.doFilter(ETagHeaderFilter.java:47) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) | |||
at com.zimbra.cs.servlet.ContextPathBasedThreadPoolBalancerFilter.doFilter(ContextPathBasedThreadPoolBalancerFilter.java:107) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) | |||
at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(ZimbraQoSFilter.java:107) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) | |||
at com.zimbra.cs.servlet.ZimbraInvalidLoginFilter.doFilter(ZimbraInvalidLoginFilter.java:117) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) | |||
at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:457) | |||
at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:326) | |||
at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:299) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) | |||
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549) | |||
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) | |||
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:544) | |||
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221) | |||
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111) | |||
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478) | |||
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183) | |||
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045) | |||
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) | |||
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:199) | |||
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109) | |||
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) | |||
at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:309) | |||
at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:81) | |||
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) | |||
at org.eclipse.jetty.server.Server.handle(Server.java:462) | |||
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279) | |||
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232) | |||
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534) | |||
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607) | |||
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536) | |||
at java.lang.Thread.run(Thread.java:745) | |||
2015-03-06 12:13:47,047 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Cleaning up data from failed restore (mid=17) | |||
2015-03-06 12:13:47,063 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - Putting mailbox 17 under maintenance. | |||
2015-03-06 12:13:47,084 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - clearing contents of mailbox 17, group 17 | |||
2015-03-06 12:13:47,331 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox for account f7f4c1a8-15d0-41a1-bd11-6e05359ea3e8 DELETED | |||
2015-03-06 12:13:47,375 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Cleaning up account created during failed restore (account=f7f4c1a8-15d0-41a1-bd11-6e05359ea3e8) | |||
2015-03-06 12:13:47,752 WARN [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Error occurred during restore account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) | |||
com.zimbra.common.service.ServiceException: system failure: Restore encountered redo log sequence error: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option | |||
ExceptionId:qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest:1425662027047:14586f805b4c1b0e | |||
Code:service.FAILURE | |||
at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:260) | |||
at com.zimbra.cs.backup.RestoreAccountSession.checkRedoLogSequenceContiguity(RestoreAccountSession.java:545) | |||
at com.zimbra.cs.backup.RestoreAccountSession.endRestore(RestoreAccountSession.java:350) | |||
at com.zimbra.cs.backup.FileBackupTarget$RestoreAcctSession.endRestore(FileBackupTarget.java:2524) | |||
at com.zimbra.cs.backup.FileBackupTarget.restore(FileBackupTarget.java:490) | |||
at com.zimbra.cs.backup.BackupManager.restore(BackupManager.java:826) | |||
at com.zimbra.cs.service.backup.Restore.handle(Restore.java:137) | |||
at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEngine.java:569) | |||
at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:432) | |||
at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:266) | |||
at com.zimbra.soap.SoapServlet.doWork(SoapServlet.java:303) | |||
at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:213) | |||
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) | |||
at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:209) | |||
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) | |||
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651) | |||
at com.zimbra.cs.servlet.RequestStringFilter.doFilter(RequestStringFilter.java:54) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) | |||
at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:59) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) | |||
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) | |||
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:351) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) | |||
at com.zimbra.cs.servlet.ETagHeaderFilter.doFilter(ETagHeaderFilter.java:47) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) | |||
at com.zimbra.cs.servlet.ContextPathBasedThreadPoolBalancerFilter.doFilter(ContextPathBasedThreadPoolBalancerFilter.java:107) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) | |||
at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(ZimbraQoSFilter.java:107) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) | |||
at com.zimbra.cs.servlet.ZimbraInvalidLoginFilter.doFilter(ZimbraInvalidLoginFilter.java:117) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) | |||
at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:457) | |||
at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:326) | |||
at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:299) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) | |||
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549) | |||
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) | |||
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:544) | |||
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221) | |||
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111) | |||
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478) | |||
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183) | |||
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045) | |||
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) | |||
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:199) | |||
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109) | |||
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) | |||
at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:309) | |||
at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:81) | |||
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) | |||
at org.eclipse.jetty.server.Server.handle(Server.java:462) | |||
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279) | |||
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232) | |||
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534) | |||
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607) | |||
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536) | |||
at java.lang.Thread.run(Thread.java:745) | |||
Caused by: com.zimbra.cs.backup.BackupServiceException: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option | |||
ExceptionId:qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest:1425662027047:14586f805b4c1b0e | |||
Code:backup.REDOLOG_OUT_OF_SEQUENCE | |||
at com.zimbra.cs.backup.BackupServiceException.REDOLOG_OUT_OF_SEQUENCE(BackupServiceException.java:91) | |||
at com.zimbra.cs.backup.util.Utils.splitRedoLogsAtSeq(Utils.java:302) | |||
at com.zimbra.cs.backup.RestoreAccountSession.checkRedoLogSequenceContiguity(RestoreAccountSession.java:534) | |||
... 56 more | |||
2015-03-06 12:13:47,755 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is shut down | |||
2015-03-06 12:13:47,755 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] SoapEngine - handler exception | |||
com.zimbra.common.service.ServiceException: system failure: Restore encountered redo log sequence error: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option | |||
ExceptionId:qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest:1425662027047:14586f805b4c1b0e | |||
Code:service.FAILURE | |||
at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:260) | |||
at com.zimbra.cs.backup.RestoreAccountSession.checkRedoLogSequenceContiguity(RestoreAccountSession.java:545) | |||
at com.zimbra.cs.backup.RestoreAccountSession.endRestore(RestoreAccountSession.java:350) | |||
at com.zimbra.cs.backup.FileBackupTarget$RestoreAcctSession.endRestore(FileBackupTarget.java:2524) | |||
at com.zimbra.cs.backup.FileBackupTarget.restore(FileBackupTarget.java:490) | |||
at com.zimbra.cs.backup.BackupManager.restore(BackupManager.java:826) | |||
at com.zimbra.cs.service.backup.Restore.handle(Restore.java:137) | |||
at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEngine.java:569) | |||
at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:432) | |||
at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:266) | |||
at com.zimbra.soap.SoapServlet.doWork(SoapServlet.java:303) | |||
at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:213) | |||
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) | |||
at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:209) | |||
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) | |||
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651) | |||
at com.zimbra.cs.servlet.RequestStringFilter.doFilter(RequestStringFilter.java:54) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) | |||
at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:59) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) | |||
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) | |||
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:351) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) | |||
at com.zimbra.cs.servlet.ETagHeaderFilter.doFilter(ETagHeaderFilter.java:47) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) | |||
at com.zimbra.cs.servlet.ContextPathBasedThreadPoolBalancerFilter.doFilter(ContextPathBasedThreadPoolBalancerFilter.java:107) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) | |||
at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(ZimbraQoSFilter.java:107) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) | |||
at com.zimbra.cs.servlet.ZimbraInvalidLoginFilter.doFilter(ZimbraInvalidLoginFilter.java:117) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) | |||
at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:457) | |||
at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:326) | |||
at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:299) | |||
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) | |||
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549) | |||
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) | |||
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:544) | |||
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221) | |||
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111) | |||
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478) | |||
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183) | |||
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045) | |||
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) | |||
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:199) | |||
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109) | |||
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) | |||
at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:309) | |||
at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:81) | |||
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) | |||
at org.eclipse.jetty.server.Server.handle(Server.java:462) | |||
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279) | |||
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232) | |||
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534) | |||
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607) | |||
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536) | |||
at java.lang.Thread.run(Thread.java:745) | |||
Caused by: com.zimbra.cs.backup.BackupServiceException: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option | |||
ExceptionId:qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest:1425662027047:14586f805b4c1b0e | |||
Code:backup.REDOLOG_OUT_OF_SEQUENCE | |||
at com.zimbra.cs.backup.BackupServiceException.REDOLOG_OUT_OF_SEQUENCE(BackupServiceException.java:91) | |||
at com.zimbra.cs.backup.util.Utils.splitRedoLogsAtSeq(Utils.java:302) | |||
at com.zimbra.cs.backup.RestoreAccountSession.checkRedoLogSequenceContiguity(RestoreAccountSession.java:534) | |||
... 56 more | |||
2015-03-06 12:13:47,755 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - RestoreRequest elapsed=2358 | |||
[zimbra@mail-172 archive]$ zmprov ga restore6_user1@`zmhostname` | head | |||
ERROR: account.NO_SUCH_ACCOUNT (no such account: restore6_user1@mail-172.example.com) | |||
</pre> | |||
Let's now see what happens if we move the missing redolog seq files back into /opt/zimbra/redolog/archive and then if we move/copy them into the backup sessions redolog directory. With restore attempts being done after each move/copy. | |||
<pre> | |||
[zimbra@mail-172 archive]$ mv /tmp/redolog-archive/* ./ | |||
[zimbra@mail-172 archive]$ pwd | |||
/opt/zimbra/redolog/archive | |||
[zimbra@mail-172 archive]$ ls | |||
redo-20150306.060015.331-seq21.log redo-20150306.060109.166-seq22.log redo-20150306.170811.794-seq23.log redo-20150306.171345.676-seq24.log | |||
[zimbra@mail-172 archive]$ zmrestore -a user1@`zmhostname` -ca -pre restore6_ | |||
Error occurred: system failure: Restore encountered redo log sequence error: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option | |||
[zimbra@mail-172 archive]$ pwd | |||
/opt/zimbra/redolog/archive | |||
[zimbra@mail-172 archive]$ cp *seq21.log *seq22.log /opt/zimbra/backup/sessions/incr-20150306.170810.793/redologs/ | |||
[zimbra@mail-172 archive]$ zmrestore -a user1@`zmhostname` -ca -pre restore6_ | |||
[zimbra@mail-172 archive]$ zmprov ga restore6_user1@`zmhostname` | head -2 | |||
# name restore6_user1@mail-172.example.com | |||
cn: user1 | |||
</pre> | |||
====Redirected Restore With No Deletes To Then Import Data From A Specific Time Range Into A Subdirectory Of Users Archive Account==== | |||
This example would be the steps one might do if you discovered that the archiving service wasn't working as expected for a given date range and you wanted to populate the users archive account with the missing data. Normally, this would involve many users or all of your users - hence, why the issue about licenses is addressed. | |||
Note, I use the following variables below but you should be replacing them to suit your needs. | |||
* $USER = users production email | |||
* recovery_$USER = the new email address that the restore created [-ca -pre recovery_] | |||
* $DOMAIN = the domain for the user | |||
* $USER-archive@$DOMAIN.archive = the production users archive email address | |||
Determine all the necessary variables for your restore command. | |||
* zmrestore -a $USER | |||
** -ca -pre recovery_ | |||
** -lb determine what full backup label to use | |||
** determine if you need to state the incremental backup label [-restoreToIncrLabel] and/or a specific time to stop the restore at [-restoreToTime] | |||
** determine if you need to use -rf or -br | |||
** --skipDeletes | |||
For example: | |||
<pre> | |||
$ zmrestore -a user1@`zmhostname` -ca -pre recovery_ -lb full-20150321.050015.999 -restoreToTime 20150323000000 -br --skipDeletes | |||
using restore to time of 2015/03/23 00:00:00 | |||
</pre> | |||
Set the restored account to not consume a normal license [notice recovery_$USER vs $USER]: | |||
<pre>$ zmprov ma recovery_$USER zimbraIsSystemResource TRUE</pre> | |||
Get what the archive accounts address is for the main account: | |||
<pre>$ zmprov ga $USER amavisArchiveQuarantineTo</pre> | |||
Set the restored account to not consume an archive license [notice recovery_$USER vs $USER]: | |||
<pre>$ zmprov ma recovery_$USER amavisArchiveQuarantineTo ""</pre> | |||
Create a directory to dump all the export data in. You might want to have this shared across your mailstores [that hold archive accounts] so you can then import them locally to the mailstore the archive account resides on. | |||
<pre> | |||
$ mkdir /tmp/export | |||
$ cd /tmp/export | |||
</pre> | |||
Example exports out all data between a given date , adjust the account and date string below. Also, the filename that is being saved. | |||
<pre> | |||
$ zmmailbox -z -m recovery_$USER@$DOMAIN gru '//?fmt=tgz&query=under:/ after:"4/8/15" AND before:"4/11/15"' > /tmp/export/$USER.tgz | |||
</pre> | |||
Example of what is dumped | |||
<pre> | |||
$ pwd | |||
/tmp/export | |||
$ ls | |||
admin.tgz | |||
$ tar zxvf $USER.tgz | |||
Inbox/0000000776-ZCS Backup Report_ SUCCESS.eml.meta | |||
Inbox/0000000776-ZCS Backup Report_ SUCCESS.eml | |||
Inbox/0000000777-Daily mail report for 2015-04-09.eml.meta | |||
Inbox/0000000777-Daily mail report for 2015-04-09.eml | |||
Inbox/0000000778-ZCS Backup Report_ SUCCESS.eml.meta | |||
Inbox/0000000778-ZCS Backup Report_ SUCCESS.eml | |||
Inbox/0000000779-Daily mail report for 2015-04-10.eml.meta | |||
Inbox/0000000779-Daily mail report for 2015-04-10.eml | |||
</pre> | |||
Now to import the data into the users Archive account. Below example will dump the imported data into a specfic folder that will be created [I'm using Imported]. Adjust the account and filename below. | |||
<pre> | |||
$ zmmailbox -z -m $USER-archive@$DOMAIN.archive pru "//?fmt=tgz&subfolder=Imported" /tmp/export/$USER.tgz | |||
$ zmmailbox -z -m $USER-archive@$DOMAIN.archive gaf | |||
Id View Unread Msg Count Path | |||
---------- ---- ---------- ---------- ---------- | |||
1 unkn 0 0 / | |||
16 docu 0 0 /Briefcase | |||
10 appo 0 0 /Calendar | |||
14 mess 0 0 /Chats | |||
7 cont 0 0 /Contacts | |||
6 mess 0 0 /Drafts | |||
13 cont 0 0 /Emailed Contacts | |||
789 unkn 0 0 /Imported | |||
790 mess 4 4 /Imported/Inbox | |||
2 mess 415 420 /Inbox | |||
4 mess 0 0 /Junk | |||
5 mess 0 0 /Sent | |||
15 task 0 0 /Tasks | |||
3 unkn 0 0 /Trash | |||
</pre> | |||
Here's a (customer-supplied) script that does the above in batch. It was used to restore archives from a date range for 29k users: https://github.com/morganlljones/zimbra_backup2archive | |||
== Quiz == | |||
# What are three methods you can locate which backups a user is in? | |||
# To stop a restore from playing all the way up to the current time, you would need to use one of these two zmrestore variables - what are they? | |||
# If you see an error about being unable to restore because of redologs, what option might you need to include? | |||
# What is the primary log file on the ZCS server you'll monitor or review for restore or backup issues? | |||
# Provide two methods to increase logging/debug output when attempting a restore? | |||
# Can you perform a full system restore from the backups done on an older ZCS version? | |||
# What are the two identify variables that zmbackup/zmrestore uses for an individual? | |||
# What user variable would you use to locate the directory a users data is stored in within a backup session? | |||
{{Article_Footer|Zimbra Collaboration Suite 8.6|01/22/2015}} | {{Article_Footer|Zimbra Collaboration Suite 8.6|01/22/2015}} | ||
[[Category:Troubleshooting_Course_Content_Rough_Drafts]] | [[Category:Troubleshooting_Course_Content_Rough_Drafts]] | ||
[[Category: Author:Ajcody]] |
Latest revision as of 20:34, 7 June 2016
Bug & RFE's I Made While Researching This Draft Write Up [Remove For Course]
- [story] Ability to search data within backup and do "item" restores or identify locations of search results
- admin console backup label view doesn't list accounts in the all accounts tab
- admin console restore - doesn't autocomplete / suggest account matches when filling out email address box
- document new restore functions / options with ZCS 8+ for admin console restore
- admin console restore - rename "Selected Servers" panel to "Restore Options"
- admin console restore - if only one mailstore in env. then state such in second panel of restore about "server for the restored accounts"
- admin console restore - expand restore To options - To full backup label, To incremental target
- admin console restore - "restore to the latest backup" incorrectly described / broken
- admin console restore - unable to restore individual accounts [sort of]
- admin console restore - reuse GAL/Contact Picker Window for "restore individual accounts"
- Default COS value [ zimbraPrefShowSelectionCheckbox ] for checkboxes in lists should be TRUE and improve 'select all' functionality
- zmschedulebackup ignores zmbackup options in auto-grouped mode
Introduction
- Will you please provide a brief paragraph introduction/overview about recovering missing data in Zimbra.
- For example: what are some scenarios wherein you need to recover missing data.
- What is the general method for recovery?
- How/Why is the backup environment relevant/important.
- Remember that this won't be obvious for all audience members.
User Issues
Questions To Ask & Address Prior To Doing A User Restore
- Can data be restored from the user's dumpster [if enabled] ?
- Is there a copy of the data in a third party client ? [POP email client that downloaded it, for example]
- Does the data maybe exist in another user's account ? [For example, the senders account or others in the To/CC]
- Does the data reside in a redolog/s ?
- Finally, restore from backups ?
- Do I need to overwrite the existing account with the restore or should I restore to a new account name?
- Using the option -ca [create account] and -pre restore_ [prefix to the new account name, restore_user@ ]
- What are your retention policies and how it might determine the backup set you need?
- Do I need to overwrite the existing account with the restore or should I restore to a new account name?
User Restores
- Admin Console
- Restore/Backup
- Redirected restore : -ca -pre restored_
- View Mail > TGZ export/import
- Crossmailboxsearch
- Importing Restored Data Into Parent Account
Restoring A User To A Specific Full Backup, Incremental Backup, Or Point In Time
One of the more common support issues around restoring a user account is the complaint that the restore didn't roll back the account to the desired state. Almost always, this is because the proper options were not passed to the zmrestore command for the restore to stop at the desired time frame. If you don't give the proper options to zmrestore, it will restore the account all the way to the most recent redolog operations on the server. For example, it replays the delete operations that caused the user to request to restore in the first place.
You MUST use the -lb [full backup label name] option when your trying to restore anything that ISN'T meant to include the latest information for the mailbox. The -lb argument should specify a full backup that took place prior to the time of the backup you wish to restore.
Find Out What Backup Session Labels You Need First
To find out what backups are associated with a particular account, you would do the following :
zmbackupquery -a user@domain
You'll want to note what is the first full that occurs before the point in time you want to restore. And then the incremental that follows right after your point in time.
Backup labels (-lb) for fulls for a particular user can be found by doing :
zmbackupquery -a user@domain --type full
Backup labels (-restoreToIncrLabel) for incrementals can be found by doing :
zmbackupquery -a user@domain --type incremental
Command Syntax Example For Restores On The CLI
Example to restore ONLY the data in the full backup session and stop :
zmrestore -a USER@DOMAIN.com -lb full-20080726.050017.306 -rf -ca -pre restore_
Example to restore only the data in the full backup session and all the incremental data after the full to a specific incremental label and stop :
zmrestore -a USER@DOMAIN.com -restoreToIncrLabel incr-20080731.060007.644 -lb full-20080726.050017.306 -br -ca -pre restore_
Example to restore to a specific time and stop :
zmrestore -a USER@DOMAIN.com -restoreToTime 20080801011800 -lb full-20080726.050017.306 -br -ca -pre restore_
Important Options You Might Want Or Need To Include
--ignoreRedoErrors : If you attempt a restore and you see an error about problems related to playing the redolog, you'll want to run the restore command again and include this option.
--skipDeletes : Please see http://bugzilla.zimbra.com/show_bug.cgi?id=31824#c5 for details on this.
-t /path/to/backup_dir : If you are restoring from another backup directory besides your current default path.
Variables that are asking for TIME rather than LABELS should follow this syntax (from zmrestore --help):
Specify date/time in one of these formats: 2008/08/06 09:55:50 2008/08/06 09:55:50 572 2008/08/06 09:55:50.572 2008/08/06-09:55:50-572 2008/08/06-09:55:50 20080806.095550.572 20080806.095550 20080806095550572 20080806095550 Specify year, month, date, hour, minute, second, and optionally millisecond. Month/date/hour/minute/second are 0-padded to 2 digits, millisecond to 3 digits. Hour must be specified in 24-hour format, and time is in local time zone.
Trouble-shooting Backup/Restore Issues And Other General Questions
Backup And Restore Compatibility Between ZCS Versions
- You can use backups from older versions of ZCS for account restores.
- You can NOT use backups from old versions of ZCS for system restores though. Disaster Recovery restores have to be done with the same version of ZCS as the backups were done with.
References:
- Upgrades And Compatibility Of Older Backups
- "Backups must be compatible across patch releases"
- "Incorrect upgrade documentation regarding backups"
- "support for restore across major versions"
- "Add conversion tool to upgrade backup versions to allow restore on later zcs versions"
- From 5.0.7 - 5.0.10 You Might See minor version upgrades moving your backups into a subdirectory
- "upgrade incorrectly invalidates backups."
Trouble-shooting
Ref:
Basic Backup Information To Submit To Support
Disk Space Usage Issues
Trend Data
If there is concerns about disks/partitions getting full, this command would be helpful for trending data on your server. Send support the resulting df.tar file . Note - adjust the tail command if you want more than 20 day's worth of trending data, the -n 20 option.
[zimbra@zcs806 tmp]$ /tmp [zimbra@zcs806 tmp]$ tar cvf /tmp/df.tar `find /opt/zimbra/zmstat -name df.cs\* | sort | tail -n 20` tar: Removing leading `/' from member names /opt/zimbra/zmstat/2014-03-10/df.csv.gz /opt/zimbra/zmstat/2014-03-11/df.csv.gz [cut - Ajcody] /opt/zimbra/zmstat/2014-03-27/df.csv.gz /opt/zimbra/zmstat/2014-03-28/df.csv.gz /opt/zimbra/zmstat/df.csv [zimbra@zcs806 tmp]$ ls -lah /tmp/df.tar -rw-r----- 1 zimbra zimbra 80K Mar 29 06:44 /tmp/df.tar [zimbra@zcs806 tmp]$ tar tvf /tmp/df.tar -rw-r----- zimbra/zimbra 2566 2014-03-11 00:00 opt/zimbra/zmstat/2014-03-10/df.csv.gz -rw-r----- zimbra/zimbra 2553 2014-03-12 00:00 opt/zimbra/zmstat/2014-03-11/df.csv.gz [cut - Ajcody] -rw-r----- zimbra/zimbra 2513 2014-03-28 00:00 opt/zimbra/zmstat/2014-03-27/df.csv.gz -rw-r----- zimbra/zimbra 2531 2014-03-29 00:00 opt/zimbra/zmstat/2014-03-28/df.csv.gz -rw-r----- zimbra/zimbra 8013 2014-03-29 06:40 opt/zimbra/zmstat/df.csv
Directory Sizes In /opt/zimbra
Please see the following and provide the output to support. Note, even though this method is faster than doing a du it still can take awhile.
* Ajcody-Server-Misc-Topics#Faster_Way_To_Get_Directory_Size_On_Filesytem_-_find_vs_du
Adjusting The Disk Alert Threshold
Note - zmlocalconfig smtp_notify must return yes if you want to receive the notifications.
If you just need to adjust the disk alert threshold, then see the following:
See current values:
zmlocalconfig | grep zmdisklog
Example adjustment:
su - zimbra zmlocalconfig -e zmdisklog_critical_threshold=98 zmlocalconfig -e zmdisklog_warn_threshold=95 zmstatctl
To exclude a partition from the checks [example of two being excluded]:
su - zimbra zmlocalconfig -e zmstat_df_excludes="/mount/point:/mount/point2" zmstatctl
Note, depending on your version of ZCS, you might be hitting a bug where you'll keep getting emails until a logrotate happens.
- Changing Zmstat-df values do not take affect until logrotate
Some things to do to confirm and share with support or in bug. As zimbra
su - zimbra ls -la /var/log/zimbra.log df -h /dev/mapper/vg_rhel664-lv_root 5.5G 3.5G 1.7G 68% / tmpfs 939M 0 939M 0% /dev/shm /dev/sda1 485M 79M 381M 18% /boot /dev/sdb1 30G 6.2G 23G 22% /opt date zmlocalconfig | grep zmdisklog zmdisklog_critical_threshold = 80 zmdisklog_warn_threshold = 85 zmlocalconfig -e zmdisklog_critical_threshold=95 zmlocalconfig -e zmdisklog_warn_threshold=90 zmlocalconfig | grep zmdisklog zmdisklog_critical_threshold = 95 zmdisklog_warn_threshold = 90 zmstatctl restart date ps -eaf | grep zmstat-df ls -la /var/log/zimbra.log date ; grep "Disk warning" /var/log/zimbra* ; zmmailbox -z -m admin@`zmhostname` s -l 100 -t message "Subject: Disk and after:yesterday" ##Note - Emails by default go out every 10 minutes - for example: [zimbra@zcs803 ~]$ date ; grep "Disk warning" /var/log/zimbra* ; zmmailbox -z -m admin@`zmhostname` s -l 100 -t message "Subject: Disk and after:yesterday" Thu May 22 09:40:08 PDT 2014 /var/log/zimbra.log:May 22 08:30:00 zcs803 zimbramon[18826]: 18826:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82% /var/log/zimbra.log:May 22 08:40:00 zcs803 zimbramon[22970]: 22970:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82% /var/log/zimbra.log:May 22 08:50:00 zcs803 zimbramon[22970]: 22970:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82% /var/log/zimbra.log:May 22 09:00:00 zcs803 zimbramon[22970]: 22970:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82% ## Note - I had readjusted the variable to not warn during this time segment ## /var/log/zimbra.log:May 22 09:20:00 zcs803 zimbramon[8322]: 8322:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82% /var/log/zimbra.log:May 22 09:30:00 zcs803 zimbramon[8322]: 8322:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82% /var/log/zimbra.log:May 22 09:40:00 zcs803 zimbramon[8322]: 8322:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82% num: 7, more: false Id Type From Subject Date ---- ---- -------------------- -------------------------------------------------- -------------- 1. 328 mess admin Disk / at 82% on zcs803.DOMAIN.com: 05/22/14 09:40 2. 327 mess admin Disk / at 82% on zcs803.DOMAIN.com: 05/22/14 09:30 3. 326 mess admin Disk / at 82% on zcs803.DOMAIN.com: 05/22/14 09:20 ## Note - I had readjusted the variable to not warn during this time segment ## 4. 325 mess admin Disk / at 82% on zcs803.DOMAIN.com: 05/22/14 09:00 5. 324 mess admin Disk / at 82% on zcs803.DOMAIN.com: 05/22/14 08:50 6. 323 mess admin Disk / at 82% on zcs803.DOMAIN.com: 05/22/14 08:40 7. 320 mess admin Disk / at 82% on zcs803.DOMAIN.com: 05/22/14 08:31
Continue to monitor your zmmailbox search results for an hour.
The Basic Information Support Needs
as root:
- cat /etc/fstab
- Shows us what is mounted upon boot
- cat /proc/mounts
- Shows us what is currently mounted and its status - you can see if a mount is read-only here.
- df -hT
- Lists current mounts using human-readable size information and also notes the filesystem type.
as zimbra:
- zmprov -l gs `zmhostname` | egrep 'Back|Redo'
- Will show us a number of variables related to backup and redologs. Also tell us if your using auto-group or the default method.
- du -sh /opt/zimbra/redolog
- Will might notice your redolog logs aren't rolling over, causing a possible issue.
- ls -latr /opt/zimbra/backup
- This is the default backup target, please adjust this path here and below if you are using a different zimbraBackupTarget value.
- zmprov gs `zmhostname` zimbraBackupTarget
- We'll be able to confirm permissions are right.
- ls -latr /opt/zimbra/backup/tmp
- This will show us if you have failed backup jobs and confirm tmp is being cleaned appropriately after the backup is done.
- ls -latr /opt/zimbra/backup/sessions
- This will show us what backup sessions are available and confirm permissions are correct.
- Adjust path if your zimbraBackupTarget value is not the default path.
- Some directory sizes in the backup directory:
- Default path first
- du -sh `find /opt/zimbra/backup -maxdepth 2 -type d`
- If your using a different backup target, check that directory also. Replace /opt/zimbra/backup above with your backup path.
- Default path first
- zmbackupquery
- This should match what's in the sessions directory and it will also tell us if status of each backup and how many accts were done.
- crontab -l | grep -i back
- This will show use when backups are support to run and with what options they are running with.
- zmlocalconfig | grep -i back
- This is useful to see a number of backup options not exposed in the crontab, things related to the zip options.
- zmvolume -l
- This is useful to see how many volumes are being used, if HSM is being used, and if compression is being done at the volume level.
Additional Log Files Support Might Need
And send the following logs:
- /var/log/messages
- Filesystem issues often times are noted here and also in syslog. This might explain an interruption in the backup process. Server restarts, filesystem going full, filesystem going read-only, etc.
- /var/log/syslog
- /opt/zimbra/log/mailbox.log
- The backup activity is logged here.
- And any other mailbox.log file that would cover the event
Some Specific Example Restore/Backup Issues
NO_SUCH_ACCOUNT_BACKUP
Example errors:
Error code: backup.NO_SUCH_ACCOUNT_BACKUP Message: no such backup for account: Missing full backup earlier than restore-to time for account 3c397edc-c015-4d82-ae8a-1084692f8a93 Details:soap:Sender Error occurred: no such backup for account: 3c397edc-c015-4d82-ae8a-1084692f8a93 Message: no such backup for account: Missing full backup earlier than restore-to time for account 3c397edc-c015-4d82-ae8a-1084692f8a93 3c397edc-c015-4d82-ae8a-1084692f8a93 is an example of the zimbraId of a user [ user1@mail-172.example.com ] .
Confirm the current email address is mapped to the appropriate zimbraId. You'll want to search what is in ldap and the mailstores database for the exiting email address and then also search the reference files in your backups. Often times, this happens when an account was deleted and then created again. Account meaning the email address. When this is does, the new account will have a new zimbraId.
$ zmprov -l ga user1@mail-172.example.com zimbraId # name user1@mail-172.example.com zimbraId: 3c397edc-c015-4d82-ae8a-1084692f8a93 $ mysql -e 'SELECT * FROM zimbra.mailbox WHERE comment LIKE "user1@mail-172.example.com"\G' $ mysql -e 'SELECT * FROM zimbra.mailbox WHERE account_id="3c397edc-c015-4d82-ae8a-1084692f8a93"\G' $ grep -C1 3c397edc-c015-4d82-ae8a-1084692f8a93 /opt/zimbra/backup/accounts.xml [using the default path; /opt/zimbra/backup/accounts.xml] $ egrep "user1@mail-172.example.com|3c397edc-c015-4d82-ae8a-1084692f8a93" /opt/zimbra/backup/accounts.xml $ egrep "user1@mail-172.example.com|3c397edc-c015-4d82-ae8a-1084692f8a93" /opt/zimbra/backup/sessions/*/session.xml
exception during auth
Example error:
LDAP backup failed: system failure: exception during auth {RemoteManager: zimbramail.example.com->zimbra@zimbramail.example.com:22} com.zimbra.common.service.ServiceException: system failure: LDAP backup failed: system failure: exception during auth {RemoteManager: zimbramail.example.com-> zimbra@zimbramail.example.com:22}
This indicates the ssh keys aren't configured correctly in the environment. You should be able to do the following to resolve the issue:
su - zimbra zmsshkeygen zmupdateauthkeys
Note - You should also confirm that you do indeed have the ssh port available. The default is port 22. If your unable to ssh into the server using port 22, then this might be the issue. If you configured ssh to listen on a different port on this server, you'll need to adjust the following variable - zmprov gcf zimbraRemoteManagementPort . You might also have the ssh port blocked - confirm your firewall, iptables, etc is not blocking the ssh port.
LICENSE_ERROR_Message: AccountsLimits exceeded
Example error:
LICENSE_ERROR_Message: AccountsLimits exceeded
You should check your license and confirm it's valid and active:
su - zimbra zmlicense -p
And confirm how many licenses are used: First, flush the cache on all servers for the license information. Then do the query via soap.
zmprov fc -a license zmsoap -z GetLicenseRequest
Four lines that are generally of interest are shown below. The first two state how many one is licensed for, the bottom two show how many are currently used :
<attr name="AccountsLimit">1000</attr> <attr name="ArchivingAccountsLimit">1000</attr> <attr name="TotalAccounts">3</attr> <attr name="ArchivingAccounts">3</attr>
Hands On How-To And Reviewing Of Log Events
Full Backup And Log Events Generated
Example of manually starting a full backup using the Standard zimbraBackupMode and doing a query to confirm it's completion and status.
[zimbra@mail-172 archive]$ zmprov gs `zmhostname` zimbraBackupMode # name mail-172.example.com zimbraBackupMode: Standard [zimbra@mail-172 archive]$ pwd ; ls *.log /opt/zimbra/redolog/archive redo-20150303.060015.816-seq9.log [zimbra@mail-172 archive]$ zmbackup -f -a all full-20150304.012000.193 [zimbra@mail-172 archive]$ zmbackupquery -lb full-20150304.012000.193 -v Label: full-20150304.012000.193 Type: full Status: completed Started: Tue, 2015/03/03 20:20:00.193 EST Ended: Tue, 2015/03/03 20:20:28.630 EST Redo log sequence range: 10 .. 10 Number of accounts: 10 out of 10 completed Accounts: admin@mail-172.example.com: completed galsync.qva4ekog0@mail-172.example.com: completed ham.jgfdb3oqjf@mail-172.example.com: completed spam.umhx3owv3s@mail-172.example.com: completed user1@mail-172.example.com: completed user2@mail-172.example.com: completed user3@mail-172.example.com: completed user4@mail-172.example.com: completed user5@mail-172.example.com: completed virus-quarantine.gusiso1cd@mail-172.example.com: completed Total space: 14020MB Free space: 6017MB
Example of what log events are generated for a full backup in /opt/zimbra/log/mailbox.log . First, you'll see the authentication/acl request and then statements about the backup request.
2015-03-03 20:19:58,499 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [] AuthProvider - Adding auth provider: zimbra com.zimbra.cs.service.ZimbraAuthProvider 2015-03-03 20:19:59,458 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] soap - AuthRequest elapsed=834 2015-03-03 20:19:59,751 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] account - Initialized access manager: com.zimbra.cs.account.accesscontrol.ACLAccessManager 2015-03-03 20:19:59,770 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup request started 2015-03-03 20:20:00,193 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Found 10 accounts on server mail-172.example.com 2015-03-03 20:20:00,464 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup request finished 2015-03-03 20:20:00,464 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] soap - BackupRequest elapsed=760
Your log events will now reference Backup in them. AsyncFileCopier will start and it will give the backup session label for this backup job.
2015-03-03 20:20:01,092 INFO [FullBackupThread] [] io - AsyncFileCopier is starting 2015-03-03 20:20:01,187 INFO [FullBackupThread] [] backup - Full backup started for backup set; label: full-20150304.012000.193
It proceeds to backing up your accounts. I'll list the backup log events for just two accounts on this system - user1@mail-172.example.com and admin@mail-172.example.com . I'll provide the zimbraId, email address, and mailboxID information for user1@mail-172.example.com below so you can confirm it's reported in the logs correctly as well.
[zimbra@mail-172 redolog]$ zmprov ga user1@mail-172.example.com zimbraId # name user1@mail-172.example.com zimbraId: 3c397edc-c015-4d82-ae8a-1084692f8a93 [zimbra@mail-172 redolog]$ zmprov gmi user1@mail-172.example.com mailboxId: 6 quotaUsed: 1209
Now, continuing with the log events. Notice the reference to the redo log file sequence, putting the account into and out of maintenance, number of blobs and index files backed up, and the statement it's finished backing up the account. Also within the section we can see the user's zimbraId [3c397edc-c015-4d82-ae8a-1084692f8a93] , email address [user1@mail-172.example.com] , and mailboxId [Mailbox 6].
[cut] 2015-03-03 20:20:06,560 INFO [FullBackupThread] [name=user1@mail-172.example.com;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/6/index/0 2015-03-03 20:20:06,560 INFO [FullBackupThread] [name=user1@mail-172.example.com;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 LOADED 2015-03-03 20:20:06,560 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - redo log file sequence is 10 at full backup for user1@mail-172.example.com 2015-03-03 20:20:06,565 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - Full backup started for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) mailbox 6 2015-03-03 20:20:06,565 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] mailbox - Putting mailbox 6 under maintenance. 2015-03-03 20:20:06,818 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - Number of blobs to backup for mailbox 6: 1 2015-03-03 20:20:06,818 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] mailbox - Ending maintenance on mailbox 6. 2015-03-03 20:20:06,818 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 AVAILABLE 2015-03-03 20:20:06,966 INFO [BackupSetWorkerThread-BLOB] [name=user1@mail-172.example.com;mid=6;] backup - Stored 1 blob files from volume message1 2015-03-03 20:20:06,966 INFO [BackupSetWorkerThread-BLOB] [name=user1@mail-172.example.com;mid=6;] backup - Stored 1 blob files 2015-03-03 20:20:07,014 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - Account user1@mail-172.example.com in backup set full-20150304.012000.193: All pending file IO completed (1 out of 1) 2015-03-03 20:20:07,016 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] mailbox - Putting mailbox 6 under maintenance. 2015-03-03 20:20:07,021 INFO [BackupSetWorkerThread-INDEX] [name=user1@mail-172.example.com;mid=6;] backup - Stored 2 index files 2015-03-03 20:20:07,496 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - Account user1@mail-172.example.com in backup set full-20150304.012000.193: All pending file IO completed (3 out of 3) 2015-03-03 20:20:07,497 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] mailbox - Ending maintenance on mailbox 6. 2015-03-03 20:20:07,497 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 AVAILABLE 2015-03-03 20:20:07,497 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - Account user1@mail-172.example.com in backup set full-20150304.012000.193: All pending file IO completed (3 out of 3) 2015-03-03 20:20:07,497 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - Full backup finished for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) mailbox 6
Events for the admin user's backup:
2015-03-03 20:20:09,894 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - redo log file sequence is 10 at full backup for admin@mail-172.example.com 2015-03-03 20:20:09,914 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - Full backup started for account admin@mail-172.example.com (7ac8de92-13fe-4d19-b9da-6872169fb5c9) mailbox 1 2015-03-03 20:20:09,915 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] mailbox - Putting mailbox 1 under maintenance. 2015-03-03 20:20:10,176 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - Number of blobs to backup for mailbox 1: 73 2015-03-03 20:20:10,177 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] mailbox - Ending maintenance on mailbox 1. 2015-03-03 20:20:10,177 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] mbxmgr - Mailbox 1 account 7ac8de92-13fe-4d19-b9da-6872169fb5c9 AVAILABLE 2015-03-03 20:20:10,343 INFO [BackupSetWorkerThread-BLOB] [name=admin@mail-172.example.com;mid=1;] backup - Stored 73 blob files from volume message1 2015-03-03 20:20:10,353 INFO [BackupSetWorkerThread-BLOB] [name=admin@mail-172.example.com;mid=1;] backup - Stored 73 blob files 2015-03-03 20:20:11,047 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - Account admin@mail-172.example.com in backup set full-20150304.012000.193: All pending file IO completed (73 out of 73) 2015-03-03 20:20:11,051 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] mailbox - Putting mailbox 1 under maintenance. 2015-03-03 20:20:11,108 INFO [BackupSetWorkerThread-INDEX] [name=admin@mail-172.example.com;mid=1;] backup - Stored 26 index files 2015-03-03 20:20:12,450 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - Account admin@mail-172.example.com in backup set full-20150304.012000.193: All pending file IO completed (99 out of 99) 2015-03-03 20:20:12,463 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] mailbox - Ending maintenance on mailbox 1. 2015-03-03 20:20:12,463 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] mbxmgr - Mailbox 1 account 7ac8de92-13fe-4d19-b9da-6872169fb5c9 AVAILABLE 2015-03-03 20:20:12,463 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - Account admin@mail-172.example.com in backup set full-20150304.012000.193: All pending file IO completed (99 out of 99) 2015-03-03 20:20:12,463 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - Full backup finished for account admin@mail-172.example.com (7ac8de92-13fe-4d19-b9da-6872169fb5c9) mailbox 1
And then a final line confirming completion and the number of accounts successfully backed up.
2015-03-03 20:20:12,463 INFO [FullBackupThread] [] backup - Finished backing up 10 of 10 accounts
One thing to point out is the log events were mentioning the redo log sequence as being number 10 but in the redolog/archive directory there is only a sequence 9 redo log. This is because the "seq 10" is actually the /opt/zimbra/redolog/redo.log file. When the redo.log file rolls over and is moved to the archive directory it will have seq10 in it's filename.
[zimbra@mail-172 archive]$ pwd ; ls -la *.log /opt/zimbra/redolog/archive -rw-r----- 1 zimbra zimbra 146723 Mar 3 20:23 redo-20150303.060015.816-seq9.log
The contents of the backup session directory. I removed the sub-directories under accounts that weren't either the admin or user1 account. Notice the zimbraId is used for the sub-directory naming scheme.
[zimbra@mail-172 full-20150304.012000.193]$ ls -R .: accounts ldap session.xml shared_blobs sys ./accounts: 3c3 423 434 75d 7ac 89f a46 af0 c53 eab ./accounts/3c3: 97e ./accounts/3c3/97e: 3c397edc-c015-4d82-ae8a-1084692f8a93 ./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93: blobs db index ldap.xml meta.xml ./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93/blobs: blobs-1.zip ./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93/db: db_schema.xml mail_item.dat mailbox.dat mailbox_metadata.dat open_conversation.dat tag.dat tagged_item.dat ./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93/index: 0 ./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93/index/0: segments.gen segments_1 [cut] ./accounts/7ac: 8de ./accounts/7ac/8de: 7ac8de92-13fe-4d19-b9da-6872169fb5c9 ./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9: blobs db index ldap.xml meta.xml ./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9/blobs: blobs-1.zip blobs-2.zip blobs-3.zip blobs-4.zip ./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9/db: db_schema.xml mail_item.dat mailbox.dat open_conversation.dat tag.dat tagged_item.dat ./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9/index: 0 ./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9/index/0: _0.fdt _0.fnm _0.nrm _0.tii _1.fdt _1.fnm _1.nrm _1.tii _2.fdt _2.fnm _2.nrm _2.tii segments.gen _0.fdx _0.frq _0.prx _0.tis _1.fdx _1.frq _1.prx _1.tis _2.fdx _2.frq _2.prx _2.tis segments_3 [cut] ./ldap: ldap-config.bak.gz ldap.bak.gz ./shared_blobs: ./sys: config.dat current_volumes.dat db_schema.xml localconfig.xml ssh ssl volume.dat ./sys/ssh: authorized_keys zimbra_identity zimbra_identity.pub ./sys/ssl: zimbra ./sys/ssl/zimbra: ca commercial jetty.pkcs12 server ./sys/ssl/zimbra/ca: ca.key ca.pem ca.srl ca.srl.old index.txt index.txt.attr index.txt.old newcerts zmssl.cnf ./sys/ssl/zimbra/ca/newcerts: 1424523586.pem 1424523591.pem 1424523597.pem 1424523603.pem ./sys/ssl/zimbra/commercial: ./sys/ssl/zimbra/server: server.crt server.csr server.key
Incremental Backup And Log Events Generated
Example of manually starting an incremental backup using the Standard zimbraBackupMode and doing a query to confirm it's completion and status.
[zimbra@mail-172 archive]$ zmprov gs `zmhostname` zimbraBackupMode # name mail-172.example.com zimbraBackupMode: Standard [zimbra@mail-172 archive]$ ls redo-20150303.060015.816-seq9.log redo-20150304.070025.476-seq10.log [zimbra@mail-172 archive]$ zmbackup -i incr-20150304.073543.151 [zimbra@mail-172 archive]$ zmbackupquery -lb incr-20150304.073543.151 -v Label: incr-20150304.073543.151 Type: incremental Status: completed Started: Wed, 2015/03/04 02:35:43.151 EST Ended: Wed, 2015/03/04 02:35:51.981 EST Redo log sequence range: 10 .. 11 Number of accounts: 10 out of 10 completed Accounts: admin@mail-172.example.com: completed galsync.qva4ekog0@mail-172.example.com: completed ham.jgfdb3oqjf@mail-172.example.com: completed spam.umhx3owv3s@mail-172.example.com: completed user1@mail-172.example.com: completed user2@mail-172.example.com: completed user3@mail-172.example.com: completed user4@mail-172.example.com: completed user5@mail-172.example.com: completed virus-quarantine.gusiso1cd@mail-172.example.com: completed Total space: 14020MB Free space: 6032MB
Notice how the log events are soap requests and differ from the full backup above and lack the INFO [*Backup*] formatting. Important to note this if your attempting to grep for backup/restore activity. Just like with the full, it starts with an authentication/acl request and states it's starting, how many accounts found, AsyncFileCopier starting, and the label name for this backup session.
From /opt/zimbra/log/mailbox.log
2015-03-04 02:35:42,356 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] soap - AuthRequest elapsed=94 2015-03-04 02:35:42,481 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup request started 2015-03-04 02:35:42,917 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Found 10 accounts on server mail-172.example.com 2015-03-04 02:35:43,254 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] io - AsyncFileCopier is starting 2015-03-04 02:35:43,255 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup started for backup set; label: incr-20150304.073543.151
The log events are again trimmed below just for the user1 and admin account for the user backup section of the log file. Notice the absence of any statements of the accounts being put into maintenance mode.
2015-03-04 02:35:44,118 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/6/index/0 2015-03-04 02:35:44,119 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 LOADED 2015-03-04 02:35:44,120 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup is started for account user1@mail-172.example.com 2015-03-04 02:35:44,120 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Saving account information from LDAP 2015-03-04 02:35:44,127 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup has ended 2015-03-04 02:35:44,127 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Account user1@mail-172.example.com in backup set incr-20150304.073543.151: All pending file IO completed (0 out of 0) [cut] 2015-03-04 02:35:44,212 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup is started for account admin@mail-172.example.com 2015-03-04 02:35:44,212 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Saving account information from LDAP 2015-03-04 02:35:44,215 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup has ended 2015-03-04 02:35:44,217 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Account admin@mail-172.example.com in backup set incr-20150304.073543.151: All pending file IO completed (0 out of 0)
The incremental will then end with log events related to copying the redologs into the backup directory and then purging the redologs that can be deleted. This example states redo log sequence 9 can be deleted. We'll confirm it is later.
2015-03-04 02:35:44,618 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] redolog - waiting for FileLogWriter.FsyncThread-1425452599560 to finish. 2015-03-04 02:35:44,628 INFO [FileLogWriter.FsyncThread-1425452599560] [] redolog - fsync thread exiting 2015-03-04 02:35:44,628 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] redolog - FileLogWriter.FsyncThread-1425452599560 finished 2015-03-04 02:35:44,757 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] redolog - Redo log rollover took 540ms 2015-03-04 02:35:44,758 INFO [FileLogWriter.FsyncThread-1425454544757] [] redolog - Starting fsync thread with interval 10 2015-03-04 02:35:44,776 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Found 3 redo logs to backup 2015-03-04 02:35:44,861 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Last backed-up redo log sequence = 9 2015-03-04 02:35:44,943 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Marking redo log /opt/zimbra/redolog/archive/redo-20150303.060015.816-seq9.log for deletion 2015-03-04 02:35:44,943 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Copying redo log /opt/zimbra/redolog/archive/redo-20150304.070025.476-seq10.log to /opt/zimbra/backup/tmp/incr-20150304.073543.151/redologs/redo-20150304.070025.476-seq10.log 2015-03-04 02:35:45,053 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Retaining recent redo log /opt/zimbra/redolog/archive/redo-20150304.070025.476-seq10.log 2015-03-04 02:35:45,053 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Copying redo log /opt/zimbra/redolog/archive/redo-20150304.073544.755-seq11.log to /opt/zimbra/backup/tmp/incr-20150304.073543.151/redologs/redo-20150304.073544.755-seq11.log 2015-03-04 02:35:45,067 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Retaining recent redo log /opt/zimbra/redolog/archive/redo-20150304.073544.755-seq11.log
The last limes confirm the backup finished successfully.
2015-03-04 02:35:51,935 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup set incr-20150304.073543.151: All pending file IO completed (0 out of 0) 2015-03-04 02:35:51,981 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] io - AsyncFileCopier is shut down 2015-03-04 02:35:51,981 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup finished for backup set; label: incr-20150304.073543.151 2015-03-04 02:35:51,982 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Deleting redo log /opt/zimbra/redolog/archive/redo-20150303.060015.816-seq9.log 2015-03-04 02:35:52,909 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup request finished 2015-03-04 02:35:52,909 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] soap - BackupRequest elapsed=10451
For an incremental, the account sub-directories will have ldap data saved also for each user but blobs and indexes aren't. That data resides in the redologs from the system under the redologs directory.
[zimbra@mail-172 incr-20150304.073543.151]$ ls -R .: accounts ldap redologs session.xml sys ./accounts: 3c3 423 434 75d 7ac 89f a46 af0 c53 eab ./accounts/3c3: 97e ./accounts/3c3/97e: 3c397edc-c015-4d82-ae8a-1084692f8a93 ./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93: ldap.xml [cut] ./accounts/7ac: 8de ./accounts/7ac/8de: 7ac8de92-13fe-4d19-b9da-6872169fb5c9 ./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9: ldap.xml [cut] ./ldap: ldap-config.bak.gz ldap.bak.gz ./redologs: redo-20150304.070025.476-seq10.log redo-20150304.073544.755-seq11.log ./sys: config.dat current_volumes.dat db_schema.xml localconfig.xml ssh ssl volume.dat ./sys/ssh: authorized_keys zimbra_identity zimbra_identity.pub ./sys/ssl: zimbra ./sys/ssl/zimbra: ca commercial jetty.pkcs12 server ./sys/ssl/zimbra/ca: ca.key ca.pem ca.srl ca.srl.old index.txt index.txt.attr index.txt.old newcerts zmssl.cnf ./sys/ssl/zimbra/ca/newcerts: 1424523586.pem 1424523591.pem 1424523597.pem 1424523603.pem ./sys/ssl/zimbra/commercial: ./sys/ssl/zimbra/server: server.crt server.csr server.key
Notice that redo log sequence 9 was indeed deleted from the redolog/archive directory.
[zimbra@mail-172 archive]$ pwd ; ls /opt/zimbra/redolog/archive redo-20150304.070025.476-seq10.log redo-20150304.073544.755-seq11.log redo-20150304.074024.058-seq12.log
If you had created new users after your last full backup, you would see that there was also a full backup started as well. For example :
[zimbra@mail-172 log]$ zmbackup -i incr-20150304.085810.220 full-20150304.085818.889 [zimbra@mail-172 log]$ zmbackupquery -a user6@mail-172.example.com Account: user6@mail-172.example.com Label: full-20150304.085818.889 Type: full Started: Wed, 2015/03/04 03:58:18.889 EST Ended: Wed, 2015/03/04 03:58:25.997 EST Acct ID: 4e898a68-1886-4a80-b3dc-ac2ab650bf8c [zimbra@mail-172 log]$ zmbackupquery -lb full-20150304.085818.889 -v Label: full-20150304.085818.889 Type: full Status: completed Started: Wed, 2015/03/04 03:58:18.889 EST Ended: Wed, 2015/03/04 03:58:25.997 EST Redo log sequence range: 14 .. 14 Number of accounts: 1 out of 1 completed Accounts: user6@mail-172.example.com: completed Total space: 14020MB Free space: 6038MB
Contents Of accounts.xml And sessions.xml From The Full And Incremental Backup Examples
First, see what zmbackupquery shows for a particular user in regards to the backup labels they are included in.
$ zmbackupquery -a user1@mail-172.example.com Account: user1@mail-172.example.com Label: incr-20150304.073543.151 Type: incremental Started: Wed, 2015/03/04 02:35:43.151 EST Ended: Wed, 2015/03/04 02:35:51.981 EST Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93 Label: full-20150304.012000.193 Type: full Started: Tue, 2015/03/03 20:20:00.193 EST Ended: Tue, 2015/03/03 20:20:28.630 EST Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93
Now let's review the accounts/session xml files that list all the users within those particular backup sessions. The /opt/zimbra/backup/accounts.xml is the referenced to determine what is the last know full backup session the user was in. If the /opt/zimbra/backup/accounts.xml is corrupted, unreadable, or incorrect - this can cause issues with restoring users.
Also notice that it also references the zimbraId for the user at the time of the backup. If you were to delete a user and then create a new account using the same email address [user1@mail-172.example.com] that new account will have a different zimbraId and therefor the backup/restore commands will not work as you might expected because of the differing zimbraId. The zmrestore command only can take an email address for the -a option, it can not use the zimbraID.
[zimbra@mail-172 backup]$ pwd /opt/zimbra/backup [zimbra@mail-172 backup]$ cat accounts.xml <?xml version="1.0" encoding="utf-8"?> <backupMetadata xmlns="urn:zimbraBackupMeta" version="7.2" dbVersion="103"> <allAccounts> <account zimbraId="af06b0b4-3160-4ea3-a55b-81163a9914a0" email="user5@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193"> </account> <account zimbraId="75d3d364-a384-4077-8ac5-87b6960b57cd" email="user3@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193"> </account> <account zimbraId="4343a299-5259-4826-b1c1-becbfbb9b8f0" email="galsync.qva4ekog0@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193"> </account> <account zimbraId="89fc5986-31a3-406e-8084-0ce0b98a251f" email="user2@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193"> </account> <account zimbraId="42380a6e-5718-4d68-ab37-e8d5bc9e6977" email="spam.umhx3owv3s@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193"> </account> <account zimbraId="eab3d56e-ffd9-4412-95de-48f102d2d252" email="user4@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193"> </account> <account zimbraId="7ac8de92-13fe-4d19-b9da-6872169fb5c9" email="admin@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193"> </account> <account zimbraId="4e898a68-1886-4a80-b3dc-ac2ab650bf8c" email="user6@mail-172.example.com" latestFullBackupLabel="full-20150304.085818.889"> </account> <account zimbraId="a46dfc59-a3e9-4f09-adfb-af8efc42aa9f" email="virus-quarantine.gusiso1cd@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193"> </account> <account zimbraId="3c397edc-c015-4d82-ae8a-1084692f8a93" email="user1@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193"> </account> <account zimbraId="c5315c5d-97cb-4d84-bc70-374a67f13309" email="ham.jgfdb3oqjf@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193"> </account> </allAccounts> </backupMetadata>
In each backup session, there will be a sessions.xml file that acts as a reference or summary file of the backup. At the top, you'll see what version of ZCS was running and what options the backup environment had set. It will then report the final status for each user for the backup attempt against the account. At the end, there's a section in regards to errors and if any were encountered.
[zimbra@mail-172 backup]$ cd sessions/ [zimbra@mail-172 sessions]$ cd full-20150304.012000.193 [zimbra@mail-172 full-20150304.012000.193]$ cat session.xml <?xml version="1.0" encoding="utf-8"?> <backupMetadata xmlns="urn:zimbraBackupMeta" version="7.2" dbVersion="103"> <backupSet label="full-20150304.012000.193" zcsRelease="8.6.0_GA_1153 20141215151258 20141215-1518 NETWORK" startTime="1425432000193" endTime="1425432028630" minRedoSeq="10" maxRedoSeq="10" sharedBlobsZipped="true" sharedBlobsZipNameDigestChars="1" sharedBlobsDirectoryDepth="5" sharedBlobsCharsPerDirectory="2" type="full" accountsDirectoryDepth="2"> <desc>Full backup</desc> <accounts> <account zimbraId="af06b0b4-3160-4ea3-a55b-81163a9914a0" email="user5@mail-172.example.com" status="COMPLETED"> </account> <account zimbraId="75d3d364-a384-4077-8ac5-87b6960b57cd" email="user3@mail-172.example.com" status="COMPLETED"> </account> <account zimbraId="89fc5986-31a3-406e-8084-0ce0b98a251f" email="user2@mail-172.example.com" status="COMPLETED"> </account> <account zimbraId="4343a299-5259-4826-b1c1-becbfbb9b8f0" email="galsync.qva4ekog0@mail-172.example.com" status="COMPLETED"> </account> <account zimbraId="eab3d56e-ffd9-4412-95de-48f102d2d252" email="user4@mail-172.example.com" status="COMPLETED"> </account> <account zimbraId="42380a6e-5718-4d68-ab37-e8d5bc9e6977" email="spam.umhx3owv3s@mail-172.example.com" status="COMPLETED"> </account> <account zimbraId="7ac8de92-13fe-4d19-b9da-6872169fb5c9" email="admin@mail-172.example.com" status="COMPLETED"> </account> <account zimbraId="a46dfc59-a3e9-4f09-adfb-af8efc42aa9f" email="virus-quarantine.gusiso1cd@mail-172.example.com" status="COMPLETED"> </account> <account zimbraId="3c397edc-c015-4d82-ae8a-1084692f8a93" email="user1@mail-172.example.com" status="COMPLETED"> </account> <account zimbraId="c5315c5d-97cb-4d84-bc70-374a67f13309" email="ham.jgfdb3oqjf@mail-172.example.com" status="COMPLETED"> </account> </accounts> <stats> <counter name="total_time" numSamples="1" sum="28437" unit="ms"> </counter> <counter name="sysdb_time" numSamples="1" sum="1111" unit="ms"> </counter> <counter name="sysdb_bytes" numSamples="1" sum="35210" unit="bytes"> </counter> <counter name="redologs_time" numSamples="0" sum="0" unit="ms"> </counter> <counter name="redologs_bytes" numSamples="0" sum="0" unit="bytes"> </counter> <counter name="accounts_time" numSamples="10" sum="4827" unit="ms"> </counter> <counter name="ldap_time" numSamples="10" sum="172" unit="ms"> </counter> <counter name="ldap_bytes" numSamples="10" sum="25665" unit="bytes"> </counter> <counter name="db_time" numSamples="10" sum="4787" unit="ms"> </counter> <counter name="db_bytes" numSamples="10" sum="78690" unit="bytes"> </counter> <counter name="db_digest_map_load_time" numSamples="10" sum="2400" unit="ms"> </counter> <counter name="msgs_time" numSamples="10" sum="451" unit="ms"> </counter> <counter name="msg_copy_time" numSamples="74" sum="25" unit="ms"> </counter> <counter name="msg_copy_bytes" numSamples="74" sum="149611" unit="bytes"> </counter> <counter name="msg_link_time" numSamples="0" sum="0" unit="ms"> </counter> <counter name="index_time" numSamples="48" sum="80" unit="ms"> </counter> <counter name="index_bytes" numSamples="38" sum="73510" unit="bytes"> </counter> </stats> <errors> </errors> </backupSet> </backupMetadata>
An incremental session.xml file is similar to a full backups session.xml .
[zimbra@mail-172 sessions]$ cd incr-20150304.073543.151 [zimbra@mail-172 incr-20150304.073543.151]$ cat session.xml <?xml version="1.0" encoding="utf-8"?> <backupMetadata xmlns="urn:zimbraBackupMeta" version="7.2" dbVersion="103"> <backupSet label="incr-20150304.073543.151" zcsRelease="8.6.0_GA_1153 20141215151258 20141215-1518 NETWORK" startTime="1425454543151" endTime="1425454551981" minRedoSeq="10" maxRedoSeq="11" sharedBlobsZipped="true" sharedBlobsZipNameDigestChars="1" sharedBlobsDirectoryDepth="5" sharedBlobsCharsPerDirectory="2" type="incremental" accountsDirectoryDepth="2"> <desc>Incremental backup</desc> <accounts> <account zimbraId="af06b0b4-3160-4ea3-a55b-81163a9914a0" email="user5@mail-172.example.com" status="COMPLETED"> </account> <account zimbraId="75d3d364-a384-4077-8ac5-87b6960b57cd" email="user3@mail-172.example.com" status="COMPLETED"> </account> <account zimbraId="89fc5986-31a3-406e-8084-0ce0b98a251f" email="user2@mail-172.example.com" status="COMPLETED"> </account> <account zimbraId="4343a299-5259-4826-b1c1-becbfbb9b8f0" email="galsync.qva4ekog0@mail-172.example.com" status="COMPLETED"> </account> <account zimbraId="eab3d56e-ffd9-4412-95de-48f102d2d252" email="user4@mail-172.example.com" status="COMPLETED"> </account> <account zimbraId="42380a6e-5718-4d68-ab37-e8d5bc9e6977" email="spam.umhx3owv3s@mail-172.example.com" status="COMPLETED"> </account> <account zimbraId="7ac8de92-13fe-4d19-b9da-6872169fb5c9" email="admin@mail-172.example.com" status="COMPLETED"> </account> <account zimbraId="a46dfc59-a3e9-4f09-adfb-af8efc42aa9f" email="virus-quarantine.gusiso1cd@mail-172.example.com" status="COMPLETED"> </account> <account zimbraId="3c397edc-c015-4d82-ae8a-1084692f8a93" email="user1@mail-172.example.com" status="COMPLETED"> </account> <account zimbraId="c5315c5d-97cb-4d84-bc70-374a67f13309" email="ham.jgfdb3oqjf@mail-172.example.com" status="COMPLETED"> </account> </accounts> <stats> <counter name="total_time" numSamples="1" sum="8830" unit="ms"> </counter> <counter name="sysdb_time" numSamples="1" sum="563" unit="ms"> </counter> <counter name="sysdb_bytes" numSamples="1" sum="35210" unit="bytes"> </counter> <counter name="redologs_time" numSamples="1" sum="310" unit="ms"> </counter> <counter name="redologs_bytes" numSamples="1" sum="126307" unit="bytes"> </counter> <counter name="accounts_time" numSamples="10" sum="196" unit="ms"> </counter> <counter name="ldap_time" numSamples="10" sum="129" unit="ms"> </counter> <counter name="ldap_bytes" numSamples="10" sum="25665" unit="bytes"> </counter> <counter name="db_time" numSamples="0" sum="0" unit="ms"> </counter> <counter name="db_bytes" numSamples="0" sum="0" unit="bytes"> </counter> <counter name="db_digest_map_load_time" numSamples="0" sum="0" unit="ms"> </counter> <counter name="msgs_time" numSamples="0" sum="0" unit="ms"> </counter> <counter name="msg_copy_time" numSamples="0" sum="0" unit="ms"> </counter> <counter name="msg_copy_bytes" numSamples="0" sum="0" unit="bytes"> </counter> <counter name="msg_link_time" numSamples="0" sum="0" unit="ms"> </counter> <counter name="index_time" numSamples="0" sum="0" unit="ms"> </counter> <counter name="index_bytes" numSamples="0" sum="0" unit="bytes"> </counter> </stats> <errors> </errors> </backupSet> </backupMetadata>
User And Log Events Generated
Please beware that prior backups were deleted and new backups were made on this ZCS test server. So the backup label names might not match prior examples.
For this example, I'm going to replicate a common situation that system administrators find themselves when trying to restore a user and they fail to include the proper restore options to have the restore stop prior to replaying a message that was deleted by a user. Proper examples of doing the restore will be shown afterwards.
Information on the account we'll restore:
[zimbra@mail-172 backup]$ zmprov gmi user1@mail-172.example.com mailboxId: 6 quotaUsed: 1209 [zimbra@mail-172 backup]$ zmprov ga user1@mail-172.example.com zimbraId # name user1@mail-172.example.com zimbraId: 3c397edc-c015-4d82-ae8a-1084692f8a93
Example of error you'll receive if you attempt to use the zimbraId rather than the email address for the zmrestore command. This is important to note if your trying to restore an account that was deleted and then later created again and the various backups and accounts.xml file has different zimbraId's associated with the same email address.
[zimbra@mail-172 backup]$ zmrestore -a 3c397edc-c015-4d82-ae8a-1084692f8a93 -ca -pre restore1_ Error occurred: invalid request: invalid account email address: 3c397edc-c015-4d82-ae8a-1084692f8a93
Showing that we have one message in the Inbox for the user we'll be restoring and also the message id of that email :
[zimbra@mail-172 backup]$ zmmailbox -z -m user1@mail-172.example.com gaf Id View Unread Msg Count Path ---------- ---- ---------- ---------- ---------- 1 unkn 0 0 / 16 docu 0 0 /Briefcase 10 appo 0 0 /Calendar 14 mess 0 0 /Chats 7 cont 0 0 /Contacts 6 mess 0 0 /Drafts 13 cont 0 0 /Emailed Contacts 2 mess 1 1 /Inbox 4 mess 0 0 /Junk 5 mess 0 0 /Sent 15 task 0 0 /Tasks 3 unkn 0 0 /Trash [zimbra@mail-172 backup]$ zmmailbox -z -m user1@mail-172.example.com s -t mess in:Inbox num: 1, more: false Id Type From Subject Date ---- ---- -------------------- -------------------------------------------------- -------------- 1. 257 mess admin Test1 02/23/15 09:03
Now to delete the email from the account:
[zimbra@mail-172 backup]$ zmmailbox -z -m user1@mail-172.example.com dm 257 And to confirm it's no longer there: [zimbra@mail-172 backup]$ zmmailbox -z -m user1@mail-172.example.com s -t mess in:Inbox num: 0, more: false
Below is an example of a delete operation being logged in the redo.log and how to query events for a particular user with the zmredodump command. Notice the DeleteItem events at the end.
[zimbra@mail-172 backup]$ zmredodump -m 6 --show-blob /opt/zimbra/redolog/redo.log VERIFYING: /opt/zimbra/redolog/redo.log HEADER ------ sequence: 14 open: 1 filesize: 512 serverId: 4407a03f-894b-4b1a-817a-684fc34c4f60 created: Wed, 2015/03/04 03:58:10.876 EST (1425459490876) first op: Wed, 2015/03/04 03:58:18.744 EST (1425459498744) last op: Wed, 2015/03/04 03:58:18.744 EST (1425459498744) (not up to date) version: 1.42 ------ [00001601 - 0000162e: 46 bytes; tstamp: 2015/03/04 04:00:26.677 EST] txn 1425452599.101 [PurgeOldMessages] ver=1.42, tstamp=1425459626677, change=1508, mailbox=6 [0000162f - 00001660: 50 bytes; tstamp: 2015/03/04 04:00:26.683 EST] txn 1425452599.101 [CommitTxn] ver=1.42, tstamp=1425459626683, mailbox=6, txnType=PurgeOldMessages [00002691 - 000026be: 46 bytes; tstamp: 2015/03/04 04:10:26.816 EST] txn 1425452599.113 [PurgeOldMessages] ver=1.42, tstamp=1425460226816, change=1509, mailbox=6 [000026bf - 000026f0: 50 bytes; tstamp: 2015/03/04 04:10:26.832 EST] txn 1425452599.113 [CommitTxn] ver=1.42, tstamp=1425460226832, mailbox=6, txnType=PurgeOldMessages [00002ab1 - 00002ade: 46 bytes; tstamp: 2015/03/04 04:21:27.376 EST] txn 1425452599.124 [PurgeOldMessages] ver=1.42, tstamp=1425460887376, change=1510, mailbox=6 [00002adf - 00002b10: 50 bytes; tstamp: 2015/03/04 04:21:27.381 EST] txn 1425452599.124 [CommitTxn] ver=1.42, tstamp=1425460887381, mailbox=6, txnType=PurgeOldMessages [00002ed1 - 00002efe: 46 bytes; tstamp: 2015/03/04 04:32:27.610 EST] txn 1425452599.135 [PurgeOldMessages] ver=1.42, tstamp=1425461547610, change=1511, mailbox=6 [00002eff - 00002f30: 50 bytes; tstamp: 2015/03/04 04:32:27.617 EST] txn 1425452599.135 [CommitTxn] ver=1.42, tstamp=1425461547617, mailbox=6, txnType=PurgeOldMessages [000032f1 - 0000331e: 46 bytes; tstamp: 2015/03/04 04:43:27.828 EST] txn 1425452599.146 [PurgeOldMessages] ver=1.42, tstamp=1425462207828, change=1512, mailbox=6 [0000331f - 00003350: 50 bytes; tstamp: 2015/03/04 04:43:27.839 EST] txn 1425452599.146 [CommitTxn] ver=1.42, tstamp=1425462207839, mailbox=6, txnType=PurgeOldMessages [000034d1 - 0000350c: 60 bytes; tstamp: 2015/03/04 04:48:18.119 EST] txn 1425452599.151 [DeleteItem] ver=1.42, tstamp=1425462498119, change=1513, mailbox=6, ids=[257], type=message [0000350d - 0000353e: 50 bytes; tstamp: 2015/03/04 04:48:18.306 EST] txn 1425452599.151 [CommitTxn] ver=1.42, tstamp=1425462498306, mailbox=6, txnType=DeleteItem
Listing of the backups that are currently associated with the account we'll be restoring:
[zimbra@mail-172 backup]$ zmbackupquery -a user1@mail-172.example.com Account: user1@mail-172.example.com Label: incr-20150304.085810.220 Type: incremental Started: Wed, 2015/03/04 03:58:10.220 EST Ended: Wed, 2015/03/04 03:58:16.393 EST Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93 Label: full-20150304.082015.535 Type: full Started: Wed, 2015/03/04 03:20:15.535 EST Ended: Wed, 2015/03/04 03:20:39.109 EST Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93
Run a zmrestore with no other options other than the account flag and the email address of the account we are restoring. Note - this example is to show why this often times isn't what you'll do in real circumstances and will require other zmrestore options to achieve what you want.
[zimbra@mail-172 backup]$ zmrestore -a user1@mail-172.example.com
From the /opt/zimbra/log/mailbox.log we have the following log events. We see the restore request kick off and the full backup label that it'll reference as the starting point. It also lists the zimbraId rather than the email address for the restore initially.
2015-03-04 04:53:02,221 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - AuthRequest elapsed=60 2015-03-04 04:53:02,519 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is starting 2015-03-04 04:53:02,565 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Using full backup full-20150304.082015.535 as starting point for restoring account 3c397edc-c015-4d82-ae8a-1084692f8a93
And then we'll see the reference to the email address and the mailboxId associated with the account/zimbraId .
2015-03-04 04:53:02,993 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - waiting for FileLogWriter.FsyncThread-1425459490899 to finish. 2015-03-04 04:53:02,997 INFO [FileLogWriter.FsyncThread-1425459490899] [] redolog - fsync thread exiting 2015-03-04 04:53:02,997 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - FileLogWriter.FsyncThread-1425459490899 finished 2015-03-04 04:53:03,028 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - Redo log rollover took 463ms 2015-03-04 04:53:03,028 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore started for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6
Since we are basically over-writing the account, the account is placed into maintenance mode first and then existing data is deleted. Only after the account has been deleted will the account be restored using the backup data. Since we didn't use the -ca and -pre options for zmrestore, this account will reuse it's existing zimbraId, mailboxId, and of course - email address.
2015-03-04 04:53:03,042 INFO [FileLogWriter.FsyncThread-1425462783028] [] redolog - Starting fsync thread with interval 10 2015-03-04 04:53:03,096 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - Putting mailbox 6 under maintenance. 2015-03-04 04:53:03,097 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - clearing contents of mailbox 6, group 6 2015-03-04 04:53:03,209 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restoring to mailbox id 6 for existing account 3c397edc-c015-4d82-ae8a-1084692f8a93. 2015-03-04 04:53:03,256 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loading schema with version 7.2 2015-03-04 04:53:03,287 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Renaming volume_id column to locator in source table mail_item 2015-03-04 04:53:03,584 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Deleting directory /opt/zimbra/store/0/6/msg 2015-03-04 04:53:03,599 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 1 blobs during restore 2015-03-04 04:53:03,622 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Deleting directory /opt/zimbra/index/0/6/index
At this point, the data for the account was restored to the state of how it was when the full backup was ran.
2015-03-04 04:53:03,664 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 2 index files during restore 2015-03-04 04:53:03,665 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 AVAILABLE 2015-03-04 04:53:03,670 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Account user1@mail-172.example.com: All pending file IO completed (2 out of 2)
Now we see the restore start playing redologs that were made after the full backup. This section is the redologs from incr-20150304.085810.220 backup session, which were : redo-20150304.070025.476-seq10.log redo-20150304.073544.755-seq11.log redo-20150304.074024.058-seq12.log redo-20150304.085810.879-seq13.log . Located in, /opt/zimbra/backup/sessions/incr-20150304.085810.220/redologs .
2015-03-04 04:53:03,671 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Run redo operations since last full backup 2015-03-04 04:53:03,733 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/6/index/0 2015-03-04 04:53:03,733 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 LOADED 2015-03-04 04:53:03,733 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. 2015-03-04 04:53:03,733 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] cache - initializing folder and tag caches for mailbox 6 2015-03-04 04:53:03,746 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. 2015-03-04 04:53:03,752 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. 2015-03-04 04:53:03,757 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
Notice it will reference the sequence number of the redo log when it is plays those in /opt/zimbra/redolog/archive/ . Currently, the server has the following in that directory : redo-20150304.085810.879-seq13.log redo-20150304.095303.026-seq14.log redo-20150304.102039.541-seq15.log .
2015-03-04 04:53:03,763 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Run redo operations from archived redo logs starting at sequence 13 2015-03-04 04:53:03,764 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. 2015-03-04 04:53:03,774 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. 2015-03-04 04:53:03,780 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. 2015-03-04 04:53:03,787 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. 2015-03-04 04:53:03,795 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages.
And while it replays redo log sequence 13, we see that it replays the delete operation for message id 257.
2015-03-04 04:53:03,824 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailop - Deleting Message (id=257).
Final log events showing restore is finished.
2015-03-04 04:53:03,832 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore finished for user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox 6 to mailbox 6 2015-03-04 04:53:03,833 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is shut down 2015-03-04 04:53:03,833 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - RestoreRequest elapsed=1535
Confirm that the message is still missing from the account:
[zimbra@mail-172 backup]$ zmmailbox -z -m user1@mail-172.example.com s -t mess in:Inbox num: 0, more: false
Confirm that the account is still using the same zimbraId, mailboxId, and email address :
[zimbra@mail-172 backup]$ zmprov gmi user1@mail-172.example.com mailboxId: 6 quotaUsed: 0 [zimbra@mail-172 backup]$ zmprov ga user1@mail-172.example.com zimbraId # name user1@mail-172.example.com zimbraId: 3c397edc-c015-4d82-ae8a-1084692f8a93
Using The -restoreToTime Option To Restore Prior To A Message Being Deleted
First, let's identify the exact time the delete operation took place. From our previous examples above I already know what redo log sequence file it was logged in.
[zimbra@mail-172 backup]$ zmredodump -m 6 --show-blob /opt/zimbra/redolog/archive/redo-20150304.095303.026-seq14.log | grep DeleteItem [000034d1 - 0000350c: 60 bytes; tstamp: 2015/03/04 04:48:18.119 EST] txn 1425452599.151 [DeleteItem] ver=1.42, tstamp=1425462498119, change=1513, mailbox=6, ids=[257], type=message [0000350d - 0000353e: 50 bytes; tstamp: 2015/03/04 04:48:18.306 EST] txn 1425452599.151 [CommitTxn] ver=1.42, tstamp=1425462498306, mailbox=6, txnType=DeleteItem
The first line above shows me the delete request was done at "2015/03/04 04:48:18.119" for mailbox=6, ids=[257], type=message . Ids is the message id, that were already used to delete the message in the above example and was shown in the zmmailbox search command.
The example uses the time above but subtracted one from it - 04:48:18.118 vs 04:48:18.119 .
[zimbra@mail-172 backup]$ zmrestore -a user1@mail-172.example.com -restoreToTime "2015/03/04 04:48:18.118" using restore to time of 2015/03/04 04:48:18
And to now confirm the message is back in the account:
[zimbra@mail-172 backup]$ zmmailbox -z -m user1@mail-172.example.com s -t mess in:Inbox num: 1, more: false Id Type From Subject Date ---- ---- -------------------- -------------------------------------------------- -------------- 1. 257 mess admin Test1 02/23/15 09:03
Note that this will also mean all other data or changes after 04:48:18.118 are lost also in this restored account. Normally in circumstances of trying to recovery data that a user is requesting because they delete messages and now need them recovered you would use the -ca and -pre options for zmrestore so you could then recover them and import the messages back into the existing account they have. Without the -ca and -pre , your over writing the existing account on the system.
The log events in /opt/zimbra/log/mailbox.log for this request were:
2015-03-04 05:20:39,194 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is starting 2015-03-04 05:20:39,238 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Using full backup full-20150304.082015.535 as starting point for restoring account 3c397edc-c015-4d82-ae8a-1084692f8a93 2015-03-04 05:20:39,526 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - waiting for FileLogWriter.FsyncThread-1425462783028 to finish. 2015-03-04 05:20:39,533 INFO [FileLogWriter.FsyncThread-1425462783028] [] redolog - fsync thread exiting 2015-03-04 05:20:39,533 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - FileLogWriter.FsyncThread-1425462783028 finished 2015-03-04 05:20:39,543 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - Redo log rollover took 305ms 2015-03-04 05:20:39,543 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore started for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6 2015-03-04 05:20:39,550 INFO [FileLogWriter.FsyncThread-1425464439542] [] redolog - Starting fsync thread with interval 10 2015-03-04 05:20:39,593 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - Putting mailbox 6 under maintenance. 2015-03-04 05:20:39,593 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - clearing contents of mailbox 6, group 6 2015-03-04 05:20:39,614 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restoring to mailbox id 6 for existing account 3c397edc-c015-4d82-ae8a-1084692f8a93. 2015-03-04 05:20:39,652 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loading schema with version 7.2 2015-03-04 05:20:39,660 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Renaming volume_id column to locator in source table mail_item 2015-03-04 05:20:39,692 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Deleting directory /opt/zimbra/store/0/6/msg 2015-03-04 05:20:39,700 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 1 blobs during restore 2015-03-04 05:20:39,703 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Deleting directory /opt/zimbra/index/0/6/index 2015-03-04 05:20:39,713 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 2 index files during restore 2015-03-04 05:20:39,715 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 AVAILABLE 2015-03-04 05:20:39,717 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Account user1@mail-172.example.com: All pending file IO completed (2 out of 2) 2015-03-04 05:20:39,717 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Run redo operations since last full backup 2015-03-04 05:20:39,732 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/6/index/0 2015-03-04 05:20:39,732 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 LOADED 2015-03-04 05:20:39,732 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. 2015-03-04 05:20:39,732 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] cache - initializing folder and tag caches for mailbox 6 2015-03-04 05:20:39,743 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. 2015-03-04 05:20:39,749 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. 2015-03-04 05:20:39,754 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. 2015-03-04 05:20:39,760 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Run redo operations from archived redo logs starting at sequence 13 2015-03-04 05:20:39,762 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. 2015-03-04 05:20:39,770 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. 2015-03-04 05:20:39,774 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. 2015-03-04 05:20:39,780 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. 2015-03-04 05:20:39,786 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. 2015-03-04 05:20:39,789 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore finished for user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox 6 to mailbox 6 2015-03-04 05:20:39,791 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is shut down 2015-03-04 05:20:39,791 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - RestoreRequest elapsed=646
Notice that this restore, unlike the first example, does NOT have the following line :
INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailop - Deleting Message (id=257).
Blobs In Backup And Blobs In Production
This might be a good time to take a look at the blobs and how they are saved/stored in the backups and how they compare to the store volume. The example below is from our user and message details from above.
[zimbra@mail-172 blobs]$ pwd /opt/zimbra/backup/sessions/full-20150304.082015.535/accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93/blobs [zimbra@mail-172 blobs]$ unzip -Z blobs-1.zip Archive: blobs-1.zip Zip file size: 1439 bytes, number of entries: 1 -rw---- 4.5 fat 1209 b- stor 15-Mar-04 03:20 1/0/sha256_mBS9YKqyNpLknKCDWY5WNEbTgotX,UNrXMA5ga+s6LA=_257-2.msg1 1 file, 1209 bytes uncompressed, 1209 bytes compressed: 0.0% [zimbra@mail-172 blobs]$ ls -la /opt/zimbra/store/0/6/msg/0/257-2.msg -rw-r----- 1 zimbra zimbra 1209 Mar 4 05:20 /opt/zimbra/store/0/6/msg/0/257-2.msg [zimbra@mail-172 blobs]$ unzip blobs-1.zip Archive: blobs-1.zip extracting: 1/0/sha256_mBS9YKqyNpLknKCDWY5WNEbTgotX,UNrXMA5ga+s6LA=_257-2.msg1 [zimbra@mail-172 blobs]$ ls 1 blobs-1.zip [zimbra@mail-172 blobs]$ ls -la 1/0/sha256_mBS9YKqyNpLknKCDWY5WNEbTgotX\,UNrXMA5ga+s6LA\=_257-2.msg1 -rw-r----- 1 zimbra zimbra 1209 Mar 4 03:20 1/0/sha256_mBS9YKqyNpLknKCDWY5WNEbTgotX,UNrXMA5ga+s6LA=_257-2.msg1 [zimbra@mail-172 blobs]$ head 1/0/sha256_mBS9YKqyNpLknKCDWY5WNEbTgotX\,UNrXMA5ga+s6LA\=_257-2.msg1 Return-Path: zimbra@mail-172.example.com Received: from mail-172.example.com (LHLO mail-172.example.com) (192.168.1.172) by mail-172.example.com with LMTP; Mon, 23 Feb 2015 09:02:58 -0500 (EST) Received: from localhost (localhost.localdomain [127.0.0.1]) by mail-172.example.com (Postfix) with ESMTP id 4F8AE8433D for <user1@mail-172.example.com>; Mon, 23 Feb 2015 09:02:58 -0500 (EST) X-Virus-Scanned: amavisd-new at mail-172.example.com X-Spam-Flag: NO X-Spam-Score: -1.9 [zimbra@mail-172 blobs]$ head /opt/zimbra/store/0/6/msg/0/257-2.msg Return-Path: zimbra@mail-172.example.com Received: from mail-172.example.com (LHLO mail-172.example.com) (192.168.1.172) by mail-172.example.com with LMTP; Mon, 23 Feb 2015 09:02:58 -0500 (EST) Received: from localhost (localhost.localdomain [127.0.0.1]) by mail-172.example.com (Postfix) with ESMTP id 4F8AE8433D for <user1@mail-172.example.com>; Mon, 23 Feb 2015 09:02:58 -0500 (EST) X-Virus-Scanned: amavisd-new at mail-172.example.com X-Spam-Flag: NO X-Spam-Score: -1.9
Knowing When The -br [To Incremental] Or -rf [Only Full] Option Is Required
Pre-Steps
Let's query current backups and then create a new full and new incremental where we also delete a message between the full and the incremental.
[zimbra@mail-172 ~]$ zmbackupquery -a user1@`zmhostname` Account: user1@mail-172.example.com Label: incr-20150305.060013.901 Type: incremental Started: Thu, 2015/03/05 01:00:13.901 EST Ended: Thu, 2015/03/05 01:00:26.273 EST Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93 Label: incr-20150304.085810.220 Type: incremental Started: Wed, 2015/03/04 03:58:10.220 EST Ended: Wed, 2015/03/04 03:58:16.393 EST Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93 Label: full-20150304.082015.535 Type: full Started: Wed, 2015/03/04 03:20:15.535 EST Ended: Wed, 2015/03/04 03:20:39.109 EST Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93 [zimbra@mail-172 ~]$ zmbackup -f -a all full-20150306.043202.011 [zimbra@mail-172 ~]$ zmmailbox -z -m user1@mail-172.example.com s -t mess in:Inbox num: 1, more: false Id Type From Subject Date ---- ---- -------------------- -------------------------------------------------- -------------- 1. 257 mess admin Test1 02/23/15 09:03 [zimbra@mail-172 ~]$ zmmailbox -z -m user1@mail-172.example.com dm 257 [zimbra@mail-172 ~]$ zmbackup -i incr-20150306.043321.683 [zimbra@mail-172 ~]$ zmbackupquery -a user1@`zmhostname` Account: user1@mail-172.example.com Label: incr-20150306.043321.683 Type: incremental Started: Thu, 2015/03/05 23:33:21.683 EST Ended: Thu, 2015/03/05 23:33:27.495 EST Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93 Label: full-20150306.043202.011 Type: full Started: Thu, 2015/03/05 23:32:02.011 EST Ended: Thu, 2015/03/05 23:32:13.316 EST Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93 Label: incr-20150305.060013.901 Type: incremental Started: Thu, 2015/03/05 01:00:13.901 EST Ended: Thu, 2015/03/05 01:00:26.273 EST Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93 Label: incr-20150304.085810.220 Type: incremental Started: Wed, 2015/03/04 03:58:10.220 EST Ended: Wed, 2015/03/04 03:58:16.393 EST Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93 Label: full-20150304.082015.535 Type: full Started: Wed, 2015/03/04 03:20:15.535 EST Ended: Wed, 2015/03/04 03:20:39.109 EST Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93
First Example - Options -lb, -ca, -pre restore_
For the following examples, we'll include the -ca [create account] and the -pre [add prefix to new account name] . This first example uses the oldest full backup [full-20150304.082015.535 vs. full-20150306.043202.011] the account has with the -lb option. The message we just deleted, existed at the time of the full backup. We'll NOT be including either the -rf or -br option in this first example.
Then we'll review the mailbox.log events. Note the following:
- That the log only mentions the parent account, there isn't a reference to the email address with the prefixed name.
- There is a RFE to include the new account name in the log, Bug 52641 - RFE: include -ca -pre new account in mailbox.log log event
- Though the log will state the new mailboxId and zimbraId for the user, though:
- "user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6" vs "Restoring to mailbox id 12 for new account 016c9f7c-e2e2-4f8f-96d0-e14a4327821d"
- Since we failed to include the -rf option, the restore continues past the full backup and goes through all the other backups that exist, hence we see:
- "backup - Run redo operations since last full backup"
- One of the redologs in our incremental had the delete operation for the message in question and we see evidence of that delete in the log:
- "mailop - Deleting Message (id=257)."
- Since we didn't include the -br option, the restore also continues past the backups and replays the redologs in /opt/zimbra/redolog and it's subdirectory, archive. Notice the reference to sequence 17.
- "backup - Run redo operations from archived redo logs starting at sequence 17"
- /opt/zimbra/redolog/archive/ has : redo-20150306.043322.070-seq17.log redo-20150306.043611.912-seq18.log
- Towards the end, we'll get a confirmation that the restore has finished with some reference information again - mailboxId's .
- "Restore finished for user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox 6 to mailbox 12"
[zimbra@mail-172 ~]$ cat /dev/null > /opt/zimbra/log/mailbox.log [zimbra@mail-172 ~]$ zmrestore -a user1@mail-172.example.com -lb full-20150304.082015.535 -ca -pre restore_ [zimbra@mail-172 ~]$ cat /opt/zimbra/log/mailbox.log 2015-03-05 23:35:40,815 INFO [MailboxPurge] [name=spam.umhx3owv3s@mail-172.example.com;mid=3;] purge - Purging messages. 2015-03-05 23:36:11,619 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - AuthRequest elapsed=3 2015-03-05 23:36:11,784 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is starting 2015-03-05 23:36:11,905 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - waiting for FileLogWriter.FsyncThread-1425616402072 to finish. 2015-03-05 23:36:11,908 INFO [FileLogWriter.FsyncThread-1425616402072] [] redolog - fsync thread exiting 2015-03-05 23:36:11,908 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - FileLogWriter.FsyncThread-1425616402072 finished 2015-03-05 23:36:11,913 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - Redo log rollover took 128ms 2015-03-05 23:36:11,913 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore started for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6 2015-03-05 23:36:11,914 INFO [FileLogWriter.FsyncThread-1425616571913] [] redolog - Starting fsync thread with interval 10 2015-03-05 23:36:13,006 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - Creating database mboxgroup12 2015-03-05 23:36:14,150 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restoring to mailbox id 12 for new account 016c9f7c-e2e2-4f8f-96d0-e14a4327821d. 2015-03-05 23:36:14,237 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loading schema with version 7.2 2015-03-05 23:36:14,276 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Renaming volume_id column to locator in source table mail_item 2015-03-05 23:36:14,315 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 1 blobs during restore 2015-03-05 23:36:14,316 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Deleting directory /opt/zimbra/index/0/12/index 2015-03-05 23:36:14,317 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 2 index files during restore 2015-03-05 23:36:14,319 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/12/index/0 2015-03-05 23:36:14,319 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox 12 account 016c9f7c-e2e2-4f8f-96d0-e14a4327821d LOADED 2015-03-05 23:36:14,319 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] cache - initializing folder and tag caches for mailbox 12 2015-03-05 23:36:14,367 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Account user1@mail-172.example.com: All pending file IO completed (2 out of 2) 2015-03-05 23:36:14,367 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Run redo operations since last full backup 2015-03-05 23:36:14,400 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. [multiple repeats of this message] 2015-03-05 23:36:14,428 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. 2015-03-05 23:36:14,465 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailop - Deleting Message (id=257). 2015-03-05 23:36:14,474 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. [multiple repeats of this message] 2015-03-05 23:36:15,152 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. 2015-03-05 23:36:15,172 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Run redo operations from archived redo logs starting at sequence 17 2015-03-05 23:36:15,172 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore finished for user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox 6 to mailbox 12 2015-03-05 23:36:15,174 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is shut down 2015-03-05 23:36:15,174 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=12;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - RestoreRequest elapsed=3481 [zimbra@mail-172 ~]$ ls /opt/zimbra/redolog/archive/ redo-20150306.043322.070-seq17.log redo-20150306.043611.912-seq18.log
Second Example - Options -lb , -ca , -pre restore2_ , -br
For the second example, we'll again use the oldest full backup [full-20150304.082015.535 vs. full-20150306.043202.011] the account has with the -lb option. The message we had deleted, existed at the time of the full backup but did NOT in the last incremental backup. Actually, the delete operation is within the redologs of the last incremental. For the second example, we'll use the -br option rather than the -rf . We'll also be adjusting the -pre option to be restore2_ vs. restore_ in the first example.
Then we'll review the mailbox.log events. Note the following:
- Though the log will state the new mailboxId and zimbraId for the user, though:
- "user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6" vs "Restoring to mailbox id 13 for new account 4c99a395-2c39-46ec-8c1b-285d97d899df."
- Notice the mailboxId and zimbraId for restore2_ is different than our first example when we used restore_ for the -pre option.
- Since we failed to include the -rf option, the restore continues past the full backup and goes through all the other backups that exist, hence we see:
- "backup - Run redo operations since last full backup"
- One of the redologs in our incremental had the delete operation for the message in question and we see evidence of that delete in the log:
- "mailop - Deleting Message (id=257)."
- Since we DID include the -br option, the restore STOPS at our last backup session. It therefor DOESN'T replay the redologs in /opt/zimbra/redolog and it's subdirectory, archive and we do NOT have the following log events like we did in our first example.
- "backup - Run redo operations from archived redo logs starting at sequence 17"
- Towards the end, we'll get a confirmation that the restore has finished with some reference information again - mailboxId's .
- "Restore finished for user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox 6 to mailbox 13"
[zimbra@mail-172 ~]$ cat /dev/null > /opt/zimbra/log/mailbox.log [zimbra@mail-172 ~]$ zmrestore -a user1@mail-172.example.com -lb full-20150304.082015.535 -br -ca -pre restore2_ [zimbra@mail-172 ~]$ cat /opt/zimbra/log/mailbox.log 2015-03-05 23:39:30,176 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - AuthRequest elapsed=2 2015-03-05 23:39:30,263 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is starting 2015-03-05 23:39:30,352 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - waiting for FileLogWriter.FsyncThread-1425616571913 to finish. 2015-03-05 23:39:30,360 INFO [FileLogWriter.FsyncThread-1425616571913] [] redolog - fsync thread exiting 2015-03-05 23:39:30,360 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - FileLogWriter.FsyncThread-1425616571913 finished 2015-03-05 23:39:30,371 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - Redo log rollover took 107ms 2015-03-05 23:39:30,371 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore started for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6 2015-03-05 23:39:30,377 INFO [FileLogWriter.FsyncThread-1425616770369] [] redolog - Starting fsync thread with interval 10 2015-03-05 23:39:30,381 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - Creating database mboxgroup13 2015-03-05 23:39:30,992 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restoring to mailbox id 13 for new account 4c99a395-2c39-46ec-8c1b-285d97d899df. 2015-03-05 23:39:30,993 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loading schema with version 7.2 2015-03-05 23:39:30,997 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Renaming volume_id column to locator in source table mail_item 2015-03-05 23:39:31,022 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 1 blobs during restore 2015-03-05 23:39:31,023 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Deleting directory /opt/zimbra/index/0/13/index 2015-03-05 23:39:31,023 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 2 index files during restore 2015-03-05 23:39:31,024 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/13/index/0 2015-03-05 23:39:31,024 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox 13 account 4c99a395-2c39-46ec-8c1b-285d97d899df LOADED 2015-03-05 23:39:31,024 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] cache - initializing folder and tag caches for mailbox 13 2015-03-05 23:39:31,050 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Account user1@mail-172.example.com: All pending file IO completed (2 out of 2) 2015-03-05 23:39:31,050 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Run redo operations since last full backup 2015-03-05 23:39:31,053 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. [multiple repeats of this message] 2015-03-05 23:39:31,070 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. 2015-03-05 23:39:31,075 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailop - Deleting Message (id=257). 2015-03-05 23:39:31,079 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. [multiple repeats of this message] 2015-03-05 23:39:31,829 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] purge - Purging messages. 2015-03-05 23:39:31,832 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore finished for user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox 6 to mailbox 13 2015-03-05 23:39:31,833 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is shut down 2015-03-05 23:39:31,834 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=13;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - RestoreRequest elapsed=1579 2015-03-05 23:39:40,932 INFO [MailboxPurge] [name=user5@mail-172.example.com;mid=7;] purge - Purging messages. [zimbra@mail-172 ~]$
Third Example - Options -lb , -ca , -pre restore3_ , -rf
For the third example, we'll again use the oldest full backup [full-20150304.082015.535 vs. full-20150306.043202.011] the account has with the -lb option. The message we had deleted, existed at the time of the full backup but did NOT in the last incremental backup. For the third example, we'll use the -rf option rather than the -br . We'll also be adjusting the -pre option to be restore3_ vs. restore_ or restore2_ in the other examples.
Then we'll review the mailbox.log events. Note the following:
- Though the log will state the new mailboxId and zimbraId for the user, though:
- "user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6" vs "Restoring to mailbox id 14 for new account d72a6d8d-8117-4a90-b8cf-f972f4af056d."
- Notice the mailboxId and zimbraId for restore3_ is different than our other examples.
- Since we INCLUDED the -rf option, the restore STOPS once the full backup is restored. Hence, we DON'T see the following line like we did in the other examples:
- "backup - Run redo operations since last full backup"
- And because it DIDN'T replay any of the redologs, there is NO reference to the following line either in the log:
- "mailop - Deleting Message (id=257)."
- NOR is there a reference to the following line either in the log:
- "backup - Run redo operations from archived redo logs starting at sequence 17"
- Towards the end, we'll get a confirmation that the restore has finished with some reference information again - mailboxId's .
- "Restore finished for user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox 6 to mailbox 14"
[zimbra@mail-172 ~]$ cat /dev/null > /opt/zimbra/log/mailbox.log [zimbra@mail-172 ~]$ zmrestore -a user1@mail-172.example.com -lb full-20150304.082015.535 -rf -ca -pre restore3_ [zimbra@mail-172 ~]$ cat /opt/zimbra/log/mailbox.log 2015-03-05 23:42:50,233 INFO [Timer-Zimbra] [] SessionCache - Removed 1 idle sessions (SOAP). 1 active sessions remain. 2015-03-05 23:42:58,194 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - AuthRequest elapsed=3 2015-03-05 23:42:58,282 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is starting 2015-03-05 23:42:58,475 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - waiting for FileLogWriter.FsyncThread-1425616770369 to finish. 2015-03-05 23:42:58,477 INFO [FileLogWriter.FsyncThread-1425616770369] [] redolog - fsync thread exiting 2015-03-05 23:42:58,477 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - FileLogWriter.FsyncThread-1425616770369 finished 2015-03-05 23:42:58,485 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - Redo log rollover took 193ms 2015-03-05 23:42:58,485 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore started for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6 2015-03-05 23:42:58,495 INFO [FileLogWriter.FsyncThread-1425616978485] [] redolog - Starting fsync thread with interval 10 2015-03-05 23:42:58,515 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - Creating database mboxgroup14 2015-03-05 23:43:06,041 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restoring to mailbox id 14 for new account d72a6d8d-8117-4a90-b8cf-f972f4af056d. 2015-03-05 23:43:06,042 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loading schema with version 7.2 2015-03-05 23:43:06,048 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Renaming volume_id column to locator in source table mail_item 2015-03-05 23:43:06,092 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 1 blobs during restore 2015-03-05 23:43:06,093 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Deleting directory /opt/zimbra/index/0/14/index 2015-03-05 23:43:06,093 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 2 index files during restore 2015-03-05 23:43:06,095 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/14/index/0 2015-03-05 23:43:06,095 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox 14 account d72a6d8d-8117-4a90-b8cf-f972f4af056d LOADED 2015-03-05 23:43:06,095 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] cache - initializing folder and tag caches for mailbox 14 2015-03-05 23:43:06,264 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Account user1@mail-172.example.com: All pending file IO completed (2 out of 2) 2015-03-05 23:43:06,265 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore finished for user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox 6 to mailbox 14 2015-03-05 23:43:06,266 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is shut down 2015-03-05 23:43:06,266 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=14;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - RestoreRequest elapsed=7999 [zimbra@mail-172 ~]$
Below is to confirm that indeed, it was only the restore with the -rf that did in fact have the message that was deleted after it.
[zimbra@mail-172 ~]$ zmmailbox -z -m restore_user1@mail-172.example.com s -t mess in:Inbox num: 0, more: false [zimbra@mail-172 ~]$ zmmailbox -z -m restore2_user1@mail-172.example.com s -t mess in:Inbox num: 0, more: false [zimbra@mail-172 ~]$ zmmailbox -z -m restore3_user1@mail-172.example.com s -t mess in:Inbox num: 1, more: false Id Type From Subject Date ---- ---- -------------------- -------------------------------------------------- -------------- 1. 257 mess admin Test1 02/23/15 09:03
Fourth Example - To Confirm -lb Label Does Matter
This example is to show that the -lb does indeed matter. Earlier examples used the oldest full we had that did include the deleted message in the account. We'll create a new full backup of the parent account, user1@ , where the message is absent and then do two restores using the differing full backup session labels with the proper option of -rf since we are wanting the restore to stop once the full session data is restored.
[zimbra@mail-172 ~]$ zmmailbox -z -m user1@mail-172.example.com s -t mess in:Inbox num: 0, more: false [zimbra@mail-172 ~]$ zmbackup -f -a all full-20150306.055845.429 [zimbra@mail-172 ~]$ zmbackupquery -a user1@`zmhostname` Account: user1@mail-172.example.com Label: full-20150306.055845.429 Type: full Started: Fri, 2015/03/06 00:58:45.429 EST Ended: Fri, 2015/03/06 00:58:52.357 EST Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93 Label: incr-20150306.043321.683 Type: incremental Started: Thu, 2015/03/05 23:33:21.683 EST Ended: Thu, 2015/03/05 23:33:27.495 EST Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93 Label: full-20150306.043202.011 Type: full Started: Thu, 2015/03/05 23:32:02.011 EST Ended: Thu, 2015/03/05 23:32:13.316 EST Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93 Label: incr-20150305.060013.901 Type: incremental Started: Thu, 2015/03/05 01:00:13.901 EST Ended: Thu, 2015/03/05 01:00:26.273 EST Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93 Label: incr-20150304.085810.220 Type: incremental Started: Wed, 2015/03/04 03:58:10.220 EST Ended: Wed, 2015/03/04 03:58:16.393 EST Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93 Label: full-20150304.082015.535 Type: full Started: Wed, 2015/03/04 03:20:15.535 EST Ended: Wed, 2015/03/04 03:20:39.109 EST Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93 [zimbra@mail-172 ~]$ zmrestore -a user1@mail-172.example.com -lb full-20150306.055845.429 -rf -ca -pre restore4_ [zimbra@mail-172 ~]$ zmrestore -a user1@mail-172.example.com -lb full-20150304.082015.535 -rf -ca -pre restore5_ [zimbra@mail-172 ~]$ zmmailbox -z -m restore4_user1@mail-172.example.com s -t mess in:Inbox num: 0, more: false [zimbra@mail-172 ~]$ zmmailbox -z -m restore5_user1@mail-172.example.com s -t mess in:Inbox num: 1, more: false Id Type From Subject Date ---- ---- -------------------- -------------------------------------------------- -------------- 1. 257 mess admin Test1 02/23/15 09:03 [zimbra@mail-172 ~]$
What Is Different Between The Parent Account And The Restored Accounts Where We Used The -ca And -pre Options [LDAP]
Below is a difference between the zmprov ga output between the parent account [user1@] and our last restore example [restore5_user1@]. Note, in our example here the parent account was created with default values and there was no customization's or changes to the defaults.
[zimbra@mail-172 ~]$ zmprov ga user1@`zmhostname` > /tmp/user1 [zimbra@mail-172 ~]$ zmprov ga restore5_user1@`zmhostname` > /tmp/restore5 [zimbra@mail-172 ~]$ diff /tmp/user1 /tmp/restore5 1c1 < # name user1@mail-172.example.com --- > # name restore5_user1@mail-172.example.com 3c3 < mail: user1@mail-172.example.com --- > mail: restore5_user1@mail-172.example.com 8c8 < uid: user1 --- > uid: restore5_user1 20,22c20 < zimbraAuthTokens: 661297364|1425702763611|8.6.0_GA_1153 < zimbraAuthTokens: 414162390|1425702787760|8.6.0_GA_1153 < zimbraAuthTokens: 2046929787|1425707910566|8.6.0_GA_1153 --- > zimbraAuthTokens: 1030982216|1425708106344|8.6.0_GA_1153 159c157 < zimbraId: 3c397edc-c015-4d82-ae8a-1084692f8a93 --- > zimbraId: e5c38a4b-6bea-45bb-9e14-dd3f39159a61 170c168 < zimbraMailDeliveryAddress: user1@mail-172.example.com --- > zimbraMailDeliveryAddress: restore5_user1@mail-172.example.com
Restore encountered redo log sequence error: Found gap in redo log sequence
Example error:
Error occurred: system failure: Restore encountered redo log sequence error: Found gap in redo log sequence; missing 3899 through 3895; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option
This error actually gives the two basic options you have when you encounter this. Depending on your circumstances, you might first do the restore using the --ignoreRedoErrors options and recovery what data you can prior to discarding the 'bad' backup sessions in question and attempting a new full backup of the users.
How-To Example - Extended Look At RedoErrors With Log Events
First, let's replicate a redolog failure situation, which can be done by moving the redologs from the /opt/zimbra/redolog/archive/ to a temporary directory.
[zimbra@mail-172 redolog]$ cd archive/ [zimbra@mail-172 archive]$ ls redo-20150306.060015.331-seq21.log redo-20150306.060109.166-seq22.log [zimbra@mail-172 archive]$ mkdir /tmp/redolog-archive [zimbra@mail-172 archive]$ mv * /tmp/redolog-archive/ [zimbra@mail-172 archive]$ cat /dev/null > /opt/zimbra/log/mailbox.log
Now that we've moved the redolog sequence files out of the redolog/archive directory path, we can attempt our zmrestore command. Note that the incremental backup label for this attempt is called incr-20150306.170810.793 . Though it fails because of the missing redologs, the backup is still created.
[zimbra@mail-172 archive]$ zmbackup -i Error occurred: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option [zimbra@mail-172 archive]$ cat /opt/zimbra/log/mailbox.log 2015-03-06 12:08:06,442 INFO [MailboxPurge] [name=ham.jgfdb3oqjf@mail-172.example.com;mid=4;] purge - Purging messages. 2015-03-06 12:08:10,350 INFO [qtp509886383-484:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] soap - AuthRequest elapsed=4 2015-03-06 12:08:10,413 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup request started 2015-03-06 12:08:10,579 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Found 16 accounts on server mail-172.example.com 2015-03-06 12:08:10,797 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] io - AsyncFileCopier is starting 2015-03-06 12:08:10,798 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup started for backup set; label: incr-20150306.170810.793 [cut] 2015-03-06 12:08:11,277 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup is started for account user1@mail-172.example.com 2015-03-06 12:08:11,277 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Saving account information from LDAP 2015-03-06 12:08:11,281 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup has ended 2015-03-06 12:08:11,281 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Account user1@mail-172.example.com in backup set incr-20150306.170810.793: All pending file IO completed (0 out of 0) [cut] 2015-03-06 12:08:11,310 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup is started for account admin@mail-172.example.com 2015-03-06 12:08:11,310 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Saving account information from LDAP 2015-03-06 12:08:11,312 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup has ended 2015-03-06 12:08:11,312 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Account admin@mail-172.example.com in backup set incr-20150306.170810.793: All pending file IO completed (0 out of 0) 2015-03-06 12:08:11,711 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] redolog - waiting for FileLogWriter.FsyncThread-1425621669168 to finish. 2015-03-06 12:08:11,712 INFO [FileLogWriter.FsyncThread-1425621669168] [] redolog - fsync thread exiting 2015-03-06 12:08:11,712 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] redolog - FileLogWriter.FsyncThread-1425621669168 finished 2015-03-06 12:08:11,799 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] redolog - Redo log rollover took 487ms 2015-03-06 12:08:11,799 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Found 1 redo logs to backup 2015-03-06 12:08:11,800 INFO [FileLogWriter.FsyncThread-1425661691796] [] redolog - Starting fsync thread with interval 10 2015-03-06 12:08:11,864 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Last backed-up redo log sequence = 21 2015-03-06 12:08:11,864 ERROR [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option 2015-03-06 12:08:11,973 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Copying redo log /opt/zimbra/redolog/archive/redo-20150306.170811.794-seq23.log to /opt/zimbra/backup/tmp/incr-20150306.170810.793/redologs/redo-20150306.170811.794-seq23.log 2015-03-06 12:08:12,100 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Retaining recent redo log /opt/zimbra/redolog/archive/redo-20150306.170811.794-seq23.log 2015-03-06 12:08:19,375 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup set incr-20150306.170810.793: All pending file IO completed (0 out of 0) 2015-03-06 12:08:19,378 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] io - AsyncFileCopier is shut down 2015-03-06 12:08:19,378 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup finished for backup set; label: incr-20150306.170810.793 2015-03-06 12:08:20,565 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] SoapEngine - handler exception com.zimbra.cs.backup.BackupServiceException: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option ExceptionId:qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest:1425661691864:14586f805b4c1b0e Code:backup.REDOLOG_OUT_OF_SEQUENCE at com.zimbra.cs.backup.BackupServiceException.REDOLOG_OUT_OF_SEQUENCE(BackupServiceException.java:91) at com.zimbra.cs.backup.util.Utils.splitRedoLogsAtSeq(Utils.java:302) at com.zimbra.cs.backup.FileBackupTarget$FileBackupSet.backupRedoLogs(FileBackupTarget.java:1360) at com.zimbra.cs.backup.BackupSet.startIncrementalBackup(BackupSet.java:817) at com.zimbra.cs.backup.FileBackupTarget$FileBackupSet.startIncrementalBackup(FileBackupTarget.java:1067) at com.zimbra.cs.backup.BackupManager.backupIncremental(BackupManager.java:336) at com.zimbra.cs.service.backup.Backup.handleNetworkRequest(Backup.java:153) at com.zimbra.cs.service.NetworkDocumentHandler.handle(NetworkDocumentHandler.java:23) at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEngine.java:569) at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:432) at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:266) at com.zimbra.soap.SoapServlet.doWork(SoapServlet.java:303) at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:213) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:209) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651) at com.zimbra.cs.servlet.RequestStringFilter.doFilter(RequestStringFilter.java:54) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:59) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:351) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) at com.zimbra.cs.servlet.ETagHeaderFilter.doFilter(ETagHeaderFilter.java:47) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) at com.zimbra.cs.servlet.ContextPathBasedThreadPoolBalancerFilter.doFilter(ContextPathBasedThreadPoolBalancerFilter.java:107) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(ZimbraQoSFilter.java:107) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) at com.zimbra.cs.servlet.ZimbraInvalidLoginFilter.doFilter(ZimbraInvalidLoginFilter.java:117) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:457) at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:326) at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:299) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:544) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:199) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:309) at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:81) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:462) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536) at java.lang.Thread.run(Thread.java:745) 2015-03-06 12:08:20,566 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] soap - BackupRequest elapsed=10154
To confirm the backup label was made and also to see how the zmbackupquery reports it, we can do the following:
[zimbra@mail-172 archive]$ zmbackupquery -lb incr-20150306.170810.793 -v Label: incr-20150306.170810.793 Type: incremental Status: completed (with errors) Started: Fri, 2015/03/06 12:08:10.793 EST Ended: Fri, 2015/03/06 12:08:19.378 EST Redo log sequence range: 23 .. 23 Number of accounts: 14 out of 14 completed Number of errors: 1 Accounts: admin@mail-172.example.com: completed galsync.qva4ekog0@mail-172.example.com: completed ham.jgfdb3oqjf@mail-172.example.com: completed restore2_user1@mail-172.example.com: completed restore3_user1@mail-172.example.com: completed restore_user1@mail-172.example.com: completed spam.umhx3owv3s@mail-172.example.com: completed user1@mail-172.example.com: completed user2@mail-172.example.com: completed user3@mail-172.example.com: completed user4@mail-172.example.com: completed user5@mail-172.example.com: completed user6@mail-172.example.com: completed virus-quarantine.gusiso1cd@mail-172.example.com: completed Error: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option com.zimbra.cs.backup.BackupServiceException: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option ExceptionId:qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/BackupRequest:1425661691864:14586f805b4c1b0e Code:backup.REDOLOG_OUT_OF_SEQUENCE at com.zimbra.cs.backup.BackupServiceException.REDOLOG_OUT_OF_SEQUENCE(BackupServiceException.java:91) at com.zimbra.cs.backup.util.Utils.splitRedoLogsAtSeq(Utils.java:302) at com.zimbra.cs.backup.FileBackupTarget$FileBackupSet.backupRedoLogs(FileBackupTarget.java:1360) at com.zimbra.cs.backup.BackupSet.startIncrementalBackup(BackupSet.java:817) at com.zimbra.cs.backup.FileBackupTarget$FileBackupSet.startIncrementalBackup(FileBackupTarget.java:1067) at com.zimbra.cs.backup.BackupManager.backupIncremental(BackupManager.java:336) at com.zimbra.cs.service.backup.Backup.handleNetworkRequest(Backup.java:153) at com.zimbra.cs.service.NetworkDocumentHandler.handle(NetworkDocumentHandler.java:23) at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEngine.java:569) at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:432) at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:266) at com.zimbra.soap.SoapServlet.doWork(SoapServlet.java:303) at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:213) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:209) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651) at com.zimbra.cs.servlet.RequestStringFilter.doFilter(RequestStringFilter.java:54) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:59) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:351) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) at com.zimbra.cs.servlet.ETagHeaderFilter.doFilter(ETagHeaderFilter.java:47) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) at com.zimbra.cs.servlet.ContextPathBasedThreadPoolBalancerFilter.doFilter(ContextPathBasedThreadPoolBalancerFilter.java:107) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(ZimbraQoSFilter.java:107) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) at com.zimbra.cs.servlet.ZimbraInvalidLoginFilter.doFilter(ZimbraInvalidLoginFilter.java:117) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:457) at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:326) at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:299) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:544) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:199) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:309) at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:81) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:462) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536) at java.lang.Thread.run(Thread.java:745) Total space: 14020MB Free space: 6031MB [zimbra@mail-172 archive]$
Let's now try doing a restore of an account.
[zimbra@mail-172 archive]$ cat /dev/null > /opt/zimbra/log/mailbox.log [zimbra@mail-172 archive]$ zmrestore -a user1@`zmhostname` -ca -pre restore6_ Error occurred: system failure: Restore encountered redo log sequence error: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option [zimbra@mail-172 archive]$ cat /opt/zimbra/log/mailbox.log 2015-03-06 12:12:31,884 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - RestoreRequest elapsed=38 2015-03-06 12:13:06,650 INFO [MailboxPurge] [name=user3@mail-172.example.com;mid=9;] purge - Purging messages. 2015-03-06 12:13:45,329 INFO [qtp509886383-484:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - AuthRequest elapsed=2 2015-03-06 12:13:45,474 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is starting 2015-03-06 12:13:45,512 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Using full backup full-20150306.055845.429 as starting point for restoring account 3c397edc-c015-4d82-ae8a-1084692f8a93 2015-03-06 12:13:45,662 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - waiting for FileLogWriter.FsyncThread-1425661691796 to finish. 2015-03-06 12:13:45,669 INFO [FileLogWriter.FsyncThread-1425661691796] [] redolog - fsync thread exiting 2015-03-06 12:13:45,669 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - FileLogWriter.FsyncThread-1425661691796 finished 2015-03-06 12:13:45,678 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=zimbra;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] redolog - Redo log rollover took 166ms 2015-03-06 12:13:45,678 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restore started for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) original mailbox id 6 2015-03-06 12:13:45,700 INFO [FileLogWriter.FsyncThread-1425662025678] [] redolog - Starting fsync thread with interval 10 2015-03-06 12:13:45,981 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - Creating database mboxgroup17 2015-03-06 12:13:46,695 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Restoring to mailbox id 17 for new account f7f4c1a8-15d0-41a1-bd11-6e05359ea3e8. 2015-03-06 12:13:46,745 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loading schema with version 7.2 2015-03-06 12:13:46,785 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Renaming volume_id column to locator in source table mail_item 2015-03-06 12:13:46,820 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 0 blobs during restore 2015-03-06 12:13:46,822 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Deleting directory /opt/zimbra/index/0/17/index 2015-03-06 12:13:46,882 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - loaded 2 index files during restore 2015-03-06 12:13:46,887 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/17/index/0 2015-03-06 12:13:46,887 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox 17 account f7f4c1a8-15d0-41a1-bd11-6e05359ea3e8 LOADED 2015-03-06 12:13:46,887 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] cache - initializing folder and tag caches for mailbox 17 2015-03-06 12:13:46,912 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Account user1@mail-172.example.com: All pending file IO completed (2 out of 2) 2015-03-06 12:13:46,912 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Run redo operations since last full backup 2015-03-06 12:13:47,047 ERROR [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option 2015-03-06 12:13:47,047 WARN [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Redo log sequence error: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option com.zimbra.cs.backup.BackupServiceException: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option ExceptionId:qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest:1425662027047:14586f805b4c1b0e Code:backup.REDOLOG_OUT_OF_SEQUENCE at com.zimbra.cs.backup.BackupServiceException.REDOLOG_OUT_OF_SEQUENCE(BackupServiceException.java:91) at com.zimbra.cs.backup.util.Utils.splitRedoLogsAtSeq(Utils.java:302) at com.zimbra.cs.backup.RestoreAccountSession.checkRedoLogSequenceContiguity(RestoreAccountSession.java:534) at com.zimbra.cs.backup.RestoreAccountSession.endRestore(RestoreAccountSession.java:350) at com.zimbra.cs.backup.FileBackupTarget$RestoreAcctSession.endRestore(FileBackupTarget.java:2524) at com.zimbra.cs.backup.FileBackupTarget.restore(FileBackupTarget.java:490) at com.zimbra.cs.backup.BackupManager.restore(BackupManager.java:826) at com.zimbra.cs.service.backup.Restore.handle(Restore.java:137) at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEngine.java:569) at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:432) at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:266) at com.zimbra.soap.SoapServlet.doWork(SoapServlet.java:303) at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:213) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:209) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651) at com.zimbra.cs.servlet.RequestStringFilter.doFilter(RequestStringFilter.java:54) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:59) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:351) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) at com.zimbra.cs.servlet.ETagHeaderFilter.doFilter(ETagHeaderFilter.java:47) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) at com.zimbra.cs.servlet.ContextPathBasedThreadPoolBalancerFilter.doFilter(ContextPathBasedThreadPoolBalancerFilter.java:107) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(ZimbraQoSFilter.java:107) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) at com.zimbra.cs.servlet.ZimbraInvalidLoginFilter.doFilter(ZimbraInvalidLoginFilter.java:117) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:457) at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:326) at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:299) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:544) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:199) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:309) at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:81) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:462) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536) at java.lang.Thread.run(Thread.java:745) 2015-03-06 12:13:47,047 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Cleaning up data from failed restore (mid=17) 2015-03-06 12:13:47,063 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - Putting mailbox 17 under maintenance. 2015-03-06 12:13:47,084 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mailbox - clearing contents of mailbox 17, group 17 2015-03-06 12:13:47,331 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] mbxmgr - Mailbox for account f7f4c1a8-15d0-41a1-bd11-6e05359ea3e8 DELETED 2015-03-06 12:13:47,375 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Cleaning up account created during failed restore (account=f7f4c1a8-15d0-41a1-bd11-6e05359ea3e8) 2015-03-06 12:13:47,752 WARN [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [name=user1@mail-172.example.com;mid=17;ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] backup - Error occurred during restore account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) com.zimbra.common.service.ServiceException: system failure: Restore encountered redo log sequence error: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option ExceptionId:qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest:1425662027047:14586f805b4c1b0e Code:service.FAILURE at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:260) at com.zimbra.cs.backup.RestoreAccountSession.checkRedoLogSequenceContiguity(RestoreAccountSession.java:545) at com.zimbra.cs.backup.RestoreAccountSession.endRestore(RestoreAccountSession.java:350) at com.zimbra.cs.backup.FileBackupTarget$RestoreAcctSession.endRestore(FileBackupTarget.java:2524) at com.zimbra.cs.backup.FileBackupTarget.restore(FileBackupTarget.java:490) at com.zimbra.cs.backup.BackupManager.restore(BackupManager.java:826) at com.zimbra.cs.service.backup.Restore.handle(Restore.java:137) at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEngine.java:569) at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:432) at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:266) at com.zimbra.soap.SoapServlet.doWork(SoapServlet.java:303) at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:213) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:209) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651) at com.zimbra.cs.servlet.RequestStringFilter.doFilter(RequestStringFilter.java:54) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:59) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:351) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) at com.zimbra.cs.servlet.ETagHeaderFilter.doFilter(ETagHeaderFilter.java:47) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) at com.zimbra.cs.servlet.ContextPathBasedThreadPoolBalancerFilter.doFilter(ContextPathBasedThreadPoolBalancerFilter.java:107) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(ZimbraQoSFilter.java:107) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) at com.zimbra.cs.servlet.ZimbraInvalidLoginFilter.doFilter(ZimbraInvalidLoginFilter.java:117) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:457) at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:326) at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:299) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:544) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:199) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:309) at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:81) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:462) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536) at java.lang.Thread.run(Thread.java:745) Caused by: com.zimbra.cs.backup.BackupServiceException: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option ExceptionId:qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest:1425662027047:14586f805b4c1b0e Code:backup.REDOLOG_OUT_OF_SEQUENCE at com.zimbra.cs.backup.BackupServiceException.REDOLOG_OUT_OF_SEQUENCE(BackupServiceException.java:91) at com.zimbra.cs.backup.util.Utils.splitRedoLogsAtSeq(Utils.java:302) at com.zimbra.cs.backup.RestoreAccountSession.checkRedoLogSequenceContiguity(RestoreAccountSession.java:534) ... 56 more 2015-03-06 12:13:47,755 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] io - AsyncFileCopier is shut down 2015-03-06 12:13:47,755 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] SoapEngine - handler exception com.zimbra.common.service.ServiceException: system failure: Restore encountered redo log sequence error: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option ExceptionId:qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest:1425662027047:14586f805b4c1b0e Code:service.FAILURE at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:260) at com.zimbra.cs.backup.RestoreAccountSession.checkRedoLogSequenceContiguity(RestoreAccountSession.java:545) at com.zimbra.cs.backup.RestoreAccountSession.endRestore(RestoreAccountSession.java:350) at com.zimbra.cs.backup.FileBackupTarget$RestoreAcctSession.endRestore(FileBackupTarget.java:2524) at com.zimbra.cs.backup.FileBackupTarget.restore(FileBackupTarget.java:490) at com.zimbra.cs.backup.BackupManager.restore(BackupManager.java:826) at com.zimbra.cs.service.backup.Restore.handle(Restore.java:137) at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEngine.java:569) at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:432) at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:266) at com.zimbra.soap.SoapServlet.doWork(SoapServlet.java:303) at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:213) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:209) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651) at com.zimbra.cs.servlet.RequestStringFilter.doFilter(RequestStringFilter.java:54) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:59) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:351) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) at com.zimbra.cs.servlet.ETagHeaderFilter.doFilter(ETagHeaderFilter.java:47) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) at com.zimbra.cs.servlet.ContextPathBasedThreadPoolBalancerFilter.doFilter(ContextPathBasedThreadPoolBalancerFilter.java:107) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(ZimbraQoSFilter.java:107) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) at com.zimbra.cs.servlet.ZimbraInvalidLoginFilter.doFilter(ZimbraInvalidLoginFilter.java:117) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:457) at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:326) at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:299) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:544) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:199) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:309) at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:81) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:462) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536) at java.lang.Thread.run(Thread.java:745) Caused by: com.zimbra.cs.backup.BackupServiceException: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option ExceptionId:qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest:1425662027047:14586f805b4c1b0e Code:backup.REDOLOG_OUT_OF_SEQUENCE at com.zimbra.cs.backup.BackupServiceException.REDOLOG_OUT_OF_SEQUENCE(BackupServiceException.java:91) at com.zimbra.cs.backup.util.Utils.splitRedoLogsAtSeq(Utils.java:302) at com.zimbra.cs.backup.RestoreAccountSession.checkRedoLogSequenceContiguity(RestoreAccountSession.java:534) ... 56 more 2015-03-06 12:13:47,755 INFO [qtp509886383-263:https://127.0.0.1:7071/service/admin/soap/RestoreRequest] [ip=127.0.0.1;ua=zmrestore/8.6.0_GA_1153;] soap - RestoreRequest elapsed=2358 [zimbra@mail-172 archive]$ zmprov ga restore6_user1@`zmhostname` | head ERROR: account.NO_SUCH_ACCOUNT (no such account: restore6_user1@mail-172.example.com)
Let's now see what happens if we move the missing redolog seq files back into /opt/zimbra/redolog/archive and then if we move/copy them into the backup sessions redolog directory. With restore attempts being done after each move/copy.
[zimbra@mail-172 archive]$ mv /tmp/redolog-archive/* ./ [zimbra@mail-172 archive]$ pwd /opt/zimbra/redolog/archive [zimbra@mail-172 archive]$ ls redo-20150306.060015.331-seq21.log redo-20150306.060109.166-seq22.log redo-20150306.170811.794-seq23.log redo-20150306.171345.676-seq24.log [zimbra@mail-172 archive]$ zmrestore -a user1@`zmhostname` -ca -pre restore6_ Error occurred: system failure: Restore encountered redo log sequence error: Found gap in redo log sequence; missing 22; To avoid future restore problems, discard all existing backups and take a full backup of all accounts; If this error occurred during restore, try the --ignoreRedoErrors option [zimbra@mail-172 archive]$ pwd /opt/zimbra/redolog/archive [zimbra@mail-172 archive]$ cp *seq21.log *seq22.log /opt/zimbra/backup/sessions/incr-20150306.170810.793/redologs/ [zimbra@mail-172 archive]$ zmrestore -a user1@`zmhostname` -ca -pre restore6_ [zimbra@mail-172 archive]$ zmprov ga restore6_user1@`zmhostname` | head -2 # name restore6_user1@mail-172.example.com cn: user1
Redirected Restore With No Deletes To Then Import Data From A Specific Time Range Into A Subdirectory Of Users Archive Account
This example would be the steps one might do if you discovered that the archiving service wasn't working as expected for a given date range and you wanted to populate the users archive account with the missing data. Normally, this would involve many users or all of your users - hence, why the issue about licenses is addressed.
Note, I use the following variables below but you should be replacing them to suit your needs.
- $USER = users production email
- recovery_$USER = the new email address that the restore created [-ca -pre recovery_]
- $DOMAIN = the domain for the user
- $USER-archive@$DOMAIN.archive = the production users archive email address
Determine all the necessary variables for your restore command.
- zmrestore -a $USER
- -ca -pre recovery_
- -lb determine what full backup label to use
- determine if you need to state the incremental backup label [-restoreToIncrLabel] and/or a specific time to stop the restore at [-restoreToTime]
- determine if you need to use -rf or -br
- --skipDeletes
For example:
$ zmrestore -a user1@`zmhostname` -ca -pre recovery_ -lb full-20150321.050015.999 -restoreToTime 20150323000000 -br --skipDeletes using restore to time of 2015/03/23 00:00:00
Set the restored account to not consume a normal license [notice recovery_$USER vs $USER]:
$ zmprov ma recovery_$USER zimbraIsSystemResource TRUE
Get what the archive accounts address is for the main account:
$ zmprov ga $USER amavisArchiveQuarantineTo
Set the restored account to not consume an archive license [notice recovery_$USER vs $USER]:
$ zmprov ma recovery_$USER amavisArchiveQuarantineTo ""
Create a directory to dump all the export data in. You might want to have this shared across your mailstores [that hold archive accounts] so you can then import them locally to the mailstore the archive account resides on.
$ mkdir /tmp/export $ cd /tmp/export
Example exports out all data between a given date , adjust the account and date string below. Also, the filename that is being saved.
$ zmmailbox -z -m recovery_$USER@$DOMAIN gru '//?fmt=tgz&query=under:/ after:"4/8/15" AND before:"4/11/15"' > /tmp/export/$USER.tgz
Example of what is dumped
$ pwd /tmp/export $ ls admin.tgz $ tar zxvf $USER.tgz Inbox/0000000776-ZCS Backup Report_ SUCCESS.eml.meta Inbox/0000000776-ZCS Backup Report_ SUCCESS.eml Inbox/0000000777-Daily mail report for 2015-04-09.eml.meta Inbox/0000000777-Daily mail report for 2015-04-09.eml Inbox/0000000778-ZCS Backup Report_ SUCCESS.eml.meta Inbox/0000000778-ZCS Backup Report_ SUCCESS.eml Inbox/0000000779-Daily mail report for 2015-04-10.eml.meta Inbox/0000000779-Daily mail report for 2015-04-10.eml
Now to import the data into the users Archive account. Below example will dump the imported data into a specfic folder that will be created [I'm using Imported]. Adjust the account and filename below.
$ zmmailbox -z -m $USER-archive@$DOMAIN.archive pru "//?fmt=tgz&subfolder=Imported" /tmp/export/$USER.tgz $ zmmailbox -z -m $USER-archive@$DOMAIN.archive gaf Id View Unread Msg Count Path ---------- ---- ---------- ---------- ---------- 1 unkn 0 0 / 16 docu 0 0 /Briefcase 10 appo 0 0 /Calendar 14 mess 0 0 /Chats 7 cont 0 0 /Contacts 6 mess 0 0 /Drafts 13 cont 0 0 /Emailed Contacts 789 unkn 0 0 /Imported 790 mess 4 4 /Imported/Inbox 2 mess 415 420 /Inbox 4 mess 0 0 /Junk 5 mess 0 0 /Sent 15 task 0 0 /Tasks 3 unkn 0 0 /Trash
Here's a (customer-supplied) script that does the above in batch. It was used to restore archives from a date range for 29k users: https://github.com/morganlljones/zimbra_backup2archive
Quiz
- What are three methods you can locate which backups a user is in?
- To stop a restore from playing all the way up to the current time, you would need to use one of these two zmrestore variables - what are they?
- If you see an error about being unable to restore because of redologs, what option might you need to include?
- What is the primary log file on the ZCS server you'll monitor or review for restore or backup issues?
- Provide two methods to increase logging/debug output when attempting a restore?
- Can you perform a full system restore from the backups done on an older ZCS version?
- What are the two identify variables that zmbackup/zmrestore uses for an individual?
- What user variable would you use to locate the directory a users data is stored in within a backup session?