Troubleshooting Course Content Rough Drafts-Recover Missing Data - Server: Difference between revisions
(Created page with "==Need To Do== * review training materials for sys admin course * determine what should be 'video' and how it can all be built upon preceding steps in a hands on training man...") |
mNo edit summary |
||
(4 intermediate revisions by the same user not shown) | |||
Line 1: | Line 1: | ||
__FORCETOC__ | |||
==Need To Do== | ==Need To Do== | ||
Line 355: | Line 357: | ||
====Server Cloning - Server Moves/Testing Purposes==== | ====Server Cloning - Server Moves/Testing Purposes==== | ||
== | ==Trouble-shooting Backup/Restore Issues And Other General Questions== | ||
=== | ===Backup And Restore Compatibility Between ZCS Versions=== | ||
* | * You can use backups from older versions of ZCS for account restores. | ||
* You can '''NOT''' use backups from old versions of ZCS for system restores though. Disaster Recovery restores have to be done with the same version of ZCS as the backups were done with. | |||
* | |||
References: | |||
* | * Upgrades And Compatibility Of Older Backups | ||
** | ** "Backups must be compatible across patch releases" | ||
** | *** http://bugzilla.zimbra.com/show_bug.cgi?id=16239 | ||
** | ** "Incorrect upgrade documentation regarding backups" | ||
* | *** http://bugzilla.zimbra.com/show_bug.cgi?id=30218 | ||
* | ** "support for restore across major versions" | ||
*** http://bugzilla.zimbra.com/show_bug.cgi?id=15750 | |||
** "Add conversion tool to upgrade backup versions to allow restore on later zcs versions" | |||
*** http://bugzilla.zimbra.com/show_bug.cgi?id=14002 | |||
* From 5.0.7 - 5.0.10 You Might See minor version upgrades moving your backups into a subdirectory | |||
** "upgrade incorrectly invalidates backups." | |||
*** http://bugzilla.zimbra.com/show_bug.cgi?id=304 | |||
=== | ===Trouble-shooting=== | ||
Ref: | |||
* http://wiki.zimbra.com/wiki/Ajcody-Backup-Restore-Issues#Information_To_Provide_When_Submitting_A_Support_Case_For_Backup_Issues | |||
====Basic Backup Information To Submit To Support==== | |||
===== | =====Disk Space Usage Issues===== | ||
======Trend Data====== | |||
If there is concerns about disks/partitions getting full, this command would be helpful for trending data on your server. Send support the resulting df.tar file . Note - adjust the tail command if you want more than 20 day's worth of trending data, the -n 20 option. | |||
<pre> | |||
[zimbra@zcs806 tmp]$ /tmp | |||
[zimbra@zcs806 tmp]$ tar cvf /tmp/df.tar `find /opt/zimbra/zmstat -name df.cs\* | sort | tail -n 20` | |||
tar: Removing leading `/' from member names | |||
/opt/zimbra/zmstat/2014-03-10/df.csv.gz | |||
/opt/zimbra/zmstat/2014-03-11/df.csv.gz | |||
[cut - Ajcody] | |||
/opt/zimbra/zmstat/2014-03-27/df.csv.gz | |||
/opt/zimbra/zmstat/2014-03-28/df.csv.gz | |||
/opt/zimbra/zmstat/df.csv | |||
[zimbra@zcs806 tmp]$ ls -lah /tmp/df.tar | |||
-rw-r----- 1 zimbra zimbra 80K Mar 29 06:44 /tmp/df.tar | |||
[zimbra@zcs806 tmp]$ tar tvf /tmp/df.tar | |||
-rw-r----- zimbra/zimbra 2566 2014-03-11 00:00 opt/zimbra/zmstat/2014-03-10/df.csv.gz | |||
-rw-r----- zimbra/zimbra 2553 2014-03-12 00:00 opt/zimbra/zmstat/2014-03-11/df.csv.gz | |||
[cut - Ajcody] | |||
-rw-r----- zimbra/zimbra 2513 2014-03-28 00:00 opt/zimbra/zmstat/2014-03-27/df.csv.gz | |||
-rw-r----- zimbra/zimbra 2531 2014-03-29 00:00 opt/zimbra/zmstat/2014-03-28/df.csv.gz | |||
-rw-r----- zimbra/zimbra 8013 2014-03-29 06:40 opt/zimbra/zmstat/df.csv | |||
</pre> | |||
======Directory Sizes In /opt/zimbra====== | |||
Please see the following and provide the output to support. Note, even though this method is faster than doing a du it still can take awhile. | |||
* [[Ajcody-Server-Misc-Topics#Faster_Way_To_Get_Directory_Size_On_Filesytem_-_find_vs_du]] | |||
======Adjusting The Disk Alert Threshold====== | |||
'''Note''' - zmlocalconfig smtp_notify must return yes if you want to receive the notifications. | |||
If you just need to adjust the disk alert threshold, then see the following: | |||
See current values: | |||
zmlocalconfig | grep zmdisklog | |||
Example adjustment: | |||
<pre> | |||
su - zimbra | |||
zmlocalconfig -e zmdisklog_critical_threshold=98 | |||
zmlocalconfig -e zmdisklog_warn_threshold=95 | |||
zmstatctl | |||
</pre> | |||
To exclude a partition from the checks [example of two being excluded]: | |||
<pre> | |||
su - zimbra | |||
zmlocalconfig -e zmstat_df_excludes="/mount/point:/mount/point2" | |||
zmstatctl | |||
</pre> | |||
' | Note, depending on your version of ZCS, you might be hitting a bug where you'll keep getting emails until a logrotate happens. | ||
* Changing Zmstat-df values do not take affect until logrotate | |||
** https://bugzilla.zimbra.com/show_bug.cgi?id=79148 | |||
Some things to do to confirm and share with support or in bug. As zimbra | |||
<pre> | <pre> | ||
su - zimbra | |||
ls -la /var/log/zimbra.log | |||
df -h | |||
/dev/mapper/vg_rhel664-lv_root | |||
5.5G 3.5G 1.7G 68% / | |||
tmpfs 939M 0 939M 0% /dev/shm | |||
/dev/sda1 485M 79M 381M 18% /boot | |||
/dev/sdb1 30G 6.2G 23G 22% /opt | |||
date | |||
== | zmlocalconfig | grep zmdisklog | ||
zmdisklog_critical_threshold = 80 | |||
zmdisklog_warn_threshold = 85 | |||
zmlocalconfig -e zmdisklog_critical_threshold=95 | |||
zmlocalconfig -e zmdisklog_warn_threshold=90 | |||
zmlocalconfig | grep zmdisklog | |||
zmdisklog_critical_threshold = 95 | |||
zmdisklog_warn_threshold = 90 | |||
zmstatctl restart | |||
date | |||
ps -eaf | grep zmstat-df | |||
ls -la /var/log/zimbra.log | |||
date ; grep "Disk warning" /var/log/zimbra* ; zmmailbox -z -m admin@`zmhostname` s -l 100 -t message "Subject: Disk and after:yesterday" | |||
##Note - Emails by default go out every 10 minutes - for example: | |||
[zimbra@zcs803 ~]$ date ; grep "Disk warning" /var/log/zimbra* ; zmmailbox -z -m admin@`zmhostname` s -l 100 -t message "Subject: Disk and after:yesterday" | |||
[zimbra@ | Thu May 22 09:40:08 PDT 2014 | ||
/var/log/zimbra.log:May 22 08:30:00 zcs803 zimbramon[18826]: 18826:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82% | |||
/var/log/zimbra.log:May 22 08:40:00 zcs803 zimbramon[22970]: 22970:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82% | |||
/var/log/zimbra.log:May 22 08:50:00 zcs803 zimbramon[22970]: 22970:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82% | |||
/var/log/zimbra.log:May 22 09:00:00 zcs803 zimbramon[22970]: 22970:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82% | |||
## Note - I had readjusted the variable to not warn during this time segment ## | |||
/var/log/zimbra.log:May 22 09:20:00 zcs803 zimbramon[8322]: 8322:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82% | |||
/var/log/zimbra.log:May 22 09:30:00 zcs803 zimbramon[8322]: 8322:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82% | |||
/var/log/zimbra.log:May 22 09:40:00 zcs803 zimbramon[8322]: 8322:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82% | |||
num: 7, more: false | |||
Id Type From Subject Date | |||
---- ---- -------------------- -------------------------------------------------- -------------- | |||
/ | 1. 328 mess admin Disk / at 82% on zcs803.DOMAIN.com: 05/22/14 09:40 | ||
/ | 2. 327 mess admin Disk / at 82% on zcs803.DOMAIN.com: 05/22/14 09:30 | ||
3. 326 mess admin Disk / at 82% on zcs803.DOMAIN.com: 05/22/14 09:20 | |||
/ | ## Note - I had readjusted the variable to not warn during this time segment ## | ||
/ | 4. 325 mess admin Disk / at 82% on zcs803.DOMAIN.com: 05/22/14 09:00 | ||
/ | 5. 324 mess admin Disk / at 82% on zcs803.DOMAIN.com: 05/22/14 08:50 | ||
6. 323 mess admin Disk / at 82% on zcs803.DOMAIN.com: 05/22/14 08:40 | |||
7. 320 mess admin Disk / at 82% on zcs803.DOMAIN.com: 05/22/14 08:31 | |||
</pre> | |||
Continue to monitor your zmmailbox search results for an hour. | |||
=====The Basic Information Support Needs===== | |||
as root: | |||
* cat /etc/fstab | |||
** Shows us what is mounted upon boot | |||
* cat /proc/mounts | |||
** Shows us what is currently mounted and its status - you can see if a mount is read-only here. | |||
* df -hT | |||
** Lists current mounts using human-readable size information and also notes the filesystem type. | |||
as zimbra: | |||
* zmprov -l gs `zmhostname` | egrep 'Back|Redo' | |||
** Will show us a number of variables related to backup and redologs. Also tell us if your using auto-group or the default method. | |||
* du -sh /opt/zimbra/redolog | |||
** Will might notice your redolog logs aren't rolling over, causing a possible issue. | |||
* ls -latr /opt/zimbra/backup | |||
** This is the default backup target, please adjust this path here and below if you are using a different zimbraBackupTarget value. | |||
If | ** zmprov gs `zmhostname` zimbraBackupTarget | ||
** We'll be able to confirm permissions are right. | |||
* ls -latr /opt/zimbra/backup/tmp | |||
** This will show us if you have failed backup jobs and confirm tmp is being cleaned appropriately after the backup is done. | |||
* ls -latr /opt/zimbra/backup/sessions | |||
** This will show us what backup sessions are available and confirm permissions are correct. | |||
** Adjust path if your zimbraBackupTarget value is not the default path. | |||
* Some directory sizes in the backup directory: | |||
** Default path first | |||
*** du -sh `find /opt/zimbra/backup -maxdepth 2 -type d` | |||
** If your using a different backup target, check that directory also. Replace /opt/zimbra/backup above with your backup path. | |||
* zmbackupquery | |||
** This should match what's in the sessions directory and it will also tell us if status of each backup and how many accts were done. | |||
* crontab -l | grep -i back | |||
** This will show use when backups are support to run and with what options they are running with. | |||
* zmlocalconfig | grep -i back | |||
** This is useful to see a number of backup options not exposed in the crontab, things related to the zip options. | |||
* zmvolume -l | |||
** This is useful to see how many volumes are being used, if HSM is being used, and if compression is being done at the volume level. | |||
=====Additional Log Files Support Might Need===== | |||
And send the following logs: | |||
* /var/log/messages | |||
** Filesystem issues often times are noted here and also in syslog. This might explain an interruption in the backup process. Server restarts, filesystem going full, filesystem going read-only, etc. | |||
* /var/log/syslog | |||
* /opt/zimbra/log/mailbox.log | |||
** The backup activity is logged here. | |||
** And any other mailbox.log file that would cover the event | |||
====Additional Checks For Performance Specific Issues==== | |||
=====If Your Using a SAN or NFS For Your Backup Target - Please Check Your IOWait===== | |||
Ideally, you would compare iowait and performance data from the target backup host as well as the stats available on the ZCS servers. To get graphs and stats on this from ZCS, please see [[Ajcody-Testing-Debugging#zmstat_and_zmstat-chart]] . You should submit this data and iowait conclusions if you still need to submit a support case about backup performance issues. | |||
* | =====Is HSM Running During Your Backup Window===== | ||
** https://bugzilla.zimbra.com/show_bug.cgi?id= | * Are you running HSM? HSM should not be ran during your backup window. | ||
** "RFE: HSM and backup should not run at the same time if initated." | |||
*** https://bugzilla.zimbra.com/show_bug.cgi?id=33452 | |||
=====Are You Using --zipStore===== | |||
--zipStore zips the blobs vs. keeping the blobs as individual files. --zipStore does not use compression either. For most circumstances, this will give the best performance, especially with NFS. This should be the default behavior of the backups, the following RFE is when it became the default [ZCS6+] : | |||
* "backup: default to the zip option" | |||
** https://bugzilla.zimbra.com/show_bug.cgi?id=31836#c6 | |||
*** Link to comment that explains options and default behavior. | |||
To see if zip's are being used for backups for example, in the backup/session directory you'll know if it is by seeing .zip files: | |||
mail:~/backup/sessions/full-20080820.160003.770/accounts/115/988/11598896-a89b-4b9d-bedb-1ed1afcb6c87/blobs | |||
zimbra$ ls blobs-1.zip blobs-2.zip blobs-3.zip blobs-4.zip | |||
To see if the zip file is using compression [-Z option for unzip will indicate whether or not the archive is actually compressed] : | |||
unzip -Z blobs-4.zip | |||
293 files, 5982984 bytes uncompressed, 5982984 bytes compressed: 0.0% | |||
Also, if your zmvolume has compression enabled the blobs will remain compressed within the zip also upon backup. The point being, they are uncompressed to be then put into a zip file when the backup is using --zipStore. | |||
====Some Specific Example Restore/Backup Issues==== | |||
=====NO_SUCH_ACCOUNT_BACKUP===== | |||
Example errors: | |||
<pre> | |||
Error code: backup.NO_SUCH_ACCOUNT_BACKUP | |||
Message: no such backup for account: Missing full backup earlier than restore-to time for account 3c397edc-c015-4d82-ae8a-1084692f8a93 | |||
Details:soap:Sender | |||
Error occurred: no such backup for account: 3c397edc-c015-4d82-ae8a-1084692f8a93 | |||
Message: no such backup for account: Missing full backup earlier than restore-to time for account 3c397edc-c015-4d82-ae8a-1084692f8a93 | |||
3c397edc-c015-4d82-ae8a-1084692f8a93 is an example of the zimbraId of a user [ user1@mail-172.example.com ] . | |||
</pre> | |||
Confirm the current email address is mapped to the appropriate zimbraId. You'll want to search what is in ldap and the mailstores database for the exiting email address and then also search the reference files in your backups. Often times, this happens when an account was deleted and then created again. Account meaning the email address. When this is does, the new account will have a new zimbraId. | |||
<pre> | |||
$ zmprov -l ga user1@mail-172.example.com zimbraId | |||
# name user1@mail-172.example.com | |||
zimbraId: 3c397edc-c015-4d82-ae8a-1084692f8a93 | |||
$ mysql -e 'SELECT * FROM zimbra.mailbox WHERE comment LIKE "user1@mail-172.example.com"\G' | |||
$ mysql -e 'SELECT * FROM zimbra.mailbox WHERE account_id="3c397edc-c015-4d82-ae8a-1084692f8a93"\G' | |||
$ grep -C1 3c397edc-c015-4d82-ae8a-1084692f8a93 /opt/zimbra/backup/accounts.xml | |||
[using the default path; /opt/zimbra/backup/accounts.xml] | |||
$ egrep "user1@mail-172.example.com|3c397edc-c015-4d82-ae8a-1084692f8a93" /opt/zimbra/backup/accounts.xml | |||
$ egrep "user1@mail-172.example.com|3c397edc-c015-4d82-ae8a-1084692f8a93" /opt/zimbra/backup/sessions/*/session.xml | |||
</pre> | |||
* | |||
=====exception during auth===== | |||
Example error: | |||
<pre> | |||
LDAP backup failed: system failure: exception during auth | |||
{RemoteManager: zimbramail.example.com->zimbra@zimbramail.example.com:22} | |||
com.zimbra.common.service.ServiceException: system failure: LDAP backup failed: | |||
system failure: exception during auth {RemoteManager: zimbramail.example.com-> | |||
zimbra@zimbramail.example.com:22} | |||
</pre> | |||
This indicates the ssh keys aren't configured correctly in the environment. You should be able to do the following to resolve the issue: | |||
su - zimbra | |||
zmsshkeygen | |||
zmupdateauthkeys | |||
'''Note''' - You should also confirm that you do indeed have the ssh port available. The default is port 22. If your unable to ssh into the server using port 22, then this might be the issue. If you configured ssh to listen on a different port on this server, you'll need to adjust the following variable - zmprov gcf zimbraRemoteManagementPort . You might also have the ssh port blocked - confirm your firewall, iptables, etc is not blocking the ssh port. | |||
=====LICENSE_ERROR_Message: AccountsLimits exceeded===== | |||
Example error: | |||
LICENSE_ERROR_Message: AccountsLimits exceeded | |||
You should check your license and confirm it's valid and active: | |||
su - zimbra | |||
zmlicense -p | |||
And confirm how many licenses are used: First, flush the cache on all servers for the license information. Then do the query via soap. | |||
zmprov fc -a license | |||
zmsoap -z GetLicenseRequest | |||
Four lines that are generally of interest are shown below. The first two state how many one is licensed for, the bottom two show how many are currently used : | |||
<pre> | |||
<attr name="AccountsLimit">1000</attr> | |||
<attr name="ArchivingAccountsLimit">1000</attr> | |||
<attr name="TotalAccounts">3</attr> | |||
<attr name="ArchivingAccounts">3</attr> | |||
</pre> | |||
To | ====Performance Issues And Time To Complete==== | ||
RFE's for increase backup performance: | |||
* "Improve zmrestore & zmrestoreoffline performance" | |||
** http://bugzilla.zimbra.com/show_bug.cgi?id=33606 | |||
* "Improvement to backup performance" | |||
** http://bugzilla.zimbra.com/show_bug.cgi?id=36220 | |||
** Above marked as a dupe of "Reduce duration of maintenance mode during backup" | |||
*** http://bugzilla.zimbra.com/show_bug.cgi?id=33583 | |||
=====What Might Be Wrong?===== | |||
'''Need to expand this section or refer to the part above about all the log/data gathering that support would normally ask for''' | |||
First thing to check is the log file and see if anything notable stands out. | |||
grep -i backup /opt/zimbra/log/mailbox.log | |||
=====Use Auto-Group Backups Rather Than Default Style Backup===== | |||
Having trouble completing that entire full backup during off-hours? Enter the hybrid auto-grouped mode, which combines the concept of full and incremental backup functions - you’re completely backing up a target number of accounts daily rather than running incrementals. | |||
Auto-grouped mode automatically pulls in the redologs since the last run so you get incremental backups of the remaining accounts; although the incremental accounts captured via the redologs are not listed specifically in the backup account list. This still allows you to do a point in time restore for any account. | |||
Please see the following for more detailed information: | |||
* [[Ajcody-Backup-Restore-Issues#Auto-Group_Backups_Rather_Than_Default_Method_Topics]] | |||
=====Need To Write Fewer Files - Add The Zip Option To Your Backup Commands===== | |||
Using the zip option will compress all those thousands of single files that exist under a user's backup, decreasing performance issues that arise from writing out thousands of small files as compared to large ones. This is often seen when one is : | |||
* Using nfs for the backup directory | |||
* Copying/rsyncing backups to a remote server | |||
* Are using some third party backup software (to tape) to archive/backup the zimbra backup sessions. | |||
Please see the following for more information about using the Zip option: | |||
* [[Ajcody-Backup-Restore-Issues#The_Zip_Option_For_Backups]] | |||
=====SAN Snapshots For Backups===== | |||
Please see: | |||
* [[Ajcody-Backup-Restore-Issues#SAN_Snapshot]] | |||
=====Cloud Backups===== | |||
Please see: | |||
* [[Ajcody-Backup-Restore-Issues#Cloud_Backups]] | |||
=====Tape Backups===== | |||
Use a 'secondary' network card to run your backups over if possible. This way the backup activity doesn't effect ZCS network performance over the default network card. Also, please review the following section on Zimbra and hard links and confirm if your third party backup software or process will handle restoring hard links correctly. | |||
Please see: | |||
* [[Ajcody-Backup-Restore-Issues#Backing_Up_Backups_-_3rd_Party_Tools_And_Software_-_Dealing_With_Directories_With_Hard_Links]] | |||
===Hands On How-To And Reviewing Of Log Events=== | |||
And | ====Full Backup And Log Events Generated==== | ||
Example of manually starting a full backup using the Standard zimbraBackupMode and doing a query to confirm it's completion and status. | |||
<pre> | <pre> | ||
[zimbra@mail-172 archive]$ zmprov gs `zmhostname` zimbraBackupMode | |||
# name mail-172.example.com | |||
zimbraBackupMode: Standard | |||
[zimbra@mail-172 archive]$ pwd ; ls *.log | |||
/opt/zimbra/redolog/archive | |||
redo-20150303.060015.816-seq9.log | |||
[zimbra@mail-172 archive]$ zmbackup -f -a all | |||
full-20150304.012000.193 | |||
[zimbra@mail-172 archive]$ zmbackupquery -lb full-20150304.012000.193 -v | |||
Label: full-20150304.012000.193 | |||
Type: full | |||
Status: completed | |||
Started: Tue, 2015/03/03 20:20:00.193 EST | |||
Ended: Tue, 2015/03/03 20:20:28.630 EST | |||
Redo log sequence range: 10 .. 10 | |||
Number of accounts: 10 out of 10 completed | |||
Accounts: | |||
admin@mail-172.example.com: completed | |||
galsync.qva4ekog0@mail-172.example.com: completed | |||
ham.jgfdb3oqjf@mail-172.example.com: completed | |||
spam.umhx3owv3s@mail-172.example.com: completed | |||
user1@mail-172.example.com: completed | |||
user2@mail-172.example.com: completed | |||
user3@mail-172.example.com: completed | |||
user4@mail-172.example.com: completed | |||
user5@mail-172.example.com: completed | |||
virus-quarantine.gusiso1cd@mail-172.example.com: completed | |||
Total space: 14020MB | |||
Free space: 6017MB | |||
</pre> | |||
Example of what log events are generated for a full backup in /opt/zimbra/log/mailbox.log . First, you'll see the authentication/acl request and then statements about the backup request. | |||
===== | <pre> | ||
2015-03-03 20:19:58,499 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [] AuthProvider - Adding auth provider: zimbra com.zimbra.cs.service.ZimbraAuthProvider | |||
2015-03-03 20:19:59,458 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] soap - AuthRequest elapsed=834 | |||
2015-03-03 20:19:59,751 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] account - Initialized access manager: com.zimbra.cs.account.accesscontrol.ACLAccessManager | |||
2015-03-03 20:19:59,770 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup request started | |||
2015-03-03 20:20:00,193 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Found 10 accounts on server mail-172.example.com | |||
2015-03-03 20:20:00,464 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup request finished | |||
2015-03-03 20:20:00,464 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] soap - BackupRequest elapsed=760 | |||
</pre> | |||
Your log events will now reference Backup in them. AsyncFileCopier will start and it will give the backup session label for this backup job. | |||
<pre> | |||
2015-03-03 20:20:01,092 INFO [FullBackupThread] [] io - AsyncFileCopier is starting | |||
2015-03-03 20:20:01,187 INFO [FullBackupThread] [] backup - Full backup started for backup set; label: full-20150304.012000.193 | |||
</pre> | |||
It proceeds to backing up your accounts. I'll list the backup log events for just two accounts on this system - user1@mail-172.example.com and admin@mail-172.example.com . I'll provide the zimbraId, email address, and mailboxID information for user1@mail-172.example.com below so you can confirm it's reported in the logs correctly as well. | |||
<pre> | |||
[zimbra@mail-172 redolog]$ zmprov ga user1@mail-172.example.com zimbraId | |||
# name user1@mail-172.example.com | |||
zimbraId: 3c397edc-c015-4d82-ae8a-1084692f8a93 | |||
[zimbra@mail-172 redolog]$ zmprov gmi user1@mail-172.example.com | |||
mailboxId: 6 | |||
quotaUsed: 1209 | |||
</pre> | |||
Now, continuing with the log events. Notice the reference to the redo log file sequence, putting the account into and out of maintenance, number of blobs and index files backed up, and the statement it's finished backing up the account. Also within the section we can see the user's zimbraId [3c397edc-c015-4d82-ae8a-1084692f8a93] , email address [user1@mail-172.example.com] , and mailboxId [Mailbox 6]. | |||
<pre> | |||
[cut] | |||
2015-03-03 20:20:06,560 INFO [FullBackupThread] [name=user1@mail-172.example.com;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/6/index/0 | |||
2015-03-03 20:20:06,560 INFO [FullBackupThread] [name=user1@mail-172.example.com;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 LOADED | |||
2015-03-03 20:20:06,560 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - redo log file sequence is 10 at full backup for user1@mail-172.example.com | |||
2015-03-03 20:20:06,565 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - Full backup started for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) mailbox 6 | |||
===== | 2015-03-03 20:20:06,565 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] mailbox - Putting mailbox 6 under maintenance. | ||
2015-03-03 20:20:06,818 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - Number of blobs to backup for mailbox 6: 1 | |||
2015-03-03 20:20:06,818 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] mailbox - Ending maintenance on mailbox 6. | |||
2015-03-03 20:20:06,818 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 AVAILABLE | |||
2015-03-03 20:20:06,966 INFO [BackupSetWorkerThread-BLOB] [name=user1@mail-172.example.com;mid=6;] backup - Stored 1 blob files from volume message1 | |||
2015-03-03 20:20:06,966 INFO [BackupSetWorkerThread-BLOB] [name=user1@mail-172.example.com;mid=6;] backup - Stored 1 blob files | |||
2015-03-03 20:20:07,014 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - Account user1@mail-172.example.com in backup set full-20150304.012000.193: All pending file IO completed (1 out of 1) | |||
2015-03-03 20:20:07,016 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] mailbox - Putting mailbox 6 under maintenance. | |||
2015-03-03 20:20:07,021 INFO [BackupSetWorkerThread-INDEX] [name=user1@mail-172.example.com;mid=6;] backup - Stored 2 index files | |||
2015-03-03 20:20:07,496 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - Account user1@mail-172.example.com in backup set full-20150304.012000.193: All pending file IO completed (3 out of 3) | |||
2015-03-03 20:20:07,497 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] mailbox - Ending maintenance on mailbox 6. | |||
2015-03-03 20:20:07,497 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 AVAILABLE | |||
2015-03-03 20:20:07,497 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - Account user1@mail-172.example.com in backup set full-20150304.012000.193: All pending file IO completed (3 out of 3) | |||
2015-03-03 20:20:07,497 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - Full backup finished for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) mailbox 6 | |||
</pre> | |||
Events for the admin user's backup: | |||
<pre> | |||
2015-03-03 20:20:09,894 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - redo log file sequence is 10 at full backup for admin@mail-172.example.com | |||
===== | 2015-03-03 20:20:09,914 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - Full backup started for account admin@mail-172.example.com (7ac8de92-13fe-4d19-b9da-6872169fb5c9) mailbox 1 | ||
2015-03-03 20:20:09,915 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] mailbox - Putting mailbox 1 under maintenance. | |||
2015-03-03 20:20:10,176 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - Number of blobs to backup for mailbox 1: 73 | |||
2015-03-03 20:20:10,177 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] mailbox - Ending maintenance on mailbox 1. | |||
2015-03-03 20:20:10,177 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] mbxmgr - Mailbox 1 account 7ac8de92-13fe-4d19-b9da-6872169fb5c9 AVAILABLE | |||
2015-03-03 20:20:10,343 INFO [BackupSetWorkerThread-BLOB] [name=admin@mail-172.example.com;mid=1;] backup - Stored 73 blob files from volume message1 | |||
2015-03-03 20:20:10,353 INFO [BackupSetWorkerThread-BLOB] [name=admin@mail-172.example.com;mid=1;] backup - Stored 73 blob files | |||
2015-03-03 20:20:11,047 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - Account admin@mail-172.example.com in backup set full-20150304.012000.193: All pending file IO completed (73 out of 73) | |||
2015-03-03 20:20:11,051 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] mailbox - Putting mailbox 1 under maintenance. | |||
2015-03-03 20:20:11,108 INFO [BackupSetWorkerThread-INDEX] [name=admin@mail-172.example.com;mid=1;] backup - Stored 26 index files | |||
2015-03-03 20:20:12,450 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - Account admin@mail-172.example.com in backup set full-20150304.012000.193: All pending file IO completed (99 out of 99) | |||
2015-03-03 20:20:12,463 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] mailbox - Ending maintenance on mailbox 1. | |||
2015-03-03 20:20:12,463 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] mbxmgr - Mailbox 1 account 7ac8de92-13fe-4d19-b9da-6872169fb5c9 AVAILABLE | |||
2015-03-03 20:20:12,463 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - Account admin@mail-172.example.com in backup set full-20150304.012000.193: All pending file IO completed (99 out of 99) | |||
2015-03-03 20:20:12,463 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - Full backup finished for account admin@mail-172.example.com (7ac8de92-13fe-4d19-b9da-6872169fb5c9) mailbox 1 | |||
</pre> | |||
And then a final line confirming completion and the number of accounts successfully backed up. | |||
<pre> | |||
2015-03-03 20:20:12,463 INFO [FullBackupThread] [] backup - Finished backing up 10 of 10 accounts | |||
</pre> | |||
One thing to point out is the log events were mentioning the redo log sequence as being number 10 but in the redolog/archive directory there is only a sequence 9 redo log. This is because the "seq 10" is actually the /opt/zimbra/redolog/redo.log file. When the redo.log file rolls over and is moved to the archive directory it will have seq10 in it's filename. | |||
* | <pre> | ||
[zimbra@mail-172 archive]$ pwd ; ls -la *.log | |||
/opt/zimbra/redolog/archive | |||
-rw-r----- 1 zimbra zimbra 146723 Mar 3 20:23 redo-20150303.060015.816-seq9.log | |||
</pre> | |||
The contents of the backup session directory. I removed the sub-directories under accounts that weren't either the admin or user1 account. Notice the zimbraId is used for the sub-directory naming scheme. | |||
<pre> | |||
[zimbra@mail-172 full-20150304.012000.193]$ ls -R | |||
.: | |||
accounts ldap session.xml shared_blobs sys | |||
./accounts: | |||
3c3 423 434 75d 7ac 89f a46 af0 c53 eab | |||
./accounts/3c3: | |||
97e | |||
./accounts/3c3/97e: | |||
3c397edc-c015-4d82-ae8a-1084692f8a93 | |||
./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93: | |||
blobs db index ldap.xml meta.xml | |||
./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93/blobs: | |||
blobs-1.zip | |||
./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93/db: | |||
db_schema.xml mail_item.dat mailbox.dat mailbox_metadata.dat open_conversation.dat tag.dat tagged_item.dat | |||
./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93/index: | |||
/ | 0 | ||
./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93/index/0: | |||
segments.gen segments_1 | |||
[cut] | |||
./accounts/7ac: | |||
8de | |||
./accounts/7ac/8de: | |||
7ac8de92-13fe-4d19-b9da-6872169fb5c9 | |||
./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9: | |||
blobs db index ldap.xml meta.xml | |||
./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9/blobs: | |||
blobs-1.zip blobs-2.zip blobs-3.zip blobs-4.zip | |||
./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9/db: | |||
db_schema.xml mail_item.dat mailbox.dat open_conversation.dat tag.dat tagged_item.dat | |||
./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9/index: | |||
0 | |||
./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9/index/0: | |||
_0.fdt _0.fnm _0.nrm _0.tii _1.fdt _1.fnm _1.nrm _1.tii _2.fdt _2.fnm _2.nrm _2.tii segments.gen | |||
_0.fdx _0.frq _0.prx _0.tis _1.fdx _1.frq _1.prx _1.tis _2.fdx _2.frq _2.prx _2.tis segments_3 | |||
[cut] | |||
./ldap: | |||
ldap-config.bak.gz ldap.bak.gz | |||
./shared_blobs: | |||
./sys: | |||
config.dat current_volumes.dat db_schema.xml localconfig.xml ssh ssl volume.dat | |||
./sys/ssh: | |||
authorized_keys zimbra_identity zimbra_identity.pub | |||
./sys/ssl: | |||
zimbra | |||
./sys/ssl/zimbra: | |||
ca commercial jetty.pkcs12 server | |||
./sys/ssl/zimbra/ca: | |||
ca.key ca.pem ca.srl ca.srl.old index.txt index.txt.attr index.txt.old newcerts zmssl.cnf | |||
./sys/ssl/zimbra/ca/newcerts: | |||
1424523586.pem 1424523591.pem 1424523597.pem 1424523603.pem | |||
./sys/ssl/zimbra/commercial: | |||
./sys/ssl/zimbra/server: | |||
server.crt server.csr server.key | |||
</pre> | |||
====Incremental Backup And Log Events Generated==== | |||
Example of manually starting an incremental backup using the Standard zimbraBackupMode and doing a query to confirm it's completion and status. | |||
<pre> | <pre> | ||
[ | [zimbra@mail-172 archive]$ zmprov gs `zmhostname` zimbraBackupMode | ||
# name mail-172.example.com | |||
zimbraBackupMode: Standard | |||
[zimbra@mail-172 archive]$ ls | |||
redo-20150303.060015.816-seq9.log redo-20150304.070025.476-seq10.log | |||
[zimbra@mail-172 archive]$ zmbackup -i | |||
incr-20150304.073543.151 | |||
[zimbra@mail-172 archive]$ zmbackupquery -lb incr-20150304.073543.151 -v | |||
Label: incr-20150304.073543.151 | |||
Type: incremental | |||
2015 | Status: completed | ||
Started: Wed, 2015/03/04 02:35:43.151 EST | |||
Ended: Wed, 2015/03/04 02:35:51.981 EST | |||
Redo log sequence range: 10 .. 11 | |||
Number of accounts: 10 out of 10 completed | |||
Accounts: | |||
admin@mail-172.example.com: completed | |||
galsync.qva4ekog0@mail-172.example.com: completed | |||
ham.jgfdb3oqjf@mail-172.example.com: completed | |||
spam.umhx3owv3s@mail-172.example.com: completed | |||
user1@mail-172.example.com: completed | |||
user2@mail-172.example.com: completed | |||
user3@mail-172.example.com: completed | |||
user4@mail-172.example.com: completed | |||
user5@mail-172.example.com: completed | |||
virus-quarantine.gusiso1cd@mail-172.example.com: completed | |||
Total space: 14020MB | |||
Free space: 6032MB | |||
</pre> | |||
Notice how the log events are soap requests and differ from the full backup above and lack the INFO [*Backup*] formatting. Important to note this if your attempting to grep for backup/restore activity. Just like with the full, it starts with an authentication/acl request and states it's starting, how many accounts found, AsyncFileCopier starting, and the label name for this backup session. | |||
[ | |||
/ | |||
From /opt/zimbra/log/mailbox.log | |||
<pre> | <pre> | ||
[zimbra | 2015-03-04 02:35:42,356 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] soap - AuthRequest elapsed=94 | ||
.: | 2015-03-04 02:35:42,481 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup request started | ||
2015-03-04 02:35:42,917 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Found 10 accounts on server mail-172.example.com | |||
2015-03-04 02:35:43,254 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] io - AsyncFileCopier is starting | |||
2015-03-04 02:35:43,255 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup started for backup set; label: incr-20150304.073543.151 | |||
</pre> | |||
. | The log events are again trimmed below just for the user1 and admin account for the user backup section of the log file. Notice the absence of any statements of the accounts being put into maintenance mode. | ||
./ | <pre> | ||
2015-03-04 02:35:44,118 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/6/index/0 | |||
2015-03-04 02:35:44,119 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 LOADED | |||
2015-03-04 02:35:44,120 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup is started for account user1@mail-172.example.com | |||
2015-03-04 02:35:44,120 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Saving account information from LDAP | |||
2015-03-04 02:35:44,127 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup has ended | |||
2015-03-04 02:35:44,127 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Account user1@mail-172.example.com in backup set incr-20150304.073543.151: All pending file IO completed (0 out of 0) | |||
[cut] | |||
2015-03-04 02:35:44,212 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup is started for account admin@mail-172.example.com | |||
2015-03-04 02:35:44,212 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Saving account information from LDAP | |||
2015-03-04 02:35:44,215 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup has ended | |||
2015-03-04 02:35:44,217 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Account admin@mail-172.example.com in backup set incr-20150304.073543.151: All pending file IO completed (0 out of 0) | |||
</pre> | |||
. | The incremental will then end with log events related to copying the redologs into the backup directory and then purging the redologs that can be deleted. This example states redo log sequence 9 can be deleted. We'll confirm it is later. | ||
./ | <pre> | ||
2015-03-04 02:35:44,618 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] redolog - waiting for FileLogWriter.FsyncThread-1425452599560 to finish. | |||
2015-03-04 02:35:44,628 INFO [FileLogWriter.FsyncThread-1425452599560] [] redolog - fsync thread exiting | |||
2015-03-04 02:35:44,628 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] redolog - FileLogWriter.FsyncThread-1425452599560 finished | |||
2015-03-04 02:35:44,757 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] redolog - Redo log rollover took 540ms | |||
2015-03-04 02:35:44,758 INFO [FileLogWriter.FsyncThread-1425454544757] [] redolog - Starting fsync thread with interval 10 | |||
2015-03-04 02:35:44,776 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Found 3 redo logs to backup | |||
2015-03-04 02:35:44,861 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Last backed-up redo log sequence = 9 | |||
2015-03-04 02:35:44,943 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Marking redo log /opt/zimbra/redolog/archive/redo-20150303.060015.816-seq9.log for deletion | |||
2015-03-04 02:35:44,943 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Copying redo log /opt/zimbra/redolog/archive/redo-20150304.070025.476-seq10.log to /opt/zimbra/backup/tmp/incr-20150304.073543.151/redologs/redo-20150304.070025.476-seq10.log | |||
2015-03-04 02:35:45,053 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Retaining recent redo log /opt/zimbra/redolog/archive/redo-20150304.070025.476-seq10.log | |||
2015-03-04 02:35:45,053 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Copying redo log /opt/zimbra/redolog/archive/redo-20150304.073544.755-seq11.log to /opt/zimbra/backup/tmp/incr-20150304.073543.151/redologs/redo-20150304.073544.755-seq11.log | |||
2015-03-04 02:35:45,067 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Retaining recent redo log /opt/zimbra/redolog/archive/redo-20150304.073544.755-seq11.log | |||
</pre> | |||
. | The last limes confirm the backup finished successfully. | ||
./ | <pre> | ||
2015-03-04 02:35:51,935 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup set incr-20150304.073543.151: All pending file IO completed (0 out of 0) | |||
2015-03-04 02:35:51,981 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] io - AsyncFileCopier is shut down | |||
./accounts/3c3/97e | 2015-03-04 02:35:51,981 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup finished for backup set; label: incr-20150304.073543.151 | ||
2015-03-04 02:35:51,982 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Deleting redo log /opt/zimbra/redolog/archive/redo-20150303.060015.816-seq9.log | |||
2015-03-04 02:35:52,909 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup request finished | |||
2015-03-04 02:35:52,909 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] soap - BackupRequest elapsed=10451 | |||
</pre> | |||
For an incremental, the account sub-directories will have ldap data saved also for each user but blobs and indexes aren't. That data resides in the redologs from the system under the redologs directory. | |||
<pre> | |||
[zimbra@mail-172 incr-20150304.073543.151]$ ls -R | |||
.: | |||
accounts ldap redologs session.xml sys | |||
./accounts: | |||
3c3 423 434 75d 7ac 89f a46 af0 c53 eab | |||
./accounts/3c3: | |||
97e | |||
./accounts/3c3/97e: | |||
3c397edc-c015-4d82-ae8a-1084692f8a93 | |||
./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93 | ./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93: | ||
ldap.xml | |||
[cut] | [cut] | ||
Line 1,000: | Line 1,082: | ||
./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9: | ./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9: | ||
ldap.xml | |||
[cut] | [cut] | ||
Line 1,020: | Line 1,089: | ||
ldap-config.bak.gz ldap.bak.gz | ldap-config.bak.gz ldap.bak.gz | ||
./ | ./redologs: | ||
redo-20150304.070025.476-seq10.log redo-20150304.073544.755-seq11.log | |||
./sys: | ./sys: | ||
Line 1,046: | Line 1,116: | ||
</pre> | </pre> | ||
Notice that redo log sequence 9 was indeed deleted from the redolog/archive directory. | |||
<pre> | |||
[zimbra@mail-172 archive]$ pwd ; ls | |||
/opt/zimbra/redolog/archive | |||
redo-20150304.070025.476-seq10.log redo-20150304.073544.755-seq11.log redo-20150304.074024.058-seq12.log | |||
</pre> | |||
If you had created new users after your last full backup, you would see that there was also a full backup started as well. For example : | |||
<pre> | <pre> | ||
[zimbra@mail-172 | [zimbra@mail-172 log]$ zmbackup -i | ||
incr-20150304.085810.220 | |||
full-20150304.085818.889 | |||
[zimbra@mail-172 | [zimbra@mail-172 log]$ zmbackupquery -a user6@mail-172.example.com | ||
Account: user6@mail-172.example.com | |||
Label: full-20150304.085818.889 | |||
Type: full | |||
Started: Wed, 2015/03/04 03:58:18.889 EST | |||
Ended: Wed, 2015/03/04 03:58:25.997 EST | |||
Acct ID: 4e898a68-1886-4a80-b3dc-ac2ab650bf8c | |||
[zimbra@mail-172 | [zimbra@mail-172 log]$ zmbackupquery -lb full-20150304.085818.889 -v | ||
Label: | Label: full-20150304.085818.889 | ||
Type: | Type: full | ||
Status: completed | Status: completed | ||
Started: Wed, 2015/03/04 | Started: Wed, 2015/03/04 03:58:18.889 EST | ||
Ended: Wed, 2015/03/04 | Ended: Wed, 2015/03/04 03:58:25.997 EST | ||
Redo log sequence range: | Redo log sequence range: 14 .. 14 | ||
Number of accounts: | Number of accounts: 1 out of 1 completed | ||
Accounts: | Accounts: | ||
user6@mail-172.example.com: completed | |||
Total space: 14020MB | Total space: 14020MB | ||
Free space: | Free space: 6038MB | ||
</pre> | </pre> | ||
====Contents Of accounts.xml And sessions.xml From The Full And Incremental Backup Examples==== | |||
First, see what zmbackupquery shows for a particular user in regards to the backup labels they are included in. | |||
<pre> | <pre> | ||
$ zmbackupquery -a user1@mail-172.example.com | |||
2015 | Account: user1@mail-172.example.com | ||
Label: incr-20150304.073543.151 | |||
Type: incremental | |||
Started: Wed, 2015/03/04 02:35:43.151 EST | |||
Ended: Wed, 2015/03/04 02:35:51.981 EST | |||
Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93 | |||
Label: full-20150304.012000.193 | |||
Type: full | |||
Started: Tue, 2015/03/03 20:20:00.193 EST | |||
Ended: Tue, 2015/03/03 20:20:28.630 EST | |||
Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93 | |||
</pre> | </pre> | ||
The | Now let's review the accounts/session xml files that list all the users within those particular backup sessions. The /opt/zimbra/backup/accounts.xml is the referenced to determine what is the last know full backup session the user was in. If the /opt/zimbra/backup/accounts.xml is corrupted, unreadable, or incorrect - this can cause issues with restoring users. | ||
Also notice that it also references the zimbraId for the user at the time of the backup. If you were to delete a user and then create a new account using the same email address [user1@mail-172.example.com] that new account will have a different zimbraId and therefor the backup/restore commands will not work as you might expected because of the differing zimbraId. The zmrestore command only can take an email address for the -a option, it can not use the zimbraID. | |||
<pre> | <pre> | ||
[zimbra@mail-172 backup]$ pwd | |||
/opt/zimbra/backup | |||
[zimbra@mail-172 backup]$ cat accounts.xml | |||
<?xml version="1.0" encoding="utf-8"?> | |||
[ | |||
<backupMetadata xmlns="urn:zimbraBackupMeta" version="7.2" dbVersion="103"> | |||
<allAccounts> | |||
< | <account zimbraId="af06b0b4-3160-4ea3-a55b-81163a9914a0" email="user5@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193"> | ||
</account> | |||
<account zimbraId="75d3d364-a384-4077-8ac5-87b6960b57cd" email="user3@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193"> | |||
</account> | |||
<account zimbraId="4343a299-5259-4826-b1c1-becbfbb9b8f0" email="galsync.qva4ekog0@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193"> | |||
</account> | |||
<account zimbraId="89fc5986-31a3-406e-8084-0ce0b98a251f" email="user2@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193"> | |||
</account> | |||
<account zimbraId="42380a6e-5718-4d68-ab37-e8d5bc9e6977" email="spam.umhx3owv3s@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193"> | |||
</account> | |||
<account zimbraId="eab3d56e-ffd9-4412-95de-48f102d2d252" email="user4@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193"> | |||
</account> | |||
<account zimbraId="7ac8de92-13fe-4d19-b9da-6872169fb5c9" email="admin@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193"> | |||
</account> | |||
<account zimbraId="4e898a68-1886-4a80-b3dc-ac2ab650bf8c" email="user6@mail-172.example.com" latestFullBackupLabel="full-20150304.085818.889"> | |||
</account> | |||
<account zimbraId="a46dfc59-a3e9-4f09-adfb-af8efc42aa9f" email="virus-quarantine.gusiso1cd@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193"> | |||
</account> | |||
<account zimbraId="3c397edc-c015-4d82-ae8a-1084692f8a93" email="user1@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193"> | |||
</account> | |||
<account zimbraId="c5315c5d-97cb-4d84-bc70-374a67f13309" email="ham.jgfdb3oqjf@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193"> | |||
</account> | |||
</allAccounts> | |||
</backupMetadata> | |||
</pre> | </pre> | ||
In each backup session, there will be a sessions.xml file that acts as a reference or summary file of the backup. At the top, you'll see what version of ZCS was running and what options the backup environment had set. It will then report the final status for each user for the backup attempt against the account. At the end, there's a section in regards to errors and if any were encountered. | |||
<pre> | <pre> | ||
[zimbra@mail-172 backup]$ cd sessions/ | |||
[zimbra@mail-172 sessions]$ cd full-20150304.012000.193 | |||
[zimbra@mail-172 full-20150304.012000.193]$ cat session.xml | |||
<?xml version="1.0" encoding="utf-8"?> | |||
<backupMetadata xmlns="urn:zimbraBackupMeta" version="7.2" dbVersion="103"> | |||
<backupSet label="full-20150304.012000.193" zcsRelease="8.6.0_GA_1153 20141215151258 20141215-1518 NETWORK" startTime="1425432000193" endTime="1425432028630" minRedoSeq="10" maxRedoSeq="10" sharedBlobsZipped="true" sharedBlobsZipNameDigestChars="1" sharedBlobsDirectoryDepth="5" sharedBlobsCharsPerDirectory="2" type="full" accountsDirectoryDepth="2"> | |||
< | <desc>Full backup</desc> | ||
<accounts> | |||
. | <account zimbraId="af06b0b4-3160-4ea3-a55b-81163a9914a0" email="user5@mail-172.example.com" status="COMPLETED"> | ||
</account> | |||
<account zimbraId="75d3d364-a384-4077-8ac5-87b6960b57cd" email="user3@mail-172.example.com" status="COMPLETED"> | |||
. | </account> | ||
<account zimbraId="89fc5986-31a3-406e-8084-0ce0b98a251f" email="user2@mail-172.example.com" status="COMPLETED"> | |||
</account> | |||
. | <account zimbraId="4343a299-5259-4826-b1c1-becbfbb9b8f0" email="galsync.qva4ekog0@mail-172.example.com" status="COMPLETED"> | ||
</account> | |||
<account zimbraId="eab3d56e-ffd9-4412-95de-48f102d2d252" email="user4@mail-172.example.com" status="COMPLETED"> | |||
</account> | |||
<account zimbraId="42380a6e-5718-4d68-ab37-e8d5bc9e6977" email="spam.umhx3owv3s@mail-172.example.com" status="COMPLETED"> | |||
</account> | |||
<account zimbraId="7ac8de92-13fe-4d19-b9da-6872169fb5c9" email="admin@mail-172.example.com" status="COMPLETED"> | |||
</account> | |||
<account zimbraId="a46dfc59-a3e9-4f09-adfb-af8efc42aa9f" email="virus-quarantine.gusiso1cd@mail-172.example.com" status="COMPLETED"> | |||
</account> | |||
<account zimbraId="3c397edc-c015-4d82-ae8a-1084692f8a93" email="user1@mail-172.example.com" status="COMPLETED"> | |||
. | </account> | ||
<account zimbraId="c5315c5d-97cb-4d84-bc70-374a67f13309" email="ham.jgfdb3oqjf@mail-172.example.com" status="COMPLETED"> | |||
</account> | |||
</accounts> | |||
<stats> | |||
<counter name="total_time" numSamples="1" sum="28437" unit="ms"> | |||
</counter> | |||
<counter name="sysdb_time" numSamples="1" sum="1111" unit="ms"> | |||
</counter> | |||
<counter name="sysdb_bytes" numSamples="1" sum="35210" unit="bytes"> | |||
</counter> | |||
<counter name="redologs_time" numSamples="0" sum="0" unit="ms"> | |||
</counter> | |||
<counter name="redologs_bytes" numSamples="0" sum="0" unit="bytes"> | |||
</counter> | |||
<counter name="accounts_time" numSamples="10" sum="4827" unit="ms"> | |||
</counter> | |||
<counter name="ldap_time" numSamples="10" sum="172" unit="ms"> | |||
</counter> | |||
<counter name="ldap_bytes" numSamples="10" sum="25665" unit="bytes"> | |||
</counter> | |||
<counter name="db_time" numSamples="10" sum="4787" unit="ms"> | |||
</counter> | |||
<counter name="db_bytes" numSamples="10" sum="78690" unit="bytes"> | |||
</counter> | |||
<counter name="db_digest_map_load_time" numSamples="10" sum="2400" unit="ms"> | |||
</counter> | |||
<counter name="msgs_time" numSamples="10" sum="451" unit="ms"> | |||
</counter> | |||
<counter name="msg_copy_time" numSamples="74" sum="25" unit="ms"> | |||
</counter> | |||
<counter name="msg_copy_bytes" numSamples="74" sum="149611" unit="bytes"> | |||
</counter> | |||
<counter name="msg_link_time" numSamples="0" sum="0" unit="ms"> | |||
</counter> | |||
<counter name="index_time" numSamples="48" sum="80" unit="ms"> | |||
</counter> | |||
<counter name="index_bytes" numSamples="38" sum="73510" unit="bytes"> | |||
</counter> | |||
</stats> | |||
<errors> | |||
</errors> | |||
</backupSet> | |||
</backupMetadata> | |||
</pre> | </pre> | ||
An incremental session.xml file is similar to a full backups session.xml . | |||
<pre> | <pre> | ||
[zimbra@mail-172 | [zimbra@mail-172 sessions]$ cd incr-20150304.073543.151 | ||
[zimbra@mail-172 incr-20150304.073543.151]$ cat session.xml | |||
<?xml version="1.0" encoding="utf-8"?> | |||
<backupMetadata xmlns="urn:zimbraBackupMeta" version="7.2" dbVersion="103"> | |||
<backupSet label="incr-20150304.073543.151" zcsRelease="8.6.0_GA_1153 20141215151258 20141215-1518 NETWORK" startTime="1425454543151" endTime="1425454551981" minRedoSeq="10" maxRedoSeq="11" sharedBlobsZipped="true" sharedBlobsZipNameDigestChars="1" sharedBlobsDirectoryDepth="5" sharedBlobsCharsPerDirectory="2" type="incremental" accountsDirectoryDepth="2"> | |||
< | <desc>Incremental backup</desc> | ||
<accounts> | |||
<account zimbraId="af06b0b4-3160-4ea3-a55b-81163a9914a0" email="user5@mail-172.example.com" status="COMPLETED"> | |||
</account> | |||
<account zimbraId="75d3d364-a384-4077-8ac5-87b6960b57cd" email="user3@mail-172.example.com" status="COMPLETED"> | |||
</account> | |||
<account zimbraId="89fc5986-31a3-406e-8084-0ce0b98a251f" email="user2@mail-172.example.com" status="COMPLETED"> | |||
</account> | |||
<account zimbraId="4343a299-5259-4826-b1c1-becbfbb9b8f0" email="galsync.qva4ekog0@mail-172.example.com" status="COMPLETED"> | |||
</account> | |||
<account zimbraId="eab3d56e-ffd9-4412-95de-48f102d2d252" email="user4@mail-172.example.com" status="COMPLETED"> | |||
</account> | |||
<account zimbraId="42380a6e-5718-4d68-ab37-e8d5bc9e6977" email="spam.umhx3owv3s@mail-172.example.com" status="COMPLETED"> | |||
</account> | |||
<account zimbraId="7ac8de92-13fe-4d19-b9da-6872169fb5c9" email="admin@mail-172.example.com" status="COMPLETED"> | |||
</account> | |||
<account zimbraId="a46dfc59-a3e9-4f09-adfb-af8efc42aa9f" email="virus-quarantine.gusiso1cd@mail-172.example.com" status="COMPLETED"> | |||
</account> | |||
<account zimbraId="3c397edc-c015-4d82-ae8a-1084692f8a93" email="user1@mail-172.example.com" status="COMPLETED"> | |||
</account> | |||
<account zimbraId="c5315c5d-97cb-4d84-bc70-374a67f13309" email="ham.jgfdb3oqjf@mail-172.example.com" status="COMPLETED"> | |||
</account> | |||
</accounts> | |||
<stats> | |||
<counter name="total_time" numSamples="1" sum="8830" unit="ms"> | |||
</counter> | |||
<counter name="sysdb_time" numSamples="1" sum="563" unit="ms"> | |||
</counter> | |||
<counter name="sysdb_bytes" numSamples="1" sum="35210" unit="bytes"> | |||
= | </counter> | ||
<counter name="redologs_time" numSamples="1" sum="310" unit="ms"> | |||
</counter> | |||
<counter name="redologs_bytes" numSamples="1" sum="126307" unit="bytes"> | |||
</counter> | |||
<counter name="accounts_time" numSamples="10" sum="196" unit="ms"> | |||
</counter> | |||
<counter name="ldap_time" numSamples="10" sum="129" unit="ms"> | |||
</counter> | |||
<counter name="ldap_bytes" numSamples="10" sum="25665" unit="bytes"> | |||
</counter> | |||
<counter name="db_time" numSamples="0" sum="0" unit="ms"> | |||
</counter> | |||
<counter name="db_bytes" numSamples="0" sum="0" unit="bytes"> | |||
</counter> | |||
<counter name="db_digest_map_load_time" numSamples="0" sum="0" unit="ms"> | |||
</counter> | |||
<counter name="msgs_time" numSamples="0" sum="0" unit="ms"> | |||
</counter> | |||
<counter name="msg_copy_time" numSamples="0" sum="0" unit="ms"> | |||
</counter> | |||
<counter name="msg_copy_bytes" numSamples="0" sum="0" unit="bytes"> | |||
</counter> | |||
<counter name="msg_link_time" numSamples="0" sum="0" unit="ms"> | |||
</counter> | |||
<counter name="index_time" numSamples="0" sum="0" unit="ms"> | |||
</counter> | |||
<counter name="index_bytes" numSamples="0" sum="0" unit="bytes"> | |||
</counter> | |||
</stats> | |||
<errors> | |||
</errors> | |||
</backupSet> | |||
<account zimbraId="af06b0b4-3160-4ea3-a55b-81163a9914a0" email="user5@mail-172.example.com" | |||
</ | |||
</ | |||
</backupMetadata> | </backupMetadata> | ||
</pre> | </pre> | ||
Line 2,681: | Line 1,377: | ||
{{Article_Footer|Zimbra Collaboration Suite 8.6|01/22/2015}} | {{Article_Footer|Zimbra Collaboration Suite 8.6|01/22/2015}} | ||
[[Category:Troubleshooting_Course_Content_Rough_Drafts]] | [[Category:Troubleshooting_Course_Content_Rough_Drafts]] | ||
[[Category: Author:Ajcody]] |
Latest revision as of 20:34, 7 June 2016
Need To Do
- review training materials for sys admin course
- determine what should be 'video' and how it can all be built upon preceding steps in a hands on training manner
Bug & RFE's I Made While Researching This Draft Write Up
- [story] Ability to search data within backup and do "item" restores or identify locations of search results
- admin console backup label view doesn't list accounts in the all accounts tab
- admin console restore - doesn't autocomplete / suggest account matches when filling out email address box
- document new restore functions / options with ZCS 8+ for admin console restore
- admin console restore - rename "Selected Servers" panel to "Restore Options"
- admin console restore - if only one mailstore in env. then state such in second panel of restore about "server for the restored accounts"
- admin console restore - expand restore To options - To full backup label, To incremental target
- admin console restore - "restore to the latest backup" incorrectly described / broken
- admin console restore - unable to restore individual accounts [sort of]
- admin console restore - reuse GAL/Contact Picker Window for "restore individual accounts"
- Default COS value [ zimbraPrefShowSelectionCheckbox ] for checkboxes in lists should be TRUE and improve 'select all' functionality
- zmschedulebackup ignores zmbackup options in auto-grouped mode
Introduction
- Will you please provide a brief paragraph introduction/overview about recovering missing data in Zimbra.
- For example: what are some scenarios wherein you need to recover missing data.
- What is the general method for recovery?
- How/Why is the backup environment relevant/important.
- Remember that this won't be obvious for all audience members.
General Overview Of Backup Environment
Backup Directory Partition & Filesystem
Confirm You Are Using Support Partition & Filesystem
It's recommended to have a dedicated partition for backups and to use only a supported partition/filesystem type:
- No CIFS
- No unsupported 'cloud' based computers or filesystems - the various Amazon offerings, Google Computing, etc.
- NFS has limited support
NFS Use For Backups
Please see this RFE:
- "Need clarity on supporting nfs mounted zimbra directories - report error/msg if nfs mount is present"
Statement that was included in the release notes following the RFE:
- ZCS & NFS:
- Zimbra will support customers that store backups (e.g. /opt/zimbra/backup) on an NFS-mounted partition. Please note that this does not relieve the customer from the responsibility of providing a storage system with a performance level appropriate to their desired backup and restore times. In our experience, network-based storage access is more likely to encounter latency or disconnects than is equivalent storage attached by fiber channel or direct SCSI.
- Zimbra continues to view NFS storage of other parts of the system as unsupported. Our testing has shown poor read and write performance for small files over NFS implementations, and as such we view it unlikely that this policy will change for the index and database stores. We will continue to evaluate support for NFS for the message store as customer demand warrants.
- When working with Zimbra Support on related issues, the customer must please disclose that the backup storage used is NFS.
Things To Check When Using NFS
- Check the /var/log/messages on both the zimbra server and the nfs server for nfs related errors during the time frame of your backup.
- Check /opt/zimbra/log/mailbox.log for error messages about folders/files not being able to be written or missing directory errors.
- Is root_squash configured on the nfs server? If it's changed to no_root_squash , does the behavior of the backup change?
- Is the */backup directory owned by zimbra:zimbra with at least 750 or 755 permissions?
- This parent directory as given in:
- zmprov gs `zmhostname` zimbraBackupTarget
- This parent directory as given in:
- Does zimbraBackupTarget have at least the subdirectories of : sessions and tmp : and are owned by zimbra:zimbra with 750 or 755 permissions?
- If not, try manually creating them and then running a test backup.
- IF USING A NAS - MAKE SURE YOUR NOT USING EXTENDED ACLS OR THAT YOU HAVE THEM CONFIGURED PROPERLY
- If your backup session directory shows a + sign in the description section like the example below, it's a sign your using extend acls.
drwxrwx---+ 2 zimbra zimbra 4096 Sep 14 00:00 TO_DELETE-full-XXXXXX
- The example above is from a customer reporting that their backups weren't being deleted and it was filling up their backup partitions. The old backup sessions were being renamed to TO_DELETE-full-XXXXXX directory naming scheme but weren't being deleted. The root cause of the problem was because extended acls were being used on their NAS server and it was preventing the delete operation from completing.
- If your backup session directory shows a + sign in the description section like the example below, it's a sign your using extend acls.
Troubleshooting Example When Using NFS
Steps to setup your ZCS server to use a NFS mount and to confirm you can save a backup to the NFS partition mount:
1. make a test partition on nfs server - /nfs-test 2. mount on zimbra server 2A. mkdir /nfs-test 2B. chmod 755 /nfs-test 2C. mount nfs-server:/nfs-test /nfs-test 2D. ls -la /nfs-test 2E. mkdir /nfs-test/backup 2F. chown zimbra:zimbra /nfs-test/backup 2G. chmod 755 /nfs-test/backup 2H. su - zimbra ; touch /nfs-test/backup/testfile 2I. ls -laR /nfs-test/ 2J. rm /nfs-test/backup/testfile 3. Set zimbraBackupTarget 3A. zmprov ms `zmhostname` zimbraBackupTarget /nfs-test/backup 4. Run a full backup against one account 4A. ex. zmbackup -f -a user@domain.com 5. ls -laR /nfs-test/ 6. If you again, run into the same problem. You could also repeat the backup after increasing the backup logging variable for the account your trying to backup. If you didn't run into the same problem, it might had to do with the initial setup of the nfs mount and permissions being used during the directory creation. 6A. zmprov aal user@domain.com zimbra.backup debug 6B. logging will show up in /opt/zimbra/log/mailbox.log 6C. Remove account logging when your done. zmprov ral user@domain.com zimbra.backup 8. Change zimbraBackupTarget back to your production path.
Setup A Fast Test NOT Using NFS
A way to "avoid" the NFS issues for testing purposes would be to setup a new zimbraBackupTarget to try doing a full backup of a couple of user accounts. I DON'T recommend this if you are using auto-group for zimbraBackupMode [ zmprov gs `zmhostname` zimbraBackupMode ] , only if you are using Standard mode.
[as root] ** adjust your new "backup" directory path as needed - mine is just an example** mkdir /mnt/usb1/backup-test chown zimbra:zimbra /mnt/usb1/backup-test chmod 750 /mnt/usb1/backup-test su - zimbra **confirm backup mode as standard** zmprov gs `zmhostname` zimbraBackupMode **if not Standard, please stop** zmprov ms `zmhostname` zimbraBackupTarget /mnt/usb1/backup-test **Find a couple of test accounts to do a full for. Make sure you'll have space to do the backup regards to need free space.** zmbackup -f -a user1@domain.com user2@domain.com user3@domain.com **Watch and confirm status of backup you just started.** zmbackupquery **Confirm files were backed up in right location** ls /mnt/usb1/backup-test/sessions/ **Failed backups would most likely results in left over directory in tmp directory** ls /mnt/usb1/backup-test/tmp/
Standard Backup Or Autogroup
Knowing what the zimbraBackupMode variable is set to on the server is important when trouble shooting and backup/restore issues. The options for zimbraBackupMode are Standard or Auto-Grouped.
- The Standard backup method is to run a weekly full backup session and daily incremental backup sessions to back up all mailboxes daily. The standard backup method is appropriate for enterprise deployments where full backups are run during non-working days.
- The Auto-Grouped backup method is recommended for large ZCS environments where running a full backup of all accounts at one time would take too long.The auto-grouped backup method runs a full backup session for a different group of mailboxes at each scheduled backup. The system administrator configures the interval that backups should run and configures the number of groups that backups are made up of. ZCS then automatically backs up mailboxes in groups over the interval specified.
zimbraBackupMode can be set at the Global and Server level, if set at the server level it overrides the Global setting if different. To see what it's currently set at you would do:
zmprov gcf zimbraBackupMode zmprov gs `zmhostname` zimbraBackupMode
Your backup scheduled and crontab should be set appropriately for the zimbraBackupMode method your using. And changes to zimbraBackupMode should also have you review the backup schedule and crontab setup.
Auto-Grouped Wants To Run A Full Backup Against All Users On The First Run
One issue that might happen when switching from Standard to Auto-Grouped for your backup method is for the first Auto-Grouped backup job it will attempt to backup all users. This can cause your disk partition to become full if your unprepared for it. To avoid this first full backup of all users when switching to the Auto-Grouped method you can consult the following bug for a work around.
- Ability to set last_backup_at to NULL or to the day before for All users, group of users, user
Backup Schedule
The backup scheduled can be set by the zmschedulebackup command or modifying the zimbra users crontab file.
The default backup schedule for a server that is using the Standard backup method is below. The example shows what's in the crontab and also the output of zmschedulebackup query command for the current schedule, the -q option :
[zimbra@mail-172 sessions]$ crontab -l | grep backup 0 1 * * 6 /opt/zimbra/bin/zmbackup -f -a all --mail-report 0 1 * * 0-5 /opt/zimbra/bin/zmbackup -i --mail-report 0 0 * * * /opt/zimbra/bin/zmbackup -del 1m --mail-report [zimbra@mail-172 ~]$ zmschedulebackup -q Current Schedule: f 0 1 * * 6 -a all --mail-report i 0 1 * * 0-5 --mail-report d 1m 0 0 * * * --mail-report
zmschedulebackup -D will set the following defaults backup scheduled based upon what zimbraBackupMode is set to on the server. The example below shows setting zimbraBackupMode to Auto-Grouped and using zmschedulebackup to set the default backup schedule in the zimbra crontab for Auto-Grouped method.
[zimbra@mail-172 sessions]$ zmschedulebackup -D Default schedule set Current Schedule: f 0 1 * * 0-6 d 1m 0 0 * * * [zimbra@mail-172 sessions]$ crontab -l | grep backup 0 1 * * 0-6 /opt/zimbra/bin/zmbackup -f 0 0 * * * /opt/zimbra/bin/zmbackup -del 1m
Notice that the Auto-Grouped method runs a "full" everyday and drops the line for incremental backups. It also drops the "-a all" option since it's an inappropriate option when using Auto-Grouped. There is currently a bug, see below, that also caused the --mail-report option to be dropped. You can include it against by manually modifying the crontab file.
- zmschedulebackup ignores zmbackup options in auto-grouped mode
Backup And Restore CLI Commands You Should Be Familiar With
- zmschedulebackup : This command is used to schedule full backups, incremental backups, and deletion of old backups.
- zmbackup : This command executes full or incremental backup of the mail server. This is run on a live server, while the mailboxd process and the mailbox server are running. This command also has an option to manually delete old backups when they are no longer needed.
- zmbackupabort : This command stops a full backup that is in process.
- zmbackupabort -r : This command stops an ongoing restore.
- zmbackupquery : This command lists the information about ongoing and completed backups, including labels and dates.
- zmrestore : This command executes a full or incremental restore to the Zimbra mail server. The zmrestore command is performed on a server that is running.
- zmrestoreoffline : This command restores the Zimbra mail server when the mailboxd process is stopped.
- zmrestoreldap : This command restores the complete LDAP directory server, including accounts, domains, servers, COS and other data.
- zmplayredo : This command allows you to play/replay a redolog file/s. It requires the mailbox service to be stopped though.
- zmredodump : This command will dump the contents of a redolog file/s so you can see the transactions that were done. This is useful if your trying to confirm an action was or wasn't done, what time it was done, and so forth. This does not require the mailbox service to be stopped.
Monitoring And Maintenance Of Backups
Monitoring backups and disk usage
- Confirm that your zimbra crontab file for your backup runs include the --mail-report option.
- Make sure your monitoring your partition space appropriately. You should have the stats service installed and enabled on your servers so it can monitor your partitions and trigger an alert if they exceed a percentage used.
[zimbra@mail-172 ~]$ zmcontrol status | grep stats stats Running [zimbra@mail-172 ~]$ zmlocalconfig | egrep -i 'zmstat_d|zmdisk' zmdisklog_critical_threshold = 95 zmdisklog_warn_threshold = 85 zmstat_df_excludes = zmstat_disk_interval = 600
- Monitor the following directories as well, they often times can be the reason behind your disk partition becoming full.
- Default path: /opt/zimbra/redolog/archive/ [zmprov gs `zmhostname` zimbraRedoLogArchiveDir] ; uses the following variable for the redolog archive directory.
- This could indicate your backups are failing and therefor your archived redologs aren't being purged with a successful backup.
- Default path: /opt/zimbra/backup/tmp/ [zmprov gs `zmhostname` zimbraBackupTarget] ; uses the following variable for the tmp directory.
- Failed backups will stay in the tmp directory and not be moved into the sessions directory.
- Default path: /opt/zimbra/redolog/archive/ [zmprov gs `zmhostname` zimbraRedoLogArchiveDir] ; uses the following variable for the redolog archive directory.
- Monitoring and reviewing the output of zmbackupquery and confirming that the "Number of accounts:" line falls within expectations for your system - most notably, for the full backups. Do NOT assume that full backup labels in your backup directory are an indication your system is backing up all your users.
- Case in point of a real support case, if the system no longer has enough space to do a full backup of all your users it will still backup new users and the label will be a full. The system administrator has assumed full backups were being done correctly because they simply listed the directory contents of the sessions directory and saw numerous full backup directories. What they didn't realize, those were just for new users and each full backup session only held one or two accounts. The server was failing on the full backup for all users because it was running out of space and would terminate prior to completing.
[zimbra@mail-172 ~]$ zmbackupquery --type full Label: full-20150228.060012.892 Type: full Status: completed Started: Sat, 2015/02/28 01:00:12.892 EST Ended: Sat, 2015/02/28 01:00:29.751 EST Redo log sequence range: 7 .. 7 Number of accounts: 10 out of 10 completed [zimbra@mail-172 ~]$ zmprov -l gaa | wc -l 10
When your using the Standard backup method, your incremental backup runs will also run a full backup against any 'new' users that currently don't have a full backup. Example of "new" accounts being backed up and also seeing the incremental being ran.
drwxr-x--- 6 zimbra zimbra 4096 Aug 20 01:04 incr-20130819.193041.672 drwxr-x--- 6 zimbra zimbra 4096 Aug 20 01:04 full-20130819.193420.421 drwxr-x--- 6 zimbra zimbra 4096 Aug 21 01:04 incr-20130820.193044.443 drwxr-x--- 6 zimbra zimbra 4096 Aug 21 01:04 full-20130820.193424.551 drwxr-x--- 6 zimbra zimbra 4096 Aug 22 01:04 incr-20130821.193039.818 drwxr-x--- 6 zimbra zimbra 4096 Aug 22 01:04 full-20130821.193427.056 drwxr-x--- 6 zimbra zimbra 4096 Aug 23 01:03 incr-20130822.193037.145 drwxr-x--- 6 zimbra zimbra 4096 Aug 23 01:04 full-20130822.193352.394
Server Issues
Questions To Ask & Address Prior To Doing A Server Restore
- Should I shutdown ZCS services or block client access until issue and resolution are identified?
- What data needs to really be restored - can you restore just that or does it need a full restore?
Server Restores
Server Restores - Disaster Recoveries
For A ZCS Server DR Restore
Please see Network_Edition_Disaster_Recovery on doing a full system restore.
To Restore Just The LDAP Date
Let's say your ldap data was 'lost/destoyed' but everything else was intact, you should look at the zmrestoreldap command.
This section should have more precaution and background information to handle this section.
The basics:
- To find the LDAP session labels type -lbs.
- zmrestoreldap -lbs
- Restore the complete LDAP directory server
- zmrestoreldap -lb full20061130135236
- Restore LDAP data for specific accounts
- zmrestoreldap -lb full20061130135236 -a tac@DOMAIN.com jane@DOMAIN.com
To Restore Just The Mysql DB
Option available from the following RFE work:
- "Allow backup of only primary message volume"
- https://bugzilla.zimbra.com/show_bug.cgi?id=35278
- Options to exclude types of data as described in the 6.0.6 Admin Guide:
- Search index : If you do not restore the search index data, the mailbox will have to be reindexed after the restore.
- zmrestore <all or account> --exclude-search-index
- Blobs : This is a useful option when all blobs for the mailbox being restored already exists.
- zmrestore <all or account>|--exclude-blobs
- HSM-blobs : This is useful when all HSM blobs for the mailbox being restored already exists.
- zmrestore <all or account> --exclude-hsm-blobs
- Search index : If you do not restore the search index data, the mailbox will have to be reindexed after the restore.
- Options to exclude types of data as described in the 6.0.6 Admin Guide:
- https://bugzilla.zimbra.com/show_bug.cgi?id=35278
Steps To Practice And Test A DR Situation - System And Mysql Data Only
This test should be done against a non-production server. Either install a 'dummy' ZCS server or create a clone of your production server to perform this test against.
Let's say your mysql data was 'lost/destoyed' but everything else was intact. This might be the solution for the situation:
The steps below are to REPRODUCE A DR situation to test.
- zmcontrol stop
- Caution - step to reproduce DR situation to test against
- mv ~/db/data ~/db/data.OLD
- Getting old mysql passwords
- zmlocalconfig -s mysql_root_password
- zmlocalconfig -s zimbra_mysql_password
- [as zimbra] /opt/zimbra/libexec/zmmyinit
- Most likely your mysql passwords will change.
- See the following article to set them back:
- ldap start
- zmconvertctl start
- mysql is already running per the zmmyinit - mysql.server status - to check.
- zmrestoreoffline -a all -br --systemData --excludeSearchIndex --excludeBlobs --excludeHsmBlobs
- Note - you most likely will want to use the -br or -rf option for this situation.
- -br,--backedupRedologsOnly : Replays the redo logs in backup only, which excludes archived and current redo logs of the system
- -rf,--restoreFullBackupOnly : Restores to last full backup only, which excludes incremental backups.
- -sys,--systemData : Restores global tables and local config.
- This restores the 'zimbra' mysql data - this stuff Ajcody-Mysql-Topics#zimbra_Database_Default_Example_for_ZCS5
- Important - the zimbra db holds information about the volumes. This needs to be restored/existing prior to user db restoration.
- Note the use of -a all , you could also do this for one account first to confirm operation is successful for your circumstances.
- Note - you most likely will want to use the -br or -rf option for this situation.
- If you used -rf or -br with your zmestoreoffline, you might also need to use zmplayredo to finish it up to get the most complete restore.
- Improve zmrestore & zmrestoreoffline performance
- Ajcody-Backup-Restore-Issues#zmplayredo_-_Replaying_Content_From_Any_Redolog_File and other 'redolog' sections on that page.
Server Cloning - Server Moves/Testing Purposes
Trouble-shooting Backup/Restore Issues And Other General Questions
Backup And Restore Compatibility Between ZCS Versions
- You can use backups from older versions of ZCS for account restores.
- You can NOT use backups from old versions of ZCS for system restores though. Disaster Recovery restores have to be done with the same version of ZCS as the backups were done with.
References:
- Upgrades And Compatibility Of Older Backups
- "Backups must be compatible across patch releases"
- "Incorrect upgrade documentation regarding backups"
- "support for restore across major versions"
- "Add conversion tool to upgrade backup versions to allow restore on later zcs versions"
- From 5.0.7 - 5.0.10 You Might See minor version upgrades moving your backups into a subdirectory
- "upgrade incorrectly invalidates backups."
Trouble-shooting
Ref:
Basic Backup Information To Submit To Support
Disk Space Usage Issues
Trend Data
If there is concerns about disks/partitions getting full, this command would be helpful for trending data on your server. Send support the resulting df.tar file . Note - adjust the tail command if you want more than 20 day's worth of trending data, the -n 20 option.
[zimbra@zcs806 tmp]$ /tmp [zimbra@zcs806 tmp]$ tar cvf /tmp/df.tar `find /opt/zimbra/zmstat -name df.cs\* | sort | tail -n 20` tar: Removing leading `/' from member names /opt/zimbra/zmstat/2014-03-10/df.csv.gz /opt/zimbra/zmstat/2014-03-11/df.csv.gz [cut - Ajcody] /opt/zimbra/zmstat/2014-03-27/df.csv.gz /opt/zimbra/zmstat/2014-03-28/df.csv.gz /opt/zimbra/zmstat/df.csv [zimbra@zcs806 tmp]$ ls -lah /tmp/df.tar -rw-r----- 1 zimbra zimbra 80K Mar 29 06:44 /tmp/df.tar [zimbra@zcs806 tmp]$ tar tvf /tmp/df.tar -rw-r----- zimbra/zimbra 2566 2014-03-11 00:00 opt/zimbra/zmstat/2014-03-10/df.csv.gz -rw-r----- zimbra/zimbra 2553 2014-03-12 00:00 opt/zimbra/zmstat/2014-03-11/df.csv.gz [cut - Ajcody] -rw-r----- zimbra/zimbra 2513 2014-03-28 00:00 opt/zimbra/zmstat/2014-03-27/df.csv.gz -rw-r----- zimbra/zimbra 2531 2014-03-29 00:00 opt/zimbra/zmstat/2014-03-28/df.csv.gz -rw-r----- zimbra/zimbra 8013 2014-03-29 06:40 opt/zimbra/zmstat/df.csv
Directory Sizes In /opt/zimbra
Please see the following and provide the output to support. Note, even though this method is faster than doing a du it still can take awhile.
* Ajcody-Server-Misc-Topics#Faster_Way_To_Get_Directory_Size_On_Filesytem_-_find_vs_du
Adjusting The Disk Alert Threshold
Note - zmlocalconfig smtp_notify must return yes if you want to receive the notifications.
If you just need to adjust the disk alert threshold, then see the following:
See current values:
zmlocalconfig | grep zmdisklog
Example adjustment:
su - zimbra zmlocalconfig -e zmdisklog_critical_threshold=98 zmlocalconfig -e zmdisklog_warn_threshold=95 zmstatctl
To exclude a partition from the checks [example of two being excluded]:
su - zimbra zmlocalconfig -e zmstat_df_excludes="/mount/point:/mount/point2" zmstatctl
Note, depending on your version of ZCS, you might be hitting a bug where you'll keep getting emails until a logrotate happens.
- Changing Zmstat-df values do not take affect until logrotate
Some things to do to confirm and share with support or in bug. As zimbra
su - zimbra ls -la /var/log/zimbra.log df -h /dev/mapper/vg_rhel664-lv_root 5.5G 3.5G 1.7G 68% / tmpfs 939M 0 939M 0% /dev/shm /dev/sda1 485M 79M 381M 18% /boot /dev/sdb1 30G 6.2G 23G 22% /opt date zmlocalconfig | grep zmdisklog zmdisklog_critical_threshold = 80 zmdisklog_warn_threshold = 85 zmlocalconfig -e zmdisklog_critical_threshold=95 zmlocalconfig -e zmdisklog_warn_threshold=90 zmlocalconfig | grep zmdisklog zmdisklog_critical_threshold = 95 zmdisklog_warn_threshold = 90 zmstatctl restart date ps -eaf | grep zmstat-df ls -la /var/log/zimbra.log date ; grep "Disk warning" /var/log/zimbra* ; zmmailbox -z -m admin@`zmhostname` s -l 100 -t message "Subject: Disk and after:yesterday" ##Note - Emails by default go out every 10 minutes - for example: [zimbra@zcs803 ~]$ date ; grep "Disk warning" /var/log/zimbra* ; zmmailbox -z -m admin@`zmhostname` s -l 100 -t message "Subject: Disk and after:yesterday" Thu May 22 09:40:08 PDT 2014 /var/log/zimbra.log:May 22 08:30:00 zcs803 zimbramon[18826]: 18826:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82% /var/log/zimbra.log:May 22 08:40:00 zcs803 zimbramon[22970]: 22970:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82% /var/log/zimbra.log:May 22 08:50:00 zcs803 zimbramon[22970]: 22970:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82% /var/log/zimbra.log:May 22 09:00:00 zcs803 zimbramon[22970]: 22970:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82% ## Note - I had readjusted the variable to not warn during this time segment ## /var/log/zimbra.log:May 22 09:20:00 zcs803 zimbramon[8322]: 8322:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82% /var/log/zimbra.log:May 22 09:30:00 zcs803 zimbramon[8322]: 8322:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82% /var/log/zimbra.log:May 22 09:40:00 zcs803 zimbramon[8322]: 8322:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82% num: 7, more: false Id Type From Subject Date ---- ---- -------------------- -------------------------------------------------- -------------- 1. 328 mess admin Disk / at 82% on zcs803.DOMAIN.com: 05/22/14 09:40 2. 327 mess admin Disk / at 82% on zcs803.DOMAIN.com: 05/22/14 09:30 3. 326 mess admin Disk / at 82% on zcs803.DOMAIN.com: 05/22/14 09:20 ## Note - I had readjusted the variable to not warn during this time segment ## 4. 325 mess admin Disk / at 82% on zcs803.DOMAIN.com: 05/22/14 09:00 5. 324 mess admin Disk / at 82% on zcs803.DOMAIN.com: 05/22/14 08:50 6. 323 mess admin Disk / at 82% on zcs803.DOMAIN.com: 05/22/14 08:40 7. 320 mess admin Disk / at 82% on zcs803.DOMAIN.com: 05/22/14 08:31
Continue to monitor your zmmailbox search results for an hour.
The Basic Information Support Needs
as root:
- cat /etc/fstab
- Shows us what is mounted upon boot
- cat /proc/mounts
- Shows us what is currently mounted and its status - you can see if a mount is read-only here.
- df -hT
- Lists current mounts using human-readable size information and also notes the filesystem type.
as zimbra:
- zmprov -l gs `zmhostname` | egrep 'Back|Redo'
- Will show us a number of variables related to backup and redologs. Also tell us if your using auto-group or the default method.
- du -sh /opt/zimbra/redolog
- Will might notice your redolog logs aren't rolling over, causing a possible issue.
- ls -latr /opt/zimbra/backup
- This is the default backup target, please adjust this path here and below if you are using a different zimbraBackupTarget value.
- zmprov gs `zmhostname` zimbraBackupTarget
- We'll be able to confirm permissions are right.
- ls -latr /opt/zimbra/backup/tmp
- This will show us if you have failed backup jobs and confirm tmp is being cleaned appropriately after the backup is done.
- ls -latr /opt/zimbra/backup/sessions
- This will show us what backup sessions are available and confirm permissions are correct.
- Adjust path if your zimbraBackupTarget value is not the default path.
- Some directory sizes in the backup directory:
- Default path first
- du -sh `find /opt/zimbra/backup -maxdepth 2 -type d`
- If your using a different backup target, check that directory also. Replace /opt/zimbra/backup above with your backup path.
- Default path first
- zmbackupquery
- This should match what's in the sessions directory and it will also tell us if status of each backup and how many accts were done.
- crontab -l | grep -i back
- This will show use when backups are support to run and with what options they are running with.
- zmlocalconfig | grep -i back
- This is useful to see a number of backup options not exposed in the crontab, things related to the zip options.
- zmvolume -l
- This is useful to see how many volumes are being used, if HSM is being used, and if compression is being done at the volume level.
Additional Log Files Support Might Need
And send the following logs:
- /var/log/messages
- Filesystem issues often times are noted here and also in syslog. This might explain an interruption in the backup process. Server restarts, filesystem going full, filesystem going read-only, etc.
- /var/log/syslog
- /opt/zimbra/log/mailbox.log
- The backup activity is logged here.
- And any other mailbox.log file that would cover the event
Additional Checks For Performance Specific Issues
If Your Using a SAN or NFS For Your Backup Target - Please Check Your IOWait
Ideally, you would compare iowait and performance data from the target backup host as well as the stats available on the ZCS servers. To get graphs and stats on this from ZCS, please see Ajcody-Testing-Debugging#zmstat_and_zmstat-chart . You should submit this data and iowait conclusions if you still need to submit a support case about backup performance issues.
Is HSM Running During Your Backup Window
- Are you running HSM? HSM should not be ran during your backup window.
- "RFE: HSM and backup should not run at the same time if initated."
Are You Using --zipStore
--zipStore zips the blobs vs. keeping the blobs as individual files. --zipStore does not use compression either. For most circumstances, this will give the best performance, especially with NFS. This should be the default behavior of the backups, the following RFE is when it became the default [ZCS6+] :
- "backup: default to the zip option"
- https://bugzilla.zimbra.com/show_bug.cgi?id=31836#c6
- Link to comment that explains options and default behavior.
- https://bugzilla.zimbra.com/show_bug.cgi?id=31836#c6
To see if zip's are being used for backups for example, in the backup/session directory you'll know if it is by seeing .zip files:
mail:~/backup/sessions/full-20080820.160003.770/accounts/115/988/11598896-a89b-4b9d-bedb-1ed1afcb6c87/blobs zimbra$ ls blobs-1.zip blobs-2.zip blobs-3.zip blobs-4.zip
To see if the zip file is using compression [-Z option for unzip will indicate whether or not the archive is actually compressed] :
unzip -Z blobs-4.zip 293 files, 5982984 bytes uncompressed, 5982984 bytes compressed: 0.0%
Also, if your zmvolume has compression enabled the blobs will remain compressed within the zip also upon backup. The point being, they are uncompressed to be then put into a zip file when the backup is using --zipStore.
Some Specific Example Restore/Backup Issues
NO_SUCH_ACCOUNT_BACKUP
Example errors:
Error code: backup.NO_SUCH_ACCOUNT_BACKUP Message: no such backup for account: Missing full backup earlier than restore-to time for account 3c397edc-c015-4d82-ae8a-1084692f8a93 Details:soap:Sender Error occurred: no such backup for account: 3c397edc-c015-4d82-ae8a-1084692f8a93 Message: no such backup for account: Missing full backup earlier than restore-to time for account 3c397edc-c015-4d82-ae8a-1084692f8a93 3c397edc-c015-4d82-ae8a-1084692f8a93 is an example of the zimbraId of a user [ user1@mail-172.example.com ] .
Confirm the current email address is mapped to the appropriate zimbraId. You'll want to search what is in ldap and the mailstores database for the exiting email address and then also search the reference files in your backups. Often times, this happens when an account was deleted and then created again. Account meaning the email address. When this is does, the new account will have a new zimbraId.
$ zmprov -l ga user1@mail-172.example.com zimbraId # name user1@mail-172.example.com zimbraId: 3c397edc-c015-4d82-ae8a-1084692f8a93 $ mysql -e 'SELECT * FROM zimbra.mailbox WHERE comment LIKE "user1@mail-172.example.com"\G' $ mysql -e 'SELECT * FROM zimbra.mailbox WHERE account_id="3c397edc-c015-4d82-ae8a-1084692f8a93"\G' $ grep -C1 3c397edc-c015-4d82-ae8a-1084692f8a93 /opt/zimbra/backup/accounts.xml [using the default path; /opt/zimbra/backup/accounts.xml] $ egrep "user1@mail-172.example.com|3c397edc-c015-4d82-ae8a-1084692f8a93" /opt/zimbra/backup/accounts.xml $ egrep "user1@mail-172.example.com|3c397edc-c015-4d82-ae8a-1084692f8a93" /opt/zimbra/backup/sessions/*/session.xml
exception during auth
Example error:
LDAP backup failed: system failure: exception during auth {RemoteManager: zimbramail.example.com->zimbra@zimbramail.example.com:22} com.zimbra.common.service.ServiceException: system failure: LDAP backup failed: system failure: exception during auth {RemoteManager: zimbramail.example.com-> zimbra@zimbramail.example.com:22}
This indicates the ssh keys aren't configured correctly in the environment. You should be able to do the following to resolve the issue:
su - zimbra zmsshkeygen zmupdateauthkeys
Note - You should also confirm that you do indeed have the ssh port available. The default is port 22. If your unable to ssh into the server using port 22, then this might be the issue. If you configured ssh to listen on a different port on this server, you'll need to adjust the following variable - zmprov gcf zimbraRemoteManagementPort . You might also have the ssh port blocked - confirm your firewall, iptables, etc is not blocking the ssh port.
LICENSE_ERROR_Message: AccountsLimits exceeded
Example error:
LICENSE_ERROR_Message: AccountsLimits exceeded
You should check your license and confirm it's valid and active:
su - zimbra zmlicense -p
And confirm how many licenses are used: First, flush the cache on all servers for the license information. Then do the query via soap.
zmprov fc -a license zmsoap -z GetLicenseRequest
Four lines that are generally of interest are shown below. The first two state how many one is licensed for, the bottom two show how many are currently used :
<attr name="AccountsLimit">1000</attr> <attr name="ArchivingAccountsLimit">1000</attr> <attr name="TotalAccounts">3</attr> <attr name="ArchivingAccounts">3</attr>
Performance Issues And Time To Complete
RFE's for increase backup performance:
- "Improve zmrestore & zmrestoreoffline performance"
- "Improvement to backup performance"
- http://bugzilla.zimbra.com/show_bug.cgi?id=36220
- Above marked as a dupe of "Reduce duration of maintenance mode during backup"
What Might Be Wrong?
Need to expand this section or refer to the part above about all the log/data gathering that support would normally ask for
First thing to check is the log file and see if anything notable stands out.
grep -i backup /opt/zimbra/log/mailbox.log
Use Auto-Group Backups Rather Than Default Style Backup
Having trouble completing that entire full backup during off-hours? Enter the hybrid auto-grouped mode, which combines the concept of full and incremental backup functions - you’re completely backing up a target number of accounts daily rather than running incrementals.
Auto-grouped mode automatically pulls in the redologs since the last run so you get incremental backups of the remaining accounts; although the incremental accounts captured via the redologs are not listed specifically in the backup account list. This still allows you to do a point in time restore for any account.
Please see the following for more detailed information:
Need To Write Fewer Files - Add The Zip Option To Your Backup Commands
Using the zip option will compress all those thousands of single files that exist under a user's backup, decreasing performance issues that arise from writing out thousands of small files as compared to large ones. This is often seen when one is :
- Using nfs for the backup directory
- Copying/rsyncing backups to a remote server
- Are using some third party backup software (to tape) to archive/backup the zimbra backup sessions.
Please see the following for more information about using the Zip option:
SAN Snapshots For Backups
Please see:
Cloud Backups
Please see:
Tape Backups
Use a 'secondary' network card to run your backups over if possible. This way the backup activity doesn't effect ZCS network performance over the default network card. Also, please review the following section on Zimbra and hard links and confirm if your third party backup software or process will handle restoring hard links correctly.
Please see:
Hands On How-To And Reviewing Of Log Events
Full Backup And Log Events Generated
Example of manually starting a full backup using the Standard zimbraBackupMode and doing a query to confirm it's completion and status.
[zimbra@mail-172 archive]$ zmprov gs `zmhostname` zimbraBackupMode # name mail-172.example.com zimbraBackupMode: Standard [zimbra@mail-172 archive]$ pwd ; ls *.log /opt/zimbra/redolog/archive redo-20150303.060015.816-seq9.log [zimbra@mail-172 archive]$ zmbackup -f -a all full-20150304.012000.193 [zimbra@mail-172 archive]$ zmbackupquery -lb full-20150304.012000.193 -v Label: full-20150304.012000.193 Type: full Status: completed Started: Tue, 2015/03/03 20:20:00.193 EST Ended: Tue, 2015/03/03 20:20:28.630 EST Redo log sequence range: 10 .. 10 Number of accounts: 10 out of 10 completed Accounts: admin@mail-172.example.com: completed galsync.qva4ekog0@mail-172.example.com: completed ham.jgfdb3oqjf@mail-172.example.com: completed spam.umhx3owv3s@mail-172.example.com: completed user1@mail-172.example.com: completed user2@mail-172.example.com: completed user3@mail-172.example.com: completed user4@mail-172.example.com: completed user5@mail-172.example.com: completed virus-quarantine.gusiso1cd@mail-172.example.com: completed Total space: 14020MB Free space: 6017MB
Example of what log events are generated for a full backup in /opt/zimbra/log/mailbox.log . First, you'll see the authentication/acl request and then statements about the backup request.
2015-03-03 20:19:58,499 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [] AuthProvider - Adding auth provider: zimbra com.zimbra.cs.service.ZimbraAuthProvider 2015-03-03 20:19:59,458 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] soap - AuthRequest elapsed=834 2015-03-03 20:19:59,751 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] account - Initialized access manager: com.zimbra.cs.account.accesscontrol.ACLAccessManager 2015-03-03 20:19:59,770 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup request started 2015-03-03 20:20:00,193 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Found 10 accounts on server mail-172.example.com 2015-03-03 20:20:00,464 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup request finished 2015-03-03 20:20:00,464 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] soap - BackupRequest elapsed=760
Your log events will now reference Backup in them. AsyncFileCopier will start and it will give the backup session label for this backup job.
2015-03-03 20:20:01,092 INFO [FullBackupThread] [] io - AsyncFileCopier is starting 2015-03-03 20:20:01,187 INFO [FullBackupThread] [] backup - Full backup started for backup set; label: full-20150304.012000.193
It proceeds to backing up your accounts. I'll list the backup log events for just two accounts on this system - user1@mail-172.example.com and admin@mail-172.example.com . I'll provide the zimbraId, email address, and mailboxID information for user1@mail-172.example.com below so you can confirm it's reported in the logs correctly as well.
[zimbra@mail-172 redolog]$ zmprov ga user1@mail-172.example.com zimbraId # name user1@mail-172.example.com zimbraId: 3c397edc-c015-4d82-ae8a-1084692f8a93 [zimbra@mail-172 redolog]$ zmprov gmi user1@mail-172.example.com mailboxId: 6 quotaUsed: 1209
Now, continuing with the log events. Notice the reference to the redo log file sequence, putting the account into and out of maintenance, number of blobs and index files backed up, and the statement it's finished backing up the account. Also within the section we can see the user's zimbraId [3c397edc-c015-4d82-ae8a-1084692f8a93] , email address [user1@mail-172.example.com] , and mailboxId [Mailbox 6].
[cut] 2015-03-03 20:20:06,560 INFO [FullBackupThread] [name=user1@mail-172.example.com;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/6/index/0 2015-03-03 20:20:06,560 INFO [FullBackupThread] [name=user1@mail-172.example.com;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 LOADED 2015-03-03 20:20:06,560 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - redo log file sequence is 10 at full backup for user1@mail-172.example.com 2015-03-03 20:20:06,565 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - Full backup started for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) mailbox 6 2015-03-03 20:20:06,565 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] mailbox - Putting mailbox 6 under maintenance. 2015-03-03 20:20:06,818 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - Number of blobs to backup for mailbox 6: 1 2015-03-03 20:20:06,818 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] mailbox - Ending maintenance on mailbox 6. 2015-03-03 20:20:06,818 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 AVAILABLE 2015-03-03 20:20:06,966 INFO [BackupSetWorkerThread-BLOB] [name=user1@mail-172.example.com;mid=6;] backup - Stored 1 blob files from volume message1 2015-03-03 20:20:06,966 INFO [BackupSetWorkerThread-BLOB] [name=user1@mail-172.example.com;mid=6;] backup - Stored 1 blob files 2015-03-03 20:20:07,014 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - Account user1@mail-172.example.com in backup set full-20150304.012000.193: All pending file IO completed (1 out of 1) 2015-03-03 20:20:07,016 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] mailbox - Putting mailbox 6 under maintenance. 2015-03-03 20:20:07,021 INFO [BackupSetWorkerThread-INDEX] [name=user1@mail-172.example.com;mid=6;] backup - Stored 2 index files 2015-03-03 20:20:07,496 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - Account user1@mail-172.example.com in backup set full-20150304.012000.193: All pending file IO completed (3 out of 3) 2015-03-03 20:20:07,497 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] mailbox - Ending maintenance on mailbox 6. 2015-03-03 20:20:07,497 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 AVAILABLE 2015-03-03 20:20:07,497 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - Account user1@mail-172.example.com in backup set full-20150304.012000.193: All pending file IO completed (3 out of 3) 2015-03-03 20:20:07,497 INFO [FullBackupThread] [name=user1@mail-172.example.com;mid=6;] backup - Full backup finished for account user1@mail-172.example.com (3c397edc-c015-4d82-ae8a-1084692f8a93) mailbox 6
Events for the admin user's backup:
2015-03-03 20:20:09,894 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - redo log file sequence is 10 at full backup for admin@mail-172.example.com 2015-03-03 20:20:09,914 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - Full backup started for account admin@mail-172.example.com (7ac8de92-13fe-4d19-b9da-6872169fb5c9) mailbox 1 2015-03-03 20:20:09,915 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] mailbox - Putting mailbox 1 under maintenance. 2015-03-03 20:20:10,176 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - Number of blobs to backup for mailbox 1: 73 2015-03-03 20:20:10,177 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] mailbox - Ending maintenance on mailbox 1. 2015-03-03 20:20:10,177 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] mbxmgr - Mailbox 1 account 7ac8de92-13fe-4d19-b9da-6872169fb5c9 AVAILABLE 2015-03-03 20:20:10,343 INFO [BackupSetWorkerThread-BLOB] [name=admin@mail-172.example.com;mid=1;] backup - Stored 73 blob files from volume message1 2015-03-03 20:20:10,353 INFO [BackupSetWorkerThread-BLOB] [name=admin@mail-172.example.com;mid=1;] backup - Stored 73 blob files 2015-03-03 20:20:11,047 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - Account admin@mail-172.example.com in backup set full-20150304.012000.193: All pending file IO completed (73 out of 73) 2015-03-03 20:20:11,051 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] mailbox - Putting mailbox 1 under maintenance. 2015-03-03 20:20:11,108 INFO [BackupSetWorkerThread-INDEX] [name=admin@mail-172.example.com;mid=1;] backup - Stored 26 index files 2015-03-03 20:20:12,450 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - Account admin@mail-172.example.com in backup set full-20150304.012000.193: All pending file IO completed (99 out of 99) 2015-03-03 20:20:12,463 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] mailbox - Ending maintenance on mailbox 1. 2015-03-03 20:20:12,463 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] mbxmgr - Mailbox 1 account 7ac8de92-13fe-4d19-b9da-6872169fb5c9 AVAILABLE 2015-03-03 20:20:12,463 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - Account admin@mail-172.example.com in backup set full-20150304.012000.193: All pending file IO completed (99 out of 99) 2015-03-03 20:20:12,463 INFO [FullBackupThread] [name=admin@mail-172.example.com;mid=1;] backup - Full backup finished for account admin@mail-172.example.com (7ac8de92-13fe-4d19-b9da-6872169fb5c9) mailbox 1
And then a final line confirming completion and the number of accounts successfully backed up.
2015-03-03 20:20:12,463 INFO [FullBackupThread] [] backup - Finished backing up 10 of 10 accounts
One thing to point out is the log events were mentioning the redo log sequence as being number 10 but in the redolog/archive directory there is only a sequence 9 redo log. This is because the "seq 10" is actually the /opt/zimbra/redolog/redo.log file. When the redo.log file rolls over and is moved to the archive directory it will have seq10 in it's filename.
[zimbra@mail-172 archive]$ pwd ; ls -la *.log /opt/zimbra/redolog/archive -rw-r----- 1 zimbra zimbra 146723 Mar 3 20:23 redo-20150303.060015.816-seq9.log
The contents of the backup session directory. I removed the sub-directories under accounts that weren't either the admin or user1 account. Notice the zimbraId is used for the sub-directory naming scheme.
[zimbra@mail-172 full-20150304.012000.193]$ ls -R .: accounts ldap session.xml shared_blobs sys ./accounts: 3c3 423 434 75d 7ac 89f a46 af0 c53 eab ./accounts/3c3: 97e ./accounts/3c3/97e: 3c397edc-c015-4d82-ae8a-1084692f8a93 ./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93: blobs db index ldap.xml meta.xml ./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93/blobs: blobs-1.zip ./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93/db: db_schema.xml mail_item.dat mailbox.dat mailbox_metadata.dat open_conversation.dat tag.dat tagged_item.dat ./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93/index: 0 ./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93/index/0: segments.gen segments_1 [cut] ./accounts/7ac: 8de ./accounts/7ac/8de: 7ac8de92-13fe-4d19-b9da-6872169fb5c9 ./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9: blobs db index ldap.xml meta.xml ./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9/blobs: blobs-1.zip blobs-2.zip blobs-3.zip blobs-4.zip ./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9/db: db_schema.xml mail_item.dat mailbox.dat open_conversation.dat tag.dat tagged_item.dat ./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9/index: 0 ./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9/index/0: _0.fdt _0.fnm _0.nrm _0.tii _1.fdt _1.fnm _1.nrm _1.tii _2.fdt _2.fnm _2.nrm _2.tii segments.gen _0.fdx _0.frq _0.prx _0.tis _1.fdx _1.frq _1.prx _1.tis _2.fdx _2.frq _2.prx _2.tis segments_3 [cut] ./ldap: ldap-config.bak.gz ldap.bak.gz ./shared_blobs: ./sys: config.dat current_volumes.dat db_schema.xml localconfig.xml ssh ssl volume.dat ./sys/ssh: authorized_keys zimbra_identity zimbra_identity.pub ./sys/ssl: zimbra ./sys/ssl/zimbra: ca commercial jetty.pkcs12 server ./sys/ssl/zimbra/ca: ca.key ca.pem ca.srl ca.srl.old index.txt index.txt.attr index.txt.old newcerts zmssl.cnf ./sys/ssl/zimbra/ca/newcerts: 1424523586.pem 1424523591.pem 1424523597.pem 1424523603.pem ./sys/ssl/zimbra/commercial: ./sys/ssl/zimbra/server: server.crt server.csr server.key
Incremental Backup And Log Events Generated
Example of manually starting an incremental backup using the Standard zimbraBackupMode and doing a query to confirm it's completion and status.
[zimbra@mail-172 archive]$ zmprov gs `zmhostname` zimbraBackupMode # name mail-172.example.com zimbraBackupMode: Standard [zimbra@mail-172 archive]$ ls redo-20150303.060015.816-seq9.log redo-20150304.070025.476-seq10.log [zimbra@mail-172 archive]$ zmbackup -i incr-20150304.073543.151 [zimbra@mail-172 archive]$ zmbackupquery -lb incr-20150304.073543.151 -v Label: incr-20150304.073543.151 Type: incremental Status: completed Started: Wed, 2015/03/04 02:35:43.151 EST Ended: Wed, 2015/03/04 02:35:51.981 EST Redo log sequence range: 10 .. 11 Number of accounts: 10 out of 10 completed Accounts: admin@mail-172.example.com: completed galsync.qva4ekog0@mail-172.example.com: completed ham.jgfdb3oqjf@mail-172.example.com: completed spam.umhx3owv3s@mail-172.example.com: completed user1@mail-172.example.com: completed user2@mail-172.example.com: completed user3@mail-172.example.com: completed user4@mail-172.example.com: completed user5@mail-172.example.com: completed virus-quarantine.gusiso1cd@mail-172.example.com: completed Total space: 14020MB Free space: 6032MB
Notice how the log events are soap requests and differ from the full backup above and lack the INFO [*Backup*] formatting. Important to note this if your attempting to grep for backup/restore activity. Just like with the full, it starts with an authentication/acl request and states it's starting, how many accounts found, AsyncFileCopier starting, and the label name for this backup session.
From /opt/zimbra/log/mailbox.log
2015-03-04 02:35:42,356 INFO [qtp509886383-19:https://127.0.0.1:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] soap - AuthRequest elapsed=94 2015-03-04 02:35:42,481 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup request started 2015-03-04 02:35:42,917 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Found 10 accounts on server mail-172.example.com 2015-03-04 02:35:43,254 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] io - AsyncFileCopier is starting 2015-03-04 02:35:43,255 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=zimbra;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup started for backup set; label: incr-20150304.073543.151
The log events are again trimmed below just for the user1 and admin account for the user backup section of the log file. Notice the absence of any statements of the accounts being put into maintenance mode.
2015-03-04 02:35:44,118 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/index/0/6/index/0 2015-03-04 02:35:44,119 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] mbxmgr - Mailbox 6 account 3c397edc-c015-4d82-ae8a-1084692f8a93 LOADED 2015-03-04 02:35:44,120 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup is started for account user1@mail-172.example.com 2015-03-04 02:35:44,120 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Saving account information from LDAP 2015-03-04 02:35:44,127 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup has ended 2015-03-04 02:35:44,127 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=user1@mail-172.example.com;mid=6;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Account user1@mail-172.example.com in backup set incr-20150304.073543.151: All pending file IO completed (0 out of 0) [cut] 2015-03-04 02:35:44,212 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup is started for account admin@mail-172.example.com 2015-03-04 02:35:44,212 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Saving account information from LDAP 2015-03-04 02:35:44,215 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup has ended 2015-03-04 02:35:44,217 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [name=admin@mail-172.example.com;mid=1;ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Account admin@mail-172.example.com in backup set incr-20150304.073543.151: All pending file IO completed (0 out of 0)
The incremental will then end with log events related to copying the redologs into the backup directory and then purging the redologs that can be deleted. This example states redo log sequence 9 can be deleted. We'll confirm it is later.
2015-03-04 02:35:44,618 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] redolog - waiting for FileLogWriter.FsyncThread-1425452599560 to finish. 2015-03-04 02:35:44,628 INFO [FileLogWriter.FsyncThread-1425452599560] [] redolog - fsync thread exiting 2015-03-04 02:35:44,628 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] redolog - FileLogWriter.FsyncThread-1425452599560 finished 2015-03-04 02:35:44,757 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] redolog - Redo log rollover took 540ms 2015-03-04 02:35:44,758 INFO [FileLogWriter.FsyncThread-1425454544757] [] redolog - Starting fsync thread with interval 10 2015-03-04 02:35:44,776 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Found 3 redo logs to backup 2015-03-04 02:35:44,861 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Last backed-up redo log sequence = 9 2015-03-04 02:35:44,943 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Marking redo log /opt/zimbra/redolog/archive/redo-20150303.060015.816-seq9.log for deletion 2015-03-04 02:35:44,943 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Copying redo log /opt/zimbra/redolog/archive/redo-20150304.070025.476-seq10.log to /opt/zimbra/backup/tmp/incr-20150304.073543.151/redologs/redo-20150304.070025.476-seq10.log 2015-03-04 02:35:45,053 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Retaining recent redo log /opt/zimbra/redolog/archive/redo-20150304.070025.476-seq10.log 2015-03-04 02:35:45,053 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Copying redo log /opt/zimbra/redolog/archive/redo-20150304.073544.755-seq11.log to /opt/zimbra/backup/tmp/incr-20150304.073543.151/redologs/redo-20150304.073544.755-seq11.log 2015-03-04 02:35:45,067 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Retaining recent redo log /opt/zimbra/redolog/archive/redo-20150304.073544.755-seq11.log
The last limes confirm the backup finished successfully.
2015-03-04 02:35:51,935 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup set incr-20150304.073543.151: All pending file IO completed (0 out of 0) 2015-03-04 02:35:51,981 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] io - AsyncFileCopier is shut down 2015-03-04 02:35:51,981 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Incremental backup finished for backup set; label: incr-20150304.073543.151 2015-03-04 02:35:51,982 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Deleting redo log /opt/zimbra/redolog/archive/redo-20150303.060015.816-seq9.log 2015-03-04 02:35:52,909 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] backup - Backup request finished 2015-03-04 02:35:52,909 INFO [qtp509886383-20:https://127.0.0.1:7071/service/admin/soap/BackupRequest] [ip=127.0.0.1;ua=zmbackup/8.6.0_GA_1153;] soap - BackupRequest elapsed=10451
For an incremental, the account sub-directories will have ldap data saved also for each user but blobs and indexes aren't. That data resides in the redologs from the system under the redologs directory.
[zimbra@mail-172 incr-20150304.073543.151]$ ls -R .: accounts ldap redologs session.xml sys ./accounts: 3c3 423 434 75d 7ac 89f a46 af0 c53 eab ./accounts/3c3: 97e ./accounts/3c3/97e: 3c397edc-c015-4d82-ae8a-1084692f8a93 ./accounts/3c3/97e/3c397edc-c015-4d82-ae8a-1084692f8a93: ldap.xml [cut] ./accounts/7ac: 8de ./accounts/7ac/8de: 7ac8de92-13fe-4d19-b9da-6872169fb5c9 ./accounts/7ac/8de/7ac8de92-13fe-4d19-b9da-6872169fb5c9: ldap.xml [cut] ./ldap: ldap-config.bak.gz ldap.bak.gz ./redologs: redo-20150304.070025.476-seq10.log redo-20150304.073544.755-seq11.log ./sys: config.dat current_volumes.dat db_schema.xml localconfig.xml ssh ssl volume.dat ./sys/ssh: authorized_keys zimbra_identity zimbra_identity.pub ./sys/ssl: zimbra ./sys/ssl/zimbra: ca commercial jetty.pkcs12 server ./sys/ssl/zimbra/ca: ca.key ca.pem ca.srl ca.srl.old index.txt index.txt.attr index.txt.old newcerts zmssl.cnf ./sys/ssl/zimbra/ca/newcerts: 1424523586.pem 1424523591.pem 1424523597.pem 1424523603.pem ./sys/ssl/zimbra/commercial: ./sys/ssl/zimbra/server: server.crt server.csr server.key
Notice that redo log sequence 9 was indeed deleted from the redolog/archive directory.
[zimbra@mail-172 archive]$ pwd ; ls /opt/zimbra/redolog/archive redo-20150304.070025.476-seq10.log redo-20150304.073544.755-seq11.log redo-20150304.074024.058-seq12.log
If you had created new users after your last full backup, you would see that there was also a full backup started as well. For example :
[zimbra@mail-172 log]$ zmbackup -i incr-20150304.085810.220 full-20150304.085818.889 [zimbra@mail-172 log]$ zmbackupquery -a user6@mail-172.example.com Account: user6@mail-172.example.com Label: full-20150304.085818.889 Type: full Started: Wed, 2015/03/04 03:58:18.889 EST Ended: Wed, 2015/03/04 03:58:25.997 EST Acct ID: 4e898a68-1886-4a80-b3dc-ac2ab650bf8c [zimbra@mail-172 log]$ zmbackupquery -lb full-20150304.085818.889 -v Label: full-20150304.085818.889 Type: full Status: completed Started: Wed, 2015/03/04 03:58:18.889 EST Ended: Wed, 2015/03/04 03:58:25.997 EST Redo log sequence range: 14 .. 14 Number of accounts: 1 out of 1 completed Accounts: user6@mail-172.example.com: completed Total space: 14020MB Free space: 6038MB
Contents Of accounts.xml And sessions.xml From The Full And Incremental Backup Examples
First, see what zmbackupquery shows for a particular user in regards to the backup labels they are included in.
$ zmbackupquery -a user1@mail-172.example.com Account: user1@mail-172.example.com Label: incr-20150304.073543.151 Type: incremental Started: Wed, 2015/03/04 02:35:43.151 EST Ended: Wed, 2015/03/04 02:35:51.981 EST Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93 Label: full-20150304.012000.193 Type: full Started: Tue, 2015/03/03 20:20:00.193 EST Ended: Tue, 2015/03/03 20:20:28.630 EST Acct ID: 3c397edc-c015-4d82-ae8a-1084692f8a93
Now let's review the accounts/session xml files that list all the users within those particular backup sessions. The /opt/zimbra/backup/accounts.xml is the referenced to determine what is the last know full backup session the user was in. If the /opt/zimbra/backup/accounts.xml is corrupted, unreadable, or incorrect - this can cause issues with restoring users.
Also notice that it also references the zimbraId for the user at the time of the backup. If you were to delete a user and then create a new account using the same email address [user1@mail-172.example.com] that new account will have a different zimbraId and therefor the backup/restore commands will not work as you might expected because of the differing zimbraId. The zmrestore command only can take an email address for the -a option, it can not use the zimbraID.
[zimbra@mail-172 backup]$ pwd /opt/zimbra/backup [zimbra@mail-172 backup]$ cat accounts.xml <?xml version="1.0" encoding="utf-8"?> <backupMetadata xmlns="urn:zimbraBackupMeta" version="7.2" dbVersion="103"> <allAccounts> <account zimbraId="af06b0b4-3160-4ea3-a55b-81163a9914a0" email="user5@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193"> </account> <account zimbraId="75d3d364-a384-4077-8ac5-87b6960b57cd" email="user3@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193"> </account> <account zimbraId="4343a299-5259-4826-b1c1-becbfbb9b8f0" email="galsync.qva4ekog0@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193"> </account> <account zimbraId="89fc5986-31a3-406e-8084-0ce0b98a251f" email="user2@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193"> </account> <account zimbraId="42380a6e-5718-4d68-ab37-e8d5bc9e6977" email="spam.umhx3owv3s@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193"> </account> <account zimbraId="eab3d56e-ffd9-4412-95de-48f102d2d252" email="user4@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193"> </account> <account zimbraId="7ac8de92-13fe-4d19-b9da-6872169fb5c9" email="admin@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193"> </account> <account zimbraId="4e898a68-1886-4a80-b3dc-ac2ab650bf8c" email="user6@mail-172.example.com" latestFullBackupLabel="full-20150304.085818.889"> </account> <account zimbraId="a46dfc59-a3e9-4f09-adfb-af8efc42aa9f" email="virus-quarantine.gusiso1cd@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193"> </account> <account zimbraId="3c397edc-c015-4d82-ae8a-1084692f8a93" email="user1@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193"> </account> <account zimbraId="c5315c5d-97cb-4d84-bc70-374a67f13309" email="ham.jgfdb3oqjf@mail-172.example.com" latestFullBackupLabel="full-20150304.012000.193"> </account> </allAccounts> </backupMetadata>
In each backup session, there will be a sessions.xml file that acts as a reference or summary file of the backup. At the top, you'll see what version of ZCS was running and what options the backup environment had set. It will then report the final status for each user for the backup attempt against the account. At the end, there's a section in regards to errors and if any were encountered.
[zimbra@mail-172 backup]$ cd sessions/ [zimbra@mail-172 sessions]$ cd full-20150304.012000.193 [zimbra@mail-172 full-20150304.012000.193]$ cat session.xml <?xml version="1.0" encoding="utf-8"?> <backupMetadata xmlns="urn:zimbraBackupMeta" version="7.2" dbVersion="103"> <backupSet label="full-20150304.012000.193" zcsRelease="8.6.0_GA_1153 20141215151258 20141215-1518 NETWORK" startTime="1425432000193" endTime="1425432028630" minRedoSeq="10" maxRedoSeq="10" sharedBlobsZipped="true" sharedBlobsZipNameDigestChars="1" sharedBlobsDirectoryDepth="5" sharedBlobsCharsPerDirectory="2" type="full" accountsDirectoryDepth="2"> <desc>Full backup</desc> <accounts> <account zimbraId="af06b0b4-3160-4ea3-a55b-81163a9914a0" email="user5@mail-172.example.com" status="COMPLETED"> </account> <account zimbraId="75d3d364-a384-4077-8ac5-87b6960b57cd" email="user3@mail-172.example.com" status="COMPLETED"> </account> <account zimbraId="89fc5986-31a3-406e-8084-0ce0b98a251f" email="user2@mail-172.example.com" status="COMPLETED"> </account> <account zimbraId="4343a299-5259-4826-b1c1-becbfbb9b8f0" email="galsync.qva4ekog0@mail-172.example.com" status="COMPLETED"> </account> <account zimbraId="eab3d56e-ffd9-4412-95de-48f102d2d252" email="user4@mail-172.example.com" status="COMPLETED"> </account> <account zimbraId="42380a6e-5718-4d68-ab37-e8d5bc9e6977" email="spam.umhx3owv3s@mail-172.example.com" status="COMPLETED"> </account> <account zimbraId="7ac8de92-13fe-4d19-b9da-6872169fb5c9" email="admin@mail-172.example.com" status="COMPLETED"> </account> <account zimbraId="a46dfc59-a3e9-4f09-adfb-af8efc42aa9f" email="virus-quarantine.gusiso1cd@mail-172.example.com" status="COMPLETED"> </account> <account zimbraId="3c397edc-c015-4d82-ae8a-1084692f8a93" email="user1@mail-172.example.com" status="COMPLETED"> </account> <account zimbraId="c5315c5d-97cb-4d84-bc70-374a67f13309" email="ham.jgfdb3oqjf@mail-172.example.com" status="COMPLETED"> </account> </accounts> <stats> <counter name="total_time" numSamples="1" sum="28437" unit="ms"> </counter> <counter name="sysdb_time" numSamples="1" sum="1111" unit="ms"> </counter> <counter name="sysdb_bytes" numSamples="1" sum="35210" unit="bytes"> </counter> <counter name="redologs_time" numSamples="0" sum="0" unit="ms"> </counter> <counter name="redologs_bytes" numSamples="0" sum="0" unit="bytes"> </counter> <counter name="accounts_time" numSamples="10" sum="4827" unit="ms"> </counter> <counter name="ldap_time" numSamples="10" sum="172" unit="ms"> </counter> <counter name="ldap_bytes" numSamples="10" sum="25665" unit="bytes"> </counter> <counter name="db_time" numSamples="10" sum="4787" unit="ms"> </counter> <counter name="db_bytes" numSamples="10" sum="78690" unit="bytes"> </counter> <counter name="db_digest_map_load_time" numSamples="10" sum="2400" unit="ms"> </counter> <counter name="msgs_time" numSamples="10" sum="451" unit="ms"> </counter> <counter name="msg_copy_time" numSamples="74" sum="25" unit="ms"> </counter> <counter name="msg_copy_bytes" numSamples="74" sum="149611" unit="bytes"> </counter> <counter name="msg_link_time" numSamples="0" sum="0" unit="ms"> </counter> <counter name="index_time" numSamples="48" sum="80" unit="ms"> </counter> <counter name="index_bytes" numSamples="38" sum="73510" unit="bytes"> </counter> </stats> <errors> </errors> </backupSet> </backupMetadata>
An incremental session.xml file is similar to a full backups session.xml .
[zimbra@mail-172 sessions]$ cd incr-20150304.073543.151 [zimbra@mail-172 incr-20150304.073543.151]$ cat session.xml <?xml version="1.0" encoding="utf-8"?> <backupMetadata xmlns="urn:zimbraBackupMeta" version="7.2" dbVersion="103"> <backupSet label="incr-20150304.073543.151" zcsRelease="8.6.0_GA_1153 20141215151258 20141215-1518 NETWORK" startTime="1425454543151" endTime="1425454551981" minRedoSeq="10" maxRedoSeq="11" sharedBlobsZipped="true" sharedBlobsZipNameDigestChars="1" sharedBlobsDirectoryDepth="5" sharedBlobsCharsPerDirectory="2" type="incremental" accountsDirectoryDepth="2"> <desc>Incremental backup</desc> <accounts> <account zimbraId="af06b0b4-3160-4ea3-a55b-81163a9914a0" email="user5@mail-172.example.com" status="COMPLETED"> </account> <account zimbraId="75d3d364-a384-4077-8ac5-87b6960b57cd" email="user3@mail-172.example.com" status="COMPLETED"> </account> <account zimbraId="89fc5986-31a3-406e-8084-0ce0b98a251f" email="user2@mail-172.example.com" status="COMPLETED"> </account> <account zimbraId="4343a299-5259-4826-b1c1-becbfbb9b8f0" email="galsync.qva4ekog0@mail-172.example.com" status="COMPLETED"> </account> <account zimbraId="eab3d56e-ffd9-4412-95de-48f102d2d252" email="user4@mail-172.example.com" status="COMPLETED"> </account> <account zimbraId="42380a6e-5718-4d68-ab37-e8d5bc9e6977" email="spam.umhx3owv3s@mail-172.example.com" status="COMPLETED"> </account> <account zimbraId="7ac8de92-13fe-4d19-b9da-6872169fb5c9" email="admin@mail-172.example.com" status="COMPLETED"> </account> <account zimbraId="a46dfc59-a3e9-4f09-adfb-af8efc42aa9f" email="virus-quarantine.gusiso1cd@mail-172.example.com" status="COMPLETED"> </account> <account zimbraId="3c397edc-c015-4d82-ae8a-1084692f8a93" email="user1@mail-172.example.com" status="COMPLETED"> </account> <account zimbraId="c5315c5d-97cb-4d84-bc70-374a67f13309" email="ham.jgfdb3oqjf@mail-172.example.com" status="COMPLETED"> </account> </accounts> <stats> <counter name="total_time" numSamples="1" sum="8830" unit="ms"> </counter> <counter name="sysdb_time" numSamples="1" sum="563" unit="ms"> </counter> <counter name="sysdb_bytes" numSamples="1" sum="35210" unit="bytes"> </counter> <counter name="redologs_time" numSamples="1" sum="310" unit="ms"> </counter> <counter name="redologs_bytes" numSamples="1" sum="126307" unit="bytes"> </counter> <counter name="accounts_time" numSamples="10" sum="196" unit="ms"> </counter> <counter name="ldap_time" numSamples="10" sum="129" unit="ms"> </counter> <counter name="ldap_bytes" numSamples="10" sum="25665" unit="bytes"> </counter> <counter name="db_time" numSamples="0" sum="0" unit="ms"> </counter> <counter name="db_bytes" numSamples="0" sum="0" unit="bytes"> </counter> <counter name="db_digest_map_load_time" numSamples="0" sum="0" unit="ms"> </counter> <counter name="msgs_time" numSamples="0" sum="0" unit="ms"> </counter> <counter name="msg_copy_time" numSamples="0" sum="0" unit="ms"> </counter> <counter name="msg_copy_bytes" numSamples="0" sum="0" unit="bytes"> </counter> <counter name="msg_link_time" numSamples="0" sum="0" unit="ms"> </counter> <counter name="index_time" numSamples="0" sum="0" unit="ms"> </counter> <counter name="index_bytes" numSamples="0" sum="0" unit="bytes"> </counter> </stats> <errors> </errors> </backupSet> </backupMetadata>
Quiz
- What are three methods you can locate which backups a user is in?
- To stop a restore from playing all the way up to the current time, you would need to use one of these two zmrestore variables - what are they?
- If you see an error about being unable to restore because of redologs, what option might you need to include?
- What is the primary log file on the ZCS server you'll monitor or review for restore or backup issues?
- Provide two methods to increase logging/debug output when attempting a restore?
- Can you perform a full system restore from the backups done on an older ZCS version?
- What are the two identify variables that zmbackup/zmrestore uses for an individual?
- What user variable would you use to locate the directory a users data is stored in within a backup session?