Difference between revisions of "Ajcody-Backup-Restore-Issues"

m (Disk Space Usage Issues)
m (Actual Backup & Restore Issues Homepage)
 
(48 intermediate revisions by 2 users not shown)
Line 1: Line 1:
{| width="100%" border="0"
+
{{BC|Zeta Alliance}}                        <!-- Note, this will also add [[Category: Zeta Alliance]] to bottom of wiki page. -->
|  bgcolor="orange" | [[Image:Attention.png]] - This article is NOT official Zimbra documentation. It is a user contribution and may include unsupported customizations, references, suggestions, or information.
+
__FORCETOC__                              <!-- Will force a TOC regards of size of article. __NOTOC__  if no TOC is wanted. -->
|}
+
<div class="col-md-12 ibox-content">
 
+
=Ajcody Backup & Restore Issues=             <!-- Normally will reflect page title. Is listed at very top of page. -->
==Backup & Restore Issues==
+
{{KB|{{ZETA}}|{{ZCS 8.5}}|{{ZCS 8.0}}|{{ZCS 7.0}}|}}            <!-- Can only handle 3 ZCS versions. -->
 +
{{WIP}}                                                <!-- For pages that are "work in progress". -->
  
 
===Actual Backup & Restore Issues Homepage===
 
===Actual Backup & Restore Issues Homepage===
  
 
Please see [[Ajcody-Backup-Restore-Issues]]
 
Please see [[Ajcody-Backup-Restore-Issues]]
 +
 +
====Update - May, 2017====
 +
 +
I'll be updating this page after the ZCS 8.8 release is done to only reflect what is true in that version and future versions. It will also include information about the [[Zimbra_Suite_Plus]] modules where appropriate. If you currently are using the backup module for ZSP, please see [[Zimbra_Suite_Plus/Zimbra_Backup_Plus]] for now.
 +
 +
====Backup-Restore Training Material Rough Drafts I Wrote====
 +
 +
* [[Ajcody-Troubleshooting-Recover_Missing_Data_-_Server]]
 +
* [[Ajcody-Troubleshooting-Recover_Missing_Data_-_User]]
 +
 +
====Bug/RFE's I Filed Against ZCS 8.6====
 +
 +
* [story] Ability to search data within backup and do "item" restores or identify locations of search results
 +
** https://bugzilla.zimbra.com/show_bug.cgi?id=97551
 +
* admin console backup label view doesn't list accounts in the all accounts tab
 +
** https://bugzilla.zimbra.com/show_bug.cgi?id=97555
 +
* admin console restore - doesn't autocomplete / suggest account matches when filling out email address box
 +
** https://bugzilla.zimbra.com/show_bug.cgi?id=97556
 +
* document new restore functions / options with ZCS 8+ for admin console restore
 +
** https://bugzilla.zimbra.com/show_bug.cgi?id=97558
 +
* admin console restore - rename "Selected Servers" panel to "Restore Options"
 +
** https://bugzilla.zimbra.com/show_bug.cgi?id=97559
 +
*admin console restore - if only one mailstore in env. then state such in second panel of restore about "server for the restored accounts"
 +
** https://bugzilla.zimbra.com/show_bug.cgi?id=97560
 +
*admin console restore - expand restore To options - To full backup label, To incremental target
 +
** https://bugzilla.zimbra.com/show_bug.cgi?id=97561
 +
*admin console restore - "restore to the latest backup" incorrectly described / broken
 +
** https://bugzilla.zimbra.com/show_bug.cgi?id=97562
 +
* admin console restore - unable to restore individual accounts [sort of]
 +
** https://bugzilla.zimbra.com/show_bug.cgi?id=97564
 +
* admin console restore - reuse GAL/Contact Picker Window for "restore individual accounts"
 +
** https://bugzilla.zimbra.com/show_bug.cgi?id=97565
  
 
====Miscellaneous Bugs & RFE's====
 
====Miscellaneous Bugs & RFE's====
Line 17: Line 50:
 
*** "add backup/restore for HSM only" [marked as duplicate of above]
 
*** "add backup/restore for HSM only" [marked as duplicate of above]
 
**** http://bugzilla.zimbra.com/show_bug.cgi?id=28200
 
**** http://bugzilla.zimbra.com/show_bug.cgi?id=28200
 +
 +
=====RFE For Live DR Restore Option=====
 +
 +
* zmrestore / zmrestoreoffline have options to --skipDeleteMailboxes - Provides Live DR option
 +
** https://bugzilla.zimbra.com/show_bug.cgi?id=94368
  
 
=====Backups And Upgrades And Prior Versions=====
 
=====Backups And Upgrades And Prior Versions=====
Line 41: Line 79:
 
======Trend Data======
 
======Trend Data======
  
If there is concerns about disks/partitions getting full, this command would be helpful for trending data on your server. Send support the resulting df.tar file .
+
If there is concerns about disks/partitions getting full, this command would be helpful for trending data on your server. Send support the resulting df.tar file . Note - adjust the tail command if you want more than 20 day's worth of trending data, the -n 20 option.
  
 
<pre>
 
<pre>
[zimbra@zcs806 zmstat]$ tar cvf /tmp/df.tar `find /opt/zimbra/zmstat -name df.csv\* -print | head -n 20`
+
[zimbra@zcs806 tmp]$ /tmp
 +
 
 +
[zimbra@zcs806 tmp]$ tar cvf /tmp/df.tar `find /opt/zimbra/zmstat -name df.cs\* | sort | tail -n 20`
 
tar: Removing leading `/' from member names
 
tar: Removing leading `/' from member names
 +
/opt/zimbra/zmstat/2014-03-10/df.csv.gz
 +
/opt/zimbra/zmstat/2014-03-11/df.csv.gz
 +
[cut - Ajcody]
 +
/opt/zimbra/zmstat/2014-03-27/df.csv.gz
 +
/opt/zimbra/zmstat/2014-03-28/df.csv.gz
 
/opt/zimbra/zmstat/df.csv
 
/opt/zimbra/zmstat/df.csv
/opt/zimbra/zmstat/2014-02-23/df.csv.gz
 
/opt/zimbra/zmstat/2014-03-27/df.csv.gz
 
[cut - Ajcody]
 
/opt/zimbra/zmstat/2014-02-10/df.csv.gz
 
/opt/zimbra/zmstat/2014-03-06/df.csv.gz
 
  
[zimbra@zcs806 zmstat]$ ls -lah /tmp/df.tar
+
[zimbra@zcs806 tmp]$ ls -lah /tmp/df.tar
-rw-r----- 1 zimbra zimbra 70K Mar 29 06:07 /tmp/df.tar
+
-rw-r----- 1 zimbra zimbra 80K Mar 29 06:44 /tmp/df.tar
  
[zimbra@zcs806 zmstat]$ tar tvf /tmp/df.tar
+
[zimbra@zcs806 tmp]$ tar tvf /tmp/df.tar
-rw-r----- zimbra/zimbra  7237 2014-03-29 06:00 opt/zimbra/zmstat/df.csv
+
-rw-r----- zimbra/zimbra  2566 2014-03-11 00:00 opt/zimbra/zmstat/2014-03-10/df.csv.gz
-rw-r----- zimbra/zimbra  2573 2014-02-24 00:00 opt/zimbra/zmstat/2014-02-23/df.csv.gz
+
-rw-r----- zimbra/zimbra  2553 2014-03-12 00:00 opt/zimbra/zmstat/2014-03-11/df.csv.gz
 +
[cut - Ajcody]
 
-rw-r----- zimbra/zimbra  2513 2014-03-28 00:00 opt/zimbra/zmstat/2014-03-27/df.csv.gz
 
-rw-r----- zimbra/zimbra  2513 2014-03-28 00:00 opt/zimbra/zmstat/2014-03-27/df.csv.gz
[cut - Ajcody]
+
-rw-r----- zimbra/zimbra  2531 2014-03-29 00:00 opt/zimbra/zmstat/2014-03-28/df.csv.gz
-rw-r----- zimbra/zimbra  2491 2014-02-11 00:00 opt/zimbra/zmstat/2014-02-10/df.csv.gz
+
-rw-r----- zimbra/zimbra  8013 2014-03-29 06:40 opt/zimbra/zmstat/df.csv
-rw-r----- zimbra/zimbra  2576 2014-03-07 00:00 opt/zimbra/zmstat/2014-03-06/df.csv.gz
 
 
</pre>
 
</pre>
  
Line 71: Line 111:
 
  * [[Ajcody-Server-Misc-Topics#Faster_Way_To_Get_Directory_Size_On_Filesytem_-_find_vs_du]]
 
  * [[Ajcody-Server-Misc-Topics#Faster_Way_To_Get_Directory_Size_On_Filesytem_-_find_vs_du]]
  
=====The Basic Information Support Needs=====
+
======Adjusting The Disk Alert Threshold======
 +
 
 +
'''Note''' - zmlocalconfig smtp_notify  must return yes if you want to receive the notifications.
 +
 
 +
If you just need to adjust the disk alert threshold, then see the following:
  
as root:
+
See current values:
  
* cat /etc/fstab
+
  zmlocalconfig | grep zmdisklog
** Shows us what is mounted upon boot
 
* cat /proc/mounts
 
** Shows us what is currently mounted and its status - you can see if a mount is read-only here.
 
* df -hT
 
** Lists current mounts using human-readable size information and also notes the filesystem type.
 
  
as zimbra:
+
Example adjustment:
  
* zmprov -l gs `zmhostname` | egrep 'Back|Redo'
+
<pre>
** Will show us a number of variables related to backup and redologs. Also tell us if your using auto-group or the default method.
+
su - zimbra
* du -sh /opt/zimbra/redolog
+
zmlocalconfig -e zmdisklog_critical_threshold=98
** Will might notice your redolog logs aren't rolling over, causing a possible issue.
+
zmlocalconfig -e zmdisklog_warn_threshold=95
* ls -latr /opt/zimbra/backup
+
zmstatctl
** This is the default backup target, please adjust this path here and below if you are using a different zimbraBackupTarget value.
+
</pre>
** zmprov gs `zmhostname` zimbraBackupTarget
 
** We'll be able to confirm permissions are right.
 
* ls -latr /opt/zimbra/backup/tmp
 
** This will show us if you have failed backup jobs and confirm tmp is being cleaned appropriately after the backup is done.
 
* ls -latr /opt/zimbra/backup/sessions
 
** This will show us what backup sessions are available and confirm permissions are correct.
 
** Adjust path if your zimbraBackupTarget value is not the default path.
 
* zmbackupquery
 
** This should match what's in the sessions directory and it will also tell us if status of each backup and how many accts were done.
 
* crontab -l | grep -i back
 
** This will show use when backups are support to run and with what options they are running with.
 
* zmlocalconfig | grep -i back
 
** This is useful to see a number of backup options not exposed in the crontab, things related to the zip options.
 
* zmvolume -l
 
** This is useful to see how many volumes are being used, if HSM is being used, and if compression is being done at the volume level.
 
  
=====Additional Log Files Support Might Need=====
+
To exclude a partition from the checks [example of two being excluded]:
  
And send the following logs:
+
<pre>
 +
su - zimbra
 +
zmlocalconfig -e zmstat_df_excludes="/mount/point:/mount/point2"
 +
zmstatctl
 +
</pre>
  
* /var/log/messages
+
They might be a bug on this, where you'll keep getting email until a logrotate happens [zimbra.log?].
** Filesystem issues often times are noted here and also in syslog. This might explain an interruption in the backup process. Server restarts, filesystem going full, filesystem going read-only, etc.
 
* /var/log/syslog
 
* /opt/zimbra/log/mailbox.log
 
** The backup activity is logged here.
 
**  And any other mailbox.log file that would cover the event
 
  
====Additional Checks For Performance Specific Issues====
+
* Changing Zmstat-df values do not take affect until logrotate
 +
** https://bugzilla.zimbra.com/show_bug.cgi?id=79148
  
=====If Your Using a SAN or NFS For Your Backup Target - Please Check Your IOWait=====
+
Some things to do to confirm and share with support or in bug. As zimbra
  
Ideally, you would compare iowait and performance data from the target backup host as well as the stats available on the ZCS servers. To get graphs and stats on this from ZCS, please see [[Ajcody-Testing-Debugging#zmstat_and_zmstat-chart]] . You should submit this data and iowait conclusions if you still need to submit a support case about backup performance issues.
+
<pre>
 +
su - zimbra
 +
 
 +
ls -la /var/log/zimbra.log
 +
 
 +
df -h
 +
/dev/mapper/vg_rhel664-lv_root
 +
                      5.5G  3.5G  1.7G  68% /
 +
tmpfs                939M    0  939M  0% /dev/shm
 +
/dev/sda1            485M  79M  381M  18% /boot
 +
/dev/sdb1              30G  6.2G  23G  22% /opt
 +
 
 +
date
  
=====Is HSM Running During Your Backup Window=====
+
zmlocalconfig | grep zmdisklog
* Are you running HSM? HSM should not be ran during your backup window.
+
zmdisklog_critical_threshold = 80
** "RFE: HSM and backup should not run at the same time if initated."
+
zmdisklog_warn_threshold = 85 
*** https://bugzilla.zimbra.com/show_bug.cgi?id=33452
 
  
=====Are You Using --zipStore=====
+
zmlocalconfig -e zmdisklog_critical_threshold=95
  
--zipStore zips the blobs vs. keeping the blobs as individual files. --zipStore does not use compression either. For most circumstances, this will give the best performance, especially with NFS. This should be the default behavior of the backups, the following RFE is when it became the default [ZCS6+] :
+
zmlocalconfig -e zmdisklog_warn_threshold=90
  
* "backup: default to the zip option"
+
zmlocalconfig | grep zmdisklog
** https://bugzilla.zimbra.com/show_bug.cgi?id=31836#c6
+
zmdisklog_critical_threshold = 95
*** Link to comment that explains options and default behavior.
+
zmdisklog_warn_threshold = 90
  
To see if zip's are being used for backups for example, in the backup/session directory you'll know if it is by seeing .zip files:
+
zmstatctl restart
  
mail:~/backup/sessions/full-20080820.160003.770/accounts/115/988/11598896-a89b-4b9d-bedb-1ed1afcb6c87/blobs
+
date
zimbra$ ls blobs-1.zip    blobs-2.zip    blobs-3.zip    blobs-4.zip
 
  
To see if the zip file is using compression [-Z option for unzip will indicate whether or not the archive is actually compressed] :
+
ps -eaf | grep zmstat-df
  
unzip -Z blobs-4.zip
+
ls -la /var/log/zimbra.log
293 files, 5982984 bytes uncompressed, 5982984 bytes compressed:  0.0%
 
  
Also, if your zmvolume has compression enabled the blobs will remain compressed within the zip also upon backup. The point being, they are uncompressed to be then put into a zip file when the backup is using --zipStore.
+
date ; grep "Disk warning" /var/log/zimbra* ; zmmailbox -z -m admin@`zmhostname` s -l 100 -t message "Subject: Disk and after:yesterday"
  
===Restore Compatibility Between ZCS Versions===
+
##Note - Emails by default go out every 10 minutes - for example:
  
Please see the following for details. In summary, user level restores should work against older ZCS backup data.
+
[zimbra@zcs803 ~]$ date ; grep "Disk warning" /var/log/zimbra* ; zmmailbox -z -m admin@`zmhostname` s -l 100 -t message "Subject: Disk and after:yesterday"
 +
Thu May 22 09:40:08 PDT 2014
 +
/var/log/zimbra.log:May 22 08:30:00 zcs803 zimbramon[18826]: 18826:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82%
 +
/var/log/zimbra.log:May 22 08:40:00 zcs803 zimbramon[22970]: 22970:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82%
 +
/var/log/zimbra.log:May 22 08:50:00 zcs803 zimbramon[22970]: 22970:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82%
 +
/var/log/zimbra.log:May 22 09:00:00 zcs803 zimbramon[22970]: 22970:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82%
 +
  ## Note - I had readjusted the variable to not warn during this time segment ##
 +
/var/log/zimbra.log:May 22 09:20:00 zcs803 zimbramon[8322]: 8322:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82%
 +
/var/log/zimbra.log:May 22 09:30:00 zcs803 zimbramon[8322]: 8322:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82%
 +
/var/log/zimbra.log:May 22 09:40:00 zcs803 zimbramon[8322]: 8322:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82%
 +
num: 7, more: false
  
* "support for restore across major versions"
+
    Id  Type  From                  Subject                                            Date
** http://bugzilla.zimbra.com/show_bug.cgi?id=15750#c18
+
  ----  ----  --------------------  --------------------------------------------------  --------------
 +
1.  328  mess  admin                Disk / at 82% on zcs803.DOMAIN.com:           05/22/14 09:40
 +
2. 327  mess  admin                Disk / at 82% on zcs803.DOMAIN.com:          05/22/14 09:30
 +
3.  326  mess  admin                Disk / at 82% on zcs803.DOMAIN.com:          05/22/14 09:20
 +
  ## Note - I had readjusted the variable to not warn during this time segment ##
 +
4.  325  mess  admin                Disk / at 82% on zcs803.DOMAIN.com:          05/22/14 09:00
 +
5.  324  mess  admin                Disk / at 82% on zcs803.DOMAIN.com:          05/22/14 08:50
 +
6.  323  mess  admin                Disk / at 82% on zcs803.DOMAIN.com:          05/22/14 08:40
 +
7.  320  mess  admin                Disk / at 82% on zcs803.DOMAIN.com:          05/22/14 08:31
 +
</pre>
  
Other related bug/rfe's:
+
Continue to monitor your zmmailbox search results for an hour.
  
* support for restore across major versions
+
=====The Basic Information Support Needs=====
** http://bugzilla.zimbra.com/show_bug.cgi?id=15750
 
* Restore across multiple versions
 
** http://bugzilla.zimbra.com/show_bug.cgi?id=25761
 
* Restore across multiple versions
 
** http://bugzilla.zimbra.com/show_bug.cgi?id=25764
 
* Backups must be compatible across patch releases
 
** http://bugzilla.zimbra.com/show_bug.cgi?id=16239
 
* write ZCS version into backup
 
** http://bugzilla.zimbra.com/show_bug.cgi?id=27819
 
* Restore should deal with database schema changes that add columns with defined default value
 
** http://bugzilla.zimbra.com/show_bug.cgi?id=26804
 
* Incorrect upgrade documentation regarding backups
 
** http://bugzilla.zimbra.com/show_bug.cgi?id=30218
 
  
===NFS Use For Backups===
+
as root:
  
Please see this RFE:
+
* cat /etc/fstab
 +
** Shows us what is mounted upon boot
 +
* cat /proc/mounts
 +
** Shows us what is currently mounted and its status - you can see if a mount is read-only here.
 +
* df -hT
 +
** Lists current mounts using human-readable size information and also notes the filesystem type.
  
* "Need clarity on supporting nfs mounted zimbra directories - report error/msg if nfs mount is present"
+
as zimbra:
** http://bugzilla.zimbra.com/show_bug.cgi?id=33221
 
  
This is the purposed statement to be included in the release notes following the RFE:
+
* zmprov -l gs `zmhostname` | egrep 'Back|Redo'
 
+
** Will show us a number of variables related to backup and redologs. Also tell us if your using auto-group or the default method.
: ZCS & NFS:
+
* du -sh /opt/zimbra/redolog
: Zimbra will support customers that store backups (e.g. /opt/zimbra/backup) on an NFS-mounted partition. Please note that this does not relieve the customer from the responsibility of providing a storage system with a performance level appropriate to their desired backup and restore times. In our experience, network-based storage access is more likely to encounter latency or disconnects than is equivalent storage attached by fiber channel or direct SCSI.
+
** Will might notice your redolog logs aren't rolling over, causing a possible issue.
 
+
* ls -latr /opt/zimbra/backup
:Zimbra continues to view NFS storage of other parts of the system as unsupported.  Our testing has shown poor read and write performance for small files over NFS implementations, and as such we view it unlikely that this policy will change for the index and database stores. We will continue to evaluate support for NFS for the message store as customer demand warrants.
+
** This is the default backup target, please adjust this path here and below if you are using a different zimbraBackupTarget value.
 +
** zmprov gs `zmhostname` zimbraBackupTarget
 +
** We'll be able to confirm permissions are right.
 +
* ls -latr /opt/zimbra/backup/tmp
 +
** This will show us if you have failed backup jobs and confirm tmp is being cleaned appropriately after the backup is done.
 +
* ls -latr /opt/zimbra/backup/sessions
 +
** This will show us what backup sessions are available and confirm permissions are correct.
 +
** Adjust path if your zimbraBackupTarget value is not the default path.
 +
*** su - zimbra
 +
*** zmjava com.zimbra.cs.backup.util.GetVersion
 +
*** cd /opt/zimbra/backup/sessions/full-[YOUR MOST RECENT FULL]/
 +
*** head -n6 session.xml
 +
*** cd /opt/zimbra/backup/sessions/incr-[YOUR MOST RECENT INCREMENTAL]/
 +
*** head -n6 session.xml
 +
* Some directory sizes in the backup directory:
 +
** Default path first
 +
*** du -sh `find /opt/zimbra/backup -maxdepth 2 -type d`
 +
** If your using a different backup target, check that directory also. Replace /opt/zimbra/backup above with your backup path.
 +
* zmbackupquery
 +
** This should match what's in the sessions directory and it will also tell us if status of each backup and how many accts were done.
 +
* crontab -l | grep -i back
 +
** This will show use when backups are support to run and with what options they are running with.
 +
* zmlocalconfig | grep -i back
 +
** This is useful to see a number of backup options not exposed in the crontab, things related to the zip options.
 +
* zmvolume -l
 +
** This is useful to see how many volumes are being used, if HSM is being used, and if compression is being done at the volume level.
  
:When working with Zimbra Support on related issues, the customer must please disclose that the backup storage used is NFS.
+
=====Additional Log Files Support Might Need=====
  
====Things To Check====
+
And send the following logs:
  
* Check the /var/log/messages on both the zimbra server and the nfs server for nfs related errors during the time frame of your backup.
+
* /var/log/messages
* Check /opt/zimbra/log/mailbox.log for error messages about folders/files not being able to be written or missing directory errors.
+
** Filesystem issues often times are noted here and also in syslog. This might explain an interruption in the backup process. Server restarts, filesystem going full, filesystem going read-only, etc.
* Is root_squash configured on the nfs server? If it's changed to no_root_squash , does the behavior of the backup change?
+
* /var/log/syslog
* Is the */backup directory owned by zimbra:zimbra with at least 750 or 755 permissions?
+
* /opt/zimbra/log/mailbox.log
** This parent directory as given in:
+
** The backup activity is logged here.
*** zmprov gs `zmhostname` zimbraBackupTarget
+
**  And any other mailbox.log file that would cover the event
* Does zimbraBackupTarget have at least the subdirectories of : sessions and tmp  : and are owned by zimbra:zimbra with 750 or 755 permissions?
 
** If not, try manually creating them and then running a test backup.
 
  
====Debugging Example====
+
====Additional Checks For Performance Specific Issues====
  
Steps I wrote for one customer, where saving out the information as you walk through all the commands would give enough information [hopefully] to submit a good rfe/bug:
+
=====If Your Using a SAN or NFS For Your Backup Target - Please Check Your IOWait=====
  
<pre>
+
Ideally, you would compare iowait and performance data from the target backup host as well as the stats available on the ZCS servers. To get graphs and stats on this from ZCS, please see [[Ajcody-Testing-Debugging#zmstat_and_zmstat-chart]] . You should submit this data and iowait conclusions if you still need to submit a support case about backup performance issues.
1. make a test partition on nfs server - /nfs-test
 
  
2. mount on zimbra server
+
=====Is HSM Running During Your Backup Window=====
2A. mkdir /nfs-test
+
* Are you running HSM? HSM should not be ran during your backup window.
2B. chmod 755 /nfs-test
+
** "RFE: HSM and backup should not run at the same time if initated."
2C. mount nfs-server:/nfs-test /nfs-test
+
*** https://bugzilla.zimbra.com/show_bug.cgi?id=33452
2D. ls -la /nfs-test
 
2E. mkdir /nfs-test/backup
 
2F. chown zimbra:zimbra /nfs-test/backup
 
2G. chmod 755 /nfs-test/backup
 
2H. su - zimbra ; touch /nfs-test/backup/testfile
 
2I. ls -laR /nfs-test/
 
2J. rm /nfs-test/backup/testfile
 
  
3. Set zimbraBackupTarget
+
=====Are You Using --zipStore=====
3A. zmprov ms `zmhostname` zimbraBackupTarget /nfs-test/backup
 
  
4. Run a full backup against one account
+
--zipStore zips the blobs vs. keeping the blobs as individual files. --zipStore does not use compression either. For most circumstances, this will give the best performance, especially with NFS. This should be the default behavior of the backups, the following RFE is when it became the default [ZCS6+] :
4A. ex. zmbackup -f -a user@domain.com
 
  
  5. ls -laR /nfs-test/
+
* "backup: default to the zip option"
 +
** https://bugzilla.zimbra.com/show_bug.cgi?id=31836#c6
 +
*** Link to comment that explains options and default behavior.
 +
 
 +
To see if zip's are being used for backups for example, in the backup/session directory you'll know if it is by seeing .zip files:
 +
 
 +
  mail:~/backup/sessions/full-20080820.160003.770/accounts/115/988/11598896-a89b-4b9d-bedb-1ed1afcb6c87/blobs
 +
zimbra$ ls blobs-1.zip    blobs-2.zip    blobs-3.zip    blobs-4.zip
 +
 
 +
To see if the zip file is using compression [-Z option for unzip will indicate whether or not the archive is actually compressed] :
  
  6. If you again, run into the same problem. You could also repeat the backup after increasing the backup
+
  unzip -Z blobs-4.zip
    logging variable for the account your trying to backup. If you didn't run into the same problem, it might
+
293 files, 5982984 bytes uncompressed, 5982984 bytes compressed: 0.0%
    had to do with the initial setup of the nfs mount and permissions being used during the directory creation.
 
  6A. zmprov aal user@domain.com zimbra.backup debug
 
6B. logging will show up in /opt/zimbra/log/mailbox.log
 
6C. Remove account logging when your done.  zmprov ral user@domain.com zimbra.backup
 
  
8. Change zimbraBackupTarget back to your production path.
+
Also, if your zmvolume has compression enabled the blobs will remain compressed within the zip also upon backup. The point being, they are uncompressed to be then put into a zip file when the backup is using --zipStore.
</pre>
 
  
====Setup A Fast Test NOT Using NFS====
+
===Restore Compatibility Between ZCS Versions===
  
A way to "avoid" the NFS issues for testing purposes would be to setup a new zimbraBackupTarget to try doing a full backup of a couple of user accounts. I DON'T recommend this if your using auto-group for zimbraBackupMode [ zmprov gs `zmhostname` zimbraBackupMode ] , only if your using Standard mode.
+
Please see the following for details. In summary, user level restores should work against older ZCS backup data.
  
<pre>
+
* "support for restore across major versions"
[as root]
+
** http://bugzilla.zimbra.com/show_bug.cgi?id=15750#c18
** adjust your new "backup" directory path as needed - mine is just an example**
 
mkdir /mnt/usb1/backup-test
 
chown zimbra:zimbra /mnt/usb1/backup-test
 
chmod 750 /mnt/usb1/backup-test
 
su - zimbra
 
**confirm backup mode as standard**
 
zmprov gs `zmhostname` zimbraBackupMode
 
**if not stand, please stop**
 
zmprov ms `zmhostname` zimbraBackupTarget /mnt/usb1/backup-test
 
**Find a couple of test accounts to do a full for. Make sure you'll have space to do the backup regards to need free space.**
 
zmbackup -f -a user1@domain.com user2@domain.com user3@domain.com
 
**Watch and confirm status of backup you just started.**
 
zmbackupquery
 
**Confirm files were backed up in right location**
 
ls /mnt/usb1/backup-test/sessions/
 
**Failed backups would most likely results in left over directory in tmp directory**
 
ls /mnt/usb1/backup-test/tmp/
 
</pre>
 
  
===Restore For Disaster Recovery===
+
Other related bug/rfe's:
  
====For Full Single ZCS Server DR Restores====
+
* support for restore across major versions
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=15750
 +
* Restore across multiple versions
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=25761
 +
* Restore across multiple versions
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=25764
 +
* Backups must be compatible across patch releases
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=16239
 +
* write ZCS version into backup
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=27819
 +
* Restore should deal with database schema changes that add columns with defined default value
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=26804
 +
* Incorrect upgrade documentation regarding backups
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=30218
  
Please see [[Network_Edition_Disaster_Recovery]]
+
===NFS Use For Backups===
  
Some additional notes I have on it - [[Ajcody-Disaster-Recovery-Specific-Notes]]
+
Please see this RFE:
  
====For Multi-Server DR Restore Specifics====
+
* "Need clarity on supporting nfs mounted zimbra directories - report error/msg if nfs mount is present"
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=33221
  
Along with the above references, please see [[Ajcody-Notes-Multi-Server-Restore-DR]]
+
This is the proposed statement to be included in the release notes following the RFE:
  
====To Restore Just The LDAP Date====
+
: ZCS & NFS:
 +
: Zimbra will support customers that store backups (e.g. /opt/zimbra/backup) on an NFS-mounted partition. Please note that this does not relieve the customer from the responsibility of providing a storage system with a performance level appropriate to their desired backup and restore times. In our experience, network-based storage access is more likely to encounter latency or disconnects than is equivalent storage attached by fiber channel or direct SCSI.
  
Let's say your ldap data was 'lost/destoyed' but everything else was intact, you should look at the zmrestoreldap command.
+
:Zimbra continues to view NFS storage of other parts of the system as unsupported.  Our testing has shown poor read and write performance for small files over NFS implementations, and as such we view it unlikely that this policy will change for the index and database stores. We will continue to evaluate support for NFS for the message store as customer demand warrants.
  
This section should have more precaution and background information to handle this section.
+
:When working with Zimbra Support on related issues, the customer must please disclose that the backup storage used is NFS.
  
The basics:
+
====Things To Check====
* To find the LDAP session labels type -lbs.
 
** zmrestoreldap -lbs
 
* Restore the complete LDAP directory server
 
** zmrestoreldap -lb full20061130135236
 
* Restore LDAP data for specific accounts
 
** zmrestoreldap -lb full20061130135236 -a tac@DOMAIN.com jane@DOMAIN.com
 
  
====To Restore Just The Mysql DB====
+
* Check the /var/log/messages on both the zimbra server and the nfs server for nfs related errors during the time frame of your backup.
 +
* Check /opt/zimbra/log/mailbox.log for error messages about folders/files not being able to be written or missing directory errors.
 +
* Is root_squash configured on the nfs server? If it's changed to no_root_squash , does the behavior of the backup change?
 +
* Is the */backup directory owned by zimbra:zimbra with at least 750 or 755 permissions?
 +
** This parent directory as given in:
 +
*** zmprov gs `zmhostname` zimbraBackupTarget
 +
* Does zimbraBackupTarget have at least the subdirectories of :  sessions and tmp  : and are owned by zimbra:zimbra with 750 or 755 permissions?
 +
** If not, try manually creating them and then running a test backup.
 +
* '''IF USING A NAS - MAKE SURE YOUR NOT USING EXTENDED ACLS OR THAT YOU HAVE THEM CONFIGURED PROPERLY'''
 +
** If your backup session directory shows something like : drwxrwx---+  2 zimbra zimbra  4096 Sep 14 00:00 TO_DELETE-full-XXXXXX , that '''+''' sign indicated extended acls are in use.
  
Option available from the following RFE work:
+
====Debugging Example====
  
* "Allow backup of only primary message volume"
+
Steps I wrote for one customer, where saving out the information as you walk through all the commands would give enough information [hopefully] to submit a good rfe/bug:
** https://bugzilla.zimbra.com/show_bug.cgi?id=35278
 
*** Options '''''to exclude types of data''''' as described in the 6.0.6 Admin Guide:
 
**** '''Search index''' : If you do not restore the search index data, the mailbox will have to be reindexed after the restore.
 
***** zmrestore <all or account> --exclude-search-index
 
**** '''Blobs''' : This is a useful option when all blobs for the mailbox being restored already exists.
 
***** zmrestore <all or account>|--exclude-blobs
 
**** '''HSM-blobs''' : This is useful when all HSM blobs for the mailbox being restored already exists.
 
***** zmrestore <all or account> --exclude-hsm-blobs
 
  
Let's say your mysql data was 'lost/destoyed' but everything else was intact. This might be the solution for the situation:
+
<pre>
 +
1. make a test partition on nfs server - /nfs-test
  
The steps below are to REPRODUCE A DR situation to test.
+
2. mount on zimbra server
# zmcontrol stop
+
2A. mkdir /nfs-test
# '''Caution''' - step to reproduce DR situation to test against
+
2B. chmod 755 /nfs-test
## mv ~/db/data ~/db/data.OLD
+
2C. mount nfs-server:/nfs-test /nfs-test
# Getting old mysql passwords
+
2D. ls -la /nfs-test
#* zmlocalconfig -s mysql_root_password
+
2E. mkdir /nfs-test/backup
#* zmlocalconfig -s zimbra_mysql_password
+
2F. chown zimbra:zimbra /nfs-test/backup
# [as zimbra] /opt/zimbra/libexec/zmmyinit
+
2G. chmod 755 /nfs-test/backup
#* Most likely your mysql passwords will change.
+
2H. su - zimbra ; touch /nfs-test/backup/testfile
#* See the following article to set them back:
+
2I. ls -laR /nfs-test/
#** [[Resetting_LDAP_%26_MySQL_Passwords]]
+
2J. rm /nfs-test/backup/testfile
#** [[Issues_with_mysql_and_logmysql_passwords#For_Mysql_Database]]
 
# ldap start
 
# zmconvertctl start
 
# mysql is already running per the zmmyinit - mysql.server status - to check.
 
# zmrestoreoffline -a all -br --systemData --excludeSearchIndex --excludeBlobs --excludeHsmBlobs
 
#* Note - you most likely will want to use the -br or -rf option for this situation.
 
#** -br,--backedupRedologsOnly : Replays the redo logs in backup only, which excludes archived and current redo logs of the system
 
#** -rf,--restoreFullBackupOnly : Restores to last full backup only, which excludes incremental backups.
 
#* -sys,--systemData : Restores global tables and local config.
 
#** This restores the 'zimbra' mysql data - this stuff [[Ajcody-Mysql-Topics#zimbra_Database_Default_Example_for_ZCS5]]
 
#** '''Important''' - the zimbra db holds information about the volumes. This needs to be restored/existing prior to user db restoration.
 
#* Note the use of -a all , you could also do this for one account first to confirm operation is successful for your circumstances.
 
# If you used -rf or -br with your zmestoreoffline, you might also need to use zmplayredo to finish it up to get the most complete restore.
 
#* Improve zmrestore & zmrestoreoffline performance
 
#*** http://bugzilla.zimbra.com/show_bug.cgi?id=33606
 
#* [[Ajcody-Backup-Restore-Issues#zmplayredo_-_Replaying_Content_From_Any_Redolog_File]] and other 'redolog' sections on that page.
 
  
===Performance Issues And Time To Complete===
+
3. Set zimbraBackupTarget
 +
3A. zmprov ms `zmhostname` zimbraBackupTarget /nfs-test/backup
  
Please see [[Ajcody-Notes-BackupPlans#What_About_Backups.3F_I_Need_A_Plan]]
+
4. Run a full backup against one account
 +
4A. ex. zmbackup -f -a user@domain.com
  
Also created an RFE for increase backup performance:
+
5. ls -laR /nfs-test/
  
* "Improve zmrestore & zmrestoreoffline performance"
+
6. If you again, run into the same problem. You could also repeat the backup after increasing the backup
** http://bugzilla.zimbra.com/show_bug.cgi?id=33606
+
    logging variable for the account your trying to backup. If you didn't run into the same problem, it might
* "Improvement to backup performance"
+
    had to do with the initial setup of the nfs mount and permissions being used during the directory creation.
** http://bugzilla.zimbra.com/show_bug.cgi?id=36220
+
6A. zmprov aal user@domain.com zimbra.backup debug
** Above marked as a dupe of "Reduce duration of maintenance mode during backup"
+
6B. logging will show up in /opt/zimbra/log/mailbox.log
*** http://bugzilla.zimbra.com/show_bug.cgi?id=33583
+
6C. Remove account logging when your done. zmprov ral user@domain.com zimbra.backup
  
===Understanding Option Flags For zmbackup & zmrestore===
+
8. Change zimbraBackupTarget back to your production path.
 +
</pre>
  
First, they don't make sense if your just reading from the help output - I will not argue this point at all.
+
====Setup A Fast Test NOT Using NFS====
  
The biggest problem with the options I point out below is that you can often include them in the command and they do nothing or you include them for a particular situation and they don't apply. Why is this a problem? Because they are silent and give no output telling you that it's not necessary, it's redundant, or it will actually cause your intended results to fail simply because you included the option.
+
A way to "avoid" the NFS issues for testing purposes would be to setup a new zimbraBackupTarget to try doing a full backup of a couple of user accounts. I DON'T recommend this if your using auto-group for zimbraBackupMode [ zmprov gs `zmhostname` zimbraBackupMode ] , only if your using Standard mode.
  
====zmrestore Options====
+
<pre>
 
+
[as root]
Problems mostly revolve around these options.
+
** adjust your new "backup" directory path as needed - mine is just an example**
 
+
mkdir /mnt/usb1/backup-test
=====To Times In The Past (If -lb Isn't Used, Implies Your Using Times/Incr/RedoSeq AFTER Last Full)=====
+
chown zimbra:zimbra /mnt/usb1/backup-test
 
+
chmod 750 /mnt/usb1/backup-test
*-restoreToIncrLabel
+
su - zimbra
**<arg> Replay redo logs up to and including this incremental backup
+
**confirm backup mode as standard**
*** Requires: --label or -lb
+
zmprov gs `zmhostname` zimbraBackupMode
*-restoreToTime
+
**if not stand, please stop**
**<arg> Replay rodo logs until this time
+
zmprov ms `zmhostname` zimbraBackupTarget /mnt/usb1/backup-test
*** Requires: --label or -lb
+
**Find a couple of test accounts to do a full for. Make sure you'll have space to do the backup regards to need free space.**
*-restoreToRedoSeq
+
zmbackup -f -a user1@domain.com user2@domain.com user3@domain.com
**<arg> Replay up to and including this redo log sequence
+
**Watch and confirm status of backup you just started.**
 +
zmbackupquery
 +
**Confirm files were backed up in right location**
 +
ls /mnt/usb1/backup-test/sessions/
 +
**Failed backups would most likely results in left over directory in tmp directory**
 +
ls /mnt/usb1/backup-test/tmp/
 +
</pre>
 +
 
 +
===Restore For Disaster Recovery===
 +
 
 +
====For Full Single ZCS Server DR Restores====
 +
 
 +
Please see [[Network_Edition_Disaster_Recovery]]
  
=====Redolog Variables=====
+
Some additional notes I have on it - [[Ajcody-Disaster-Recovery-Specific-Notes]]
  
*--backedupRedologs or -br
+
====For Multi-Server DR Restore Specifics====
**Replays the redo logs in backup only, which excludes archived and current redo logs of the system
+
 
*** Only useful when restoring against latest full backup (NOT using the -lb option).
+
Along with the above references, please see [[Ajcody-Notes-Multi-Server-Restore-DR]]
*** Will restore using incremental backup data after the last full backup as well but NOT including any redolog activity.
 
*--restorefullBackup or -rf
 
**Restores to the full backup only, not any incremental backups since that backup.
 
*** The default behavior of zmrestore in general is to always play from a "full" and "incrementals" that are associated with it UNLESS you state otherwise.
 
**** If you do: <pre>zmrestore -a user@domain.com -lb full6monthsago</pre>
 
**** It will playback the incremental data associated with that full from 6 months ago.
 
**** If you do: <pre>zmrestore -a user@domain.com -lb full6monthsago -rf</pre>
 
**** It will ONLY playback the data in the full from 6 months ago.
 
*** Will not progress past the data that's in the last full backup, no incremental backups after it in other words.
 
*** Implies NO redolog play, so there's no need to use -br.
 
  
=====Targets And Labels=====
+
====To Restore Just The LDAP Date====
  
*--label or -lb
+
Let's say your ldap data was 'lost/destoyed' but everything else was intact, you should look at the zmrestoreldap command.
**<arg> The label of the full backup to restore. Restores to the latest full backup if this is omitted.
 
*--target or -t
 
**<arg> Specifies the backup target location. The default is <zimbra_home>/backup.
 
  
=====Impact Of AutoGroup Option Being Used=====
+
This section should have more precaution and background information to handle this section.
  
Place for notes about how autogroup backup option might impact or limit command options.
+
The basics:
 +
* To find the LDAP session labels type -lbs.
 +
** zmrestoreldap -lbs
 +
* Restore the complete LDAP directory server
 +
** zmrestoreldap -lb full20061130135236
 +
* Restore LDAP data for specific accounts
 +
** zmrestoreldap -lb full20061130135236 -a tac@DOMAIN.com jane@DOMAIN.com
  
====zmbackup Options====
+
====To Restore Just The Mysql DB====
  
Problems mostly revolve around these options:
+
Option available from the following RFE work:
  
*--target or -t
+
* "Allow backup of only primary message volume"
**<arg> Specifies the target backup location. The default is <zimbra_home>/backup.
+
** https://bugzilla.zimbra.com/show_bug.cgi?id=35278
*--zip or -z
+
*** Options '''''to exclude types of data''''' as described in the 6.0.6 Admin Guide:
**Zips email blobs in backup - using compression
+
**** '''Search index''' : If you do not restore the search index data, the mailbox will have to be reindexed after the restore.
*--zipStore
+
***** zmrestore <all or account> --exclude-search-index
**Zips email blobs in backup - does '''NOT''' use compression
+
**** '''Blobs''' : This is a useful option when all blobs for the mailbox being restored already exists.
 +
***** zmrestore <all or account>|--exclude-blobs
 +
**** '''HSM-blobs''' : This is useful when all HSM blobs for the mailbox being restored already exists.
 +
***** zmrestore <all or account> --exclude-hsm-blobs
  
=====Deleting Old Backups -del=====
+
Let's say your mysql data was 'lost/destoyed' but everything else was intact. This might be the solution for the situation:
  
'''Caution''' You want to delete from the oldest label to newest. The -del option will automatically purge all older sessions prior to the label you used. To find out the label names, use zmbackupquery.
+
The steps below are to REPRODUCE A DR situation to test.
 
+
# zmcontrol stop
Format example:
+
# '''Caution''' - step to reproduce DR situation to test against
 
+
## mv ~/db/data ~/db/data.OLD
zmbackup -del <oldest_backup_label>
+
# Getting old mysql passwords
 
+
#* zmlocalconfig -s mysql_root_password
=====Impact Of AutoGroup Option Being Used=====
+
#* zmlocalconfig -s zimbra_mysql_password
 
+
# [as zimbra] /opt/zimbra/libexec/zmmyinit
Place for notes about how autogroup backup option might impact or limit command options.
+
#* Most likely your mysql passwords will change.
 +
#* See the following article to set them back:
 +
#** [[Resetting_LDAP_%26_MySQL_Passwords]]
 +
#** [[Issues_with_mysql_and_logmysql_passwords#For_Mysql_Database]]
 +
# ldap start
 +
# zmconvertctl start
 +
# mysql is already running per the zmmyinit - mysql.server status - to check.
 +
# zmrestoreoffline -a all -br --systemData --excludeSearchIndex --excludeBlobs --excludeHsmBlobs
 +
#* Note - you most likely will want to use the -br or -rf option for this situation.
 +
#** -br,--backedupRedologsOnly : Replays the redo logs in backup only, which excludes archived and current redo logs of the system
 +
#** -rf,--restoreFullBackupOnly : Restores to last full backup only, which excludes incremental backups.
 +
#* -sys,--systemData : Restores global tables and local config.
 +
#** This restores the 'zimbra' mysql data - this stuff [[Ajcody-Mysql-Topics#zimbra_Database_Default_Example_for_ZCS5]]
 +
#** '''Important''' - the zimbra db holds information about the volumes. This needs to be restored/existing prior to user db restoration.
 +
#* Note the use of -a all , you could also do this for one account first to confirm operation is successful for your circumstances.
 +
# If you used -rf or -br with your zmestoreoffline, you might also need to use zmplayredo to finish it up to get the most complete restore.
 +
#* Improve zmrestore & zmrestoreoffline performance
 +
#*** http://bugzilla.zimbra.com/show_bug.cgi?id=33606
 +
#* [[Ajcody-Backup-Restore-Issues#zmplayredo_-_Replaying_Content_From_Any_Redolog_File]] and other 'redolog' sections on that page.
  
====Changing Default Backup Target====
+
===Performance Issues And Time To Complete===
  
To find out what the current backup target is, do:
+
Please see [[Ajcody-Notes-BackupPlans#What_About_Backups.3F_I_Need_A_Plan]]
  
<pre> zmprov gacf | grep zimbraBackupTarget
+
Also created an RFE for increase backup performance:
zimbraBackupTarget: /opt/zimbra/backup</pre>
 
  
This is also configurable at the "server" level:
+
* "Improve zmrestore & zmrestoreoffline performance"
 
+
** http://bugzilla.zimbra.com/show_bug.cgi?id=33606
<pre> zmprov gs `zmhostname` | grep zimbraBackupTarget
+
* "Improvement to backup performance"
zimbraBackupTarget: /opt/zimbra/backup</pre>
+
** http://bugzilla.zimbra.com/show_bug.cgi?id=36220
 +
** Above marked as a dupe of "Reduce duration of maintenance mode during backup"
 +
*** http://bugzilla.zimbra.com/show_bug.cgi?id=33583
  
For example:
+
===Understanding Option Flags For zmbackup & zmrestore===
  
<pre> zmprov ms `zmhostname` zimbraBackupTarget /san/mount/backup</pre>
+
First, they don't make sense if your just reading from the help output - I will not argue this point at all.
  
Issues of changing the default path in regards to the admin web console. Please see:
+
The biggest problem with the options I point out below is that you can often include them in the command and they do nothing or you include them for a particular situation and they don't apply. Why is this a problem? Because they are silent and give no output telling you that it's not necessary, it's redundant, or it will actually cause your intended results to fail simply because you included the option.
* "Work backup path if it was changed in admin interface"
 
** http://bugzilla.zimbra.com/show_bug.cgi?id=28235
 
  
Another way to change the backup path is described at [[Changing_Backup_directory_and_General_Information]]. I recommend reviewing it as well.
+
====zmrestore Options====
  
See [[Ajcody-Backup-Restore-Issues#Change_.22Location.22_For_Backup_Or_Restore_Source_Data|Change Location For Backup Or Restore Source Data]] for non-default adjustments to CLI commands in regards to a non-default path for backups.
+
Problems mostly revolve around these options.
  
====RFE & Bugs Concerning Option Flags For Restore And Backup====
+
=====To Times In The Past (If -lb Isn't Used, Implies Your Using Times/Incr/RedoSeq AFTER Last Full)=====
  
=====Setting Account Status After Restore Is Done=====
+
*-restoreToIncrLabel
 +
**<arg> Replay redo logs up to and including this incremental backup
 +
*** Requires: --label or -lb
 +
*-restoreToTime
 +
**<arg> Replay rodo logs until this time
 +
*** Requires: --label or -lb
 +
*-restoreToRedoSeq
 +
**<arg> Replay up to and including this redo log sequence
  
I filed an RFE for this:
+
=====Redolog Variables=====
  
* "RFE: zmrestore & zmrestoreoffline have option to set account status"
+
*--backedupRedologs or -br
** http://bugzilla.zimbra.com/show_bug.cgi?id=52460
+
**Replays the redo logs in backup only, which excludes archived and current redo logs of the system
 +
*** Only useful when restoring against latest full backup (NOT using the -lb option).
 +
*** Will restore using incremental backup data after the last full backup as well but NOT including any redolog activity.
 +
*--restorefullBackup or -rf
 +
**Restores to the full backup only, not any incremental backups since that backup.
 +
*** The default behavior of zmrestore in general is to always play from a "full" and "incrementals" that are associated with it UNLESS you state otherwise.
 +
**** If you do: <pre>zmrestore -a user@domain.com -lb full6monthsago</pre>
 +
**** It will playback the incremental data associated with that full from 6 months ago.
 +
**** If you do: <pre>zmrestore -a user@domain.com -lb full6monthsago -rf</pre>
 +
**** It will ONLY playback the data in the full from 6 months ago.
 +
*** Will not progress past the data that's in the last full backup, no incremental backups after it in other words.
 +
*** Implies NO redolog play, so there's no need to use -br.
  
=====Use DL COS Status To Generate User List Of Accounts=====
+
=====Targets And Labels=====
  
I filed an RFE for this:
+
*--label or -lb
 +
**<arg> The label of the full backup to restore. Restores to the latest full backup if this is omitted.
 +
*--target or -t
 +
**<arg> Specifies the backup target location. The default is <zimbra_home>/backup.
  
* "RFE: backup/restore should allow -a option to use DL's COS Status"
+
=====Impact Of AutoGroup Option Being Used=====
** http://bugzilla.zimbra.com/show_bug.cgi?id=52489
 
  
===An Overview Of Some Backup/Restore Items===
+
Place for notes about how autogroup backup option might impact or limit command options.
  
====Shared Blobs (messages)====
+
====zmbackup Options====
  
I believe we are a little light on describing the shared blobs situation. Shared blobs can cause different corrections to a problem as compared to a normal message issue that isn't shared. I'll start some notes on this here.
+
Problems mostly revolve around these options:
  
From [http://www.zimbra.com/docs/ne/latest/administration_guide/10_Backup_Restore.17.01.html Backup and Restore]
+
*--target or -t
 +
**<arg> Specifies the target backup location. The default is <zimbra_home>/backup.
 +
*--zip or -z
 +
**Zips email blobs in backup - using compression
 +
*--zipStore
 +
**Zips email blobs in backup - does '''NOT''' use compression
  
*''"When backing up shared messages, the backup process looks to see whether a Binary Large Object file (BLOB) representing a message already exists in the backup. If it does, it simply flags this object as such and does not copy its content again."''
+
=====Deleting Old Backups -del=====
*''"Keeping the same backup target saves disk space, because shared binary large object files (BLOB) and other files do not have to be duplicated every time the backup process runs.''
 
  
Bugs/RFE's
+
'''Caution''' You want to delete from the oldest label to newest. The -del option will automatically purge all older sessions prior to the label you used. To find out the label names, use zmbackupquery.
  
* "Use zip files for shared blobs of a full backup made with --zip option"
+
Format example:
** http://bugzilla.zimbra.com/show_bug.cgi?id=26624
 
  
====Remote copies of backup data for DR use====
+
zmbackup -del <oldest_backup_label>
  
You will want to copy over /opt/zimbra/redolog/archive/* and /opt/zimbra/redolog/redo.log frequently in order to stay current.  The redo.log file being open is not a problem since the crash recovery step can work with redo.log file in any state.
+
=====Impact Of AutoGroup Option Being Used=====
  
The redolog/archive/ contains logs that have not yet been backed up by an incremental (or by a full in auto-grouped mode)
+
Place for notes about how autogroup backup option might impact or limit command options.
  
The redo.log rolls over when it reaches zimbraRedoLogRolloverFileSizeKB (by default 100MB prior to ZCS 5.0.11 and 1GB after). When ZCS restarts after a crash, it seems to work through the current redo.log ok regardless of its state, if the current log really must be copied."
+
====Changing Default Backup Target====
  
=====My Initial Thoughts On This=====
+
To find out what the current backup target is, do:
  
Start of process:
+
<pre> zmprov gacf | grep zimbraBackupTarget
*Weekend full on prod
+
zimbraBackupTarget: /opt/zimbra/backup</pre>
*rsync full-xxx on prod > remote sessions/
 
*rsync redolog/* > remote redolog/
 
** through non-full and incremental times every x about of minutes
 
*weekday nights 10pm incre-xxx on prod
 
*rsync incr-xxx on prod > remote sessions/
 
*rsync redlog/* > remote redolog/
 
  
Created three separate rsync cron rules.
+
This is also configurable at the "server" level:
 +
 
 +
<pre> zmprov gs `zmhostname` | grep zimbraBackupTarget
 +
zimbraBackupTarget: /opt/zimbra/backup</pre>
  
* Full - once a week
+
For example:
** Confirms full is done and then looks for latest full-xxxx and rsyncs that specific directory
 
* Incre - once a night except for full schedule night
 
** Confirms incre is done and then looks for latest incr-xxx and rsyncs that specific directory
 
* Redolog/ - every x amount of minutes (outside of full and incr backups sessions)
 
** Soes full rsync of redolog/ - probably want delete/remove option?
 
** lsof will report /opt/zimbra/redolog/redo.log is open.
 
  
Somewhere we need to account for accounts.xml in this process. And also confirm what else might be missing. Also, steps on the actually restore process depending on when/where the DR event took place.
+
<pre> zmprov ms `zmhostname` zimbraBackupTarget /san/mount/backup</pre>
  
====What's Needed For Later Restores====
+
Issues of changing the default path in regards to the admin web console. Please see:
 +
* "Work backup path if it was changed in admin interface"
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=28235
  
In regards to what is moved and what is needed for later restores, you must remember this "flow" of the backups:
+
Another way to change the backup path is described at [[Changing_Backup_directory_and_General_Information]]. I recommend reviewing it as well.
  
* Full backup files that contains all the information needed to restore mailboxes (to that point in time)
+
See [[Ajcody-Backup-Restore-Issues#Change_.22Location.22_For_Backup_Or_Restore_Source_Data|Change Location For Backup Or Restore Source Data]] for non-default adjustments to CLI commands in regards to a non-default path for backups.
* Incremental backup files that contains the LDAP directory server files and all the redo log transactions written since the last backup (to that point in time)
 
* Redo logs that contains current and archived transactions processed by the Zimbra server since the last incremental backup (to that point in time - more about this topic is above)
 
  
Variables to be aware of in regards to backup/restore [ viewed with - zmprov gs [server-name] | grep Backup ]:
+
====RFE & Bugs Concerning Option Flags For Restore And Backup====
* This is that gui path option - note, it doesn't change default to redo file path [see below]
 
<pre>
 
zimbraBackupTarget
 
  
zimbraBackupMode
+
=====Setting Account Status After Restore Is Done=====
zimbraBackupAutoGroupedInterval
 
zimbraBackupAutoGroupedNumGroups
 
zimbraBackupAutoGroupedThrottled
 
  
zimbraBackupReportEmailSubjectPrefix
+
I filed an RFE for this:
zimbraBackupReportEmailSender
 
zimbraBackupReportEmailRecipients
 
</pre>
 
* Variables to be aware of in regards to redo files [ viewed with - zmprov gs [server-name] | grep Redo ]:
 
<pre>
 
zimbraRedoLogArchiveDir
 
zimbraRedoLogDeleteOnRollover
 
zimbraRedoLogEnabled
 
zimbraRedoLogFsyncIntervalslMS
 
zimbraRedoLogLogPath
 
zimbraRedoLogRolloverFileSizeKB
 
</pre>
 
  
====Mysql Table That References Most Recent Backup Session Of Users (AutoGroup Backup Mode)====
+
* "RFE: zmrestore & zmrestoreoffline have option to set account status"
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=52460
  
During the backup we update the zimbra.mailbox table for each mailbox to record the most recent backup time.  This is in the "last_backup_at" column within Mysql. 
+
=====Use DL COS Status To Generate User List Of Accounts=====
  
This data is used by auto-grouped backup to figure out which mailboxes to backup.
+
I filed an RFE for this:
  
=====Possible Issue That A Failed Or Interrupted Backup Causes=====
+
* "RFE: backup/restore should allow -a option to use DL's COS Status"
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=52489
  
An interrupted backup can cause an issue because the table currently gets updated right off the bat rather than waiting for backup to be successfully completed.
+
===An Overview Of Some Backup/Restore Items===
  
Possible RFE: To update zimbra.mailbox.last_backup_at column for successfully backed-up mailboxes to the very end of the backup process, to either just before or just after renaming the /opt/zimbra/backup/tmp/<backup label> directory to /opt/zimbra/backup/sessions/<backup label>.
+
====Shared Blobs (messages)====
  
======Setting To Null To Cause A New Backup For User======
+
I believe we are a little light on describing the shared blobs situation. Shared blobs can cause different corrections to a problem as compared to a normal message issue that isn't shared. I'll start some notes on this here.
  
To undo what was done by an interrupted backup for example, you need to clear this column (set it to null) for the affected mailboxes. By clearing the column, you're forcing the next AG backup to choose these mailboxes because they look like they have never been backed up. If you don't clear this column, you have to wait until the next cycle. (7 days)
+
From [http://www.zimbra.com/docs/ne/latest/administration_guide/10_Backup_Restore.17.01.html Backup and Restore]
  
Example syntax to view:
+
*''"When backing up shared messages, the backup process looks to see whether a Binary Large Object file (BLOB) representing a message already exists in the backup. If it does, it simply flags this object as such and does not copy its content again."''
 +
*''"Keeping the same backup target saves disk space, because shared binary large object files (BLOB) and other files do not have to be duplicated every time the backup process runs.''
  
mysql zimbra -e "select last_backup_at from mailbox where id=27"
+
Bugs/RFE's
  
Example syntax to change data of the last_backup_at to NULL"
+
* "Use zip files for shared blobs of a full backup made with --zip option"
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=26624
  
mysql zimbra -e "UPDATE mailbox SET last_backup_at = NULL WHERE id = 27"
+
====Remote copies of backup data for DR use====
  
======Related Bugs RFE's======
+
You will want to copy over /opt/zimbra/redolog/archive/* and /opt/zimbra/redolog/redo.log frequently in order to stay current.  The redo.log file being open is not a problem since the crash recovery step can work with redo.log file in any state.
  
* "In auto-grouped backup, delay the update of mailboxes' last_backup_at timestamp to the very end of backup"
+
The redolog/archive/ contains logs that have not yet been backed up by an incremental (or by a full in auto-grouped mode)
** http://bugzilla.zimbra.com/show_bug.cgi?id=33275
 
* "Partial backup should be able to finish successfully when backup volume runs out of space"
 
** http://bugzilla.zimbra.com/show_bug.cgi?id=33276
 
  
====Restore Requires accounts.xml File====
+
The redo.log rolls over when it reaches zimbraRedoLogRolloverFileSizeKB (by default 100MB prior to ZCS 5.0.11 and 1GB after). When ZCS restarts after a crash, it seems to work through the current redo.log ok regardless of its state, if the current log really must be copied."
  
* "accounts.xml file dependency needed for zmrestore"
+
=====My Initial Thoughts On This=====
** http://bugzilla.zimbra.com/show_bug.cgi?id=30979
 
  
Also, see:
+
Start of process:
 
+
*Weekend full on prod
* "accounts.xml file improperly updated when a full backup is deleted"
+
*rsync full-xxx on prod > remote sessions/
** http://bugzilla.zimbra.com/show_bug.cgi?id=31591
+
*rsync redolog/* > remote redolog/
 +
** through non-full and incremental times every x about of minutes
 +
*weekday nights 10pm incre-xxx on prod
 +
*rsync incr-xxx on prod > remote sessions/
 +
*rsync redlog/* > remote redolog/
  
====Change "Location" For Backup Or Restore Source Data====
+
Created three separate rsync cron rules.
  
Remember that zmbackup and zmrestore can take flags as well in regards to the location of items.
+
* Full - once a week
 +
** Confirms full is done and then looks for latest full-xxxx and rsyncs that specific directory
 +
* Incre - once a night except for full schedule night
 +
** Confirms incre is done and then looks for latest incr-xxx and rsyncs that specific directory
 +
* Redolog/ - every x amount of minutes (outside of full and incr backups sessions)
 +
** Soes full rsync of redolog/ - probably want delete/remove option?
 +
** lsof will report /opt/zimbra/redolog/redo.log is open.
  
* zmrestore & zmbackup can both take : -t,--target (default <zimbra_home/backup)
+
Somewhere we need to account for accounts.xml in this process. And also confirm what else might be missing. Also, steps on the actually restore process depending on when/where the DR event took place.
  
'''''You can't state a different location with redo logs though'''''. There's a command called '''zmplayredo''' [for newer versions of ZCS] and it has a variable to point to the redologs to play from [ --logfiles ]. It will replay into the default redolog directory or redolog file. The mailbox has to be stop to run zmplayredo . This is a command to manual kick off a replay of a redo log. This is normally done with the zmrestore when options about to a specific time aren't included.
+
====What's Needed For Later Restores====
  
====Manual Removal Of Older Backup Sessions====
+
In regards to what is moved and what is needed for later restores, you must remember this "flow" of the backups:
  
General Situation:
+
* Full backup files that contains all the information needed to restore mailboxes (to that point in time)
*Keep in mind every restore requires starting with data from a full backup. 
+
* Incremental backup files that contains the LDAP directory server files and all the redo log transactions written since the last backup (to that point in time)
*For each account on the server, there must be at least 1 full backup after the deletion is complete. 
+
* Redo logs that contains current and archived transactions processed by the Zimbra server since the last incremental backup (to that point in time - more about this topic is above)
*You should also make sure all incremental backups made after the oldest of the remaining backups are retained. 
 
**This basically gets reduced to deleting only those backups that are old enough, based on your full/incremental schedule.
 
  
More Specific Issues:
+
Variables to be aware of in regards to backup/restore [ viewed with - zmprov gs [server-name] | grep Backup ]:
*Does the accounts.xml dependency cause issues with this?
+
* This is that gui path option - note, it doesn't change default to redo file path [see below]
**No.  Just don't delete it.
+
<pre>
*What about the contents of the backup/tmp/ data or shared blobs type references?
+
zimbraBackupTarget
**Don't touch this directory either.  It is used during backup and restore.  You don't want to change its content while an operation is going on, so best to leave it alone.
 
*What if a zimbra server is running some type of restore of backup command while the manual removal is running on the nfs server?
 
**You shouldn't remove the backups that are being used in a restore currently underway.  You are responsible for avoiding the race condition.
 
*** Please understand you are responsible for avoiding the race condition.  Make sure no backup or restore is happening at the moment, then rename the directories that will be deleted, preferably move them to another subdirectory, e.g. /opt/zimbra/backup/sessions_to_delete.  Then delete.
 
  
===LDAP Backup Related Items===
+
zimbraBackupMode
 +
zimbraBackupAutoGroupedInterval
 +
zimbraBackupAutoGroupedNumGroups
 +
zimbraBackupAutoGroupedThrottled
  
====Backup Schedule On LDAP Only Server [non-Mailstore]====
+
zimbraBackupReportEmailSubjectPrefix
 +
zimbraBackupReportEmailSender
 +
zimbraBackupReportEmailRecipients
 +
</pre>
 +
* Variables to be aware of in regards to redo files [ viewed with - zmprov gs [server-name] | grep Redo ]:
 +
<pre>
 +
zimbraRedoLogArchiveDir
 +
zimbraRedoLogDeleteOnRollover
 +
zimbraRedoLogEnabled
 +
zimbraRedoLogFsyncIntervalslMS
 +
zimbraRedoLogLogPath
 +
zimbraRedoLogRolloverFileSizeKB
 +
</pre>
  
If you look at the code in  /opt/zimbra/bin/zmschedulebackup  :
+
====Mysql Table That References Most Recent Backup Session Of Users (AutoGroup Backup Mode)====
  
<pre>
+
During the backup we update the zimbra.mailbox table for each mailbox to record the most recent backup time.  This is in the "last_backup_at" column within Mysql. 
if ($BACKUP_MODE eq 'Standard') {
+
 
    # default schedule: full backup 1am every sunday, incr backup 1am every weekday
+
This data is used by auto-grouped backup to figure out which mailboxes to backup.
    # deletes backups older than a month at 12am everyday
+
 
  if (isLdapOnly()) {
+
=====Creating A List Of Last Backup Of Users=====
      @default = ("0 1 * * 6 /opt/zimbra/bin/zmbackup $target\n",
 
            "0 0 * * * /opt/zimbra/bin/zmbackup -del 1m\n");
 
  } else {
 
      @default = ("0 1 * * 6 /opt/zimbra/bin/zmbackup -f $account $target $compress \n",
 
          "0 1 * * 0-5 /opt/zimbra/bin/zmbackup -i $compress\n",
 
            "0 0 * * * /opt/zimbra/bin/zmbackup -del 1m\n");
 
  }
 
    %commandmap = ("f", "/opt/zimbra/bin/zmbackup -f $target $compress $account",
 
                  "i", "/opt/zimbra/bin/zmbackup -i $compress",
 
                  "d", "/opt/zimbra/bin/zmbackup -del");
 
} else {  # Auto-Grouped mode
 
    # default schedule: full backup 1am everyday, no incr backup
 
    # deletes backups older than a month at 12am everyday
 
    @default = ("0 1 * * 0-6 /opt/zimbra/bin/zmbackup -f $target $compress\n",
 
                "0 0 * * * /opt/zimbra/bin/zmbackup -del 1m\n");
 
    %commandmap = ("f", "/opt/zimbra/bin/zmbackup -f $target $compress",
 
                  "i", "/opt/zimbra/bin/zmbackup -i $compress",
 
                  "d", "/opt/zimbra/bin/zmbackup -del");
 
}
 
</pre>
 
  
Notice the specific check and then format for ldapOnly:
+
Remove the , | head , below to get a full listing of all your accounts. Note, this reports on the users that exist on the mailstore your running the command.
  
 
<pre>
 
<pre>
  if (isLdapOnly()) {
+
[zimbra@zcs806 tmp]$ mysql zimbra -NBe 'select from_unixtime(last_backup_at), comment from mailbox' | sort | head
      @default = ("0 1 * * 6 /opt/zimbra/bin/zmbackup $target\n",
+
2014-06-05 01:00:24    dluser558@zcs806.DOMAIN.com
            "0 0 * * * /opt/zimbra/bin/zmbackup -del 1m\n");
+
2014-06-05 01:00:26    dluser557@zcs806.DOMAIN.com
 +
2014-06-05 01:00:28    dluser556@zcs806.DOMAIN.com
 +
2014-06-05 01:00:30    dluser555@zcs806.DOMAIN.com
 +
2014-06-05 01:00:31    dluser554@zcs806.DOMAIN.com
 +
2014-06-05 01:00:32    dluser551@zcs806.DOMAIN.com
 +
2014-06-05 01:00:32    dluser552@zcs806.DOMAIN.com
 +
2014-06-05 01:00:32    dluser553@zcs806.DOMAIN.com
 +
2014-06-05 01:00:33    dluser550@zcs806.DOMAIN.com
 +
2014-06-05 01:00:34    dluser549@zcs806.DOMAIN.com
 
</pre>
 
</pre>
  
====Related Bugs To LDAP Backups====
+
I include also below the results from the zmbackupquery for the first user:
  
Some bugs to be aware of - most are resolved/fixed:
+
<pre>
 +
[zimbra@zcs806 tmp]$ date ; zmbackupquery -a dluser558@zcs806.DOMAIN.com
 +
Wed Jun 11 14:29:48 PDT 2014
 +
Account: dluser558@zcs806.DOMAIN.com
  
* "Zmbackup ldap data is not backed up if target server is not hosting it."
+
    Label:   full-20140605.080023.296
** http://bugzilla.zimbra.com/show_bug.cgi?id=7420
+
    Type:   full
* "System LDAP is backuped only when -a all is specified"
+
    Started: Thu, 2014/06/05 01:00:23.296 PDT
** http://bugzilla.zimbra.com/show_bug.cgi?id=8365
+
    Ended:   Thu, 2014/06/05 01:02:37.739 PDT
* "LDAP backup directory format difference between full install and ldap only install"
+
    Acct ID: 85d3c8f4-eea8-4cf2-8a84-8f1fcc87d555
** http://bugzilla.zimbra.com/show_bug.cgi?id=9551
 
* "backups in the admin console should back up LDAP data"
 
** http://bugzilla.zimbra.com/show_bug.cgi?id=14069
 
* "LDAP backup failure due to library version mismatch"
 
** http://bugzilla.zimbra.com/show_bug.cgi?id=20275
 
* "ldap backup not kept with rest of backup"
 
** http://bugzilla.zimbra.com/show_bug.cgi?id=21753
 
* "zmbackup ldap backup is not logged"
 
** http://bugzilla.zimbra.com/show_bug.cgi?id=22303
 
* "Backup default cronjob is wrong for ldap only install"
 
** http://bugzilla.zimbra.com/show_bug.cgi?id=23513
 
* "zmbackupldap fails to rename directory on NFS"
 
** http://bugzilla.zimbra.com/show_bug.cgi?id=25451
 
* "default ldap server only backup schedule is weekly"
 
** http://bugzilla.zimbra.com/show_bug.cgi?id=25681
 
* "zmbackupldap fails when using month intervals and succeeding month has more days"
 
** http://bugzilla.zimbra.com/show_bug.cgi?id=32834
 
* "zmbackup on LDAP-only hosts should have help/usage argument, or at least reject invalid arguments"
 
** http://bugzilla.zimbra.com/show_bug.cgi?id=32878
 
* "corp: ldap backups fail"
 
** http://bugzilla.zimbra.com/show_bug.cgi?id=38558
 
  
===A Way To Verify Backup Integrity===
+
    Label:  full-20140529.080016.546
 +
    Type:    full
 +
    Started: Thu, 2014/05/29 01:00:16.546 PDT
 +
    Ended:  Thu, 2014/05/29 01:02:51.556 PDT
 +
    Acct ID: 85d3c8f4-eea8-4cf2-8a84-8f1fcc87d555
  
I filed an RFE for this:
+
    Label:   full-20140522.080015.818
 +
    Type:    full
 +
    Started: Thu, 2014/05/22 01:00:15.818 PDT
 +
    Ended:  Thu, 2014/05/22 01:02:42.126 PDT
 +
    Acct ID: 85d3c8f4-eea8-4cf2-8a84-8f1fcc87d555
  
* "A way to verify backup integrity"
+
    Label:  full-20140515.080016.160
** http://bugzilla.zimbra.com/show_bug.cgi?id=28025
+
    Type:   full
 +
    Started: Thu, 2014/05/15 01:00:16.160 PDT
 +
    Ended:  Thu, 2014/05/15 01:02:34.204 PDT
 +
    Acct ID: 85d3c8f4-eea8-4cf2-8a84-8f1fcc87d555
 +
</pre>
  
====Negative Seek Offset Error & RFE====
+
=====Possible Issue That A Failed Or Interrupted Backup Causes=====
  
Explanation of negative seek offset error during a restore attempt and manual fixes are located here:
+
An interrupted backup can cause an issue because the table currently gets updated right off the bat rather than waiting for backup to be successfully completed.
  
* "RFE: Monitor/test backup *.zip files for corruption & repair tools"
+
Possible RFE: To update zimbra.mailbox.last_backup_at column for successfully backed-up mailboxes to the very end of the backup process, to either just before or just after renaming the /opt/zimbra/backup/tmp/<backup label> directory to /opt/zimbra/backup/sessions/<backup label>.
** https://bugzilla.zimbra.com/show_bug.cgi?id=55148
 
  
===Auto-Group Backups Rather Than Default Method Topics===
+
======Setting To Null To Cause A New Backup For User======
  
====General Description And Official References====
+
To undo what was done by an interrupted backup for example, you need to clear this column (set it to null) for the affected mailboxes.  By clearing the column, you're forcing the next AG backup to choose these mailboxes because they look like they have never been backed up. If you don't clear this column, you have to wait until the next cycle. (7 days)
  
Having trouble completing that entire full backup during off-hours? Enter the hybrid auto-grouped mode, which combines the concept of full and incremental backup functions - you’re completely backing up a target number of accounts daily rather than running incrementals.
+
Example syntax to view:
  
Auto-grouped mode automatically pulls in the redologs since the last run so you get incremental backups of the remaining accounts; although the incremental accounts captured via the redologs are not listed specifically in the backup account list. This still allows you to do a point in time restore for any account.
+
mysql zimbra -e "select last_backup_at from mailbox where id=27"
  
'''Administrative manual page:'''
+
Example syntax to change data of the last_backup_at to NULL"
  
http://www.zimbra.com/docs/ne/latest/administration_guide/10_Backup_Restore.17.01.html
+
mysql zimbra -e "UPDATE mailbox SET last_backup_at = NULL WHERE id = 27"
  
'''Compare the sections called "Standard Backup Method" & "Auto-Grouped Backup Method"'''
+
======Related Bugs RFE's======
  
http://www.zimbra.com/docs/ne/latest/administration_guide/10_Backup_Restore.17.02.html
+
* "In auto-grouped backup, delay the update of mailboxes' last_backup_at timestamp to the very end of backup"
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=33275
 +
* "Partial backup should be able to finish successfully when backup volume runs out of space"
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=33276
  
'''Configuration details are here on that page:'''
+
====Restore Requires accounts.xml File====
  
http://www.zimbra.com/docs/ne/latest/administration_guide/10_Backup_Restore.17.08.html
+
* "accounts.xml file dependency needed for zmrestore"
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=30979
  
'''Good explanation:'''
+
Also, see:
  
http://www.zimbrablog.com/blog/archives/2008/08/recent-admin-tidbits-part-1.html
+
* "accounts.xml file improperly updated when a full backup is deleted"
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=31591
  
Simply divide your total accounts by the number of groups you choose (zimbraBackupAutoGroupedNumGroups is 7 by default) and that’s how many will get a full backup session each night. Newly provisioned accounts, and accounts whose last backup is a specified number of days older are picked first. (zimbraBackupAutoGroupedInterval is defaulted to 1d)
+
====Change "Location" For Backup Or Restore Source Data====
  
Think of auto-grouped mode as a full backup for the scheduled group as well as an incremental (via redologs) for the all other accounts at the same time.
+
Remember that zmbackup and zmrestore can take flags as well in regards to the location of items.
  
====Bugs - RFE's To Review For Auto-Group====
+
* zmrestore & zmbackup can both take : -t,--target (default <zimbra_home/backup)
  
Please see:
+
'''''You can't state a different location with redo logs though'''''. There's a command called '''zmplayredo''' [for newer versions of ZCS] and it has a variable to point to the redologs to play from [ --logfiles ]. It will replay into the default redolog directory or redolog file. The mailbox has to be stop to run zmplayredo . This is a command to manual kick off a replay of a redo log. This is normally done with the zmrestore when options about to a specific time aren't included.
  
* "In auto-grouped backup, delay the update of mailboxes' last_backup_at timestamp to the very end of backup"
+
====Manual Removal Of Older Backup Sessions====
** http://bugzilla.zimbra.com/show_bug.cgi?id=33275
 
* "improve error reporting to ignore harmless missing sequences"
 
** http://bugzilla.zimbra.com/show_bug.cgi?id=34008
 
* "unbalanced auto-grouped backups"
 
** http://bugzilla.zimbra.com/show_bug.cgi?id=34028
 
  
====Enabling Auto-Group For Backups - Schedule In Crontab====
+
General Situation:
 +
*Keep in mind every restore requires starting with data from a full backup. 
 +
*For each account on the server, there must be at least 1 full backup after the deletion is complete. 
 +
*You should also make sure all incremental backups made after the oldest of the remaining backups are retained. 
 +
**This basically gets reduced to deleting only those backups that are old enough, based on your full/incremental schedule.
  
* You'll need to make the variable changes as listed in the administrative guide and then the following for set the crontab correctly.
+
More Specific Issues:
* Run zmschedulebackup --help to see a list of options.
+
*Does the accounts.xml dependency cause issues with this?
* Run zmschedulebackup -D , which will now set a new default schedule (crontab) that uses auto-group settings now that your variable are set for it.
+
**No. Just don't delete it.
** If you want the [[Ajcody-Notes-BackupPlans#Need_To_Write_Fewer_Files_-_Add_The_Zip_Option_To_Your_Backup_Commands|zip option]] to also be used do:
+
*What about the contents of the backup/tmp/ data or shared blobs type references?
*** <pre>zmschedulebackup -D -z</pre>
+
**Don't touch this directory either.  It is used during backup and restore.  You don't want to change its content while an operation is going on, so best to leave it alone.
*** See this bug about zmschedulebackup not being able to pass -zipStore option.
+
*What if a zimbra server is running some type of restore of backup command while the manual removal is running on the nfs server?
**** http://bugzilla.zimbra.com/show_bug.cgi?id=30981
+
**You shouldn't remove the backups that are being used in a restore currently underway. You are responsible for avoiding the race condition.
* Incremental backups aren't performed as they were.  
+
*** Please understand you are responsible for avoiding the race condition.  Make sure no backup or restore is happening at the moment, then rename the directories that will be deleted, preferably move them to another subdirectory, e.g. /opt/zimbra/backup/sessions_to_delete.  Then delete.
** You'll see there's no longer the -i option in cron.
+
 
** Incrementals are performed, in a manner, but the redologs are just copied into the full backup session
+
===LDAP Backup Related Items===
  
Two bugs to look at as well:
+
====Backup Schedule On LDAP Only Server [non-Mailstore]====
  
* cannot backup using admin ui in autogrouped mode
+
If you look at the code in /opt/zimbra/bin/zmschedulebackup  :
** http://bugzilla.zimbra.com/show_bug.cgi?id=29421
 
* Expose autogroup backup configuration to admin UI
 
** http://bugzilla.zimbra.com/show_bug.cgi?id=28198
 
  
======Standard Mode Default Cron Setup======
 
 
* Here is a schedule without auto-grouped backups enabled:
 
** The Default Schedule For Normal Backups [ zmschedulebackup --help ]:
 
 
<pre>
 
<pre>
f    0 1 * * 6
+
if ($BACKUP_MODE eq 'Standard') {
i    0 1 * * 0-5
+
    # default schedule: full backup 1am every sunday, incr backup 1am every weekday
  d 1m 0 0 * * *
+
    # deletes backups older than a month at 12am everyday
 +
  if (isLdapOnly()) {
 +
      @default = ("0 1 * * 6 /opt/zimbra/bin/zmbackup $target\n",
 +
            "0 0 * * * /opt/zimbra/bin/zmbackup -del 1m\n");
 +
  } else {
 +
      @default = ("0 1 * * 6 /opt/zimbra/bin/zmbackup -f $account $target $compress \n",
 +
          "0 1 * * 0-5 /opt/zimbra/bin/zmbackup -i $compress\n",
 +
            "0 0 * * * /opt/zimbra/bin/zmbackup -del 1m\n");
 +
  }
 +
    %commandmap = ("f", "/opt/zimbra/bin/zmbackup -f $target $compress $account",
 +
                  "i", "/opt/zimbra/bin/zmbackup -i $compress",
 +
                  "d", "/opt/zimbra/bin/zmbackup -del");
 +
} else { # Auto-Grouped mode
 +
    # default schedule: full backup 1am everyday, no incr backup
 +
    # deletes backups older than a month at 12am everyday
 +
    @default = ("0 1 * * 0-6 /opt/zimbra/bin/zmbackup -f $target $compress\n",
 +
                "0 0 * * * /opt/zimbra/bin/zmbackup -del 1m\n");
 +
    %commandmap = ("f", "/opt/zimbra/bin/zmbackup -f $target $compress",
 +
                  "i", "/opt/zimbra/bin/zmbackup -i $compress",
 +
                  "d", "/opt/zimbra/bin/zmbackup -del");
 +
}
 
</pre>
 
</pre>
**To set the backup to standard mode and use the default schedule one would:
+
 
***<pre>zmprov mcf zimbraBackupMode Standard</pre>
+
Notice the specific check and then format for ldapOnly:
***<pre>zmschedulebackup -D</pre>
+
 
**The crontab would look like:
 
 
<pre>
 
<pre>
0 1 * * 6 /opt/zimbra/bin/zmbackup -f -a all    
+
   if (isLdapOnly()) {
0 1 * * 0-5 /opt/zimbra/bin/zmbackup -i -a all
+
      @default = ("0 1 * * 6 /opt/zimbra/bin/zmbackup $target\n",
0 0 * * * /opt/zimbra/bin/zmbackup -del 1m
+
            "0 0 * * * /opt/zimbra/bin/zmbackup -del 1m\n");
 
</pre>
 
</pre>
  
======Auto-Group Mode Cron Setup======
+
====Related Bugs To LDAP Backups====
  
* Here is a schedule with auto-grouped enabled:
+
Some bugs to be aware of - most are resolved/fixed:
** The Default Schedule For Auto-Group Backups [ zmschedulebackup --help ]:
 
::<pre>
 
:: f    0 1 * * 0-6
 
:: d 1m 0 0 * * *
 
</pre>
 
**To set the backup to auto-group mode and use the default schedule one would:
 
***<pre>zmprov mcf zimbraBackupMode Auto-Grouped</pre>
 
***<pre>zmschedulebackup -D</pre>
 
**The crontab would look like:
 
::<pre>
 
::0 1 * * 0-6 /opt/zimbra/bin/zmbackup -f 
 
::0 0 * * * /opt/zimbra/bin/zmbackup -del 1m
 
</pre>
 
  
====Some Variables For Auto-Group====
+
* "Zmbackup ldap data is not backed up if target server is not hosting it."
 
+
** http://bugzilla.zimbra.com/show_bug.cgi?id=7420
The below might not be complete or the defaults, I just wanted to save this before I forget them. Try to get more complete details on these later.
+
* "System LDAP is backuped only when -a all is specified"
 
+
** http://bugzilla.zimbra.com/show_bug.cgi?id=8365
zmprov gacf | grep Backup
+
* "LDAP backup directory format difference between full install and ldap only install"
zimbraBackupAutoGroupedInterval: 1d
+
** http://bugzilla.zimbra.com/show_bug.cgi?id=9551
zimbraBackupAutoGroupedNumGroups: 7
+
* "backups in the admin console should back up LDAP data"
zimbraBackupAutoGroupedThrottled: FALSE
+
** http://bugzilla.zimbra.com/show_bug.cgi?id=14069
zimbraBackupMode: Auto-Grouped
+
* "LDAP backup failure due to library version mismatch"
 
+
** http://bugzilla.zimbra.com/show_bug.cgi?id=20275
====Auto-group And Redologs====
+
* "ldap backup not kept with rest of backup"
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=21753
 +
* "zmbackup ldap backup is not logged"
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=22303
 +
* "Backup default cronjob is wrong for ldap only install"
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=23513
 +
* "zmbackupldap fails to rename directory on NFS"
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=25451
 +
* "default ldap server only backup schedule is weekly"
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=25681
 +
* "zmbackupldap fails when using month intervals and succeeding month has more days"
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=32834
 +
* "zmbackup on LDAP-only hosts should have help/usage argument, or at least reject invalid arguments"
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=32878
 +
* "corp: ldap backups fail"
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=38558
  
Please see [[Ajcody-Backup-Restore-Issues#Redologs_And_Auto-group_In_Regards_To_Backups]]
+
===A Way To Verify Backup Integrity===
  
====Problems Switching To Auto-Groups Because It Wants To Run A Full Against All Accounts====
+
I filed an RFE for this:
  
Please see the following bug/rfe made about problems switching over to Auto-group when the first backup run of it tries to backup ALL of he accounts. I have the full how-to within the bug. It basically manipulates the last_backup_at for each account.
+
* "A way to verify backup integrity"
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=28025
  
* Ability to set last_backup_at to NULL or to the day before for All users, group of users, user
+
====Negative Seek Offset Error & RFE====
** https://bugzilla.zimbra.com/show_bug.cgi?id=87834
 
  
===Backup And Deletion Schedule - zmschedulebackup===
+
Explanation of negative seek offset error during a restore attempt and manual fixes are located here:
  
====How To Adjust The Deletion Schedule====
+
* "RFE: Monitor/test backup *.zip files for corruption & repair tools"
 +
** https://bugzilla.zimbra.com/show_bug.cgi?id=55148
  
For my example below, I first set the backup to the "default" schedule. And I then adjust that "default" to have backups delete with a 14 day interval rather than 1 month.
+
===Auto-Group Backups Rather Than Default Method Topics===
  
<pre>
+
====General Description And Official References====
[zimbra@mail3 ~]$ zmschedulebackup -D
 
Default schedule set
 
  
Current Schedule:
+
Having trouble completing that entire full backup during off-hours? Enter the hybrid auto-grouped mode, which combines the concept of full and incremental backup functions - you’re completely backing up a target number of accounts daily rather than running incrementals.
  
        f 0 1 * * 6 -a all
+
Auto-grouped mode automatically pulls in the redologs since the last run so you get incremental backups of the remaining accounts; although the incremental accounts captured via the redologs are not listed specifically in the backup account list. This still allows you to do a point in time restore for any account.
        i 0 1 * * 0-5
 
        d 1m 0 0 * * *
 
  
[zimbra@mail3 ~]$ zmschedulebackup -q
+
'''Administrative manual page:'''
Current Schedule:
 
  
        f 0 1 * * 6 -a all
+
http://www.zimbra.com/docs/ne/latest/administration_guide/10_Backup_Restore.17.01.html
        i 0 1 * * 0-5
 
        d 1m 0 0 * * *
 
  
[zimbra@mail3 ~]$ zmschedulebackup -R f "0 1 * * 6" i "0 1 * * 0-5" d 14d "0 0 * * *"
+
'''Compare the sections called "Standard Backup Method" & "Auto-Grouped Backup Method"'''
Schedule replaced
 
  
Current Schedule:
+
http://www.zimbra.com/docs/ne/latest/administration_guide/10_Backup_Restore.17.02.html
  
        f 0 1 * * 6 -a all
+
'''Configuration details are here on that page:'''
        i 0 1 * * 0-5
 
        d 14d 0 0 * * *
 
  
[zimbra@mail3 ~]$ zmschedulebackup -q
+
http://www.zimbra.com/docs/ne/latest/administration_guide/10_Backup_Restore.17.08.html
Current Schedule:
 
  
        f 0 1 * * 6 -a all
+
'''Good explanation:'''
        i 0 1 * * 0-5
+
 
        d 14d 0 0 * * *
+
http://www.zimbrablog.com/blog/archives/2008/08/recent-admin-tidbits-part-1.html
  
[zimbra@mail3 ~]$ crontab -l | grep backup
+
Simply divide your total accounts by the number of groups you choose (zimbraBackupAutoGroupedNumGroups is 7 by default) and that’s how many will get a full backup session each night. Newly provisioned accounts, and accounts whose last backup is a specified number of days older are picked first. (zimbraBackupAutoGroupedInterval is defaulted to 1d)
0 1 * * 6 /opt/zimbra/bin/zmbackup -f  -a all
 
0 1 * * 0-5 /opt/zimbra/bin/zmbackup -i 
 
0 0 * * * /opt/zimbra/bin/zmbackup -del 14d
 
</pre>
 
  
===The Zip - Compression Option For Backups===
+
Think of auto-grouped mode as a full backup for the scheduled group as well as an incremental (via redologs) for the all other accounts at the same time.
  
Using the zip option will compress all those thousands of single files that exist under a user's backup, decreasing performance issues that arise from writing out thousands of small files as compared to large ones. This is often seen when one is :
+
====Bugs - RFE's To Review For Auto-Group====
  
* Using nfs for the backup directory
+
Please see:
* Copying/rsyncing backups to a remote server
 
* Are using some third party backup software (to tape) to archive/backup the zimbra backup sessions.
 
  
====Optional Tweaks To The Zip Options====
+
* "In auto-grouped backup, delay the update of mailboxes' last_backup_at timestamp to the very end of backup"
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=33275
 +
* "improve error reporting to ignore harmless missing sequences"
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=34008
 +
* "unbalanced auto-grouped backups"
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=34028
  
Please see this comment and those underneath it within this RFE:
+
====Enabling Auto-Group For Backups - Schedule In Crontab====
  
* "Use zip files for shared blobs of a full backup made with --zip option"
+
* You'll need to make the variable changes as listed in the administrative guide and then the following for set the crontab correctly.
** http://bugzilla.zimbra.com/show_bug.cgi?id=26624#c6
+
* Run zmschedulebackup --help to see a list of options.
*** Use zmlocalconfig. To set:
+
* Run zmschedulebackup -D , which will now set a new default schedule (crontab) that uses auto-group settings now that your variable are set for it.
**** $ zmlocalconfig -e key=val
+
** If you want the [[Ajcody-Notes-BackupPlans#Need_To_Write_Fewer_Files_-_Add_The_Zip_Option_To_Your_Backup_Commands|zip option]] to also be used do:
*** To unset:
+
*** <pre>zmschedulebackup -D -z</pre>
**** $ zmlocalconfig -u key
+
*** See this bug about zmschedulebackup not being able to pass -zipStore option.
*** Once you set the key you will be able to view it.
+
**** http://bugzilla.zimbra.com/show_bug.cgi?id=30981
**** $ zmlocalconfig
+
* Incremental backups aren't performed as they were.  
*** backup_zip_copier_private_blob_zips
+
** You'll see there's no longer the -i option in cron.
**** How many zip files to distribute a mailbox's private (unshared) blobs over; default 4 (blobs-1.zip through blobs-4.zip); range 1 to 10,000
+
** Incrementals are performed, in a manner, but the redologs are just copied into the full backup session
*** backup_zip_copier_copy_buffer_size
 
**** File copy buffer size; default 16384 (16KB); range 4KB to 1MB
 
*** backup_zip_copier_queue_capacity
 
**** Each zip file gets a queue of files to add.  This key sets the queue size.  Default is 10.  Range is 1 to 10,000.
 
*** backup_zip_copier_deflate_level
 
**** Compression level. Default is -1. (same as in java.util.zip.ZipOutputStream).  -1 is same as level 6.  This behavior comes from zlib library which the JVM uses to implement zip.  Other than the special default value, the level can range from 0 to 9.  0 means no compression.  1 means fastest compression and 9 means best compression.
 
*** backup_disable_shared_blobs
 
**** This one isn't limited to zip backups.  When this is set to true, all blobs are backed up as private backups.  Default is false.
 
*** backup_debug_use_old_zip_format
 
**** If true, backup will behave like ZCS 5.0.4 and earlier.  Shared blobs are never zipped, and private blobs are added to a single blobs.zip file in zip backup.  Default is false.
 
  
====Need To Write Fewer Files - Add The Zip Option To Your Backup Commands====
+
Two bugs to look at as well:
  
RFE to make zip option the default for backups:
+
* cannot backup using admin ui in autogrouped mode
* "backup: default to the zip option"
+
** http://bugzilla.zimbra.com/show_bug.cgi?id=29421
** http://bugzilla.zimbra.com/show_bug.cgi?id=31836
+
* Expose autogroup backup configuration to admin UI
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=28198
  
There is very little details in the official documentation on this option unfortunately. This does have a really good explanation though:
+
======Standard Mode Default Cron Setup======
 +
 +
* Here is a schedule without auto-grouped backups enabled:
 +
** The Default Schedule For Normal Backups [ zmschedulebackup --help ]:
 +
<pre>
 +
f    0 1 * * 6
 +
i    0 1 * * 0-5
 +
d 1m 0 0 * * *
 +
</pre>
 +
**To set the backup to standard mode and use the default schedule one would:
 +
***<pre>zmprov mcf zimbraBackupMode Standard</pre>
 +
***<pre>zmschedulebackup -D</pre>
 +
**The crontab would look like:
 +
<pre>
 +
0 1 * * 6 /opt/zimbra/bin/zmbackup -f -a all 
 +
0 1 * * 0-5 /opt/zimbra/bin/zmbackup -i -a all
 +
0 0 * * * /opt/zimbra/bin/zmbackup -del 1m
 +
</pre>
  
http://www.zimbrablog.com/blog/archives/2008/08/recent-admin-tidbits-part-1.html
+
======Auto-Group Mode Cron Setup======
  
From the administrative manual on the Backup section:
+
* Here is a schedule with auto-grouped enabled:
 
+
** The Default Schedule For Auto-Group Backups [ zmschedulebackup --help ]:
http://www.zimbra.com/docs/ne/latest/administration_guide/10_Backup_Restore.15.1.html
+
<pre>
 
+
f 0 1 * * 0-6
It says,
+
d 1m 0 0 * * *
 +
</pre>
 +
**To set the backup to auto-group mode and use the default schedule one would:
 +
***<pre>zmprov mcf zimbraBackupMode Auto-Grouped</pre>
 +
***<pre>zmschedulebackup -D</pre>
 +
**The crontab would look like:
 +
<pre>
 +
0 1 * * 0-6 /opt/zimbra/bin/zmbackup -f 
 +
0 0 * * * /opt/zimbra/bin/zmbackup -del 1m
 +
</pre>
  
:: "-zip can be added to the command line to zip the message files during backup. Zipping these can save backup storage space."
+
====Some Variables For Auto-Group====
  
It's implied that instead of having all the individual message files in the backup that it will bunch them all together into zip files. '''The body of a shared blob is added once to a shared-blobs zip file, then a small pointer-only entry is added to a mailbox's zip file.  Same effect as in non-zipped case.''' This will be useful when the number of message files is causing disk i/o issues. Maybe your trying to rsync the backup session directories off to another server or your running a third party backup on it to save to tape. The default use of -zip will use compression, if this also causes overhead that you need to avoid you can use the -zipStore option.  
+
The below might not be complete or the defaults, I just wanted to save this before I forget them. Try to get more complete details on these later.
  
Note about -zipStore:
+
zmprov gacf | grep Backup
:: "when used with the -zip option, it allows the backup to write fewer files (-zip), but not incur the compression overhead as well"
+
zimbraBackupAutoGroupedInterval: 1d
 +
zimbraBackupAutoGroupedNumGroups: 7
 +
zimbraBackupAutoGroupedThrottled: FALSE
 +
zimbraBackupMode: Auto-Grouped
  
'''The zip options effect backups that are in blob formats (full's). Incremental backups are bascially redologs, not the full message store of the user. In summary, the zip option will not impact the increment type backups. Auto-group backups are a mixture of both fulls and incrementals.'''
+
====Auto-group And Redologs====
  
=====How To Use As A Default Option?=====
+
Please see [[Ajcody-Backup-Restore-Issues#Redologs_And_Auto-group_In_Regards_To_Backups]]
  
You'll add the options to the zimbra crontab file. This can be done with the zmschedulebackup command.
+
====Problems Switching To Auto-Groups Because It Wants To Run A Full Against All Accounts====
  
Run zmschedulebackup with help option:
+
Please see the following bug/rfe made about problems switching over to Auto-group when the first backup run of it tries to backup ALL of he accounts. I have the full how-to within the bug. It basically manipulates the last_backup_at for each account.
  
zmschedulebackup --help
+
* Ability to set last_backup_at to NULL or to the day before for All users, group of users, user
+
** https://bugzilla.zimbra.com/show_bug.cgi?id=87834
You'll see:
 
  
-z: compress - compress email blobs with zip
+
===Backup And Deletion Schedule - zmschedulebackup===
  
It appears that you'll need to manually add the options about -zipStore , if you want it, to the crontab file.
+
====How To Adjust The Deletion Schedule====
  
See bug :
+
For my example below, I first set the backup to the "default" schedule. And I then adjust that "default" to have backups delete with a 14 day interval rather than 1 month.
  
http://bugzilla.zimbra.com/show_bug.cgi?id=30981
+
<pre>
 +
[zimbra@mail3 ~]$ zmschedulebackup -D
 +
Default schedule set
  
=====What Does It Look Like When I Use Zip?=====
+
Current Schedule:
  
Shared blobs are zipped and blobs (messages) are zipped per root store directory.
+
        f 0 1 * * 6 -a all
 +
        i 0 1 * * 0-5
 +
        d 1m 0 0 * * *
  
mail:~/backup/sessions/full-20080820.160003.770/accounts/115/988/11598896-a89b-4b9d-bedb-1ed1afcb6c87/blobs
+
[zimbra@mail3 ~]$ zmschedulebackup -q
zimbra$ ls blobs-1.zip    blobs-2.zip    blobs-3.zip    blobs-4.zip
+
Current Schedule:
  
===General Backup & Restore Debugging===
+
        f 0 1 * * 6 -a all
 +
        i 0 1 * * 0-5
 +
        d 1m 0 0 * * *
  
'''You'll be monitoring the /opt/zimbra/log/mailbox.log file'''
+
[zimbra@mail3 ~]$ zmschedulebackup -R f "0 1 * * 6" i "0 1 * * 0-5" d 14d "0 0 * * *"
 +
Schedule replaced
  
Include the -d / --debug option on the CLI for either zmrestore or zmbackup .
+
Current Schedule:
  
To increasing logging for backup/restore-related logs - /opt/zimbra/log/mailbox.log . Enable DEBUG log level for "zimbra.backup" logger in :
+
        f 0 1 * * 6 -a all
 +
        i 0 1 * * 0-5
 +
        d 14d 0 0 * * *
  
* /opt/zimbra/conf/log4j.properties for "temporary" change - until next restart. This could take a couple of minutes before jetty "sees" the changes.
+
[zimbra@mail3 ~]$ zmschedulebackup -q
* /opt/zimbra/conf/log4.properties.in for "permament" change that will stick after restart. A restart of jetty/mailbox would be required for this change - zmmailboxctl restart .
+
Current Schedule:
  
    log4j.logger.zimbra.backup=DEBUG
+
        f 0 1 * * 6 -a all
 +
        i 0 1 * * 0-5
 +
        d 14d 0 0 * * *
  
 +
[zimbra@mail3 ~]$ crontab -l | grep backup
 +
0 1 * * 6 /opt/zimbra/bin/zmbackup -f  -a all
 +
0 1 * * 0-5 /opt/zimbra/bin/zmbackup -i 
 +
0 0 * * * /opt/zimbra/bin/zmbackup -del 14d
 +
</pre>
  
For incremental backups, this should log each redolog being copied to the backup and also log which ones will be deleted out of archive directory.  Those not deleted are kept because they are newer than 1 hour (default).  The kept logs are deleted (but not copied again) during the next incremental backup.
+
===The Zip - Compression Option For Backups===
  
===Redolog Files===
+
Using the zip option will compress all those thousands of single files that exist under a user's backup, decreasing performance issues that arise from writing out thousands of small files as compared to large ones. This is often seen when one is :
  
----
+
* Using nfs for the backup directory
 +
* Copying/rsyncing backups to a remote server
 +
* Are using some third party backup software (to tape) to archive/backup the zimbra backup sessions.
  
====Redologs Copied To Backup Session And When Deleted====
+
====Optional Tweaks To The Zip Options====
  
Archived logs that are less than an hour old at the time of incremental backup are copied to the backup but aren't deleted to support post-crash waitset reinitialization mechanism.  The interval is set in localconfig key backup_archived_redolog_keep_time, which is in seconds, default=3600.
+
Please see this comment and those underneath it within this RFE:
  
=====An Outline Of The Step=====
+
* "Use zip files for shared blobs of a full backup made with --zip option"
 
+
** http://bugzilla.zimbra.com/show_bug.cgi?id=26624#c6
# /opt/zimbra/redolog/redo.log (starts and then grows to zimbraRedoLogRolloverFileSizeKB size - default 100MB)
+
*** Use zmlocalconfig.  To set:
# This flushes to /opt/zimbra/redolog/archive/[file] upon hitting the zimbraRedoLogRolloverFileSizeKB.
+
**** $ zmlocalconfig -e key=val
# 1 & 2 keep repeating when zimbraRedoLogRolloverFileSizeKB is hit.
+
*** To unset:
# When a backup is done, the archive/* files are copiedThe redo.log file is not moved.
+
**** $ zmlocalconfig -u key
## When the backup processes archive/* logs, it first figures out the last sequence copied to backupAll newer logs are copied to the current backupThen, all logs are deleted except those that are too new, determined by localconfig parameter backup_archived_redolog_keep_time, which defaults to 1 hour(This is part of the waitset feature.)
+
*** Once you set the key you will be able to view it.
##* In standard backup mode, only incremental backups move the redologs. 
+
**** $ zmlocalconfig
##* In auto-grouped mode, every backup is a hybrid of full and incremental and thus redologs are moved.
+
*** backup_zip_copier_private_blob_zips
 +
**** How many zip files to distribute a mailbox's private (unshared) blobs over; default 4 (blobs-1.zip through blobs-4.zip); range 1 to 10,000
 +
*** backup_zip_copier_copy_buffer_size
 +
**** File copy buffer size; default 16384 (16KB); range 4KB to 1MB
 +
*** backup_zip_copier_queue_capacity
 +
**** Each zip file gets a queue of files to add.  This key sets the queue size. Default is 10. Range is 1 to 10,000.
 +
*** backup_zip_copier_deflate_level
 +
**** Compression level.  Default is -1(same as in java.util.zip.ZipOutputStream). -1 is same as level 6.  This behavior comes from zlib library which the JVM uses to implement zip. Other than the special default value, the level can range from 0 to 9.  0 means no compression1 means fastest compression and 9 means best compression.
 +
*** backup_disable_shared_blobs
 +
**** This one isn't limited to zip backupsWhen this is set to true, all blobs are backed up as private backupsDefault is false.
 +
*** backup_debug_use_old_zip_format
 +
**** If true, backup will behave like ZCS 5.0.4 and earlier.  Shared blobs are never zipped, and private blobs are added to a single blobs.zip file in zip backup.  Default is false.
  
=====Redologs And Auto-group In Regards To Backups=====
+
====Need To Write Fewer Files - Add The Zip Option To Your Backup Commands====
  
http://www.zimbra.com/forums/administrators/21360-auto-grouped-backups.html
+
RFE to make zip option the default for backups:
 +
* "backup: default to the zip option"
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=31836
  
<blockquote>Think of auto-grouped mode as a full backup for the scheduled group as well as an incremental (via redologs) for the all other accounts at the same time. Auto-grouped mode automatically pulls in the redologs since the last run, so you get incremental backups of the remaining accounts.  Although the incremental accounts captured via the redologs are not listed specifically in the backup account list. This still allows you to do a point in time restore for any account.</blockquote>
+
There is very little details in the official documentation on this option unfortunately. This does have a really good explanation though:
  
=====If You Have Older Redologs Not Being Deleted=====
+
http://www.zimbrablog.com/blog/archives/2008/08/recent-admin-tidbits-part-1.html
  
According to the code, only archived logs newer than 1 hour old (default for backup_archived_redolog_keep_time) should remain after an incremental backup.  It is a bug if you are seeing older logs sticking around.  If so, look at mailbox.log and see if any error was logged.  If you enable DEBUG logging for "zimbra.backup" logger in log4j.properties you will see log statements for each copy and deletion.
+
From the administrative manual on the Backup section:
  
=====The zimbraRedoLogDeleteOnRollover variable=====
+
http://www.zimbra.com/docs/ne/latest/administration_guide/10_Backup_Restore.15.1.html
  
zimbraRedoLogDeleteOnRollover shouldn't have an effect on "If you have older redologs not being deleted". By default it's FALSE and affects whether or not stuff makes it into /opt/zimbra/redolog/archive at all. With it set to TRUE there's just /opt/zimbra/redolog/redo.log and it's deleted/not rolled over into archive. As discussed above old redologs are deleted after the incremental; thus if you don't take incremental backups you should set this value to TRUE or periodically script manual deletion of /opt/zimbra/redolog/archive. (And with zimbraRedoLogEnabled FALSE there's no redo.log at all.)
+
It says,
  
======If You Don't Run Incremental Backups Or Don't Need Archive Redologs======
+
:: "-zip can be added to the command line to zip the message files during backup. Zipping these can save backup storage space."
  
You would set zimbraRedoLogDeleteOnRollover to TRUE.
+
It's implied that instead of having all the individual message files in the backup that it will bunch them all together into zip files. '''The body of a shared blob is added once to a shared-blobs zip file, then a small pointer-only entry is added to a mailbox's zip file.  Same effect as in non-zipped case.''' This will be useful when the number of message files is causing disk i/o issues. Maybe your trying to rsync the backup session directories off to another server or your running a third party backup on it to save to tape. The default use of -zip will use compression, if this also causes overhead that you need to avoid you can use the -zipStore option.  
  
(Auto-Grouped backups you can still leave this to the default of FALSE.)
+
Note about -zipStore:
 +
:: "when used with the -zip option, it allows the backup to write fewer files (-zip), but not incur the compression overhead as well"
  
====Redolog Sequence And The Backup Session====
+
'''The zip options effect backups that are in blob formats (full's). Incremental backups are bascially redologs, not the full message store of the user. In summary, the zip option will not impact the increment type backups. Auto-group backups are a mixture of both fulls and incrementals.'''
  
Redologs will exist in the incremental backup sessions. The zmbackupquery command will reference the redologs associated with the backup. For example"
+
=====How To Use As A Default Option?=====
  
<pre>
+
You'll add the options to the zimbra crontab file. This can be done with the zmschedulebackup command.
[zimbra@mail3 ~]$ ls /opt/zimbra/backup/sessions/incr-20080925.224528.230/redologs/
 
redo-20080925.213136.165-seq53.log  redo-20080925.220726.521-seq55.log  redo-20080925.224209.287-seq57.log
 
redo-20080925.215516.450-seq54.log  redo-20080925.221749.133-seq56.log
 
  
[zimbra@mail3 ~]$ zmbackupquery -lb incr-20080925.224528.230
+
Run zmschedulebackup with help option:
Label:   incr-20080925.224528.230
 
Type:    incremental
 
Status:  completed
 
Started: Thu, 2008/09/25 18:45:28.230 EDT
 
Ended:  Thu, 2008/09/25 18:45:39.099 EDT
 
Redo log sequence range: 53 .. 57
 
Number of accounts: 2
 
</pre>
 
  
In the above example, we see the sequence range of "53 .. 57" is referring to the files in the backup session directory called redologs.
+
zmschedulebackup --help
 +
 +
You'll see:
 +
 
 +
-z: compress - compress email blobs with zip
  
 +
It appears that you'll need to manually add the options about -zipStore , if you want it, to the crontab file.
  
 +
See bug :
  
====RedoLog Variables====
+
http://bugzilla.zimbra.com/show_bug.cgi?id=30981
  
=====Changing Redolog File Size And Location=====
+
=====What Does It Look Like When I Use Zip?=====
  
The /opt/zimbra/redolog/redo.log rolls over when it reaches zimbraRedoLogRolloverFileSizeKB (by default 100mb).
+
Shared blobs are zipped and blobs (messages) are zipped per root store directory.
  
The "roll overs" then goto /opt/zimbra/redolog/archive/
+
mail:~/backup/sessions/full-20080820.160003.770/accounts/115/988/11598896-a89b-4b9d-bedb-1ed1afcb6c87/blobs
 +
zimbra$ ls blobs-1.zip    blobs-2.zip    blobs-3.zip    blobs-4.zip
  
zmprov gacf | grep Redo
+
===General Backup & Restore Debugging===
zimbraRedoLogArchiveDir: redolog/archive
 
zimbraRedoLogDeleteOnRollover: FALSE
 
zimbraRedoLogEnabled: TRUE
 
zimbraRedoLogFsyncIntervalMS: 10
 
zimbraRedoLogLogPath: redolog/redo.log
 
zimbraRedoLogRolloverFileSizeKB: 102400
 
  
======Need To Move Redologs Because Partition Getting Full======
+
'''You'll be monitoring the /opt/zimbra/log/mailbox.log file'''
  
Let's say you have a partition getting full and you need to move the redolog to another partition or nfs mount temporary to deal with the potential crisis that will happen when the partition becomes full. '''You'll need to reallocate the complete redolog/ directory and the archive subdirectory to the same partition''' because the roll over from redo.log to the archive directory happens with a rename function within the java code. '''This will require downtime''' since you'll need to move the actual redo.log file and zimbra can't be running while you do this. You can use a symlink to your new partition path. For example:
+
Include the -d / --debug option on the CLI for either zmrestore or zmbackup .
  
su - zimbra
+
To increasing logging for backup/restore-related logs - /opt/zimbra/log/mailbox.log . Enable DEBUG log level for "zimbra.backup" logger in :
zmcontrol stop
 
su -
 
  ** as root
 
mkdir /data/redolog
 
chown zimbra:zimbra /data/redolog
 
mount /dev/sdb1 /data/redolog
 
mv /opt/zimbra/redolog/* /data/redolog/
 
  ** or use rsync
 
rmdir /opt/zimbra/redolog
 
ln -s /opt/zimbra/redolog /data/redolog
 
ls -laR /data/redolog
 
** confirm ownership is with zimbra and double check zimbra can write in this directory
 
su - zimbra
 
touch /data/redolog/testfile
 
rm /data/redolog/testfile
 
zmcontrol start
 
  
=====Automatic Deleting Of Redo Logs On Rollover=====
+
* /opt/zimbra/conf/log4j.properties for "temporary" change - until next restart. This could take a couple of minutes before jetty "sees" the changes.
 +
* /opt/zimbra/conf/log4.properties.in for "permament" change that will stick after restart. A restart of jetty/mailbox would be required for this change - zmmailboxctl restart .
  
This variable (zimbraRedoLogDeleteOnRollover) is set TRUE or FALSE.
+
    log4j.logger.zimbra.backup=DEBUG
  
zmprov gacf | grep zimbraRedoLogDeleteOnRollover
 
  
To modify it
+
For incremental backups, this should log each redolog being copied to the backup and also log which ones will be deleted out of archive directory.  Those not deleted are kept because they are newer than 1 hour (default).  The kept logs are deleted (but not copied again) during the next incremental backup.
  
zmprov mcf zimbraRedoLogDeleteOnRollover TRUE
+
===Redolog Files===
  
====Want To See What's In Redolog Files====
+
----
  
'''This is for older versions of ZCS - newer versions should use zmredodump if it's available.'''
+
====Redologs Copied To Backup Session And When Deleted====
  
If you suspect there's too much redolog activity during a time window or have another need to inspect the contents of the redolog, dump it and examine it:
+
Archived logs that are less than an hour old at the time of incremental backup are copied to the backup but aren't deleted to support post-crash waitset reinitialization mechanism.  The interval is set in localconfig key backup_archived_redolog_keep_time, which is in seconds, default=3600.
  
$ zmjava com.zimbra.cs.redolog.util.RedoLogVerify /opt/zimbra/redolog/redo.log > out.file
+
=====An Outline Of The Step=====
  
Pick the right redolog file, either redo.log or one of the files under archive/, based on timestamp.
+
# /opt/zimbra/redolog/redo.log (starts and then grows to zimbraRedoLogRolloverFileSizeKB size - default 100MB)
 +
# This flushes to /opt/zimbra/redolog/archive/[file] upon hitting the zimbraRedoLogRolloverFileSizeKB.
 +
# 1 & 2 keep repeating when zimbraRedoLogRolloverFileSizeKB is hit.
 +
# When a backup is done, the archive/* files are copied.  The redo.log file is not moved.
 +
## When the backup processes archive/* logs, it first figures out the last sequence copied to backup.  All newer logs are copied to the current backup.  Then, all logs are deleted except those that are too new, determined by localconfig parameter backup_archived_redolog_keep_time, which defaults to 1 hour.  (This is part of the waitset feature.)
 +
##* In standard backup mode, only incremental backups move the redologs. 
 +
##* In auto-grouped mode, every backup is a hybrid of full and incremental and thus redologs are moved.
  
====zmplayredo And zmredodump====
+
=====Redologs And Auto-group In Regards To Backups=====
  
=====zmplayredo - Replaying Content From Any Redolog File=====
+
http://www.zimbra.com/forums/administrators/21360-auto-grouped-backups.html
  
zmplayredo is a newer command, first introduced in 5.0.5 I believe. The mailbox has to be stop to run zmplayredo.
+
<blockquote>Think of auto-grouped mode as a full backup for the scheduled group as well as an incremental (via redologs) for the all other accounts at the same time. Auto-grouped mode automatically pulls in the redologs since the last run, so you get incremental backups of the remaining accounts.  Although the incremental accounts captured via the redologs are not listed specifically in the backup account list. This still allows you to do a point in time restore for any account.</blockquote>
  
The help output from 6.0.8:
+
=====If You Have Older Redologs Not Being Deleted=====
<pre>
 
$ zmplayredo --help
 
usage: zmplayredo <options>
 
    --fromSeq <arg>        Replay from this redolog sequence (inclusive)
 
    --fromTime <arg>        Replay from this time (inclusive)
 
-h,--help                  Show help (this output)
 
    --logfiles <arg>        Replay these logfiles, in order
 
    --mailboxId <arg>      Replay for this mailbox only
 
    --queueCapacity <arg>  Queue capacity per player thread; default=100
 
    --stopOnError          Stop replay on any error
 
    --threads <arg>        Number of parallel redo threads; default=50
 
    --toSeq <arg>          Replay to this redolog sequence (inclusive)
 
    --toTime <arg>          Replay to this time (inclusive)
 
  
Specify date/time in one of these formats:
+
According to the code, only archived logs newer than 1 hour old (default for backup_archived_redolog_keep_time) should remain after an incremental backup.  It is a bug if you are seeing older logs sticking around.  If so, look at mailbox.log and see if any error was logged.  If you enable DEBUG logging for "zimbra.backup" logger in log4j.properties you will see log statements for each copy and deletion.
  
    2010/11/19 13:55:08
+
=====The zimbraRedoLogDeleteOnRollover variable=====
    2010/11/19 13:55:08 802
+
 
    2010/11/19 13:55:08.802
+
zimbraRedoLogDeleteOnRollover shouldn't have an effect on "If you have older redologs not being deleted". By default it's FALSE and affects whether or not stuff makes it into /opt/zimbra/redolog/archive at all. With it set to TRUE there's just /opt/zimbra/redolog/redo.log and it's deleted/not rolled over into archive. As discussed above old redologs are deleted after the incremental; thus if you don't take incremental backups you should set this value to TRUE or periodically script manual deletion of /opt/zimbra/redolog/archive. (And with zimbraRedoLogEnabled FALSE there's no redo.log at all.)
    2010/11/19-13:55:08-802
+
 
    2010/11/19-13:55:08
+
======If You Don't Run Incremental Backups Or Don't Need Archive Redologs======
    20101119.135508.802
+
 
    20101119.135508
+
You would set zimbraRedoLogDeleteOnRollover to TRUE.
    20101119135508802
 
    20101119135508
 
  
Specify year, month, date, hour, minute, second, and optionally millisecond.
+
(Auto-Grouped backups you can still leave this to the default of FALSE.)
Month/date/hour/minute/second are 0-padded to 2 digits, millisecond to 3 digits.
 
Hour must be specified in 24-hour format, and time is in local time zone.
 
</pre>
 
  
=====zmredodump - Replaying Content From Any Redolog File=====
+
====Redolog Sequence And The Backup Session====
  
zmplayredo is a newer command and very useful. It does not require mailboxd to be stopped like zmplayredo does.
+
Redologs will exist in the incremental backup sessions. The zmbackupquery command will reference the redologs associated with the backup. For example"
  
The help output from 6.0.8:
 
 
<pre>
 
<pre>
$ zmredodump --help
+
[zimbra@mail3 ~]$ ls /opt/zimbra/backup/sessions/incr-20080925.224528.230/redologs/
usage: zmredodump [options] <redolog file/directory> [...]
+
redo-20080925.213136.165-seq53.log  redo-20080925.220726.521-seq55.log  redo-20080925.224209.287-seq57.log
where [options] are:
+
redo-20080925.215516.450-seq54.log  redo-20080925.221749.133-seq56.log
  
-h,--help        show this output
+
[zimbra@mail3 ~]$ zmbackupquery -lb incr-20080925.224528.230
    --m <arg>    one or more mailbox ids separated by comma or white
+
Label:  incr-20080925.224528.230
                  space. The entire list must be quoted if using space as separator. If
+
Type:    incremental
                  this option is given, only redo ops for the specified mailboxes are
+
Status: completed
                  dumped. Omit this option to dump redo ops for all mailboxes.
+
Started: Thu, 2008/09/25 18:45:28.230 EDT
    --no-offset  don't show file offsets and size for each redo op
+
Ended:   Thu, 2008/09/25 18:45:39.099 EDT
-q,--quiet      quiet mode.  Only print the log filename and any errors.
+
Redo log sequence range: 53 .. 57
                  This option can be used to verify the integrity of redologs with minimal
+
Number of accounts: 2
                  output.
 
    --show-blob   show blob content.  Item's blob is printed, surrounded
 
                  by <START OF BLOB> and <END OF BLOB> markers. The last newline before end
 
                  marker is not part of the blob.
 
 
 
Multiple log files/directories can be specified. For each directory, all
 
redolog files directly under it are processed, sorted in ascending redolog
 
sequence order.
 
 
</pre>
 
</pre>
  
=====Using zmredodump To Get Message Blobs To Inject With zmlmtpinject - RFE=====
+
In the above example, we see the sequence range of "53 .. 57" is referring to the files in the backup session directory called redologs.
  
Please see:
 
  
* "RFE: zmredodump blobs to single files for zmlmtpinject [for example]"
 
** http://bugzilla.zimbra.com/show_bug.cgi?id=52930
 
* See also:
 
** "RFE: zmplayredo option for --frommailboxId --tomailboxId"
 
*** http://bugzilla.zimbra.com/show_bug.cgi?id=52642
 
  
=====Expand Ability To Parse Redologs - RFE=====
+
====RedoLog Variables====
  
Please see:
+
=====Changing Redolog File Size And Location=====
  
* "expand ability to parse redologs for recovery purposes - zmredodump / zmplayredo"
+
The /opt/zimbra/redolog/redo.log rolls over when it reaches zimbraRedoLogRolloverFileSizeKB (by default 100mb).
** http://bugzilla.zimbra.com/show_bug.cgi?id=37743
 
  
====Getting A Sequence or Time Variable For Restore Or Replay====
+
The "roll overs" then goto /opt/zimbra/redolog/archive/
  
You can see the changes within the redo logs with the command below. You can point it to any redolog.
+
zmprov gacf | grep Redo
 +
zimbraRedoLogArchiveDir: redolog/archive
 +
zimbraRedoLogDeleteOnRollover: FALSE
 +
zimbraRedoLogEnabled: TRUE
 +
zimbraRedoLogFsyncIntervalMS: 10
 +
zimbraRedoLogLogPath: redolog/redo.log
 +
zimbraRedoLogRolloverFileSizeKB: 102400
  
<pre>zmjava com.zimbra.cs.redolog.util.RedoLogVerify /opt/zimbra/redolog/redo.log > out.file</pre>
+
======Need To Move Redologs Because Partition Getting Full======
  
You'll get output like this:
+
Let's say you have a partition getting full and you need to move the redolog to another partition or nfs mount temporary to deal with the potential crisis that will happen when the partition becomes full. '''You'll need to reallocate the complete redolog/ directory and the archive subdirectory to the same partition''' because the roll over from redo.log to the archive directory happens with a rename function within the java code. '''This will require downtime''' since you'll need to move the actual redo.log file and zimbra can't be running while you do this. You can use a symlink to your new partition path. For example:
  
<pre>
+
su - zimbra
VERIFYING: redo.log
+
zmcontrol stop
HEADER
+
su -
------
+
  ** as root
sequence: 59
+
mkdir /data/redolog
open: 1
+
chown zimbra:zimbra /data/redolog
filesize: 512
+
mount /dev/sdb1 /data/redolog
serverId: d5c5d6a7-b82f-4c29-b0cd-91818057196b
+
mv /opt/zimbra/redolog/* /data/redolog/
firstOpTstamp: 1222385426273
+
  ** or use rsync
lastOpTstamp: 1222385426273
+
rmdir /opt/zimbra/redolog
version: 1.22
+
ln -s /opt/zimbra/redolog /data/redolog
 +
ls -laR /data/redolog
 +
** confirm ownership is with zimbra and double check zimbra can write in this directory
 +
su - zimbra
 +
touch /data/redolog/testfile
 +
rm /data/redolog/testfile
 +
  zmcontrol start
 +
 
 +
=====Automatic Deleting Of Redo Logs On Rollover=====
 +
 
 +
This variable (zimbraRedoLogDeleteOnRollover) is set TRUE or FALSE.
 +
 
 +
zmprov gacf | grep zimbraRedoLogDeleteOnRollover
 +
 
 +
To modify it
  
------
+
zmprov mcf zimbraRedoLogDeleteOnRollover TRUE
txn 1222383600.1 [PurgeOldMessages] ver=1.22, tstamp=1222385426273, change=20200, mailbox=1
 
txn 1222383600.1 [CommitTxn] ver=1.22, tstamp=1222385426329, mailbox=1, txnType=PurgeOldMessages
 
txn 1222383600.2 [PurgeOldMessages] ver=1.22, tstamp=1222385486337, change=13500, mailbox=3
 
txn 1222383600.2 [CommitTxn] ver=1.22, tstamp=1222385486351, mailbox=3, txnType=PurgeOldMessages
 
txn 1222383600.3 [PurgeOldMessages] ver=1.22, tstamp=1222385546357, change=20201, mailbox=1
 
txn 1222383600.3 [CommitTxn] ver=1.22, tstamp=1222385546383, mailbox=1, txnType=PurgeOldMessages
 
txn 1222383600.4 [PurgeOldMessages] ver=1.22, tstamp=1222385606391, change=13501, mailbox=3
 
txn 1222383600.4 [CommitTxn] ver=1.22, tstamp=1222385606404, mailbox=3, txnType=PurgeOldMessages
 
txn 1222383600.5 [PurgeOldMessages] ver=1.22, tstamp=1222385666416, change=20202, mailbox=1
 
txn 1222383600.5 [CommitTxn] ver=1.22, tstamp=1222385666428, mailbox=1, txnType=PurgeOldMessages
 
txn 1222383600.6 [PurgeOldMessages] ver=1.22, tstamp=1222385726435, change=13502, mailbox=3
 
txn 1222383600.6 [CommitTxn] ver=1.22, tstamp=1222385726459, mailbox=3, txnType=PurgeOldMessages
 
txn 1222383600.7 [PurgeOldMessages] ver=1.22, tstamp=1222385786476, change=20203, mailbox=1
 
txn 1222383600.7 [CommitTxn] ver=1.22, tstamp=1222385786486, mailbox=1, txnType=PurgeOldMessages
 
txn 1222383600.8 [PurgeOldMessages] ver=1.22, tstamp=1222385846493, change=13503, mailbox=3
 
txn 1222383600.8 [CommitTxn] ver=1.22, tstamp=1222385846506, mailbox=3, txnType=PurgeOldMessages
 
txn 1222383600.9 [PurgeOldMessages] ver=1.22, tstamp=1222385906739, change=20204, mailbox=1
 
txn 1222383600.9 [CommitTxn] ver=1.22, tstamp=1222385906775, mailbox=1, txnType=PurgeOldMessages
 
txn 1222383600.10 [PurgeOldMessages] ver=1.22, tstamp=1222385966944, change=13504, mailbox=3
 
txn 1222383600.10 [CommitTxn] ver=1.22, tstamp=1222385966963, mailbox=3, txnType=PurgeOldMessages
 
txn 1222383600.11 [PurgeOldMessages] ver=1.22, tstamp=1222386026972, change=20205, mailbox=1
 
txn 1222383600.11 [CommitTxn] ver=1.22, tstamp=1222386026990, mailbox=1, txnType=PurgeOldMessages
 
...
 
</pre>
 
  
====How Do I Figure Out Which Sequence or Time Variable To Use For Restore Or Replay====
+
====Want To See What's In Redolog Files====
  
:'''''In 5.0.10+ we'll have a CLI wrapper (zmredodump) with a slightly different command line syntax, but the below long syntax works in earlier versions.'''''
+
'''This is for older versions of ZCS - newer versions should use zmredodump if it's available.'''
  
To locate the correct restore-to time, you have to start with an approximate time the message was added/deleted.  Look at the redolog files.  The filename contains the GMT time when the file was rolled over, which is roughly the tstamp of the last operation in the file.  If your time data is accurate you can find the specific file.  Or you have a range of files to examine.
+
If you suspect there's too much redolog activity during a time window or have another need to inspect the contents of the redolog, dump it and examine it:
  
Use the redolog verify tool to dump the contents into text form, the -m / --message option to show message body data:
+
$ zmjava com.zimbra.cs.redolog.util.RedoLogVerify /opt/zimbra/redolog/redo.log > out.file
  
<pre>zmjava com.zimbra.cs.redolog.util.RedoLogVerify -m <filename or directory> ... > out.file</pre>
+
Pick the right redolog file, either redo.log or one of the files under archive/, based on timestamp.
  
If the message was deleted and you don't know the id, you must go by some other clue such as the subject. Search the file to locate your message.  You can cut/paste the message and lmtp-inject it to recover the message.  No need to go through with a restore if this is all you needed.
+
====zmplayredo And zmredodump====
  
====Are You Messages Really Gone - Things To Check If zmplayredo Isn't Doing What You Expect====
+
=====zmplayredo - Replaying Content From Any Redolog File=====
  
Here's something I found out testing zmplayredo for a customer case. Testing on a ZCS 6.0.8 single ZCS server.
+
zmplayredo is a newer command, first introduced in 5.0.5 I believe. The mailbox has to be stop to run zmplayredo.
  
Created a test account and sent it one message that is in the Inbox. I delete the msg in zwc but don't purge the Trash - msg is in Trash now.
+
The help output from 6.0.8:
 
 
Log events of above action:
 
 
<pre>
 
<pre>
2010-10-27 15:07:13,375 INFO [btpool0-3://192.168.0.71/service/soap/ConvActionRequest]
+
$ zmplayredo --help
[name=ajcody@mail71.DOMAIN.com;mid=17;ip=192.168.0.17;ua=ZimbraWebClient
+
usage: zmplayredo <options>
- FF3.0 (Linux)/6.0.8_GA_2661;] mailop - Moving VirtualConversation (id=-257) to  
+
    --fromSeq <arg>        Replay from this redolog sequence (inclusive)
Folder Trash (id=3). Affected message ids: 257.
+
    --fromTime <arg>        Replay from this time (inclusive)
</pre>
+
-h,--help                  Show help (this output)
 +
    --logfiles <arg>        Replay these logfiles, in order
 +
    --mailboxId <arg>      Replay for this mailbox only
 +
    --queueCapacity <arg>  Queue capacity per player thread; default=100
 +
    --stopOnError          Stop replay on any error
 +
    --threads <arg>        Number of parallel redo threads; default=50
 +
    --toSeq <arg>          Replay to this redolog sequence (inclusive)
 +
    --toTime <arg>          Replay to this time (inclusive)
  
Stop mailboxd so I can use zmplayredo and then start mailboxd back up after zmplayredo is finished :
+
Specify date/time in one of these formats:
  
<pre>
+
    2010/11/19 13:55:08
zmmailboxdctl stop
+
    2010/11/19 13:55:08 802
zmplayredo --mailboxId 17 --logfiles /opt/zimbra/redolog/archive/* --fromTime 20101027090000 --toTime 20101027140000
+
    2010/11/19 13:55:08.802
zmmailboxdctl start
+
    2010/11/19-13:55:08-802
</pre>
+
    2010/11/19-13:55:08
 +
    20101119.135508.802
 +
    20101119.135508
 +
    20101119135508802
 +
    20101119135508
  
Log event for above:
+
Specify year, month, date, hour, minute, second, and optionally millisecond.
 
+
Month/date/hour/minute/second are 0-padded to 2 digits, millisecond to 3 digits.
<pre>
+
Hour must be specified in 24-hour format, and time is in local time zone.
2010-10-27 15:07:50,383 INFO [main] [] mailop - Adding Message: id=257, Message-ID=
 
<1604937630.920784.1288201208382.JavaMail.root@corp.zimbra.com>, parentId=-1, folderId=2, folderName=Inbox.
 
2010-10-27 15:07:50,404 INFO [main] [] RedoableOp - Message 257 is already in mailbox 17
 
 
</pre>
 
</pre>
  
Log into ZWC with the test account. The msg is not in the Inbox, but it's still in Trash folder. I purge it from Trash.
+
=====zmredodump - Replaying Content From Any Redolog File=====
  
Log event for deletion.
+
zmredodump is a newer command and very useful. It does not require mailboxd to be stopped like zmplayredo does.
  
 +
The help output from 6.0.8:
 
<pre>
 
<pre>
2010-10-27 15:09:38,761 INFO [btpool0-2://192.168.0.71/service/soap/ConvActionRequest]
+
$ zmredodump --help
[name=ajcody@mail71.DOMAIN.com;mid=17;ip=192.168.0.17;ua=ZimbraWebClient
+
usage: zmredodump [options] <redolog file/directory> [...]
- FF3.0 (Linux)/6.0.8_GA_2661;] mailop - Deleting Message (id=257).
+
where [options] are:
</pre>
 
  
Then I redo the stop/start of mailboxd and zmplayredo again.
+
-h,--help        show this output
 
+
    --m <arg>     one or more mailbox ids separated by comma or white
<pre>
+
                  space.  The entire list must be quoted if using space as separator. If
  zmmailboxdctl stop
+
                  this option is given, only redo ops for the specified mailboxes are
  zmplayredo --mailboxId 17 --logfiles /opt/zimbra/redolog/archive/* --fromTime 20101027090000 --toTime 20101027140000
+
                  dumped. Omit this option to dump redo ops for all mailboxes.
  zmmailboxdctl start
+
    --no-offset  don't show file offsets and size for each redo op
 +
-q,--quiet      quiet mode.  Only print the log filename and any errors.
 +
                  This option can be used to verify the integrity of redologs with minimal
 +
                  output.
 +
    --show-blob  show blob content.  Item's blob is printed, surrounded
 +
                  by <START OF BLOB> and <END OF BLOB> markers.  The last newline before end
 +
                  marker is not part of the blob.
 +
 
 +
Multiple log files/directories can be specified. For each directory, all
 +
redolog files directly under it are processed, sorted in ascending redolog
 +
sequence order.
 
</pre>
 
</pre>
  
Log event for the above:
+
=====Using zmredodump To Get Message Blobs To Inject With zmlmtpinject - RFE=====
  
<pre>
+
Please see:
2010-10-27 15:10:29,192 INFO [main] [] mailop - Adding Message: id=257, Message-ID=
 
<1604937630.920784.1288201208382.JavaMail.root@corp.zimbra.com>, parentId=-1,
 
folderId=2, folderName=Inbox.
 
</pre>
 
  
Log back into ZWC with test account and now I can confirm that the msg is not in Trash and it is now showing in Inbox.
+
* "RFE: zmredodump blobs to single files for zmlmtpinject [for example]"
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=52930
 +
* See also:
 +
** "RFE: zmplayredo option for --frommailboxId --tomailboxId"
 +
*** http://bugzilla.zimbra.com/show_bug.cgi?id=52642
  
====Gap In Redo Log====
+
=====Expand Ability To Parse Redologs - RFE=====
  
The error message from either a backup or restore command:
+
Please see:
  
:::'''''"Error occurred: Found gap in redo log sequence; missing 5965 through 6149;'''''
+
* "expand ability to parse redologs for recovery purposes - zmredodump / zmplayredo"
:::'''''To avoid future restore problems, discard all existing backups and take a'''''
+
** http://bugzilla.zimbra.com/show_bug.cgi?id=37743
:::'''''full backup of all accounts; If this error occurred during restore,'''''
 
:::'''''try the --ignoreRedoErrors option"'''''
 
  
The output is pretty accurate in how to handle the situation.
+
====Getting A Sequence or Time Variable For Restore Or Replay====
  
If you get the error during a backup, the recommendation is to move your old backups out. The directories in /opt/zimbra/backup/sessions/* . You'll want to keep them around just in case and then proceed to do a full backup.
+
You can see the changes within the redo logs with the command below. You can point it to any redolog.
  
If you get the error during a restore, you would add the flag --ignoreRedoErrors to your restore command.
+
<pre>zmjava com.zimbra.cs.redolog.util.RedoLogVerify /opt/zimbra/redolog/redo.log > out.file</pre>
  
Another possible related issue is if your /tmp or /opt/zimbra/redolog/ is filling up.
+
You'll get output like this:
  
====Error Executing redoOp====
+
<pre>
 +
VERIFYING: redo.log
 +
HEADER
 +
------
 +
sequence: 59
 +
open: 1
 +
filesize: 512
 +
serverId: d5c5d6a7-b82f-4c29-b0cd-91818057196b
 +
firstOpTstamp: 1222385426273
 +
lastOpTstamp:  1222385426273
 +
version: 1.22
  
Errors with restores that involve the message 'error executing redoOp' will not show up in the admin console but will when you attempt the restore from CLI. This can also be the cause when you use the RestoreToTime option from the admin console and it doesn't seem to work correctly - the restore stopping prematurely from the specified date/time.  
+
------
 
+
txn 1222383600.1 [PurgeOldMessages] ver=1.22, tstamp=1222385426273, change=20200, mailbox=1
I created the following RFE in regards to the admin console issue:
+
txn 1222383600.1 [CommitTxn] ver=1.22, tstamp=1222385426329, mailbox=1, txnType=PurgeOldMessages
 +
txn 1222383600.2 [PurgeOldMessages] ver=1.22, tstamp=1222385486337, change=13500, mailbox=3
 +
txn 1222383600.2 [CommitTxn] ver=1.22, tstamp=1222385486351, mailbox=3, txnType=PurgeOldMessages
 +
txn 1222383600.3 [PurgeOldMessages] ver=1.22, tstamp=1222385546357, change=20201, mailbox=1
 +
txn 1222383600.3 [CommitTxn] ver=1.22, tstamp=1222385546383, mailbox=1, txnType=PurgeOldMessages
 +
txn 1222383600.4 [PurgeOldMessages] ver=1.22, tstamp=1222385606391, change=13501, mailbox=3
 +
txn 1222383600.4 [CommitTxn] ver=1.22, tstamp=1222385606404, mailbox=3, txnType=PurgeOldMessages
 +
txn 1222383600.5 [PurgeOldMessages] ver=1.22, tstamp=1222385666416, change=20202, mailbox=1
 +
txn 1222383600.5 [CommitTxn] ver=1.22, tstamp=1222385666428, mailbox=1, txnType=PurgeOldMessages
 +
txn 1222383600.6 [PurgeOldMessages] ver=1.22, tstamp=1222385726435, change=13502, mailbox=3
 +
txn 1222383600.6 [CommitTxn] ver=1.22, tstamp=1222385726459, mailbox=3, txnType=PurgeOldMessages
 +
txn 1222383600.7 [PurgeOldMessages] ver=1.22, tstamp=1222385786476, change=20203, mailbox=1
 +
txn 1222383600.7 [CommitTxn] ver=1.22, tstamp=1222385786486, mailbox=1, txnType=PurgeOldMessages
 +
txn 1222383600.8 [PurgeOldMessages] ver=1.22, tstamp=1222385846493, change=13503, mailbox=3
 +
txn 1222383600.8 [CommitTxn] ver=1.22, tstamp=1222385846506, mailbox=3, txnType=PurgeOldMessages
 +
txn 1222383600.9 [PurgeOldMessages] ver=1.22, tstamp=1222385906739, change=20204, mailbox=1
 +
txn 1222383600.9 [CommitTxn] ver=1.22, tstamp=1222385906775, mailbox=1, txnType=PurgeOldMessages
 +
txn 1222383600.10 [PurgeOldMessages] ver=1.22, tstamp=1222385966944, change=13504, mailbox=3
 +
txn 1222383600.10 [CommitTxn] ver=1.22, tstamp=1222385966963, mailbox=3, txnType=PurgeOldMessages
 +
txn 1222383600.11 [PurgeOldMessages] ver=1.22, tstamp=1222386026972, change=20205, mailbox=1
 +
txn 1222383600.11 [CommitTxn] ver=1.22, tstamp=1222386026990, mailbox=1, txnType=PurgeOldMessages
 +
...
 +
</pre>
  
* "Include --ignoreRedoErrors option and error feedback in Admin Console for restore"
+
====How Do I Figure Out Which Sequence or Time Variable To Use For Restore Or Replay====
** http://bugzilla.zimbra.com/show_bug.cgi?id=52358
 
*** This could explain why your restore to time isn't working in the Admin Console but does from CLI when you see an error about redologs and then reattempt restore with the --ignoreRedoErrors and it works.
 
  
Another RFE that was made but marked as 'WONTFIX' that gives a background story to the issue is:
+
:'''''In 5.0.10+ we'll have a CLI wrapper (zmredodump) with a slightly different command line syntax, but the below long syntax works in earlier versions.'''''
  
* "Need more robust redolog serialization format"
+
To locate the correct restore-to time, you have to start with an approximate time the message was added/deleted.  Look at the redolog files.  The filename contains the GMT time when the file was rolled over, which is roughly the tstamp of the last operation in the file.  If your time data is accurate you can find the specific file. Or you have a range of files to examine.
** http://bugzilla.zimbra.com/show_bug.cgi?id=7080
+
 
 +
Use the redolog verify tool to dump the contents into text form, the -m / --message option to show message body data:
  
When you hit this error with your backup data during restore attempts, there's basically only a couple of options to handle this that are recommended by support:
+
<pre>zmjava com.zimbra.cs.redolog.util.RedoLogVerify -m <filename or directory> ... > out.file</pre>
  
# Try to get full backups of your accounts or the accounts in question and then test against them and the preceding incrementals after the full.
+
If the message was deleted and you don't know the id, you must go by some other clue such as the subject. Search the file to locate your message. You can cut/paste the message and lmtp-inject it to recover the message. No need to go through with a restore if this is all you needed.
# Attempt restores via the CLI using the additional option of :  --ignoreRedoErrors
 
#* Similar to the various steps described here - [[Ajcody-Backup-Restore-Issues#Restore_To_Time_Problems]] - but with the --ignoreRedoErrors option included.
 
# Do your restore against the latest full backup of the account in question and then use the zmplayredo command against the redologs in the incrementals and/or the /opt/zimbra/redologs/* directory . This will give you more control to walk the restored account up to the point in time you want it at. One should really read through the whole section above, [[Ajcody-Backup-Restore-Issues#Redolog_Files]] , to understand the whole concept of redologs and then the use of zmplayredo.
 
  
Generally, "fixing" the redolog itself is not an option.
+
====Are You Messages Really Gone - Things To Check If zmplayredo Isn't Doing What You Expect====
  
===Why Do My Fulls Not Report All Accounts?===
+
Here's something I found out testing zmplayredo for a customer case. Testing on a ZCS 6.0.8 single ZCS server.
  
Are you sure it was a full backup that was ran or just a full session that was generated from your incremental backup job? When an incremental is ran, it will create a "full" session for any new accounts it discovers after the last actual full backup job.
+
Created a test account and sent it one message that is in the Inbox. I delete the msg in zwc but don't purge the Trash - msg is in Trash now.
  
For example, here's a full session that was created by an incremental backup job:
+
Log events of above action:
 
<pre>
 
<pre>
Label: full-20081010.060126.559
+
2010-10-27 15:07:13,375 INFO [btpool0-3://192.168.0.71/service/soap/ConvActionRequest]
Type: full
+
[name=ajcody@mail71.DOMAIN.com;mid=17;ip=192.168.0.17;ua=ZimbraWebClient
Status: completed
+
- FF3.0 (Linux)/6.0.8_GA_2661;] mailop - Moving VirtualConversation (id=-257) to
Started: Fri, 2008/10/10 01:01:26.559 CDT
+
Folder Trash (id=3). Affected message ids: 257.
Ended: Fri, 2008/10/10 01:01:28.988 CDT
+
</pre>
Redo log sequence range: 705 .. 705
 
Number of accounts: 1
 
  
Label: incr-20081010.060009.420
+
Stop mailboxd so I can use zmplayredo and then start mailboxd back up after zmplayredo is finished :
Type: incremental
 
Status: completed
 
Started: Fri, 2008/10/10 01:00:09.420 CDT
 
Ended: Fri, 2008/10/10 01:01:26.413 CDT
 
Redo log sequence range: 700 .. 704
 
Number of accounts: 392
 
  
 
+
<pre>
[zimbra@servername ~]$ du -sh /opt/zimbra/backup/sessions/full-20081010.060126.559
+
zmmailboxdctl stop
1.2M /opt/zimbra/backup/sessions/full-20081010.060126.559
+
zmplayredo --mailboxId 17 --logfiles /opt/zimbra/redolog/archive/* --fromTime 20101027090000 --toTime 20101027140000
[zimbra@servername ~]$ du -sh /opt/zimbra/backup/sessions/incr-20081003.060010.622
+
zmmailboxdctl start
452M /opt/zimbra/backup/sessions/incr-20081003.060010.622
 
 
</pre>
 
</pre>
  
Notice the Start and End times, this will show that the full is related to the incremental job.
+
Log event for above:
  
You'll want to run zmbackupquery against your full labels to see your "main" full backup session - assuming you can't simply guess based upon the cron entry for it [ su - zimbra ; crontab -l | grep backup ]. For example, to see all your fulls from today's date back to October 01, 2008 and the accounts within each session - you would do:
+
<pre>
 +
2010-10-27 15:07:50,383 INFO [main] [] mailop - Adding Message: id=257, Message-ID=
 +
<1604937630.920784.1288201208382.JavaMail.root@corp.zimbra.com>, parentId=-1, folderId=2, folderName=Inbox.
 +
2010-10-27 15:07:50,404 INFO [main] [] RedoableOp - Message 257 is already in mailbox 17
 +
</pre>
  
zmbackupquery -v --from 20081001.000000 --type full
+
Log into ZWC with the test account. The msg is not in the Inbox, but it's still in Trash folder. I purge it from Trash.
  
The -v flag outputs the accounts, the --from uses YearMonthDay.HourMinuteSecond , and the --type can be full or incremental. To just see one particular sessions date, you would use the lb [label] flag:
+
Log event for deletion.
  
zmbackupquery -v -lb [your full label, ex. full-20081001.000000]
+
<pre>
 +
2010-10-27 15:09:38,761 INFO [btpool0-2://192.168.0.71/service/soap/ConvActionRequest]
 +
[name=ajcody@mail71.DOMAIN.com;mid=17;ip=192.168.0.17;ua=ZimbraWebClient
 +
- FF3.0 (Linux)/6.0.8_GA_2661;] mailop - Deleting Message (id=257).
 +
</pre>
  
===Issues After /opt/zimbra/backup Became Full===
+
Then I redo the stop/start of mailboxd and zmplayredo again.
  
Bugs/RFE's on this issue:
+
<pre>
* "backup should give better error when disk is full"
+
zmmailboxdctl stop
** http://bugzilla.zimbra.com/show_bug.cgi?id=16482
+
zmplayredo --mailboxId 17 --logfiles /opt/zimbra/redolog/archive/* --fromTime 20101027090000 --toTime 20101027140000
* Was marked dup'd of above, "Issues when backup partition becomes full"
+
zmmailboxdctl start
** http://bugzilla.zimbra.com/show_bug.cgi?id=32017
+
</pre>
  
See also[[Ajcody-Backup-Restore-Issues#Possible_Issue_That_A_Failed_Or_Interrupted_Backup_Causes]] and the Bugs & RFE's under that section.
+
Log event for the above:
  
You can run into numerous issues if you allow your backup directory to become full.
+
<pre>
 +
2010-10-27 15:10:29,192 INFO [main] [] mailop - Adding Message: id=257, Message-ID=
 +
<1604937630.920784.1288201208382.JavaMail.root@corp.zimbra.com>, parentId=-1,
 +
folderId=2, folderName=Inbox.
 +
</pre>
  
Confirm your /opt/zimbra/backup/accounts.xml is being updated after a backup. You might see that the newer account.xml* file is accounts.xml.new . This is a sign of problems.
+
Log back into ZWC with test account and now I can confirm that the msg is not in Trash and it is now showing in Inbox.
  
Confirm that the files in /opt/zimbra/backup/tmp/* don't have 0 byte lengths. There might be files like 1.xml and 3.xml in there. If they show 0 bytes, you need to remove them. The backup/restore commands if the file exist and they are empty. Your errors might look like this:
+
====Gap In Redo Log====
  
[zimbra@mailb ~]$ zmrestore -a USER@DOMAIN -ca -pre restore_
+
The error message from either a backup or restore command:
Error occurred: system failure: Unable to parse XML file /opt/zimbra/backup/tmp/restore/shared_blobs/1.xml
 
  
If you tried doing restores (redirected -ca -pre) before to clear up the above issues, you might find you can't do a successful restore AND you can't delete the account afterwards.
+
:::'''''"Error occurred: Found gap in redo log sequence; missing 5965 through 6149;'''''
 
+
:::'''''To avoid future restore problems, discard all existing backups and take a'''''
If you get errors like :
+
:::'''''full backup of all accounts; If this error occurred during restore,'''''
 +
:::'''''try the --ignoreRedoErrors option"'''''
  
zmprov da restore3_tester3@XXXXXX.edu
+
The output is pretty accurate in how to handle the situation.
ERROR: service.FAILURE (system failure: writing new mailbox row for account 89e7d9f4-013e-4cf1-a352-7b2f0a00d5af)
 
 
zmprov gmi restored_USER
 
ERROR: service.FAILURE (system failure: writing new mailbox row for account 56a7f654-f85b-45cc-931a-81d9bb9076bf)
 
  
You'll need to delete the account via a ldapdelete command.
+
If you get the error during a backup, the recommendation is to move your old backups out. The directories in /opt/zimbra/backup/sessions/* . You'll want to keep them around just in case and then proceed to do a full backup.
  
===Query And Stopping A Backup or Restore In Progress===
+
If you get the error during a restore, you would add the flag --ignoreRedoErrors to your restore command.
  
Please see the url below for Backup In Progress - topic name "Aborting Full Backup In Progress":
+
Another possible related issue is if your /tmp or /opt/zimbra/redolog/ is filling up.
  
* A matter of using zmbackupquery & zmbackupabort .
+
====Error Executing redoOp====
  
Please see url below for Restore In Progress - topic name "Stopping a Restore Process" :
+
Errors with restores that involve the message 'error executing redoOp' will not show up in the admin console but will when you attempt the restore from CLI. This can also be the cause when you use the RestoreToTime option from the admin console and it doesn't seem to work correctly - the restore stopping prematurely from the specified date/time.
  
* A matter of using :  ''zmbackupabort -r''
+
I created the following RFE in regards to the admin console issue:
** Latest ZCS Admin Guide Table of Contents, search for the topic name mentioned above.
 
*** http://www.zimbra.com/docs/ne/latest/administration_guide/toc.html
 
** ZCS 4.5.10 Admin Guide with static links to back section, you should review info in latest admin guide though.
 
*** http://www.zimbra.com/docs/ne/4.5.10/administration_guide/10_Backup_Restore.13.1.html#1042283
 
** '''Note''' - what is absent is an easy way to query restores in progress though. Please see:
 
*** "Status for Zmrestore"
 
**** http://bugzilla.zimbra.com/show_bug.cgi?id=47238
 
**** Currently, status of a restore can only be monitored via the logging events in the mailbox.log file.
 
  
====A -del Delete In Progress====
+
* "Include --ignoreRedoErrors option and error feedback in Admin Console for restore"
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=52358
 +
*** This could explain why your restore to time isn't working in the Admin Console but does from CLI when you see an error about redologs and then reattempt restore with the --ignoreRedoErrors and it works.
  
Please see the bug / rfe's I filed
+
Another RFE that was made but marked as 'WONTFIX' that gives a background story to the issue is:
* "need way to kill/recover from zmbackup -del when in progress"
 
** http://bugzilla.zimbra.com/show_bug.cgi?id=31869
 
* "zmbackup -del shouldn't prevent zmbackup -f/-i from running for ALL cases"
 
** http://bugzilla.zimbra.com/show_bug.cgi?id=33294
 
  
====zmbackupabort syntax====
+
* "Need more robust redolog serialization format"
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=7080
  
From ZCS 6.0.8
+
When you hit this error with your backup data during restore attempts, there's basically only a couple of options to handle this that are recommended by support:
  
<pre>
+
# Try to get full backups of your accounts or the accounts in question and then test against them and the preceding incrementals after the full.
$ zmbackupabort -h
+
# Attempt restores via the CLI using the additional option of :  --ignoreRedoErrors
usage: zmbackupabort <options>
+
#* Similar to the various steps described here - [[Ajcody-Backup-Restore-Issues#Restore_To_Time_Problems]] - but with the --ignoreRedoErrors option included.
  -d,--debug          Display diagnostics for debugging purposes.
+
# Do your restore against the latest full backup of the account in question and then use the zmplayredo command against the redologs in the incrementals and/or the /opt/zimbra/redologs/* directory . This will give you more control to walk the restored account up to the point in time you want it at. One should really read through the whole section above, [[Ajcody-Backup-Restore-Issues#Redolog_Files]] , to understand the whole concept of redologs and then the use of zmplayredo.
-h,--help          Displays this help message.
 
-lb,--label <arg>  Label of full backup set to abort.
 
-r,--restore        Abort the restore in progress.
 
-s,--server <arg>  Mail server hostname. Default is localhost.
 
-t,--target <arg>  Backup target location (default
 
                    <zimbra_home>/backup).
 
</pre>
 
  
====zmbackupquery syntax====
+
Generally, "fixing" the redolog itself is not an option.
  
From ZCS 6.0.8
+
===Why Do My Fulls Not Report All Accounts?===
  
 +
Are you sure it was a full backup that was ran or just a full session that was generated from your incremental backup job? When an incremental is ran, it will create a "full" session for any new accounts it discovers after the last actual full backup job.
 +
 +
For example, here's a full session that was created by an incremental backup job:
 
<pre>
 
<pre>
$ zmbackupquery -h
+
Label: full-20081010.060126.559
usage: zmbackupquery <options>
+
Type: full
-a,--account <arg>  Account email addresses seperated by white space or
+
Status: completed
                      "all" for all accounts.
+
Started: Fri, 2008/10/10 01:01:26.559 CDT
-d,--debug          Display diagnostics for debugging purposes.
+
Ended: Fri, 2008/10/10 01:01:28.988 CDT
    --from <arg>      List backups whose start date/time is at or after
+
Redo log sequence range: 705 .. 705
                      this date/time.
+
Number of accounts: 1
-h,--help            Displays this help message.
+
 
-lb,--label <arg>    The label of full backup to query.
+
Label: incr-20081010.060009.420
-s,--server <arg>    Mail server hostname. Default is localhost.
+
Type: incremental
-t,--target <arg>    Backup target location (default
+
Status: completed
                      <zimbra_home>/backup).
+
Started: Fri, 2008/10/10 01:00:09.420 CDT
    --to <arg>        List backups whose start date/time is at or before
+
Ended: Fri, 2008/10/10 01:01:26.413 CDT
                      this date/time.
+
Redo log sequence range: 700 .. 704
    --type <arg>      Backup set type to query. "full" or "incremental";
+
Number of accounts: 392
                      both if unspecified.
 
-v,--verbose        Show account list in each backup.
 
  
Specify date/time in one of these formats:
 
 
    2010/11/19 14:06:22
 
    2010/11/19 14:06:22 923
 
    2010/11/19 14:06:22.923
 
    2010/11/19-14:06:22-923
 
    2010/11/19-14:06:22
 
    20101119.140622.923
 
    20101119.140622
 
    20101119140622923
 
    20101119140622
 
  
Specify year, month, date, hour, minute, second, and optionally millisecond.
+
[zimbra@servername ~]$ du -sh /opt/zimbra/backup/sessions/full-20081010.060126.559
Month/date/hour/minute/second are 0-padded to 2 digits, millisecond to 3 digits.
+
1.2M /opt/zimbra/backup/sessions/full-20081010.060126.559
Hour must be specified in 24-hour format, and time is in local time zone.
+
[zimbra@servername ~]$ du -sh /opt/zimbra/backup/sessions/incr-20081003.060010.622
 +
452M /opt/zimbra/backup/sessions/incr-20081003.060010.622
 
</pre>
 
</pre>
  
===Restore To Time Problems===
+
Notice the Start and End times, this will show that the full is related to the incremental job.
  
There's seems to be some syntax issues when using this variable. Please review the following to confirm your syntax.
+
You'll want to run zmbackupquery against your full labels to see your "main" full backup session - assuming you can't simply guess based upon the cron entry for it [ su - zimbra ; crontab -l | grep backup ]. For example, to see all your fulls from today's date back to October 01, 2008 and the accounts within each session - you would do:
  
http://wiki.zimbra.com/index.php?title=CLI_zmrestore_restoreToTime_Network_Edition_only
+
zmbackupquery -v --from 20081001.000000 --type full
  
The gist, you '''MUST''' use the -lb full-200xxxxxx option when your trying to restore anything that ISN'T meant to include the latest information of the mailbox. The -lb argument should specify a full backup that took place prior to the time of the backup you wish to restore.
+
The -v flag outputs the accounts, the --from uses YearMonthDay.HourMinuteSecond , and the --type can be full or incremental. To just see one particular sessions date, you would use the lb [label] flag:
  
Example:
+
zmbackupquery -v -lb [your full label, ex. full-20081001.000000]
  
zmrestore -a USER@DOMAIN.com -restoreToIncrLabel incr-20080731.060007.644 -lb full-20080726.050017.306 -br -ca -pre restore_
+
===Issues After /opt/zimbra/backup Became Full===
  
zmrestore -a USER@DOMAIN.com -restoreToTime 20080801011800 -lb full-20080726.050017.306 -br -ca -pre restore_
+
Bugs/RFE's on this issue:
 +
* "backup should give better error when disk is full"
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=16482
 +
* Was marked dup'd of above, "Issues when backup partition becomes full"
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=32017
 +
 
 +
See also[[Ajcody-Backup-Restore-Issues#Possible_Issue_That_A_Failed_Or_Interrupted_Backup_Causes]] and the Bugs & RFE's under that section.
  
Backup label (-lb) for fulls can be found by:
+
You can run into numerous issues if you allow your backup directory to become full.
  
zmbackupquery | grep full
+
Confirm your /opt/zimbra/backup/accounts.xml is being updated after a backup. You might see that the newer account.xml* file is accounts.xml.new . This is a sign of problems.
  
Backup labels (-restoreToIncrLabel) for incrementals can be found by:
+
Confirm that the files in /opt/zimbra/backup/tmp/* don't have 0 byte lengths. There might be files like 1.xml and 3.xml in there. If they show 0 bytes, you need to remove them. The backup/restore commands if the file exist and they are empty. Your errors might look like this:
  
  zmbackupquery | grep incr
+
  [zimbra@mailb ~]$ zmrestore -a USER@DOMAIN -ca -pre restore_
 +
Error occurred: system failure: Unable to parse XML file /opt/zimbra/backup/tmp/restore/shared_blobs/1.xml
  
'''Important Options You Might Want Or Need To Include'''
+
If you tried doing restores (redirected -ca -pre) before to clear up the above issues, you might find you can't do a successful restore AND you can't delete the account afterwards.
  
'''--ignoreRedoErrors''' : If you attempt a restore and you see an error about problems related to playing the redolog, you'll want to run the restore command again and include this option.
+
If you get errors like :
  
 +
zmprov da restore3_tester3@XXXXXX.edu
 +
ERROR: service.FAILURE (system failure: writing new mailbox row for account 89e7d9f4-013e-4cf1-a352-7b2f0a00d5af)
 +
 +
zmprov gmi restored_USER
 +
ERROR: service.FAILURE (system failure: writing new mailbox row for account 56a7f654-f85b-45cc-931a-81d9bb9076bf)
  
'''--skipDeletes''' : Please see http://bugzilla.zimbra.com/show_bug.cgi?id=31824#c5 for details on this.
+
You'll need to delete the account via a ldapdelete command.
  
 +
===Query And Stopping A Backup or Restore In Progress===
  
'''-t /path/to/backup_dir''' : If you are restoring from another backup directory besides your current default path.
+
Please see the url below for Backup In Progress - topic name "Aborting Full Backup In Progress":
  
 +
* A matter of using zmbackupquery & zmbackupabort .
  
Variables that are asking for TIME rather than LABELS should follow this syntax (from zmrestore --help):
+
Please see url below for Restore In Progress - topic name "Stopping a Restore Process" :
<pre>
 
Specify date/time in one of these formats:
 
  
    2008/08/06 09:55:50
+
* A matter of using :  ''zmbackupabort -r''
    2008/08/06 09:55:50 572
+
** Latest ZCS Admin Guide Table of Contents, search for the topic name mentioned above.
    2008/08/06 09:55:50.572
+
*** http://www.zimbra.com/docs/ne/latest/administration_guide/toc.html
    2008/08/06-09:55:50-572
+
** ZCS 4.5.10 Admin Guide with static links to back section, you should review info in latest admin guide though.
    2008/08/06-09:55:50
+
*** http://www.zimbra.com/docs/ne/4.5.10/administration_guide/10_Backup_Restore.13.1.html#1042283
    20080806.095550.572
+
** '''Note''' - what is absent is an easy way to query restores in progress though. Please see:
    20080806.095550
+
*** "Status for Zmrestore"
    20080806095550572
+
**** http://bugzilla.zimbra.com/show_bug.cgi?id=47238
    20080806095550
+
**** Currently, status of a restore can only be monitored via the logging events in the mailbox.log file.
 +
 
 +
====A -del Delete In Progress====
  
Specify year, month, date, hour, minute, second, and optionally millisecond.
+
Please see the bug / rfe's I filed
Month/date/hour/minute/second are 0-padded to 2 digits, millisecond to 3 digits.
+
* "need way to kill/recover from zmbackup -del when in progress"
Hour must be specified in 24-hour format, and time is in local time zone.
+
** http://bugzilla.zimbra.com/show_bug.cgi?id=31869
</pre>
+
* "zmbackup -del shouldn't prevent zmbackup -f/-i from running for ALL cases"
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=33294
  
====Bugs And RFE's To Review====
+
====zmbackupabort syntax====
  
* zmrestore using restoreToTime option restores data after the specified time
+
From ZCS 6.0.8
** See: http://bugzilla.zimbra.com/show_bug.cgi?id=15320
 
*** Resulting in documentation changes : [[CLI_zmrestore_Network_Edition_only]]
 
* zmrestore -restoreToTime should fail if no backup label is passed
 
** See: http://bugzilla.zimbra.com/show_bug.cgi?id=28320
 
*** "unable to restore to point in time/incremental" '''Admin GUI'''
 
**** http://bugzilla.zimbra.com/show_bug.cgi?id=33135
 
* Admin console issues and other details I've posted are in this bug:
 
** http://bugzilla.zimbra.com/show_bug.cgi?id=27746
 
** Added for search terms, admin console can't restore to point in time
 
* Include --ignoreRedoErrors option and error feedback in Admin Console for restore
 
** http://bugzilla.zimbra.com/show_bug.cgi?id=52358
 
*** This could explain why your restore to time isn't working in the Admin Console but does from CLI when you see an error about redologs and then reattempt restore with the --ignoreRedoErrors and it works.
 
  
===Restore An Individual Message===
+
<pre>
 +
$ zmbackupabort -h
 +
usage: zmbackupabort <options>
 +
-d,--debug          Display diagnostics for debugging purposes.
 +
-h,--help          Displays this help message.
 +
-lb,--label <arg>  Label of full backup set to abort.
 +
-r,--restore        Abort the restore in progress.
 +
-s,--server <arg>  Mail server hostname. Default is localhost.
 +
-t,--target <arg>  Backup target location (default
 +
                    <zimbra_home>/backup).
 +
</pre>
  
The zmrestore command is at a mailbox level.
+
====zmbackupquery syntax====
  
An RFE was filed already to expand this. It is currently targeted for the Helix release.
+
From ZCS 6.0.8
  
* "More Granular Restore: per folder & per-message"
+
<pre>
** http://bugzilla.zimbra.com/show_bug.cgi?id=8849
+
$ zmbackupquery -h
 +
usage: zmbackupquery <options>
 +
-a,--account <arg>  Account email addresses seperated by white space or
 +
                      "all" for all accounts.
 +
-d,--debug          Display diagnostics for debugging purposes.
 +
    --from <arg>      List backups whose start date/time is at or after
 +
                      this date/time.
 +
-h,--help            Displays this help message.
 +
-lb,--label <arg>    The label of full backup to query.
 +
-s,--server <arg>    Mail server hostname. Default is localhost.
 +
-t,--target <arg>    Backup target location (default
 +
                      <zimbra_home>/backup).
 +
    --to <arg>        List backups whose start date/time is at or before
 +
                      this date/time.
 +
    --type <arg>      Backup set type to query.  "full" or "incremental";
 +
                      both if unspecified.
 +
-v,--verbose        Show account list in each backup.
  
A way around the current limitations would be to use lmtpinject. Please see the following for details on that:
+
Specify date/time in one of these formats:
  
* http://www.zimbra.com/forums/installation/12617-recover-data-store-folders.html#post64962
+
    2010/11/19 14:06:22
* http://www.zimbra.com/forums/administrators/729-using-zmlmtpinject.html#post4017
+
    2010/11/19 14:06:22 923
 +
    2010/11/19 14:06:22.923
 +
    2010/11/19-14:06:22-923
 +
    2010/11/19-14:06:22
 +
    20101119.140622.923
 +
    20101119.140622
 +
    20101119140622923
 +
    20101119140622
  
The difficultly would be determining the message your trying to find within your backups that was "deleted" in prod.
+
Specify year, month, date, hour, minute, second, and optionally millisecond.
 +
Month/date/hour/minute/second are 0-padded to 2 digits, millisecond to 3 digits.
 +
Hour must be specified in 24-hour format, and time is in local time zone.
 +
</pre>
 +
 
 +
===Restore To Time Problems===
 +
 
 +
There's seems to be some syntax issues when using this variable. Please review the following to confirm your syntax.
 +
 
 +
http://wiki.zimbra.com/index.php?title=CLI_zmrestore_restoreToTime_Network_Edition_only
 +
 
 +
The gist, you '''MUST''' use the -lb full-200xxxxxx option when your trying to restore anything that ISN'T meant to include the latest information for the mailbox. '''''The -lb argument should specify a full backup that took place prior to the time of the backup you wish to restore.'''''
 +
 
 +
====Find Out What Backup Session Labels You Need First====
 +
 
 +
To find out what backups are associated with a particular account, you would do the following :
 +
 
 +
zmbackupquery -a user@domain
 +
 
 +
You'll want to note what is the first full that occurs before the point in time you want to restore. And then the incremental that follows right after your point in time.
 +
 
 +
Backup label (-lb) for fulls can be found by doing [include the -v option if you want to see a listing of the user accounts within the backups] :
 +
 
 +
zmbackupquery --type full
 +
 
 +
Backup labels (-restoreToIncrLabel) for incrementals can be found by:
  
===User Deleted A Bunch Of Data And Notified You Hours Later Wanting It Restored===
+
zmbackupquery --type incremental
  
# To determine the time of the delete, use the ''zmredodump'' command.
+
====Command Syntax Example For Restores On The CLI====
#* You'll use this "time" for the restore command.
 
#* Example:
 
#** ''$ zmprov gmi USER@DOMAIN.com''  ::: gives me mailboxid 17
 
#** ''$ zmredodump --show-blob --m 17 /opt/zimbra/redolog/ | grep Delete''  ::: returns:
 
#*** ''[0000f311 - 0000f350: 64 bytes; tstamp: 2010/11/19 11:21:30.852 CST] txn 1290033504.2544 [DeleteItem] ver=1.28, tstamp=1290187290852, change=3913, mailbox=17, ids=[304, 308], type=5''
 
#*** ''[0000f351 - 0000f382: 50 bytes; tstamp: 2010/11/19 11:21:30.869 CST] txn 1290033504.2544 [CommitTxn] ver=1.28, tstamp=1290187290869, mailbox=17, txnType=DeleteItem''
 
# See [[Ajcody-Backup-Restore-Issues#Restore_To_Time_Problems]] about setting up the proper restore syntax.
 
#* You'll want to include the ''-ca -pre restore_'' options since we'll first get the old data into a temporary account.
 
#* Remember to use ''zmbackupquery -a USER@DOMAIN.com --type full'' to find what full label will be needed for the restore.
 
#* Also, if you need/want the incremental label, do a ''zmbackupquery -a USER@DOMAIN.com --type incremental'' .
 
#** Example:
 
#*** ''$ zmbackupquery -a USER@DOMAIN.com --type full''  ::: shows the latest as being:
 
#**** ''Label:  full-20101119.070018.607''
 
#**** ''Type:    full''
 
#**** ''Started: Fri, 2010/11/19 01:00:18.607 CST''
 
#**** ''Ended:  Fri, 2010/11/19 01:00:29.751 CST''
 
#**** ''Acct ID: f33d6daf-8875-4496-8bee-6df345f295e7''
 
#* Do the restore now with the proper information for the variable flags.
 
#** '''''Note''''': You might need to include -br , -rf , or neither depending on the time frames involved.
 
#** Example:
 
#*** ''$ zmrestore -a USER@DOMAIN.com -restoreToTime 20101119112000 -lb full-20101119.070018.607 -ca -pre restore_''
 
#** It would be best to log into the restore_USER@DOMAIN.com account to confirm the data is as you expect it.
 
# Use ''zmmailbox'' with the ''getRestUrl'' option against the "restore_USER" account now to export the data.
 
#* Examples:
 
#** Export ALL the 'old' data from the restored account
 
#*** ''$ /opt/zimbra/bin/zmmailbox -z -m restore_USER@DOMAIN.com getRestURL "//?fmt=tgz" > /var/tmp/user-export.tgz''
 
#** You can also export SOME of the 'old' data from the restore account using other options. One option is with the ''before'' and ''after'' variables. '''NOTE''' - We have to set the query string as a variable to get around some of the shell issues.
 
#** For example:
 
#*** ''$ query='before:11/20/2010 after:11/1/2010'''
 
#**** ZCS5 might require you to have a %20 rather than the actual space character.
 
#*** ''$ /opt/zimbra/bin/zmmailbox -z -m restore_USER@DOMAIN.com getRestURL "//?fmt=tgz&query=$query" > /var/tmp/user-export.tgz''
 
#**** '''Note''': the $query above will be the expansion of the variable you set prior in the shell, ''query='before:11/20/2010 after:11/1/2010''' .
 
# Use ''zmmailbox'' with the ''postRestUrl'' option to IMPORT the RESTORE_USER account into the "USER" account now with appropriate options.
 
#* Examples:
 
#** ''$ /opt/zimbra/bin/zmmailbox -z -m USER@DOMAIN.com postRestURL “//?fmt=tgz&resolve=replace” /var/tmp/user-export.tgz''
 
#* '''Note''' : A critical option in the above command is the ''&resolve=replace'' one. There are various ways you can handle the importing of data. Please review the following to determine what is best for you needs.
 
#** [[Ajcody-Migration-Notes#ZCS_User_to_Another_ZCS_Server_-_With_Rest_.26_TGZ]]
 
  
===Restore Deleted Items - skipDeletes Option - ZCS6+===
+
Example to full label and stop :
  
* "skip delete operations flag to zmrestore"
+
zmrestore -a USER@DOMAIN.com -lb full-20080726.050017.306 -rf -ca -pre restore_
** http://bugzilla.zimbra.com/show_bug.cgi?id=31824#c5
 
  
From the RFE comment:
+
Example to incremental label and stop :
<pre>
 
        Added new option --skipDeletes to zmrestore.  If specified, skip
 
        over delete operations during redo replay.  Delete ops are:
 
  
        DeleteItem (hard delete)
+
zmrestore -a USER@DOMAIN.com -restoreToIncrLabel incr-20080731.060007.644 -lb full-20080726.050017.306 -br -ca -pre restore_
        DeleteMailbox
 
        EmptyFolder
 
        MoveItem, if moving item to Trash folder
 
        PurgeImapDeleted
 
        PurgeOldMessages
 
  
        Skipping these deletes can lead to other problems later, such as
+
Example to specific time and stop :
        conflicting paths, but it is assumed the priority is recovering as
 
        much data as possible when using this option.
 
</pre>
 
  
===Restore Account Not Yet In Backups===
+
zmrestore -a USER@DOMAIN.com -restoreToTime 20080801011800 -lb full-20080726.050017.306 -br -ca -pre restore_
  
Please see:
 
* "add ability to restore accounts not yet backed up (but still in redologs)"
 
** http://bugzilla.zimbra.com/show_bug.cgi?id=31536
 
  
===How To Restore To Events Within The Same Day===
+
'''Important Options You Might Want Or Need To Include'''
  
Work in progress and investigation.
+
'''--ignoreRedoErrors''' : If you attempt a restore and you see an error about problems related to playing the redolog, you'll want to run the restore command again and include this option.
  
====Users Trash Items====
 
  
=====User Ability To Recover Trash Purge=====
+
'''--skipDeletes''' : Please see http://bugzilla.zimbra.com/show_bug.cgi?id=31824#c5 for details on this.
  
Please see RFE:
 
* "RFE: Recover Deleted Items ability for users"
 
** http://bugzilla.zimbra.com/show_bug.cgi?id=30582
 
  
=====Retention Policy About Purges=====
+
'''-t /path/to/backup_dir''' : If you are restoring from another backup directory besides your current default path.
  
Please see [[Mailbox_Purge]]
 
  
===Can't Restore Or Find An Account That Was Renamed===
+
Variables that are asking for TIME rather than LABELS should follow this syntax (from zmrestore --help):
 +
<pre>
 +
Specify date/time in one of these formats:
  
When an account is "renamed", the old account name will no longer be "found" is your "default" type restore or backup queries. This can cause some confusions when one needs to restore to a time frame of when the account was under it's older name.
+
    2008/08/06 09:55:50
 +
    2008/08/06 09:55:50 572
 +
    2008/08/06 09:55:50.572
 +
    2008/08/06-09:55:50-572
 +
    2008/08/06-09:55:50
 +
    20080806.095550.572
 +
    20080806.095550
 +
    20080806095550572
 +
    20080806095550
  
====Restoring From CLI====
+
Specify year, month, date, hour, minute, second, and optionally millisecond.
 +
Month/date/hour/minute/second are 0-padded to 2 digits, millisecond to 3 digits.
 +
Hour must be specified in 24-hour format, and time is in local time zone.
 +
</pre>
  
First, identify what sessions labels hold the differing account names:
+
====Bugs And RFE's To Review====
  
zmbackupquery -a USER@DOMAIN
+
'''Update - Bug/RFE's I Filed Against ZCS 8.6'''
  
zmbackupquery -a USER-RENAMED@DOMAIN
+
* [story] Ability to search data within backup and do "item" restores or identify locations of search results
 +
** https://bugzilla.zimbra.com/show_bug.cgi?id=97551
 +
* admin console backup label view doesn't list accounts in the all accounts tab
 +
** https://bugzilla.zimbra.com/show_bug.cgi?id=97555
 +
* admin console restore - doesn't autocomplete / suggest account matches when filling out email address box
 +
** https://bugzilla.zimbra.com/show_bug.cgi?id=97556
 +
* document new restore functions / options with ZCS 8+ for admin console restore
 +
** https://bugzilla.zimbra.com/show_bug.cgi?id=97558
 +
* admin console restore - rename "Selected Servers" panel to "Restore Options"
 +
** https://bugzilla.zimbra.com/show_bug.cgi?id=97559
 +
*admin console restore - if only one mailstore in env. then state such in second panel of restore about "server for the restored accounts"
 +
** https://bugzilla.zimbra.com/show_bug.cgi?id=97560
 +
*admin console restore - expand restore To options - To full backup label, To incremental target
 +
** https://bugzilla.zimbra.com/show_bug.cgi?id=97561
 +
*admin console restore - "restore to the latest backup" incorrectly described / broken
 +
** https://bugzilla.zimbra.com/show_bug.cgi?id=97562
 +
* admin console restore - unable to restore individual accounts [sort of]
 +
** https://bugzilla.zimbra.com/show_bug.cgi?id=97564
 +
* admin console restore - reuse GAL/Contact Picker Window for "restore individual accounts"
 +
** https://bugzilla.zimbra.com/show_bug.cgi?id=97565
  
You should know have what you need to do follow the steps in the [[Ajcody-Backup-Restore-Issues#Restore_To_Time_Problems]] section.
+
'''Older Bug/RFE's'''
 
+
* zmrestore using restoreToTime option restores data after the specified time
====Restoring From Admin Console GUI====
+
** See: http://bugzilla.zimbra.com/show_bug.cgi?id=15320
 
+
*** Resulting in documentation changes : [[CLI_zmrestore_Network_Edition_only]]
I'll give a detail explanation of the situation when working around restores of renamed accounts in the admin console [web GUI]:
+
* zmrestore -restoreToTime should fail if no backup label is passed
 
+
** See: http://bugzilla.zimbra.com/show_bug.cgi?id=28320
# If you in the GUI goto the "Restore" button, it first asks for an account rather than giving an option for date/time/session. I think you already stated, that "renamed" accounts don't show up in this query window. Therefore, one wouldn't really progress to the next window that would allow you to change the backup session label.
+
*** "unable to restore to point in time/incremental" '''Admin GUI'''
# They way you get around this is, you actually double click on the full session listing that you see on the backup page in the admin UI. This will bring you to another page, that is specific to that session. In there, you should see the old account name prior to the rename. You can then highlight that account listing and click on the "Restore" button. This will bring up the restore dialog, which will now have the date/time/session label auto-filled out.
+
**** http://bugzilla.zimbra.com/show_bug.cgi?id=33135
 +
* Admin console issues and other details I've posted are in this bug:
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=27746
 +
** Added for search terms, admin console can't restore to point in time
 +
* Include --ignoreRedoErrors option and error feedback in Admin Console for restore
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=52358
 +
*** This could explain why your restore to time isn't working in the Admin Console but does from CLI when you see an error about redologs and then reattempt restore with the --ignoreRedoErrors and it works.
  
===Quota Is Stopping A Redirected Restore===
+
===Restore An Individual Message===
  
'''Update'''
+
The zmrestore command is at a mailbox level.
  
* "accounts with quota set by COS fail to restore when over default quota"
+
An RFE was filed already to expand this. It is currently targeted for the Helix release.
** http://bugzilla.zimbra.com/show_bug.cgi?id=41875
 
  
Reasoning for need, maybe the msg files coming from the restore are no longer "shared message blobs" and therefore increase the mailbox to a size it wasn't in the past. Changes to HSM maybe?  
+
* "More Granular Restore: per folder & per-message"
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=8849
 +
*** '''Note''' - the above was marked as a duplicate of our work on "dumpster" option. I disagree with this choice and created the following RFE below.
 +
* "Ability to search data within backup and do "item" restores or identify locations of search results"
 +
** https://bugzilla.zimbra.com/show_bug.cgi?id=97551
  
I think I'll need to create a RFE about adding an option to the zmrestore command to also include an option to set the COS value on a created account. Until then... Create a new COS and set it up to NOT have any quota. Once you kick off the backup and you see the account is created you can then apply the COS to that account. Call the cos something like no-quota. Here's the steps below.
+
A way around the current limitations would be to use lmtpinject. Please see the following for details on that:
  
To copy your cos (assuming your quota cause is the default one, change default to match your production cos your using).
+
* http://www.zimbra.com/forums/installation/12617-recover-data-store-folders.html#post64962
 +
* http://www.zimbra.com/forums/administrators/729-using-zmlmtpinject.html#post4017
  
To see all cos's
+
The difficultly would be determining the message your trying to find within your backups that was "deleted" in prod.
  
zmprov gac
+
===User Deleted A Bunch Of Data And Notified You Hours Later Wanting It Restored===
  
Copies cos called default to cos named no-quota
+
# To determine the time of the delete, use the ''zmredodump'' command.
 
+
#* You'll use this "time" for the restore command.
zmprov cpc default no-quota
+
#* Example:
 
+
#** ''$ zmprov gmi USER@DOMAIN.com''  ::: gives me mailboxid 17
To remove quota to the new cos no-quota
+
#** ''$ zmredodump --show-blob --m 17 /opt/zimbra/redolog/ | grep Delete''  ::: returns:
 
+
#*** ''[0000f311 - 0000f350: 64 bytes; tstamp: 2010/11/19 11:21:30.852 CST] txn 1290033504.2544 [DeleteItem] ver=1.28, tstamp=1290187290852, change=3913, mailbox=17, ids=[304, 308], type=5''
  zmprov mc no-quota zimbraMailQuota 0
+
#*** ''[0000f351 - 0000f382: 50 bytes; tstamp: 2010/11/19 11:21:30.869 CST] txn 1290033504.2544 [CommitTxn] ver=1.28, tstamp=1290187290869, mailbox=17, txnType=DeleteItem''
 
+
# See [[Ajcody-Backup-Restore-Issues#Restore_To_Time_Problems]] about setting up the proper restore syntax.
To confirm
+
#* You'll want to include the ''-ca -pre restore_'' options since we'll first get the old data into a temporary account.
 
+
#* Remember to use ''zmbackupquery -a USER@DOMAIN.com --type full'' to find what full label will be needed for the restore.
zmprov gc no-quota | grep -i quota
+
#* Also, if you need/want the incremental label, do a ''zmbackupquery -a USER@DOMAIN.com --type incremental'' .
 
+
#** Example:
Now you would start your redirected restore and once you see the account is created, run the below in a separate shell. (example)
+
#*** ''$ zmbackupquery -a USER@DOMAIN.com --type full'' ::: shows the latest as being:
 
+
#**** ''Label:  full-20101119.070018.607''
zmrestore -a USER@DOMAIN.com -restoreToTime 20080801011800 -lb full-20080726.050017.306 -br -ca -pre restore-
+
#**** ''Type:    full''
 
+
#**** ''Started: Fri, 2010/11/19 01:00:18.607 CST''
Once restore has kicked off - To apply the no-quota cos to the restored account:
+
#**** ''Ended:  Fri, 2010/11/19 01:00:29.751 CST''
 
+
#**** ''Acct ID: f33d6daf-8875-4496-8bee-6df345f295e7''
zmprov sac restore-USER@DOMAIN.com no-quota
+
#* Do the restore now with the proper information for the variable flags.
 +
#** '''''Note''''': You might need to include -br , -rf , or neither depending on the time frames involved.
 +
#** Example:
 +
#*** ''$ zmrestore -a USER@DOMAIN.com -restoreToTime 20101119112000 -lb full-20101119.070018.607 -ca -pre restore_''
 +
#** It would be best to log into the restore_USER@DOMAIN.com account to confirm the data is as you expect it.
 +
# Use ''zmmailbox'' with the ''getRestUrl'' option against the "restore_USER" account now to export the data.
 +
#* Examples:
 +
#** Export ALL the 'old' data from the restored account
 +
#*** ''$ /opt/zimbra/bin/zmmailbox -z -m restore_USER@DOMAIN.com getRestURL "//?fmt=tgz" > /var/tmp/user-export.tgz''
 +
#** You can also export SOME of the 'old' data from the restore account using other options. One option is with the ''before'' and ''after'' variables. '''NOTE''' - We have to set the query string as a variable to get around some of the shell issues.
 +
#** For example:
 +
#*** ''$ query='before:11/20/2010 after:11/1/2010'''
 +
#**** ZCS5 might require you to have a %20 rather than the actual space character.
 +
#*** ''$ /opt/zimbra/bin/zmmailbox -z -m restore_USER@DOMAIN.com getRestURL "//?fmt=tgz&query=$query" > /var/tmp/user-export.tgz''
 +
#**** '''Note''': the $query above will be the expansion of the variable you set prior in the shell, ''query='before:11/20/2010 after:11/1/2010''' .
 +
# Use ''zmmailbox'' with the ''postRestUrl'' option to IMPORT the RESTORE_USER account into the "USER" account now with appropriate options.
 +
#* Examples:
 +
#** ''$ /opt/zimbra/bin/zmmailbox -z -m USER@DOMAIN.com postRestURL “//?fmt=tgz&resolve=replace” /var/tmp/user-export.tgz''
 +
#* '''Note''' : A critical option in the above command is the ''&resolve=replace'' one. There are various ways you can handle the importing of data. Please review the following to determine what is best for you needs.
 +
#** [[Ajcody-Migration-Notes#ZCS_User_to_Another_ZCS_Server_-_With_Rest_.26_TGZ]]
  
===Restore Of Non-Account Items - Example - COS DL Etc ===
+
===Restore Deleted Items - skipDeletes Option - ZCS6+===  
  
Cos and DL's are ldap entries basically.
+
* "skip delete operations flag to zmrestore"
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=31824#c5
  
From [http://www.zimbra.com/docs/ne/latest/administration_guide/10_Backup_Restore.17.01.html Backup and Restore]
+
From the RFE comment:
 +
<pre>
 +
        Added new option --skipDeletes to zmrestore. If specified, skip
 +
        over delete operations during redo replay. Delete ops are:
  
*zmrestoreldap. This command restores the complete LDAP directory server, including accounts, domains, servers, COS and other data."
+
        DeleteItem (hard delete)
 +
        DeleteMailbox
 +
        EmptyFolder
 +
        MoveItem, if moving item to Trash folder
 +
        PurgeImapDeleted
 +
        PurgeOldMessages
  
You'll see in a DR process, [[Network_Edition_Disaster_Recovery]] , that the zmrestoreldap is done before the zmrestoreoffline .
+
        Skipping these deletes can lead to other problems later, such as
 +
        conflicting paths, but it is assumed the priority is recovering as
 +
        much data as possible when using this option.
 +
</pre>
  
*zmrestoreldap doesn't have options that allow specific items to be restored (COS, DL's, etc.). It only has option for named accounts (-a). One could try a ldapadd with a ldif of the COS or DL details. One could also take the information on the COS or DL within the ldap file in the backup session to at least have all the variables to manually add it back (via the zmprov command). Your looking at the backups on the LDAP master if your in a multi-server configuration.
+
===Restore Account Not Yet In Backups===
  
<pre>/opt/zimbra/backup/sessions/full-xxxxxx/ldap/ldap.bak</pre>
+
Please see:
 +
* "add ability to restore accounts not yet backed up (but still in redologs)"
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=31536
  
Start of ldap entry example to search for:
+
===How To Restore To Events Within The Same Day===
  
*Cos example
+
Work in progress and investigation.
**dn: cn=default,cn=cos,cn=zimbra
 
*DL example
 
**dn: uid=dl-group,ou=people,dc=mail,dc=domain,dc=com
 
  
To compare a current DL with past details, just save out the ldap entry from the backup to a txt file. And then do:
+
====Users Trash Items====
  
<pre>zmprov gdl maillist@domain.com</pre>
+
=====User Ability To Recover Trash Purge=====
  
Make the necessary changes after comparing the two.
+
Please see RFE:
 +
* "RFE: Recover Deleted Items ability for users"
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=30582
  
===Restoring A Calendar (ics)===
+
=====Retention Policy About Purges=====
 +
 
 +
Please see [[Mailbox_Purge]]
 +
 
 +
===Can't Restore Or Find An Account That Was Renamed===
 +
 
 +
When an account is "renamed", the old account name will no longer be "found" is your "default" type restore or backup queries. This can cause some confusions when one needs to restore to a time frame of when the account was under it's older name.
 +
 
 +
====Restoring From CLI====
  
There seems to be a bug or odd expectation on how this command is currently working. If the appointment exists in the Calendar and the time is different with the same appointment in the ics file your importing - the time of the appointment will not change to the imported ics one. If you delete the event first, then the imported appointment will reflect the correct time.
+
First, identify what sessions labels hold the differing account names:
  
Here's what I did to reproduce this situation. It seems this has been true for sometime, customer was on 4.5.11 and I was on 5.0.8
+
zmbackupquery -a USER@DOMAIN
  
<pre>
+
zmbackupquery -a USER-RENAMED@DOMAIN
Created test account and made two appointments on friday - 9am and 4pm.
 
Did a full backup.
 
Restored test account to restore_user
 
Ran :
 
    zmmailbox -z -m user@domain.com gru /Calendar > /tmp/calendarA.ics
 
    zmmailbox -z -m restore_user@domain.com gru /Calendar > /tmp/calendarB.ics
 
And then
 
    diff /tmp/calendarA.ics /tmp/calendarB.ics [no differences]
 
  
Now some tests.
+
You should know have what you need to do follow the steps in the [[Ajcody-Backup-Restore-Issues#Restore_To_Time_Problems]] section.
As user, I deleted the two appointments and then:
 
    zmmailbox -z -m user@domain.com pru /Calendar /tmp/calendarB.ics
 
Refreshed Calendar as User in webclient.
 
    9am and 4pm appointment shows up.
 
I then moved in the webclient the 9am appointment to 11am
 
Did another restore:
 
    zmmailbox -z -m user@domain.com pru /Calendar /tmp/calendarB.ics
 
Refreshed Calendar as User in webclient.
 
    11am and 4pm appointment shows up.
 
    ** The restore did not move the 11am appointment back to the 9am slot as in /tmp/calendarB.ics
 
    ** Assumption, this process will not over-write an appointment if it's there - it does not look to the time.
 
        Let's do a diff of the state of the calendar
 
        zmmailbox -z -m user@domain.com gru /Calendar > /tmp/calendarC.ics
 
        diff /tmp/calendarB.ics /tmp/calendarC.ics
 
          The DTSTAMP and SEQUENCE shows the difference in the time.
 
If I delete the 11am appointment and then do the calendarB.ics restore the appointment shows up again at 9am.
 
</pre>
 
  
I see this same behavior if I also use the web interface to export/import the calendar between the restored account and user one. Even when I import it into a NEW calendar, it even changes the two appointments to reflect the new calendar rather than the default one.
+
====Restoring From Admin Console GUI====
  
====One Fix====
+
I'll give a detail explanation of the situation when working around restores of renamed accounts in the admin console [web GUI]:
One fix, if the situation allows, is to purge the current Calendar and then import the full ics file. This would be done like this:
 
  
<pre>
+
# If you in the GUI goto the "Restore" button, it first asks for an account rather than giving an option for date/time/session. I think you already stated, that "renamed" accounts don't show up in this query window. Therefore, one wouldn't really progress to the next window that would allow you to change the backup session label.
zmmailbox -z -v -m ajcody@mail3.internal.homeunix.com ef /Calendar
+
# They way you get around this is, you actually double click on the full session listing that you see on the backup page in the admin UI. This will bring you to another page, that is specific to that session. In there, you should see the old account name prior to the rename. You can then highlight that account listing and click on the "Restore" button. This will bring up the restore dialog, which will now have the date/time/session label auto-filled out.
* ef is for emptyFolder  zmmailbox help folder
 
*webclient shows all events gone but Calendars are still listed.
 
zmmailbox -z -v -m ajcody@mail3.internal.homeunix.com pru /Calendar /tmp/calendarB.ics
 
*webclient show all events with times as expected.
 
</pre>
 
  
====Second Fix====
+
===Quota Is Stopping A Redirected Restore===
  
Haven't tried yet, but someone said you should be able to adjust the SEQUENCE number in the appointment and the import process will use that data (data/time) of the newer sequence number appointment.
+
'''Update'''
  
===Archiving User Accounts Out Of Production Use In Zimbra===
+
* "accounts with quota set by COS fail to restore when over default quota"
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=41875
  
====Backup - Restore Method====
+
Reasoning for need, maybe the msg files coming from the restore are no longer "shared message blobs" and therefore increase the mailbox to a size it wasn't in the past. Changes to HSM maybe?
  
To "archive" the user data with the ability of Zimbra later restoring if needed, one would rely on the backup/restore tools.  
+
I think I'll need to create a RFE about adding an option to the zmrestore command to also include an option to set the COS value on a created account. Until then... Create a new COS and set it up to NOT have any quota. Once you kick off the backup and you see the account is created you can then apply the COS to that account. Call the cos something like no-quota. Here's the steps below.
  
For example:
+
To copy your cos (assuming your quota cause is the default one, change default to match your production cos your using).
  
zmbackup -f -a user@domain
+
To see all cos's
  
If you want to make this more "intelligent" later, one could do this:
+
zmprov gac
  
mkdir /nfs-mount/archived-users/[user-name]
+
Copies cos called default to cos named no-quota
  
  zmbackup -f -a userA@domain -t /nfs-mount/archived-users/userA-domain/
+
  zmprov cpc default no-quota
  
This would give some "intelligent" information for recovery later. Confirms it was an archived account, shows the "real" username, the date of the full backup, and so forth.
+
To remove quota to the new cos no-quota
  
  ls /nfs-mount/archived-users/userA-domain/
+
  zmprov mc no-quota zimbraMailQuota 0
  full-20081104.131643.006
 
ls /nfs-mount/archived-users/userA-domain/full-20081104.131643.006
 
  accounts/ session.xml shared_blobs sys
 
  
If one would need to "restore" the account later, it would consume a license if the account was "deleted".
+
To confirm
  
  zmrestore -a userA@domain -ca -pre restored- -t /nfs-mount/archived-users/userA-domain/ -lb full-20081104.131643.006
+
  zmprov gc no-quota | grep -i quota
  
=====Setup A Secondary Zimbra Box For Restores Of Archive Accounts=====
+
Now you would start your redirected restore and once you see the account is created, run the below in a separate shell. (example)
  
Your Zimbra license can be installed on multiple machines. One idea that might prove useful in handling these "archive" accounts for those situations when you need to investigate something is to setup a "archive" Zimbra box. You'll want to isolate this box from any "production" use. It will need to be configured to have the "domains" of the archive accounts. You can then use this box to restore the "archive" account and then use the administration tools to investigate the user data.
+
zmrestore -a USER@DOMAIN.com -restoreToTime 20080801011800 -lb full-20080726.050017.306 -br -ca -pre restore-
  
===Use Of REST And Other Tools For Specific User Data===
+
Once restore has kicked off - To apply the no-quota cos to the restored account:
 +
 
 +
zmprov sac restore-USER@DOMAIN.com no-quota
  
The following page, [[User_Migration]] , will shows numerous examples of how to export different types of data from a user account into a neutral file format that one could use for "archive" purposes.
+
===Restore Of Non-Account Items - Example - COS DL Etc ===
  
===Use Of The REST Command To Export ALL User Data - Version Dependant, 5.0.9+ [I believe]===
+
Cos and DL's are ldap entries basically.
  
Example of the commend syntax:
+
From [http://www.zimbra.com/docs/ne/latest/administration_guide/10_Backup_Restore.17.01.html Backup and Restore]
  
  /opt/zimbra/bin/zmmailbox -z -m user@domain.com getRestURL “//?fmt=tgz” > /tmp/account.tgz
+
*zmrestoreldap. This command restores the complete LDAP directory server, including accounts, domains, servers, COS and other data."
  
===Backup And HSM===
+
You'll see in a DR process, [[Network_Edition_Disaster_Recovery]] , that the zmrestoreldap is done before the zmrestoreoffline .
  
Please see Bugs/RFE:
+
*zmrestoreldap doesn't have options that allow specific items to be restored (COS, DL's, etc.). It only has option for named accounts (-a). One could try a ldapadd with a ldif of the COS or DL details. One could also take the information on the COS or DL within the ldap file in the backup session to at least have all the variables to manually add it back (via the zmprov command). Your looking at the backups on the LDAP master if your in a multi-server configuration.
  
* "RFE: HSM and backup should not run at the same time if initated."
+
<pre>/opt/zimbra/backup/sessions/full-xxxxxx/ldap/ldap.bak</pre>
** http://bugzilla.zimbra.com/show_bug.cgi?id=33452
 
  
===Scripting Out Individual Backups Of Accounts===
+
Start of ldap entry example to search for:
  
If you want to do individual backups of accounts using a for-loop, for example, you might want to include the -sync option from zmbackup. zmbackup without this will normal give an error as it passing the next zmbackup command stating that there's a current backup in progress.
+
*Cos example
 +
**dn: cn=default,cn=cos,cn=zimbra
 +
*DL example
 +
**dn: uid=dl-group,ou=people,dc=mail,dc=domain,dc=com
  
Example command in some for-loop script is ['''without the -sync option''']:
+
To compare a current DL with past details, just save out the ldap entry from the backup to a txt file. And then do:
  
sudo -u zimbra /opt/zimbra/bin/zmbackup --fullBackup --account "$acct" --server "$host" --target "$dest"
+
<pre>zmprov gdl maillist@domain.com</pre>
  
zmbackup --help shows:
+
Make the necessary changes after comparing the two.
  
--sync  -sync Runs full backup synchronously.
+
===Restoring A Calendar (ics)===
  
The admin guide mentions:
+
There seems to be a bug or odd expectation on how this command is currently working. If the appointment exists in the Calendar and the time is different with the same appointment in the ics file your importing - the time of the appointment will not change to the imported ics one. If you delete the event first, then the imported appointment will reflect the correct time.
 
+
 
:"Full backup is usually run asynchronously. When you begin the full backup, the label of the ongoing backup process is immediately displayed. The backup continues in the background. You can use the zmbackupquery command to check the status of the running backup at any time."
+
Here's what I did to reproduce this situation. It seems this has been true for sometime, customer was on 4.5.11 and I was on 5.0.8
 
+
 
I couldn't find any other indication beyond that to explain in more details the purpose of that flag. But from what is stated above, it does look like the -sync flag will resolve the issues of "backup in progress" when scripting out multiple zmbackup commands.
+
<pre>
 
+
Created test account and made two appointments on friday - 9am and 4pm.
If not, you could query for "Status:  in progress" from the zmbackupquery command.
+
Did a full backup.
 
+
Restored test account to restore_user
You can give the zmbackupquery command flags for date/time, label, account, -t target, and so forth [Do a zmbackupquery --help to see the options format].
+
Ran :
 
+
    zmmailbox -z -m user@domain.com gru /Calendar > /tmp/calendarA.ics
===Backing Up Backups - 3rd Party Tools And Software - Dealing With Directories With Hard Links===
+
    zmmailbox -z -m restore_user@domain.com gru /Calendar > /tmp/calendarB.ics
 
+
And then
====Description Of Hard Links====
+
    diff /tmp/calendarA.ics /tmp/calendarB.ics [no differences]
 
+
 
Zimbra uses hard links and special attention needs to be given to this fact. See [http://en.wikipedia.org/wiki/Hard_link hard links] if your not familiar with hard links and their difference to [http://en.wikipedia.org/wiki/Symbolic_link symbolic links]. Not all 3rd party backup software will handle or respect hard links. Many unix commands will need special flags to maintain hard links. When hard links are respected and also "copied" to the new location you could find your data usage become a large multiplication of the original size.
+
Now some tests.
 
+
As user, I deleted the two appointments and then:
An good thread I found on the topic of preserving hard links for copy/move/backup operations is here:
+
    zmmailbox -z -m user@domain.com pru /Calendar /tmp/calendarB.ics
 
+
Refreshed Calendar as User in webclient.
* "How to Copy a Filesystem and Preserve Hard Links in Linux - Some random bits scribbled by Jeremy Zawodny"
+
    9am and 4pm appointment shows up.
**  http://jeremy.zawodny.com/blog/archives/010037.html
+
I then moved in the webclient the 9am appointment to 11am
 
+
Did another restore:
I'll be summarizing the comments and including additional information I find on the topic below, based upon the command or software being used.
+
    zmmailbox -z -m user@domain.com pru /Calendar /tmp/calendarB.ics
 
+
Refreshed Calendar as User in webclient.
=====Zimbra and Single Instance Storage - Hard Links=====
+
    11am and 4pm appointment shows up.
 
+
    ** The restore did not move the 11am appointment back to the 9am slot as in /tmp/calendarB.ics
If hard links are possible, we use them. When hard links aren't possible, we'll have to save the 'msg' again and then internally to that mailstore/disk partition we can use hard links to that initial save.
+
    ** Assumption, this process will not over-write an appointment if it's there - it does not look to the time.
 
+
        Let's do a diff of the state of the calendar
Mike gives a good description of it here:
+
        zmmailbox -z -m user@domain.com gru /Calendar > /tmp/calendarC.ics
* http://www.zimbra.com/forums/administrators/14435-single-instance-storage.html#post73746
+
        diff /tmp/calendarB.ics /tmp/calendarC.ics
** Here's the one wiki page Mike mentions in there: [[Account_mailbox_database_structure]]
+
          The DTSTAMP and SEQUENCE shows the difference in the time.
 
+
If I delete the 11am appointment and then do the calendarB.ics restore the appointment shows up again at 9am.
======Redoing Hard Links RFE======
+
</pre>
 
+
 
For 8.0.2+ , see this option - zmdedupe :
+
I see this same behavior if I also use the web interface to export/import the calendar between the restored account and user one. Even when I import it into a NEW calendar, it even changes the two appointments to reflect the new calendar rather than the default one.
 
+
 
* RFE: Shipping a tool to convert duplicate blobs to hardlinks
+
====One Fix====
** https://bugzilla.zimbra.com/show_bug.cgi?id=77597
+
One fix, if the situation allows, is to purge the current Calendar and then import the full ics file. This would be done like this:
 
+
 
======Easy Way To SEE Hard Links In Use======
+
<pre>
 +
zmmailbox -z -v -m ajcody@mail3.internal.homeunix.com ef /Calendar
 +
* ef is for emptyFolder  zmmailbox help folder
 +
*webclient shows all events gone but Calendars are still listed.
 +
zmmailbox -z -v -m ajcody@mail3.internal.homeunix.com pru /Calendar /tmp/calendarB.ics
 +
*webclient show all events with times as expected.
 +
</pre>
 +
 
 +
====Second Fix====
 +
 
 +
Haven't tried yet, but someone said you should be able to adjust the SEQUENCE number in the appointment and the import process will use that data (data/time) of the newer sequence number appointment.
 +
 
 +
===Archiving User Accounts Out Of Production Use In Zimbra===
 +
 
 +
====Backup - Restore Method====
 +
 
 +
To "archive" the user data with the ability of Zimbra later restoring if needed, one would rely on the backup/restore tools.
 +
 
 +
For example:
 +
 
 +
zmbackup -f -a user@domain
 +
 
 +
If you want to make this more "intelligent" later, one could do this:
 +
 
 +
mkdir /nfs-mount/archived-users/[user-name]
 +
 
 +
zmbackup -f -a userA@domain -t /nfs-mount/archived-users/userA-domain/
 +
 
 +
This would give some "intelligent" information for recovery later. Confirms it was an archived account, shows the "real" username, the date of the full backup, and so forth.
 +
 
 +
ls /nfs-mount/archived-users/userA-domain/
 +
  full-20081104.131643.006
 +
ls /nfs-mount/archived-users/userA-domain/full-20081104.131643.006
 +
  accounts/ session.xml shared_blobs sys
 +
 
 +
If one would need to "restore" the account later, it would consume a license if the account was "deleted".
 +
 
 +
zmrestore -a userA@domain -ca -pre restored- -t /nfs-mount/archived-users/userA-domain/ -lb full-20081104.131643.006
 +
 
 +
=====Setup A Secondary Zimbra Box For Restores Of Archive Accounts=====
 +
 
 +
Your Zimbra license can be installed on multiple machines. One idea that might prove useful in handling these "archive" accounts for those situations when you need to investigate something is to setup a "archive" Zimbra box. You'll want to isolate this box from any "production" use. It will need to be configured to have the "domains" of the archive accounts. You can then use this box to restore the "archive" account and then use the administration tools to investigate the user data.
 +
 
 +
===Use Of REST And Other Tools For Specific User Data===
 +
 
 +
The following page, [[User_Migration]] , will shows numerous examples of how to export different types of data from a user account into a neutral file format that one could use for "archive" purposes.
 +
 
 +
===Use Of The REST Command To Export ALL User Data - Version Dependant, 5.0.9+ [I believe]===
 +
 
 +
Example of the commend syntax:
 +
 
 +
  /opt/zimbra/bin/zmmailbox -z -m user@domain.com getRestURL “//?fmt=tgz” > /tmp/account.tgz
 +
 
 +
===Backup And HSM===
 +
 
 +
Please see Bugs/RFE:
 +
 
 +
* "RFE: HSM and backup should not run at the same time if initated."
 +
** http://bugzilla.zimbra.com/show_bug.cgi?id=33452
 +
 
 +
===Using zmprov -l gaa To Create User Listing for zmbackup & zmrestore -a option===
 +
 
 +
For example, see below. '''Note''' the use of the egrep -v option below, this would ''remove'' any matches from the grep. Useful if you want to only backup certain domains for example.
 +
 
 +
<pre>
 +
[zimbra@ldap2 sessions]$ zmbackup -f -a `zmprov -l gaa | egrep -v "ham|spam|virus|admin|galsync"`
 +
full-20150130.210821.124
 +
 
 +
[zimbra@ldap2 sessions]$ zmbackup -f -a `zmprov -l gaa`
 +
full-20150130.210922.637
 +
 
 +
[zimbra@ldap2 sessions]$ zmbackupquery -lb full-20150130.210821.124 -v
 +
Label:  full-20150130.210821.124
 +
Type:    full
 +
Status:  completed
 +
Started: Fri, 2015/01/30 16:08:21.124 EST
 +
Ended:  Fri, 2015/01/30 16:08:50.181 EST
 +
Redo log sequence range: 23 .. 23
 +
Number of accounts: 3 out of 3 completed
 +
Accounts:
 +
  archive-search-results@ldap2.zimbra.DOMAIN.com: completed
 +
  user1-20150115@ldap2.zimbra.DOMAIN.com.archive: completed
 +
  user1@ldap2.zimbra.DOMAIN.com: completed
 +
 
 +
Total space: 14020MB
 +
Free space: 5977MB
 +
 
 +
[zimbra@ldap2 sessions]$ zmbackupquery -lb full-20150130.210922.637 -v
 +
Label:  full-20150130.210922.637
 +
Type:    full
 +
Status:  completed
 +
Started: Fri, 2015/01/30 16:09:22.637 EST
 +
Ended:  Fri, 2015/01/30 16:09:33.083 EST
 +
Redo log sequence range: 23 .. 23
 +
Number of accounts: 9 out of 9 completed
 +
Accounts:
 +
  admin-20150122@ldap2.zimbra.DOMAIN.com.archive: completed
 +
  admin@ldap2.zimbra.DOMAIN.com: completed
 +
  archive-search-results@ldap2.zimbra.DOMAIN.com: completed
 +
  galsync.tfnapjb9@ldap2.zimbra.DOMAIN.com: completed
 +
  ham.yeygcogdd2@ldap2.zimbra.DOMAIN.com: completed
 +
  spam.dpxyqjnm6t@ldap2.zimbra.DOMAIN.com: completed
 +
  user1-20150115@ldap2.zimbra.DOMAIN.com.archive: completed
 +
  user1@ldap2.zimbra.DOMAIN.com: completed
 +
  virus-quarantine.wfkd4vvm1g@ldap2.zimbra.DOMAIN.com: completed
 +
 
 +
Total space: 14020MB
 +
Free space: 5977MB
 +
</pre>
 +
 
 +
===Scripting Out Individual Backups Of Accounts===
 +
 
 +
If you want to do individual backups of accounts using a for-loop, for example, you might want to include the -sync option from zmbackup. zmbackup without this will normal give an error as it passing the next zmbackup command stating that there's a current backup in progress.
 +
 
 +
Example command in some for-loop script is ['''without the -sync option''']:
 +
 
 +
sudo -u zimbra /opt/zimbra/bin/zmbackup --fullBackup --account "$acct" --server "$host" --target "$dest"
 +
 
 +
zmbackup --help shows:
 +
 
 +
--sync  -sync Runs full backup synchronously.
 +
 
 +
The admin guide mentions:
 +
 
 +
:"Full backup is usually run asynchronously. When you begin the full backup, the label of the ongoing backup process is immediately displayed. The backup continues in the background. You can use the zmbackupquery command to check the status of the running backup at any time."
 +
 
 +
I couldn't find any other indication beyond that to explain in more details the purpose of that flag. But from what is stated above, it does look like the -sync flag will resolve the issues of "backup in progress" when scripting out multiple zmbackup commands.
 +
 
 +
If not, you could query for "Status:  in progress" from the zmbackupquery command.
 +
 
 +
You can give the zmbackupquery command flags for date/time, label, account, -t target, and so forth [Do a zmbackupquery --help to see the options format].
 +
 
 +
===Finding Message Blob In Users Backup===
 +
 
 +
Get the zimbraId of the user, this is what is used in the backup session directory layout:
 +
<pre>
 +
$ zmprov ga admin@`zmhostname` zimbraId
 +
# name admin@zcs806.DOMAIN.com
 +
zimbraId: e46a828b-cdda-4635-98ab-31b8ac0129b6
 +
</pre>
 +
 
 +
Get the mailboxId, this is what is used in the zmvolume primary message volume directory layout:
 +
<pre>
 +
$ zmprov gmi admin@`zmhostname`
 +
mailboxId: 1
 +
quotaUsed: 77794874
 +
</pre>
 +
 
 +
Finding a message in the primary message volume:
 +
<pre>
 +
$ cd /opt/zimbra/store/0/1/msg/9/
 +
 
 +
$ ls -la 40511-49376.msg
 +
-rw-r----- 1 zimbra zimbra 1372 Mar 16 02:01 40511-49376.msg
 +
</pre>
 +
 
 +
Finding the users directory in a full backup session directory layout:
 +
<pre>
 +
$ find /opt/zimbra/backup/sessions/full-20140426.080019.153/ -name e46a828b-cdda-4635-98ab-31b8ac0129b6 -print
 +
/opt/zimbra/backup/sessions/full-20140426.080019.153/accounts/e46/a82/e46a828b-cdda-4635-98ab-31b8ac0129b6
 +
</pre>
 +
 
 +
Locating the message in the backup session - this example is where the backups are using the zip option:
 +
<pre>
 +
$ cd /opt/zimbra/backup/sessions/full-20140426.080019.153/accounts/e46/a82/e46a828b-cdda-4635-98ab-31b8ac0129b6/blobs/
 +
 
 +
$ ls
 +
blobs-1.zip  blobs-2.zip  blobs-3.zip  blobs-4.zip
 +
 
 +
$ for i in `ls /opt/zimbra/backup/sessions/full-20140426.080019.153/accounts/e46/a82/e46a828b-cdda-4635-98ab-31b8ac0129b6/blobs/`; do echo $i ; unzip -l $i | grep 40511-49376.msg; done
 +
 
 +
blobs-1.zip
 +
    1372  04-26-2014 01:00  1/9/sha256_bKH8aeWJSh9bhzv5zsXrweA7+jZ88NUllkfV+7m9cJo=_40511-49376.msg1
 +
</pre>
 +
 
 +
===Backing Up Backups - 3rd Party Tools And Software - Dealing With Directories With Hard Links===
 +
 
 +
====Description Of Hard Links====
 +
 
 +
Zimbra uses hard links and special attention needs to be given to this fact. See [http://en.wikipedia.org/wiki/Hard_link hard links] if your not familiar with hard links and their difference to [http://en.wikipedia.org/wiki/Symbolic_link symbolic links]. Not all 3rd party backup software will handle or respect hard links. Many unix commands will need special flags to maintain hard links. When hard links are respected and also "copied" to the new location you could find your data usage become a large multiplication of the original size.
 +
 
 +
An good thread I found on the topic of preserving hard links for copy/move/backup operations is here:
 +
 
 +
* "How to Copy a Filesystem and Preserve Hard Links in Linux - Some random bits scribbled by Jeremy Zawodny"
 +
**  http://jeremy.zawodny.com/blog/archives/010037.html
 +
 
 +
I'll be summarizing the comments and including additional information I find on the topic below, based upon the command or software being used.
 +
 
 +
=====Zimbra and Single Instance Storage - Hard Links=====
 +
 
 +
If hard links are possible, we use them. The message must be identical and on to users on the same mailstore and stored on zmvolumes where hard linking is possible. Hard links exist only on the same partition. Postfix has a veriable, default_destination_recipient_limit , which will cause large recipient emails to be delivered in a way where they aren't identical [ see [[Ajcody-Hardlinks-And-Postfix-default_destination_recipient_limit]] for more details ].
 +
 
 +
Mike gives a good description of it here:
 +
* http://www.zimbra.com/forums/administrators/14435-single-instance-storage.html#post73746
 +
** Here's the one wiki page Mike mentions in there: [[Account_mailbox_database_structure]]
 +
 
 +
======Redoing Hard Links RFE======
 +
 
 +
For 8.0.2+ , see this option - zmdedupe :
 +
 
 +
* RFE: Shipping a tool to convert duplicate blobs to hardlinks
 +
** https://bugzilla.zimbra.com/show_bug.cgi?id=77597
 +
 
 +
======Easy Way To SEE Hard Links In Use======
  
 
I sent a test email with 5 accounts listed in the To field, that is shown below with the inode listing of 2133394. the first column of the ls [because of the i option] is listing the inode number of the file. I included the -type f because the . and .. will show directories using the same inode as the 'name' of the directory as listed.
 
I sent a test email with 5 accounts listed in the To field, that is shown below with the inode listing of 2133394. the first column of the ls [because of the i option] is listing the inode number of the file. I included the -type f because the . and .. will show directories using the same inode as the 'name' of the directory as listed.
Line 1,879: Line 2,216:
 
2133404 -rw-r----- 2 zimbra zimbra 1518 May  7 05:22 ./0/27/msg/0/260-3018.msg
 
2133404 -rw-r----- 2 zimbra zimbra 1518 May  7 05:22 ./0/27/msg/0/260-3018.msg
 
</pre>
 
</pre>
 +
 +
======Why Does A Message Blob With The Same Message-ID Have Multiple Inodes======
 +
 +
Most likely, the message has a large recipient list and because of a variable in postfix, default_destination_recipient_limit  , causes multiple deliveries of the message to have slight differences between them - for example, the times in the headers.
 +
 +
Please see the following:
 +
* https://wiki.zimbra.com/wiki/Ajcody-Hardlinks-And-Postfix-default_destination_recipient_limit
  
 
====Hard Links Used Within Zimbra Backup Directory - Sessions====
 
====Hard Links Used Within Zimbra Backup Directory - Sessions====
Line 2,040: Line 2,384:
  
 
* [[5.0.x_Network_Edition_Backup_and_Restore#Using_snapshots_to_backup_and_restore]]
 
* [[5.0.x_Network_Edition_Backup_and_Restore#Using_snapshots_to_backup_and_restore]]
 +
* [[Individual_Mailbox_Restore_from_Snapshot]]
 
* Developer discussion on the issue
 
* Developer discussion on the issue
 
** http://bugzilla.zimbra.com/show_bug.cgi?id=21273
 
** http://bugzilla.zimbra.com/show_bug.cgi?id=21273
Line 2,265: Line 2,610:
 
* "Installing a DLT, SDLT, VS, LTO, or DAT Tape Drive Into a Linux Operating System"
 
* "Installing a DLT, SDLT, VS, LTO, or DAT Tape Drive Into a Linux Operating System"
 
** [PDF warning] http://cp-quantum.talismaonline.com/display/2/kb/article.aspx?aid=8040&n=1&docid=72010
 
** [PDF warning] http://cp-quantum.talismaonline.com/display/2/kb/article.aspx?aid=8040&n=1&docid=72010
 +
 +
----
  
 
[[Category: Community Sandbox]]
 
[[Category: Community Sandbox]]
 
[[Category:Backup and Restore]]
 
[[Category:Backup and Restore]]
 +
[[Category: Author:Ajcody]]

Latest revision as of 13:49, 17 May 2017

Contents

Ajcody Backup & Restore Issues

   KB 2549        Last updated on 2017-05-17  




0.00
(0 votes)
24px ‎  - This is Zeta Alliance Certified Documentation. The content has been tested by the Community.


Actual Backup & Restore Issues Homepage

Please see Ajcody-Backup-Restore-Issues

Update - May, 2017

I'll be updating this page after the ZCS 8.8 release is done to only reflect what is true in that version and future versions. It will also include information about the Zimbra_Suite_Plus modules where appropriate. If you currently are using the backup module for ZSP, please see Zimbra_Suite_Plus/Zimbra_Backup_Plus for now.

Backup-Restore Training Material Rough Drafts I Wrote

Bug/RFE's I Filed Against ZCS 8.6

Miscellaneous Bugs & RFE's

For those that address other specific issues described on this page:

RFE For Live DR Restore Option
Backups And Upgrades And Prior Versions

Information To Provide When Submitting A Support Case For Backup Issues

Basic Backup Information To Submit To Support

Disk Space Usage Issues
Trend Data

If there is concerns about disks/partitions getting full, this command would be helpful for trending data on your server. Send support the resulting df.tar file . Note - adjust the tail command if you want more than 20 day's worth of trending data, the -n 20 option.

[zimbra@zcs806 tmp]$ /tmp

[zimbra@zcs806 tmp]$ tar cvf /tmp/df.tar `find /opt/zimbra/zmstat -name df.cs\* | sort | tail -n 20`
tar: Removing leading `/' from member names
/opt/zimbra/zmstat/2014-03-10/df.csv.gz
/opt/zimbra/zmstat/2014-03-11/df.csv.gz
 [cut - Ajcody]
/opt/zimbra/zmstat/2014-03-27/df.csv.gz
/opt/zimbra/zmstat/2014-03-28/df.csv.gz
/opt/zimbra/zmstat/df.csv

[zimbra@zcs806 tmp]$ ls -lah /tmp/df.tar
-rw-r----- 1 zimbra zimbra 80K Mar 29 06:44 /tmp/df.tar

[zimbra@zcs806 tmp]$ tar tvf /tmp/df.tar
-rw-r----- zimbra/zimbra  2566 2014-03-11 00:00 opt/zimbra/zmstat/2014-03-10/df.csv.gz
-rw-r----- zimbra/zimbra  2553 2014-03-12 00:00 opt/zimbra/zmstat/2014-03-11/df.csv.gz
 [cut - Ajcody]
-rw-r----- zimbra/zimbra  2513 2014-03-28 00:00 opt/zimbra/zmstat/2014-03-27/df.csv.gz
-rw-r----- zimbra/zimbra  2531 2014-03-29 00:00 opt/zimbra/zmstat/2014-03-28/df.csv.gz
-rw-r----- zimbra/zimbra  8013 2014-03-29 06:40 opt/zimbra/zmstat/df.csv
Directory Sizes In /opt/zimbra

Please see the following and provide the output to support. Note, even though this method is faster than doing a du it still can take awhile.

* Ajcody-Server-Misc-Topics#Faster_Way_To_Get_Directory_Size_On_Filesytem_-_find_vs_du
Adjusting The Disk Alert Threshold

Note - zmlocalconfig smtp_notify must return yes if you want to receive the notifications.

If you just need to adjust the disk alert threshold, then see the following:

See current values:

 zmlocalconfig | grep zmdisklog

Example adjustment:

 su - zimbra
 zmlocalconfig -e zmdisklog_critical_threshold=98
 zmlocalconfig -e zmdisklog_warn_threshold=95
 zmstatctl

To exclude a partition from the checks [example of two being excluded]:

 su - zimbra
 zmlocalconfig -e zmstat_df_excludes="/mount/point:/mount/point2"
 zmstatctl

They might be a bug on this, where you'll keep getting email until a logrotate happens [zimbra.log?].

Some things to do to confirm and share with support or in bug. As zimbra

su - zimbra

ls -la /var/log/zimbra.log

df -h
 /dev/mapper/vg_rhel664-lv_root
                      5.5G  3.5G  1.7G  68% /
 tmpfs                 939M     0  939M   0% /dev/shm
 /dev/sda1             485M   79M  381M  18% /boot
 /dev/sdb1              30G  6.2G   23G  22% /opt

date

zmlocalconfig | grep zmdisklog
 zmdisklog_critical_threshold = 80
 zmdisklog_warn_threshold = 85  

zmlocalconfig -e zmdisklog_critical_threshold=95

zmlocalconfig -e zmdisklog_warn_threshold=90

zmlocalconfig | grep zmdisklog
 zmdisklog_critical_threshold = 95
 zmdisklog_warn_threshold = 90

zmstatctl restart

date

ps -eaf | grep zmstat-df

ls -la /var/log/zimbra.log

date ; grep "Disk warning" /var/log/zimbra* ; zmmailbox -z -m admin@`zmhostname` s -l 100 -t message "Subject: Disk and after:yesterday"

##Note - Emails by default go out every 10 minutes - for example:

[zimbra@zcs803 ~]$ date ; grep "Disk warning" /var/log/zimbra* ; zmmailbox -z -m admin@`zmhostname` s -l 100 -t message "Subject: Disk and after:yesterday"
Thu May 22 09:40:08 PDT 2014
/var/log/zimbra.log:May 22 08:30:00 zcs803 zimbramon[18826]: 18826:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82%
/var/log/zimbra.log:May 22 08:40:00 zcs803 zimbramon[22970]: 22970:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82%
/var/log/zimbra.log:May 22 08:50:00 zcs803 zimbramon[22970]: 22970:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82%
/var/log/zimbra.log:May 22 09:00:00 zcs803 zimbramon[22970]: 22970:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82%
  ## Note - I had readjusted the variable to not warn during this time segment ##
/var/log/zimbra.log:May 22 09:20:00 zcs803 zimbramon[8322]: 8322:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82%
/var/log/zimbra.log:May 22 09:30:00 zcs803 zimbramon[8322]: 8322:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82%
/var/log/zimbra.log:May 22 09:40:00 zcs803 zimbramon[8322]: 8322:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82%
num: 7, more: false

     Id  Type   From                  Subject                                             Date
   ----  ----   --------------------  --------------------------------------------------  --------------
1.  328  mess   admin                 Disk / at 82% on zcs803.DOMAIN.com:           05/22/14 09:40
2.  327  mess   admin                 Disk / at 82% on zcs803.DOMAIN.com:           05/22/14 09:30
3.  326  mess   admin                 Disk / at 82% on zcs803.DOMAIN.com:           05/22/14 09:20
  ## Note - I had readjusted the variable to not warn during this time segment ##
4.  325  mess   admin                 Disk / at 82% on zcs803.DOMAIN.com:           05/22/14 09:00
5.  324  mess   admin                 Disk / at 82% on zcs803.DOMAIN.com:           05/22/14 08:50
6.  323  mess   admin                 Disk / at 82% on zcs803.DOMAIN.com:           05/22/14 08:40
7.  320  mess   admin                 Disk / at 82% on zcs803.DOMAIN.com:           05/22/14 08:31

Continue to monitor your zmmailbox search results for an hour.

The Basic Information Support Needs

as root:

  • cat /etc/fstab
    • Shows us what is mounted upon boot
  • cat /proc/mounts
    • Shows us what is currently mounted and its status - you can see if a mount is read-only here.
  • df -hT
    • Lists current mounts using human-readable size information and also notes the filesystem type.

as zimbra:

  • zmprov -l gs `zmhostname` | egrep 'Back|Redo'
    • Will show us a number of variables related to backup and redologs. Also tell us if your using auto-group or the default method.
  • du -sh /opt/zimbra/redolog
    • Will might notice your redolog logs aren't rolling over, causing a possible issue.
  • ls -latr /opt/zimbra/backup
    • This is the default backup target, please adjust this path here and below if you are using a different zimbraBackupTarget value.
    • zmprov gs `zmhostname` zimbraBackupTarget
    • We'll be able to confirm permissions are right.
  • ls -latr /opt/zimbra/backup/tmp
    • This will show us if you have failed backup jobs and confirm tmp is being cleaned appropriately after the backup is done.
  • ls -latr /opt/zimbra/backup/sessions
    • This will show us what backup sessions are available and confirm permissions are correct.
    • Adjust path if your zimbraBackupTarget value is not the default path.
      • su - zimbra
      • zmjava com.zimbra.cs.backup.util.GetVersion
      • cd /opt/zimbra/backup/sessions/full-[YOUR MOST RECENT FULL]/
      • head -n6 session.xml
      • cd /opt/zimbra/backup/sessions/incr-[YOUR MOST RECENT INCREMENTAL]/
      • head -n6 session.xml
  • Some directory sizes in the backup directory:
    • Default path first
      • du -sh `find /opt/zimbra/backup -maxdepth 2 -type d`
    • If your using a different backup target, check that directory also. Replace /opt/zimbra/backup above with your backup path.
  • zmbackupquery
    • This should match what's in the sessions directory and it will also tell us if status of each backup and how many accts were done.
  • crontab -l | grep -i back
    • This will show use when backups are support to run and with what options they are running with.
  • zmlocalconfig | grep -i back
    • This is useful to see a number of backup options not exposed in the crontab, things related to the zip options.
  • zmvolume -l
    • This is useful to see how many volumes are being used, if HSM is being used, and if compression is being done at the volume level.
Additional Log Files Support Might Need

And send the following logs:

  • /var/log/messages
    • Filesystem issues often times are noted here and also in syslog. This might explain an interruption in the backup process. Server restarts, filesystem going full, filesystem going read-only, etc.
  • /var/log/syslog
  • /opt/zimbra/log/mailbox.log
    • The backup activity is logged here.
    • And any other mailbox.log file that would cover the event

Additional Checks For Performance Specific Issues

If Your Using a SAN or NFS For Your Backup Target - Please Check Your IOWait

Ideally, you would compare iowait and performance data from the target backup host as well as the stats available on the ZCS servers. To get graphs and stats on this from ZCS, please see Ajcody-Testing-Debugging#zmstat_and_zmstat-chart . You should submit this data and iowait conclusions if you still need to submit a support case about backup performance issues.

Is HSM Running During Your Backup Window
Are You Using --zipStore

--zipStore zips the blobs vs. keeping the blobs as individual files. --zipStore does not use compression either. For most circumstances, this will give the best performance, especially with NFS. This should be the default behavior of the backups, the following RFE is when it became the default [ZCS6+] :

To see if zip's are being used for backups for example, in the backup/session directory you'll know if it is by seeing .zip files:

mail:~/backup/sessions/full-20080820.160003.770/accounts/115/988/11598896-a89b-4b9d-bedb-1ed1afcb6c87/blobs 
zimbra$ ls blobs-1.zip    blobs-2.zip    blobs-3.zip    blobs-4.zip

To see if the zip file is using compression [-Z option for unzip will indicate whether or not the archive is actually compressed] :

unzip -Z blobs-4.zip
293 files, 5982984 bytes uncompressed, 5982984 bytes compressed:  0.0%

Also, if your zmvolume has compression enabled the blobs will remain compressed within the zip also upon backup. The point being, they are uncompressed to be then put into a zip file when the backup is using --zipStore.

Restore Compatibility Between ZCS Versions

Please see the following for details. In summary, user level restores should work against older ZCS backup data.

Other related bug/rfe's:

NFS Use For Backups

Please see this RFE:

This is the proposed statement to be included in the release notes following the RFE:

ZCS & NFS:
Zimbra will support customers that store backups (e.g. /opt/zimbra/backup) on an NFS-mounted partition. Please note that this does not relieve the customer from the responsibility of providing a storage system with a performance level appropriate to their desired backup and restore times. In our experience, network-based storage access is more likely to encounter latency or disconnects than is equivalent storage attached by fiber channel or direct SCSI.
Zimbra continues to view NFS storage of other parts of the system as unsupported. Our testing has shown poor read and write performance for small files over NFS implementations, and as such we view it unlikely that this policy will change for the index and database stores. We will continue to evaluate support for NFS for the message store as customer demand warrants.
When working with Zimbra Support on related issues, the customer must please disclose that the backup storage used is NFS.

Things To Check

  • Check the /var/log/messages on both the zimbra server and the nfs server for nfs related errors during the time frame of your backup.
  • Check /opt/zimbra/log/mailbox.log for error messages about folders/files not being able to be written or missing directory errors.
  • Is root_squash configured on the nfs server? If it's changed to no_root_squash , does the behavior of the backup change?
  • Is the */backup directory owned by zimbra:zimbra with at least 750 or 755 permissions?
    • This parent directory as given in:
      • zmprov gs `zmhostname` zimbraBackupTarget
  • Does zimbraBackupTarget have at least the subdirectories of : sessions and tmp  : and are owned by zimbra:zimbra with 750 or 755 permissions?
    • If not, try manually creating them and then running a test backup.
  • IF USING A NAS - MAKE SURE YOUR NOT USING EXTENDED ACLS OR THAT YOU HAVE THEM CONFIGURED PROPERLY
    • If your backup session directory shows something like : drwxrwx---+ 2 zimbra zimbra 4096 Sep 14 00:00 TO_DELETE-full-XXXXXX , that + sign indicated extended acls are in use.

Debugging Example

Steps I wrote for one customer, where saving out the information as you walk through all the commands would give enough information [hopefully] to submit a good rfe/bug:

 1. make a test partition on nfs server - /nfs-test

 2. mount on zimbra server
 2A. mkdir /nfs-test
 2B. chmod 755 /nfs-test
 2C. mount nfs-server:/nfs-test /nfs-test
 2D. ls -la /nfs-test
 2E. mkdir /nfs-test/backup
 2F. chown zimbra:zimbra /nfs-test/backup
 2G. chmod 755 /nfs-test/backup
 2H. su - zimbra ; touch /nfs-test/backup/testfile 
 2I. ls -laR /nfs-test/
 2J. rm /nfs-test/backup/testfile

 3. Set zimbraBackupTarget
 3A. zmprov ms `zmhostname` zimbraBackupTarget /nfs-test/backup

 4. Run a full backup against one account
 4A. ex. zmbackup -f -a user@domain.com

 5. ls -laR /nfs-test/

 6. If you again, run into the same problem. You could also repeat the backup after increasing the backup 
    logging variable for the account your trying to backup. If you didn't run into the same problem, it might 
    had to do with the initial setup of the nfs mount and permissions being used during the directory creation.
 6A. zmprov aal user@domain.com zimbra.backup debug
 6B. logging will show up in /opt/zimbra/log/mailbox.log
 6C. Remove account logging when your done.  zmprov ral user@domain.com zimbra.backup

 8. Change zimbraBackupTarget back to your production path.

Setup A Fast Test NOT Using NFS

A way to "avoid" the NFS issues for testing purposes would be to setup a new zimbraBackupTarget to try doing a full backup of a couple of user accounts. I DON'T recommend this if your using auto-group for zimbraBackupMode [ zmprov gs `zmhostname` zimbraBackupMode ] , only if your using Standard mode.

[as root]
** adjust your new "backup" directory path as needed - mine is just an example**
mkdir /mnt/usb1/backup-test
chown zimbra:zimbra /mnt/usb1/backup-test
chmod 750 /mnt/usb1/backup-test
su - zimbra
**confirm backup mode as standard**
zmprov gs `zmhostname` zimbraBackupMode
**if not stand, please stop**
zmprov ms `zmhostname` zimbraBackupTarget /mnt/usb1/backup-test
**Find a couple of test accounts to do a full for. Make sure you'll have space to do the backup regards to need free space.**
zmbackup -f -a user1@domain.com user2@domain.com user3@domain.com
**Watch and confirm status of backup you just started.**
zmbackupquery
**Confirm files were backed up in right location**
ls /mnt/usb1/backup-test/sessions/
**Failed backups would most likely results in left over directory in tmp directory**
ls /mnt/usb1/backup-test/tmp/

Restore For Disaster Recovery

For Full Single ZCS Server DR Restores

Please see Network_Edition_Disaster_Recovery

Some additional notes I have on it - Ajcody-Disaster-Recovery-Specific-Notes

For Multi-Server DR Restore Specifics

Along with the above references, please see Ajcody-Notes-Multi-Server-Restore-DR

To Restore Just The LDAP Date

Let's say your ldap data was 'lost/destoyed' but everything else was intact, you should look at the zmrestoreldap command.

This section should have more precaution and background information to handle this section.

The basics:

  • To find the LDAP session labels type -lbs.
    • zmrestoreldap -lbs
  • Restore the complete LDAP directory server
    • zmrestoreldap -lb full20061130135236
  • Restore LDAP data for specific accounts
    • zmrestoreldap -lb full20061130135236 -a tac@DOMAIN.com jane@DOMAIN.com

To Restore Just The Mysql DB

Option available from the following RFE work:

  • "Allow backup of only primary message volume"
    • https://bugzilla.zimbra.com/show_bug.cgi?id=35278
      • Options to exclude types of data as described in the 6.0.6 Admin Guide:
        • Search index : If you do not restore the search index data, the mailbox will have to be reindexed after the restore.
          • zmrestore <all or account> --exclude-search-index
        • Blobs : This is a useful option when all blobs for the mailbox being restored already exists.
          • zmrestore <all or account>|--exclude-blobs
        • HSM-blobs : This is useful when all HSM blobs for the mailbox being restored already exists.
          • zmrestore <all or account> --exclude-hsm-blobs

Let's say your mysql data was 'lost/destoyed' but everything else was intact. This might be the solution for the situation:

The steps below are to REPRODUCE A DR situation to test.

  1. zmcontrol stop
  2. Caution - step to reproduce DR situation to test against
    1. mv ~/db/data ~/db/data.OLD
  3. Getting old mysql passwords
    • zmlocalconfig -s mysql_root_password
    • zmlocalconfig -s zimbra_mysql_password
  4. [as zimbra] /opt/zimbra/libexec/zmmyinit
  5. ldap start
  6. zmconvertctl start
  7. mysql is already running per the zmmyinit - mysql.server status - to check.
  8. zmrestoreoffline -a all -br --systemData --excludeSearchIndex --excludeBlobs --excludeHsmBlobs
    • Note - you most likely will want to use the -br or -rf option for this situation.
      • -br,--backedupRedologsOnly : Replays the redo logs in backup only, which excludes archived and current redo logs of the system
      • -rf,--restoreFullBackupOnly : Restores to last full backup only, which excludes incremental backups.
    • -sys,--systemData : Restores global tables and local config.
    • Note the use of -a all , you could also do this for one account first to confirm operation is successful for your circumstances.
  9. If you used -rf or -br with your zmestoreoffline, you might also need to use zmplayredo to finish it up to get the most complete restore.

Performance Issues And Time To Complete

Please see Ajcody-Notes-BackupPlans#What_About_Backups.3F_I_Need_A_Plan

Also created an RFE for increase backup performance:

Understanding Option Flags For zmbackup & zmrestore

First, they don't make sense if your just reading from the help output - I will not argue this point at all.

The biggest problem with the options I point out below is that you can often include them in the command and they do nothing or you include them for a particular situation and they don't apply. Why is this a problem? Because they are silent and give no output telling you that it's not necessary, it's redundant, or it will actually cause your intended results to fail simply because you included the option.

zmrestore Options

Problems mostly revolve around these options.

To Times In The Past (If -lb Isn't Used, Implies Your Using Times/Incr/RedoSeq AFTER Last Full)
  • -restoreToIncrLabel
    • <arg> Replay redo logs up to and including this incremental backup
      • Requires: --label or -lb
  • -restoreToTime
    • <arg> Replay rodo logs until this time
      • Requires: --label or -lb
  • -restoreToRedoSeq
    • <arg> Replay up to and including this redo log sequence
Redolog Variables
  • --backedupRedologs or -br
    • Replays the redo logs in backup only, which excludes archived and current redo logs of the system
      • Only useful when restoring against latest full backup (NOT using the -lb option).
      • Will restore using incremental backup data after the last full backup as well but NOT including any redolog activity.
  • --restorefullBackup or -rf
    • Restores to the full backup only, not any incremental backups since that backup.
      • The default behavior of zmrestore in general is to always play from a "full" and "incrementals" that are associated with it UNLESS you state otherwise.
        • If you do:
          zmrestore -a user@domain.com -lb full6monthsago
        • It will playback the incremental data associated with that full from 6 months ago.
        • If you do:
          zmrestore -a user@domain.com -lb full6monthsago -rf
        • It will ONLY playback the data in the full from 6 months ago.
      • Will not progress past the data that's in the last full backup, no incremental backups after it in other words.
      • Implies NO redolog play, so there's no need to use -br.
Targets And Labels
  • --label or -lb
    • <arg> The label of the full backup to restore. Restores to the latest full backup if this is omitted.
  • --target or -t
    • <arg> Specifies the backup target location. The default is <zimbra_home>/backup.
Impact Of AutoGroup Option Being Used

Place for notes about how autogroup backup option might impact or limit command options.

zmbackup Options

Problems mostly revolve around these options:

  • --target or -t
    • <arg> Specifies the target backup location. The default is <zimbra_home>/backup.
  • --zip or -z
    • Zips email blobs in backup - using compression
  • --zipStore
    • Zips email blobs in backup - does NOT use compression
Deleting Old Backups -del

Caution You want to delete from the oldest label to newest. The -del option will automatically purge all older sessions prior to the label you used. To find out the label names, use zmbackupquery.

Format example:

zmbackup -del <oldest_backup_label>
Impact Of AutoGroup Option Being Used

Place for notes about how autogroup backup option might impact or limit command options.

Changing Default Backup Target

To find out what the current backup target is, do:

 zmprov gacf | grep zimbraBackupTarget
 zimbraBackupTarget: /opt/zimbra/backup

This is also configurable at the "server" level:

 zmprov gs `zmhostname` | grep zimbraBackupTarget
 zimbraBackupTarget: /opt/zimbra/backup

For example:

 zmprov ms `zmhostname` zimbraBackupTarget /san/mount/backup

Issues of changing the default path in regards to the admin web console. Please see:

Another way to change the backup path is described at Changing_Backup_directory_and_General_Information. I recommend reviewing it as well.

See Change Location For Backup Or Restore Source Data for non-default adjustments to CLI commands in regards to a non-default path for backups.

RFE & Bugs Concerning Option Flags For Restore And Backup

Setting Account Status After Restore Is Done

I filed an RFE for this:

Use DL COS Status To Generate User List Of Accounts

I filed an RFE for this:

An Overview Of Some Backup/Restore Items

Shared Blobs (messages)

I believe we are a little light on describing the shared blobs situation. Shared blobs can cause different corrections to a problem as compared to a normal message issue that isn't shared. I'll start some notes on this here.

From Backup and Restore

  • "When backing up shared messages, the backup process looks to see whether a Binary Large Object file (BLOB) representing a message already exists in the backup. If it does, it simply flags this object as such and does not copy its content again."
  • "Keeping the same backup target saves disk space, because shared binary large object files (BLOB) and other files do not have to be duplicated every time the backup process runs.

Bugs/RFE's

Remote copies of backup data for DR use

You will want to copy over /opt/zimbra/redolog/archive/* and /opt/zimbra/redolog/redo.log frequently in order to stay current. The redo.log file being open is not a problem since the crash recovery step can work with redo.log file in any state.

The redolog/archive/ contains logs that have not yet been backed up by an incremental (or by a full in auto-grouped mode)

The redo.log rolls over when it reaches zimbraRedoLogRolloverFileSizeKB (by default 100MB prior to ZCS 5.0.11 and 1GB after). When ZCS restarts after a crash, it seems to work through the current redo.log ok regardless of its state, if the current log really must be copied."

My Initial Thoughts On This

Start of process:

  • Weekend full on prod
  • rsync full-xxx on prod > remote sessions/
  • rsync redolog/* > remote redolog/
    • through non-full and incremental times every x about of minutes
  • weekday nights 10pm incre-xxx on prod
  • rsync incr-xxx on prod > remote sessions/
  • rsync redlog/* > remote redolog/

Created three separate rsync cron rules.

  • Full - once a week
    • Confirms full is done and then looks for latest full-xxxx and rsyncs that specific directory
  • Incre - once a night except for full schedule night
    • Confirms incre is done and then looks for latest incr-xxx and rsyncs that specific directory
  • Redolog/ - every x amount of minutes (outside of full and incr backups sessions)
    • Soes full rsync of redolog/ - probably want delete/remove option?
    • lsof will report /opt/zimbra/redolog/redo.log is open.

Somewhere we need to account for accounts.xml in this process. And also confirm what else might be missing. Also, steps on the actually restore process depending on when/where the DR event took place.

What's Needed For Later Restores

In regards to what is moved and what is needed for later restores, you must remember this "flow" of the backups:

  • Full backup files that contains all the information needed to restore mailboxes (to that point in time)
  • Incremental backup files that contains the LDAP directory server files and all the redo log transactions written since the last backup (to that point in time)
  • Redo logs that contains current and archived transactions processed by the Zimbra server since the last incremental backup (to that point in time - more about this topic is above)

Variables to be aware of in regards to backup/restore [ viewed with - zmprov gs [server-name] | grep Backup ]:

  • This is that gui path option - note, it doesn't change default to redo file path [see below]
zimbraBackupTarget

zimbraBackupMode
zimbraBackupAutoGroupedInterval
zimbraBackupAutoGroupedNumGroups
zimbraBackupAutoGroupedThrottled

zimbraBackupReportEmailSubjectPrefix
zimbraBackupReportEmailSender
zimbraBackupReportEmailRecipients
  • Variables to be aware of in regards to redo files [ viewed with - zmprov gs [server-name] | grep Redo ]:
zimbraRedoLogArchiveDir
zimbraRedoLogDeleteOnRollover
zimbraRedoLogEnabled
zimbraRedoLogFsyncIntervalslMS
zimbraRedoLogLogPath
zimbraRedoLogRolloverFileSizeKB

Mysql Table That References Most Recent Backup Session Of Users (AutoGroup Backup Mode)

During the backup we update the zimbra.mailbox table for each mailbox to record the most recent backup time. This is in the "last_backup_at" column within Mysql.

This data is used by auto-grouped backup to figure out which mailboxes to backup.

Creating A List Of Last Backup Of Users

Remove the , | head , below to get a full listing of all your accounts. Note, this reports on the users that exist on the mailstore your running the command.

[zimbra@zcs806 tmp]$ mysql zimbra -NBe 'select from_unixtime(last_backup_at), comment from mailbox' | sort | head
2014-06-05 01:00:24     dluser558@zcs806.DOMAIN.com
2014-06-05 01:00:26     dluser557@zcs806.DOMAIN.com
2014-06-05 01:00:28     dluser556@zcs806.DOMAIN.com
2014-06-05 01:00:30     dluser555@zcs806.DOMAIN.com
2014-06-05 01:00:31     dluser554@zcs806.DOMAIN.com
2014-06-05 01:00:32     dluser551@zcs806.DOMAIN.com
2014-06-05 01:00:32     dluser552@zcs806.DOMAIN.com
2014-06-05 01:00:32     dluser553@zcs806.DOMAIN.com
2014-06-05 01:00:33     dluser550@zcs806.DOMAIN.com
2014-06-05 01:00:34     dluser549@zcs806.DOMAIN.com

I include also below the results from the zmbackupquery for the first user:

[zimbra@zcs806 tmp]$ date ; zmbackupquery -a dluser558@zcs806.DOMAIN.com
Wed Jun 11 14:29:48 PDT 2014
Account: dluser558@zcs806.DOMAIN.com

    Label:   full-20140605.080023.296
    Type:    full
    Started: Thu, 2014/06/05 01:00:23.296 PDT
    Ended:   Thu, 2014/06/05 01:02:37.739 PDT
    Acct ID: 85d3c8f4-eea8-4cf2-8a84-8f1fcc87d555

    Label:   full-20140529.080016.546
    Type:    full
    Started: Thu, 2014/05/29 01:00:16.546 PDT
    Ended:   Thu, 2014/05/29 01:02:51.556 PDT
    Acct ID: 85d3c8f4-eea8-4cf2-8a84-8f1fcc87d555

    Label:   full-20140522.080015.818
    Type:    full
    Started: Thu, 2014/05/22 01:00:15.818 PDT
    Ended:   Thu, 2014/05/22 01:02:42.126 PDT
    Acct ID: 85d3c8f4-eea8-4cf2-8a84-8f1fcc87d555

    Label:   full-20140515.080016.160
    Type:    full
    Started: Thu, 2014/05/15 01:00:16.160 PDT
    Ended:   Thu, 2014/05/15 01:02:34.204 PDT
    Acct ID: 85d3c8f4-eea8-4cf2-8a84-8f1fcc87d555
Possible Issue That A Failed Or Interrupted Backup Causes

An interrupted backup can cause an issue because the table currently gets updated right off the bat rather than waiting for backup to be successfully completed.

Possible RFE: To update zimbra.mailbox.last_backup_at column for successfully backed-up mailboxes to the very end of the backup process, to either just before or just after renaming the /opt/zimbra/backup/tmp/<backup label> directory to /opt/zimbra/backup/sessions/<backup label>.

Setting To Null To Cause A New Backup For User

To undo what was done by an interrupted backup for example, you need to clear this column (set it to null) for the affected mailboxes. By clearing the column, you're forcing the next AG backup to choose these mailboxes because they look like they have never been backed up. If you don't clear this column, you have to wait until the next cycle. (7 days)

Example syntax to view:

mysql zimbra -e "select last_backup_at from mailbox where id=27"

Example syntax to change data of the last_backup_at to NULL"

mysql zimbra -e "UPDATE mailbox SET last_backup_at = NULL WHERE id = 27"
Related Bugs RFE's

Restore Requires accounts.xml File

Also, see:

Change "Location" For Backup Or Restore Source Data

Remember that zmbackup and zmrestore can take flags as well in regards to the location of items.

  • zmrestore & zmbackup can both take : -t,--target (default <zimbra_home/backup)

You can't state a different location with redo logs though. There's a command called zmplayredo [for newer versions of ZCS] and it has a variable to point to the redologs to play from [ --logfiles ]. It will replay into the default redolog directory or redolog file. The mailbox has to be stop to run zmplayredo . This is a command to manual kick off a replay of a redo log. This is normally done with the zmrestore when options about to a specific time aren't included.

Manual Removal Of Older Backup Sessions

General Situation:

  • Keep in mind every restore requires starting with data from a full backup.
  • For each account on the server, there must be at least 1 full backup after the deletion is complete.
  • You should also make sure all incremental backups made after the oldest of the remaining backups are retained.
    • This basically gets reduced to deleting only those backups that are old enough, based on your full/incremental schedule.

More Specific Issues:

  • Does the accounts.xml dependency cause issues with this?
    • No. Just don't delete it.
  • What about the contents of the backup/tmp/ data or shared blobs type references?
    • Don't touch this directory either. It is used during backup and restore. You don't want to change its content while an operation is going on, so best to leave it alone.
  • What if a zimbra server is running some type of restore of backup command while the manual removal is running on the nfs server?
    • You shouldn't remove the backups that are being used in a restore currently underway. You are responsible for avoiding the race condition.
      • Please understand you are responsible for avoiding the race condition. Make sure no backup or restore is happening at the moment, then rename the directories that will be deleted, preferably move them to another subdirectory, e.g. /opt/zimbra/backup/sessions_to_delete. Then delete.

LDAP Backup Related Items

Backup Schedule On LDAP Only Server [non-Mailstore]

If you look at the code in /opt/zimbra/bin/zmschedulebackup  :

if ($BACKUP_MODE eq 'Standard') {
    # default schedule: full backup 1am every sunday, incr backup 1am every weekday
    # deletes backups older than a month at 12am everyday
  if (isLdapOnly()) {
      @default = ("0 1 * * 6 /opt/zimbra/bin/zmbackup $target\n",
            "0 0 * * * /opt/zimbra/bin/zmbackup -del 1m\n"); 
  } else { 
      @default = ("0 1 * * 6 /opt/zimbra/bin/zmbackup -f $account $target $compress \n",
          "0 1 * * 0-5 /opt/zimbra/bin/zmbackup -i $compress\n",
            "0 0 * * * /opt/zimbra/bin/zmbackup -del 1m\n");
  }
    %commandmap = ("f", "/opt/zimbra/bin/zmbackup -f $target $compress $account",
                   "i", "/opt/zimbra/bin/zmbackup -i $compress",
                   "d", "/opt/zimbra/bin/zmbackup -del");
} else {  # Auto-Grouped mode
    # default schedule: full backup 1am everyday, no incr backup
    # deletes backups older than a month at 12am everyday
    @default = ("0 1 * * 0-6 /opt/zimbra/bin/zmbackup -f $target $compress\n",
                "0 0 * * * /opt/zimbra/bin/zmbackup -del 1m\n");
    %commandmap = ("f", "/opt/zimbra/bin/zmbackup -f $target $compress",
                   "i", "/opt/zimbra/bin/zmbackup -i $compress",
                   "d", "/opt/zimbra/bin/zmbackup -del");
}

Notice the specific check and then format for ldapOnly:

  if (isLdapOnly()) {
      @default = ("0 1 * * 6 /opt/zimbra/bin/zmbackup $target\n",
            "0 0 * * * /opt/zimbra/bin/zmbackup -del 1m\n"); 

Related Bugs To LDAP Backups

Some bugs to be aware of - most are resolved/fixed:

A Way To Verify Backup Integrity

I filed an RFE for this:

Negative Seek Offset Error & RFE

Explanation of negative seek offset error during a restore attempt and manual fixes are located here:

Auto-Group Backups Rather Than Default Method Topics

General Description And Official References

Having trouble completing that entire full backup during off-hours? Enter the hybrid auto-grouped mode, which combines the concept of full and incremental backup functions - you’re completely backing up a target number of accounts daily rather than running incrementals.

Auto-grouped mode automatically pulls in the redologs since the last run so you get incremental backups of the remaining accounts; although the incremental accounts captured via the redologs are not listed specifically in the backup account list. This still allows you to do a point in time restore for any account.

Administrative manual page:

http://www.zimbra.com/docs/ne/latest/administration_guide/10_Backup_Restore.17.01.html

Compare the sections called "Standard Backup Method" & "Auto-Grouped Backup Method"

http://www.zimbra.com/docs/ne/latest/administration_guide/10_Backup_Restore.17.02.html

Configuration details are here on that page:

http://www.zimbra.com/docs/ne/latest/administration_guide/10_Backup_Restore.17.08.html

Good explanation:

http://www.zimbrablog.com/blog/archives/2008/08/recent-admin-tidbits-part-1.html

Simply divide your total accounts by the number of groups you choose (zimbraBackupAutoGroupedNumGroups is 7 by default) and that’s how many will get a full backup session each night. Newly provisioned accounts, and accounts whose last backup is a specified number of days older are picked first. (zimbraBackupAutoGroupedInterval is defaulted to 1d)

Think of auto-grouped mode as a full backup for the scheduled group as well as an incremental (via redologs) for the all other accounts at the same time.

Bugs - RFE's To Review For Auto-Group

Please see:

Enabling Auto-Group For Backups - Schedule In Crontab

  • You'll need to make the variable changes as listed in the administrative guide and then the following for set the crontab correctly.
  • Run zmschedulebackup --help to see a list of options.
  • Run zmschedulebackup -D , which will now set a new default schedule (crontab) that uses auto-group settings now that your variable are set for it.
  • Incremental backups aren't performed as they were.
    • You'll see there's no longer the -i option in cron.
    • Incrementals are performed, in a manner, but the redologs are just copied into the full backup session

Two bugs to look at as well:

Standard Mode Default Cron Setup
  • Here is a schedule without auto-grouped backups enabled:
    • The Default Schedule For Normal Backups [ zmschedulebackup --help ]:
 f    0 1 * * 6
 i    0 1 * * 0-5
 d 1m 0 0 * * *
    • To set the backup to standard mode and use the default schedule one would:
      • zmprov mcf zimbraBackupMode Standard
      • zmschedulebackup -D
    • The crontab would look like:
 0 1 * * 6 /opt/zimbra/bin/zmbackup -f -a all   
 0 1 * * 0-5 /opt/zimbra/bin/zmbackup -i -a all
 0 0 * * * /opt/zimbra/bin/zmbackup -del 1m
Auto-Group Mode Cron Setup
  • Here is a schedule with auto-grouped enabled:
    • The Default Schedule For Auto-Group Backups [ zmschedulebackup --help ]:
f 0 1 * * 0-6
d 1m 0 0 * * *
    • To set the backup to auto-group mode and use the default schedule one would:
      • zmprov mcf zimbraBackupMode Auto-Grouped
      • zmschedulebackup -D
    • The crontab would look like:
0 1 * * 0-6 /opt/zimbra/bin/zmbackup -f  
0 0 * * * /opt/zimbra/bin/zmbackup -del 1m

Some Variables For Auto-Group

The below might not be complete or the defaults, I just wanted to save this before I forget them. Try to get more complete details on these later.

zmprov gacf | grep Backup
zimbraBackupAutoGroupedInterval: 1d
zimbraBackupAutoGroupedNumGroups: 7
zimbraBackupAutoGroupedThrottled: FALSE
zimbraBackupMode: Auto-Grouped

Auto-group And Redologs

Please see Ajcody-Backup-Restore-Issues#Redologs_And_Auto-group_In_Regards_To_Backups

Problems Switching To Auto-Groups Because It Wants To Run A Full Against All Accounts

Please see the following bug/rfe made about problems switching over to Auto-group when the first backup run of it tries to backup ALL of he accounts. I have the full how-to within the bug. It basically manipulates the last_backup_at for each account.

Backup And Deletion Schedule - zmschedulebackup

How To Adjust The Deletion Schedule

For my example below, I first set the backup to the "default" schedule. And I then adjust that "default" to have backups delete with a 14 day interval rather than 1 month.

[zimbra@mail3 ~]$ zmschedulebackup -D
Default schedule set

Current Schedule:

        f 0 1 * * 6 -a all
        i 0 1 * * 0-5
        d 1m 0 0 * * *

[zimbra@mail3 ~]$ zmschedulebackup -q
Current Schedule:

        f 0 1 * * 6 -a all
        i 0 1 * * 0-5
        d 1m 0 0 * * *

[zimbra@mail3 ~]$ zmschedulebackup -R f "0 1 * * 6" i "0 1 * * 0-5" d 14d "0 0 * * *"
Schedule replaced

Current Schedule:

        f 0 1 * * 6 -a all
        i 0 1 * * 0-5
        d 14d 0 0 * * *

[zimbra@mail3 ~]$ zmschedulebackup -q
Current Schedule:

        f 0 1 * * 6 -a all
        i 0 1 * * 0-5
        d 14d 0 0 * * *

[zimbra@mail3 ~]$ crontab -l | grep backup
0 1 * * 6 /opt/zimbra/bin/zmbackup -f   -a all
0 1 * * 0-5 /opt/zimbra/bin/zmbackup -i  
0 0 * * * /opt/zimbra/bin/zmbackup -del 14d

The Zip - Compression Option For Backups

Using the zip option will compress all those thousands of single files that exist under a user's backup, decreasing performance issues that arise from writing out thousands of small files as compared to large ones. This is often seen when one is :

  • Using nfs for the backup directory
  • Copying/rsyncing backups to a remote server
  • Are using some third party backup software (to tape) to archive/backup the zimbra backup sessions.

Optional Tweaks To The Zip Options

Please see this comment and those underneath it within this RFE:

  • "Use zip files for shared blobs of a full backup made with --zip option"
    • http://bugzilla.zimbra.com/show_bug.cgi?id=26624#c6
      • Use zmlocalconfig. To set:
        • $ zmlocalconfig -e key=val
      • To unset:
        • $ zmlocalconfig -u key
      • Once you set the key you will be able to view it.
        • $ zmlocalconfig
      • backup_zip_copier_private_blob_zips
        • How many zip files to distribute a mailbox's private (unshared) blobs over; default 4 (blobs-1.zip through blobs-4.zip); range 1 to 10,000
      • backup_zip_copier_copy_buffer_size
        • File copy buffer size; default 16384 (16KB); range 4KB to 1MB
      • backup_zip_copier_queue_capacity
        • Each zip file gets a queue of files to add. This key sets the queue size. Default is 10. Range is 1 to 10,000.
      • backup_zip_copier_deflate_level
        • Compression level. Default is -1. (same as in java.util.zip.ZipOutputStream). -1 is same as level 6. This behavior comes from zlib library which the JVM uses to implement zip. Other than the special default value, the level can range from 0 to 9. 0 means no compression. 1 means fastest compression and 9 means best compression.
      • backup_disable_shared_blobs
        • This one isn't limited to zip backups. When this is set to true, all blobs are backed up as private backups. Default is false.
      • backup_debug_use_old_zip_format
        • If true, backup will behave like ZCS 5.0.4 and earlier. Shared blobs are never zipped, and private blobs are added to a single blobs.zip file in zip backup. Default is false.

Need To Write Fewer Files - Add The Zip Option To Your Backup Commands

RFE to make zip option the default for backups:

There is very little details in the official documentation on this option unfortunately. This does have a really good explanation though:

http://www.zimbrablog.com/blog/archives/2008/08/recent-admin-tidbits-part-1.html

From the administrative manual on the Backup section:

http://www.zimbra.com/docs/ne/latest/administration_guide/10_Backup_Restore.15.1.html

It says,

"-zip can be added to the command line to zip the message files during backup. Zipping these can save backup storage space."

It's implied that instead of having all the individual message files in the backup that it will bunch them all together into zip files. The body of a shared blob is added once to a shared-blobs zip file, then a small pointer-only entry is added to a mailbox's zip file. Same effect as in non-zipped case. This will be useful when the number of message files is causing disk i/o issues. Maybe your trying to rsync the backup session directories off to another server or your running a third party backup on it to save to tape. The default use of -zip will use compression, if this also causes overhead that you need to avoid you can use the -zipStore option.

Note about -zipStore:

"when used with the -zip option, it allows the backup to write fewer files (-zip), but not incur the compression overhead as well"

The zip options effect backups that are in blob formats (full's). Incremental backups are bascially redologs, not the full message store of the user. In summary, the zip option will not impact the increment type backups. Auto-group backups are a mixture of both fulls and incrementals.

How To Use As A Default Option?

You'll add the options to the zimbra crontab file. This can be done with the zmschedulebackup command.

Run zmschedulebackup with help option:

zmschedulebackup --help

You'll see:

-z: compress - compress email blobs with zip

It appears that you'll need to manually add the options about -zipStore , if you want it, to the crontab file.

See bug :

http://bugzilla.zimbra.com/show_bug.cgi?id=30981

What Does It Look Like When I Use Zip?

Shared blobs are zipped and blobs (messages) are zipped per root store directory.

mail:~/backup/sessions/full-20080820.160003.770/accounts/115/988/11598896-a89b-4b9d-bedb-1ed1afcb6c87/blobs 
zimbra$ ls blobs-1.zip    blobs-2.zip    blobs-3.zip    blobs-4.zip

General Backup & Restore Debugging

You'll be monitoring the /opt/zimbra/log/mailbox.log file

Include the -d / --debug option on the CLI for either zmrestore or zmbackup .

To increasing logging for backup/restore-related logs - /opt/zimbra/log/mailbox.log . Enable DEBUG log level for "zimbra.backup" logger in :

  • /opt/zimbra/conf/log4j.properties for "temporary" change - until next restart. This could take a couple of minutes before jetty "sees" the changes.
  • /opt/zimbra/conf/log4.properties.in for "permament" change that will stick after restart. A restart of jetty/mailbox would be required for this change - zmmailboxctl restart .
   log4j.logger.zimbra.backup=DEBUG


For incremental backups, this should log each redolog being copied to the backup and also log which ones will be deleted out of archive directory. Those not deleted are kept because they are newer than 1 hour (default). The kept logs are deleted (but not copied again) during the next incremental backup.

Redolog Files


Redologs Copied To Backup Session And When Deleted

Archived logs that are less than an hour old at the time of incremental backup are copied to the backup but aren't deleted to support post-crash waitset reinitialization mechanism. The interval is set in localconfig key backup_archived_redolog_keep_time, which is in seconds, default=3600.

An Outline Of The Step
  1. /opt/zimbra/redolog/redo.log (starts and then grows to zimbraRedoLogRolloverFileSizeKB size - default 100MB)
  2. This flushes to /opt/zimbra/redolog/archive/[file] upon hitting the zimbraRedoLogRolloverFileSizeKB.
  3. 1 & 2 keep repeating when zimbraRedoLogRolloverFileSizeKB is hit.
  4. When a backup is done, the archive/* files are copied. The redo.log file is not moved.
    1. When the backup processes archive/* logs, it first figures out the last sequence copied to backup. All newer logs are copied to the current backup. Then, all logs are deleted except those that are too new, determined by localconfig parameter backup_archived_redolog_keep_time, which defaults to 1 hour. (This is part of the waitset feature.)
      • In standard backup mode, only incremental backups move the redologs.
      • In auto-grouped mode, every backup is a hybrid of full and incremental and thus redologs are moved.
Redologs And Auto-group In Regards To Backups

http://www.zimbra.com/forums/administrators/21360-auto-grouped-backups.html

Think of auto-grouped mode as a full backup for the scheduled group as well as an incremental (via redologs) for the all other accounts at the same time. Auto-grouped mode automatically pulls in the redologs since the last run, so you get incremental backups of the remaining accounts. Although the incremental accounts captured via the redologs are not listed specifically in the backup account list. This still allows you to do a point in time restore for any account.
If You Have Older Redologs Not Being Deleted

According to the code, only archived logs newer than 1 hour old (default for backup_archived_redolog_keep_time) should remain after an incremental backup. It is a bug if you are seeing older logs sticking around. If so, look at mailbox.log and see if any error was logged. If you enable DEBUG logging for "zimbra.backup" logger in log4j.properties you will see log statements for each copy and deletion.

The zimbraRedoLogDeleteOnRollover variable

zimbraRedoLogDeleteOnRollover shouldn't have an effect on "If you have older redologs not being deleted". By default it's FALSE and affects whether or not stuff makes it into /opt/zimbra/redolog/archive at all. With it set to TRUE there's just /opt/zimbra/redolog/redo.log and it's deleted/not rolled over into archive. As discussed above old redologs are deleted after the incremental; thus if you don't take incremental backups you should set this value to TRUE or periodically script manual deletion of /opt/zimbra/redolog/archive. (And with zimbraRedoLogEnabled FALSE there's no redo.log at all.)

If You Don't Run Incremental Backups Or Don't Need Archive Redologs

You would set zimbraRedoLogDeleteOnRollover to TRUE.

(Auto-Grouped backups you can still leave this to the default of FALSE.)

Redolog Sequence And The Backup Session

Redologs will exist in the incremental backup sessions. The zmbackupquery command will reference the redologs associated with the backup. For example"

[zimbra@mail3 ~]$ ls /opt/zimbra/backup/sessions/incr-20080925.224528.230/redologs/
redo-20080925.213136.165-seq53.log  redo-20080925.220726.521-seq55.log  redo-20080925.224209.287-seq57.log
redo-20080925.215516.450-seq54.log  redo-20080925.221749.133-seq56.log

[zimbra@mail3 ~]$ zmbackupquery -lb incr-20080925.224528.230
Label:   incr-20080925.224528.230
Type:    incremental
Status:  completed
Started: Thu, 2008/09/25 18:45:28.230 EDT
Ended:   Thu, 2008/09/25 18:45:39.099 EDT
Redo log sequence range: 53 .. 57
Number of accounts: 2

In the above example, we see the sequence range of "53 .. 57" is referring to the files in the backup session directory called redologs.


RedoLog Variables

Changing Redolog File Size And Location

The /opt/zimbra/redolog/redo.log rolls over when it reaches zimbraRedoLogRolloverFileSizeKB (by default 100mb).

The "roll overs" then goto /opt/zimbra/redolog/archive/

zmprov gacf | grep Redo
zimbraRedoLogArchiveDir: redolog/archive
zimbraRedoLogDeleteOnRollover: FALSE
zimbraRedoLogEnabled: TRUE
zimbraRedoLogFsyncIntervalMS: 10
zimbraRedoLogLogPath: redolog/redo.log
zimbraRedoLogRolloverFileSizeKB: 102400
Need To Move Redologs Because Partition Getting Full

Let's say you have a partition getting full and you need to move the redolog to another partition or nfs mount temporary to deal with the potential crisis that will happen when the partition becomes full. You'll need to reallocate the complete redolog/ directory and the archive subdirectory to the same partition because the roll over from redo.log to the archive directory happens with a rename function within the java code. This will require downtime since you'll need to move the actual redo.log file and zimbra can't be running while you do this. You can use a symlink to your new partition path. For example:

su - zimbra
zmcontrol stop
su -
 ** as root
mkdir /data/redolog
chown zimbra:zimbra /data/redolog
mount /dev/sdb1 /data/redolog
mv /opt/zimbra/redolog/* /data/redolog/
 ** or use rsync
rmdir /opt/zimbra/redolog
ln -s /opt/zimbra/redolog /data/redolog
ls -laR /data/redolog
** confirm ownership is with zimbra and double check zimbra can write in this directory
su - zimbra
touch /data/redolog/testfile
rm /data/redolog/testfile
zmcontrol start
Automatic Deleting Of Redo Logs On Rollover

This variable (zimbraRedoLogDeleteOnRollover) is set TRUE or FALSE.

zmprov gacf | grep zimbraRedoLogDeleteOnRollover

To modify it

zmprov mcf zimbraRedoLogDeleteOnRollover TRUE

Want To See What's In Redolog Files

This is for older versions of ZCS - newer versions should use zmredodump if it's available.

If you suspect there's too much redolog activity during a time window or have another need to inspect the contents of the redolog, dump it and examine it:

$ zmjava com.zimbra.cs.redolog.util.RedoLogVerify /opt/zimbra/redolog/redo.log > out.file

Pick the right redolog file, either redo.log or one of the files under archive/, based on timestamp.

zmplayredo And zmredodump

zmplayredo - Replaying Content From Any Redolog File

zmplayredo is a newer command, first introduced in 5.0.5 I believe. The mailbox has to be stop to run zmplayredo.

The help output from 6.0.8:

$ zmplayredo --help
usage: zmplayredo <options>
    --fromSeq <arg>         Replay from this redolog sequence (inclusive)
    --fromTime <arg>        Replay from this time (inclusive)
 -h,--help                  Show help (this output)
    --logfiles <arg>        Replay these logfiles, in order
    --mailboxId <arg>       Replay for this mailbox only
    --queueCapacity <arg>   Queue capacity per player thread; default=100
    --stopOnError           Stop replay on any error
    --threads <arg>         Number of parallel redo threads; default=50
    --toSeq <arg>           Replay to this redolog sequence (inclusive)
    --toTime <arg>          Replay to this time (inclusive)

Specify date/time in one of these formats:

    2010/11/19 13:55:08
    2010/11/19 13:55:08 802
    2010/11/19 13:55:08.802
    2010/11/19-13:55:08-802
    2010/11/19-13:55:08
    20101119.135508.802
    20101119.135508
    20101119135508802
    20101119135508

Specify year, month, date, hour, minute, second, and optionally millisecond.
Month/date/hour/minute/second are 0-padded to 2 digits, millisecond to 3 digits.
Hour must be specified in 24-hour format, and time is in local time zone.
zmredodump - Replaying Content From Any Redolog File

zmredodump is a newer command and very useful. It does not require mailboxd to be stopped like zmplayredo does.

The help output from 6.0.8:

$ zmredodump --help
usage: zmredodump [options] <redolog file/directory> [...]
where [options] are:

 -h,--help        show this output
    --m <arg>     one or more mailbox ids separated by comma or white
                  space.  The entire list must be quoted if using space as separator.  If
                  this option is given, only redo ops for the specified mailboxes are
                  dumped.  Omit this option to dump redo ops for all mailboxes.
    --no-offset   don't show file offsets and size for each redo op
 -q,--quiet       quiet mode.  Only print the log filename and any errors.
                  This option can be used to verify the integrity of redologs with minimal
                  output.
    --show-blob   show blob content.  Item's blob is printed, surrounded
                  by <START OF BLOB> and <END OF BLOB> markers.  The last newline before end
                  marker is not part of the blob.

Multiple log files/directories can be specified.  For each directory, all
redolog files directly under it are processed, sorted in ascending redolog
sequence order.
Using zmredodump To Get Message Blobs To Inject With zmlmtpinject - RFE

Please see:

Expand Ability To Parse Redologs - RFE

Please see:

Getting A Sequence or Time Variable For Restore Or Replay

You can see the changes within the redo logs with the command below. You can point it to any redolog.

zmjava com.zimbra.cs.redolog.util.RedoLogVerify /opt/zimbra/redolog/redo.log > out.file

You'll get output like this:

VERIFYING: redo.log
HEADER
------
sequence: 59
open: 1
filesize: 512
serverId: d5c5d6a7-b82f-4c29-b0cd-91818057196b
firstOpTstamp: 1222385426273
lastOpTstamp:  1222385426273
version: 1.22

------
txn 1222383600.1 [PurgeOldMessages] ver=1.22, tstamp=1222385426273, change=20200, mailbox=1
txn 1222383600.1 [CommitTxn] ver=1.22, tstamp=1222385426329, mailbox=1, txnType=PurgeOldMessages
txn 1222383600.2 [PurgeOldMessages] ver=1.22, tstamp=1222385486337, change=13500, mailbox=3
txn 1222383600.2 [CommitTxn] ver=1.22, tstamp=1222385486351, mailbox=3, txnType=PurgeOldMessages
txn 1222383600.3 [PurgeOldMessages] ver=1.22, tstamp=1222385546357, change=20201, mailbox=1
txn 1222383600.3 [CommitTxn] ver=1.22, tstamp=1222385546383, mailbox=1, txnType=PurgeOldMessages
txn 1222383600.4 [PurgeOldMessages] ver=1.22, tstamp=1222385606391, change=13501, mailbox=3
txn 1222383600.4 [CommitTxn] ver=1.22, tstamp=1222385606404, mailbox=3, txnType=PurgeOldMessages
txn 1222383600.5 [PurgeOldMessages] ver=1.22, tstamp=1222385666416, change=20202, mailbox=1
txn 1222383600.5 [CommitTxn] ver=1.22, tstamp=1222385666428, mailbox=1, txnType=PurgeOldMessages
txn 1222383600.6 [PurgeOldMessages] ver=1.22, tstamp=1222385726435, change=13502, mailbox=3
txn 1222383600.6 [CommitTxn] ver=1.22, tstamp=1222385726459, mailbox=3, txnType=PurgeOldMessages
txn 1222383600.7 [PurgeOldMessages] ver=1.22, tstamp=1222385786476, change=20203, mailbox=1
txn 1222383600.7 [CommitTxn] ver=1.22, tstamp=1222385786486, mailbox=1, txnType=PurgeOldMessages
txn 1222383600.8 [PurgeOldMessages] ver=1.22, tstamp=1222385846493, change=13503, mailbox=3
txn 1222383600.8 [CommitTxn] ver=1.22, tstamp=1222385846506, mailbox=3, txnType=PurgeOldMessages
txn 1222383600.9 [PurgeOldMessages] ver=1.22, tstamp=1222385906739, change=20204, mailbox=1
txn 1222383600.9 [CommitTxn] ver=1.22, tstamp=1222385906775, mailbox=1, txnType=PurgeOldMessages
txn 1222383600.10 [PurgeOldMessages] ver=1.22, tstamp=1222385966944, change=13504, mailbox=3
txn 1222383600.10 [CommitTxn] ver=1.22, tstamp=1222385966963, mailbox=3, txnType=PurgeOldMessages
txn 1222383600.11 [PurgeOldMessages] ver=1.22, tstamp=1222386026972, change=20205, mailbox=1
txn 1222383600.11 [CommitTxn] ver=1.22, tstamp=1222386026990, mailbox=1, txnType=PurgeOldMessages
...

How Do I Figure Out Which Sequence or Time Variable To Use For Restore Or Replay

In 5.0.10+ we'll have a CLI wrapper (zmredodump) with a slightly different command line syntax, but the below long syntax works in earlier versions.

To locate the correct restore-to time, you have to start with an approximate time the message was added/deleted. Look at the redolog files. The filename contains the GMT time when the file was rolled over, which is roughly the tstamp of the last operation in the file. If your time data is accurate you can find the specific file. Or you have a range of files to examine.

Use the redolog verify tool to dump the contents into text form, the -m / --message option to show message body data:

zmjava com.zimbra.cs.redolog.util.RedoLogVerify -m <filename or directory> ... > out.file

If the message was deleted and you don't know the id, you must go by some other clue such as the subject. Search the file to locate your message. You can cut/paste the message and lmtp-inject it to recover the message. No need to go through with a restore if this is all you needed.

Are You Messages Really Gone - Things To Check If zmplayredo Isn't Doing What You Expect

Here's something I found out testing zmplayredo for a customer case. Testing on a ZCS 6.0.8 single ZCS server.

Created a test account and sent it one message that is in the Inbox. I delete the msg in zwc but don't purge the Trash - msg is in Trash now.

Log events of above action:

2010-10-27 15:07:13,375 INFO [btpool0-3://192.168.0.71/service/soap/ConvActionRequest] 
[name=ajcody@mail71.DOMAIN.com;mid=17;ip=192.168.0.17;ua=ZimbraWebClient 
- FF3.0 (Linux)/6.0.8_GA_2661;] mailop - Moving VirtualConversation (id=-257) to 
Folder Trash (id=3). Affected message ids: 257.

Stop mailboxd so I can use zmplayredo and then start mailboxd back up after zmplayredo is finished :

 zmmailboxdctl stop
 zmplayredo --mailboxId 17 --logfiles /opt/zimbra/redolog/archive/* --fromTime 20101027090000 --toTime 20101027140000
 zmmailboxdctl start

Log event for above:

2010-10-27 15:07:50,383 INFO [main] [] mailop - Adding Message: id=257, Message-ID=
<1604937630.920784.1288201208382.JavaMail.root@corp.zimbra.com>, parentId=-1, folderId=2, folderName=Inbox.
2010-10-27 15:07:50,404 INFO [main] [] RedoableOp - Message 257 is already in mailbox 17

Log into ZWC with the test account. The msg is not in the Inbox, but it's still in Trash folder. I purge it from Trash.

Log event for deletion.

2010-10-27 15:09:38,761 INFO [btpool0-2://192.168.0.71/service/soap/ConvActionRequest]
[name=ajcody@mail71.DOMAIN.com;mid=17;ip=192.168.0.17;ua=ZimbraWebClient 
- FF3.0 (Linux)/6.0.8_GA_2661;] mailop - Deleting Message (id=257).

Then I redo the stop/start of mailboxd and zmplayredo again.

 zmmailboxdctl stop 
 zmplayredo --mailboxId 17 --logfiles /opt/zimbra/redolog/archive/* --fromTime 20101027090000 --toTime 20101027140000 
 zmmailboxdctl start 

Log event for the above:

2010-10-27 15:10:29,192 INFO [main] [] mailop - Adding Message: id=257, Message-ID=
<1604937630.920784.1288201208382.JavaMail.root@corp.zimbra.com>, parentId=-1, 
folderId=2, folderName=Inbox.

Log back into ZWC with test account and now I can confirm that the msg is not in Trash and it is now showing in Inbox.

Gap In Redo Log

The error message from either a backup or restore command:

"Error occurred: Found gap in redo log sequence; missing 5965 through 6149;
To avoid future restore problems, discard all existing backups and take a
full backup of all accounts; If this error occurred during restore,
try the --ignoreRedoErrors option"

The output is pretty accurate in how to handle the situation.

If you get the error during a backup, the recommendation is to move your old backups out. The directories in /opt/zimbra/backup/sessions/* . You'll want to keep them around just in case and then proceed to do a full backup.

If you get the error during a restore, you would add the flag --ignoreRedoErrors to your restore command.

Another possible related issue is if your /tmp or /opt/zimbra/redolog/ is filling up.

Error Executing redoOp

Errors with restores that involve the message 'error executing redoOp' will not show up in the admin console but will when you attempt the restore from CLI. This can also be the cause when you use the RestoreToTime option from the admin console and it doesn't seem to work correctly - the restore stopping prematurely from the specified date/time.

I created the following RFE in regards to the admin console issue:

  • "Include --ignoreRedoErrors option and error feedback in Admin Console for restore"
    • http://bugzilla.zimbra.com/show_bug.cgi?id=52358
      • This could explain why your restore to time isn't working in the Admin Console but does from CLI when you see an error about redologs and then reattempt restore with the --ignoreRedoErrors and it works.

Another RFE that was made but marked as 'WONTFIX' that gives a background story to the issue is:

When you hit this error with your backup data during restore attempts, there's basically only a couple of options to handle this that are recommended by support:

  1. Try to get full backups of your accounts or the accounts in question and then test against them and the preceding incrementals after the full.
  2. Attempt restores via the CLI using the additional option of : --ignoreRedoErrors
  3. Do your restore against the latest full backup of the account in question and then use the zmplayredo command against the redologs in the incrementals and/or the /opt/zimbra/redologs/* directory . This will give you more control to walk the restored account up to the point in time you want it at. One should really read through the whole section above, Ajcody-Backup-Restore-Issues#Redolog_Files , to understand the whole concept of redologs and then the use of zmplayredo.

Generally, "fixing" the redolog itself is not an option.

Why Do My Fulls Not Report All Accounts?

Are you sure it was a full backup that was ran or just a full session that was generated from your incremental backup job? When an incremental is ran, it will create a "full" session for any new accounts it discovers after the last actual full backup job.

For example, here's a full session that was created by an incremental backup job:

Label: full-20081010.060126.559
Type: full
Status: completed
Started: Fri, 2008/10/10 01:01:26.559 CDT
Ended: Fri, 2008/10/10 01:01:28.988 CDT
Redo log sequence range: 705 .. 705
Number of accounts: 1

Label: incr-20081010.060009.420
Type: incremental
Status: completed
Started: Fri, 2008/10/10 01:00:09.420 CDT
Ended: Fri, 2008/10/10 01:01:26.413 CDT
Redo log sequence range: 700 .. 704
Number of accounts: 392


[zimbra@servername ~]$ du -sh /opt/zimbra/backup/sessions/full-20081010.060126.559
1.2M /opt/zimbra/backup/sessions/full-20081010.060126.559
[zimbra@servername ~]$ du -sh /opt/zimbra/backup/sessions/incr-20081003.060010.622
452M /opt/zimbra/backup/sessions/incr-20081003.060010.622

Notice the Start and End times, this will show that the full is related to the incremental job.

You'll want to run zmbackupquery against your full labels to see your "main" full backup session - assuming you can't simply guess based upon the cron entry for it [ su - zimbra ; crontab -l | grep backup ]. For example, to see all your fulls from today's date back to October 01, 2008 and the accounts within each session - you would do:

zmbackupquery -v --from 20081001.000000 --type full

The -v flag outputs the accounts, the --from uses YearMonthDay.HourMinuteSecond , and the --type can be full or incremental. To just see one particular sessions date, you would use the lb [label] flag:

zmbackupquery -v -lb [your full label, ex. full-20081001.000000]

Issues After /opt/zimbra/backup Became Full

Bugs/RFE's on this issue:

See alsoAjcody-Backup-Restore-Issues#Possible_Issue_That_A_Failed_Or_Interrupted_Backup_Causes and the Bugs & RFE's under that section.

You can run into numerous issues if you allow your backup directory to become full.

Confirm your /opt/zimbra/backup/accounts.xml is being updated after a backup. You might see that the newer account.xml* file is accounts.xml.new . This is a sign of problems.

Confirm that the files in /opt/zimbra/backup/tmp/* don't have 0 byte lengths. There might be files like 1.xml and 3.xml in there. If they show 0 bytes, you need to remove them. The backup/restore commands if the file exist and they are empty. Your errors might look like this:

[zimbra@mailb ~]$ zmrestore -a USER@DOMAIN -ca -pre restore_
Error occurred: system failure: Unable to parse XML file /opt/zimbra/backup/tmp/restore/shared_blobs/1.xml

If you tried doing restores (redirected -ca -pre) before to clear up the above issues, you might find you can't do a successful restore AND you can't delete the account afterwards.

If you get errors like :

zmprov da restore3_tester3@XXXXXX.edu
ERROR: service.FAILURE (system failure: writing new mailbox row for account 89e7d9f4-013e-4cf1-a352-7b2f0a00d5af)

zmprov gmi restored_USER
ERROR: service.FAILURE (system failure: writing new mailbox row for account 56a7f654-f85b-45cc-931a-81d9bb9076bf)

You'll need to delete the account via a ldapdelete command.

Query And Stopping A Backup or Restore In Progress

Please see the url below for Backup In Progress - topic name "Aborting Full Backup In Progress":

  • A matter of using zmbackupquery & zmbackupabort .

Please see url below for Restore In Progress - topic name "Stopping a Restore Process" :

A -del Delete In Progress

Please see the bug / rfe's I filed

zmbackupabort syntax

From ZCS 6.0.8

$ zmbackupabort -h
usage: zmbackupabort <options>
 -d,--debug          Display diagnostics for debugging purposes.
 -h,--help           Displays this help message.
 -lb,--label <arg>   Label of full backup set to abort.
 -r,--restore        Abort the restore in progress.
 -s,--server <arg>   Mail server hostname. Default is localhost.
 -t,--target <arg>   Backup target location (default
                     <zimbra_home>/backup).

zmbackupquery syntax

From ZCS 6.0.8

$ zmbackupquery -h
usage: zmbackupquery <options>
 -a,--account <arg>   Account email addresses seperated by white space or
                      "all" for all accounts.
 -d,--debug           Display diagnostics for debugging purposes.
    --from <arg>      List backups whose start date/time is at or after
                      this date/time.
 -h,--help            Displays this help message.
 -lb,--label <arg>    The label of full backup to query.
 -s,--server <arg>    Mail server hostname. Default is localhost.
 -t,--target <arg>    Backup target location (default
                      <zimbra_home>/backup).
    --to <arg>        List backups whose start date/time is at or before
                      this date/time.
    --type <arg>      Backup set type to query.  "full" or "incremental";
                      both if unspecified.
 -v,--verbose         Show account list in each backup.

Specify date/time in one of these formats:

    2010/11/19 14:06:22
    2010/11/19 14:06:22 923
    2010/11/19 14:06:22.923
    2010/11/19-14:06:22-923
    2010/11/19-14:06:22
    20101119.140622.923
    20101119.140622
    20101119140622923
    20101119140622

Specify year, month, date, hour, minute, second, and optionally millisecond.
Month/date/hour/minute/second are 0-padded to 2 digits, millisecond to 3 digits.
Hour must be specified in 24-hour format, and time is in local time zone.

Restore To Time Problems

There's seems to be some syntax issues when using this variable. Please review the following to confirm your syntax.

http://wiki.zimbra.com/index.php?title=CLI_zmrestore_restoreToTime_Network_Edition_only

The gist, you MUST use the -lb full-200xxxxxx option when your trying to restore anything that ISN'T meant to include the latest information for the mailbox. The -lb argument should specify a full backup that took place prior to the time of the backup you wish to restore.

Find Out What Backup Session Labels You Need First

To find out what backups are associated with a particular account, you would do the following :

zmbackupquery -a user@domain

You'll want to note what is the first full that occurs before the point in time you want to restore. And then the incremental that follows right after your point in time.

Backup label (-lb) for fulls can be found by doing [include the -v option if you want to see a listing of the user accounts within the backups] :

zmbackupquery --type full

Backup labels (-restoreToIncrLabel) for incrementals can be found by:

zmbackupquery --type incremental

Command Syntax Example For Restores On The CLI

Example to full label and stop :

zmrestore -a USER@DOMAIN.com -lb full-20080726.050017.306 -rf -ca -pre restore_

Example to incremental label and stop :

zmrestore -a USER@DOMAIN.com -restoreToIncrLabel incr-20080731.060007.644 -lb full-20080726.050017.306 -br -ca -pre restore_

Example to specific time and stop :

zmrestore -a USER@DOMAIN.com -restoreToTime 20080801011800 -lb full-20080726.050017.306 -br -ca -pre restore_


Important Options You Might Want Or Need To Include

--ignoreRedoErrors : If you attempt a restore and you see an error about problems related to playing the redolog, you'll want to run the restore command again and include this option.


--skipDeletes : Please see http://bugzilla.zimbra.com/show_bug.cgi?id=31824#c5 for details on this.


-t /path/to/backup_dir : If you are restoring from another backup directory besides your current default path.


Variables that are asking for TIME rather than LABELS should follow this syntax (from zmrestore --help):

Specify date/time in one of these formats:

    2008/08/06 09:55:50
    2008/08/06 09:55:50 572
    2008/08/06 09:55:50.572
    2008/08/06-09:55:50-572
    2008/08/06-09:55:50
    20080806.095550.572
    20080806.095550
    20080806095550572
    20080806095550

Specify year, month, date, hour, minute, second, and optionally millisecond.
Month/date/hour/minute/second are 0-padded to 2 digits, millisecond to 3 digits.
Hour must be specified in 24-hour format, and time is in local time zone.

Bugs And RFE's To Review

Update - Bug/RFE's I Filed Against ZCS 8.6

Older Bug/RFE's

Restore An Individual Message

The zmrestore command is at a mailbox level.

An RFE was filed already to expand this. It is currently targeted for the Helix release.

A way around the current limitations would be to use lmtpinject. Please see the following for details on that:

The difficultly would be determining the message your trying to find within your backups that was "deleted" in prod.

User Deleted A Bunch Of Data And Notified You Hours Later Wanting It Restored

  1. To determine the time of the delete, use the zmredodump command.
    • You'll use this "time" for the restore command.
    • Example:
      • $ zmprov gmi USER@DOMAIN.com  ::: gives me mailboxid 17
      • $ zmredodump --show-blob --m 17 /opt/zimbra/redolog/ | grep Delete  ::: returns:
        • [0000f311 - 0000f350: 64 bytes; tstamp: 2010/11/19 11:21:30.852 CST] txn 1290033504.2544 [DeleteItem] ver=1.28, tstamp=1290187290852, change=3913, mailbox=17, ids=[304, 308], type=5
        • [0000f351 - 0000f382: 50 bytes; tstamp: 2010/11/19 11:21:30.869 CST] txn 1290033504.2544 [CommitTxn] ver=1.28, tstamp=1290187290869, mailbox=17, txnType=DeleteItem
  2. See Ajcody-Backup-Restore-Issues#Restore_To_Time_Problems about setting up the proper restore syntax.
    • You'll want to include the -ca -pre restore_ options since we'll first get the old data into a temporary account.
    • Remember to use zmbackupquery -a USER@DOMAIN.com --type full to find what full label will be needed for the restore.
    • Also, if you need/want the incremental label, do a zmbackupquery -a USER@DOMAIN.com --type incremental .
      • Example:
        • $ zmbackupquery -a USER@DOMAIN.com --type full  ::: shows the latest as being:
          • Label: full-20101119.070018.607
          • Type: full
          • Started: Fri, 2010/11/19 01:00:18.607 CST
          • Ended: Fri, 2010/11/19 01:00:29.751 CST
          • Acct ID: f33d6daf-8875-4496-8bee-6df345f295e7
    • Do the restore now with the proper information for the variable flags.
      • Note: You might need to include -br , -rf , or neither depending on the time frames involved.
      • Example:
        • $ zmrestore -a USER@DOMAIN.com -restoreToTime 20101119112000 -lb full-20101119.070018.607 -ca -pre restore_
      • It would be best to log into the restore_USER@DOMAIN.com account to confirm the data is as you expect it.
  3. Use zmmailbox with the getRestUrl option against the "restore_USER" account now to export the data.
    • Examples:
      • Export ALL the 'old' data from the restored account
        • $ /opt/zimbra/bin/zmmailbox -z -m restore_USER@DOMAIN.com getRestURL "//?fmt=tgz" > /var/tmp/user-export.tgz
      • You can also export SOME of the 'old' data from the restore account using other options. One option is with the before and after variables. NOTE - We have to set the query string as a variable to get around some of the shell issues.
      • For example:
        • $ query='before:11/20/2010 after:11/1/2010'
          • ZCS5 might require you to have a %20 rather than the actual space character.
        • $ /opt/zimbra/bin/zmmailbox -z -m restore_USER@DOMAIN.com getRestURL "//?fmt=tgz&query=$query" > /var/tmp/user-export.tgz
          • Note': the $query above will be the expansion of the variable you set prior in the shell, query='before:11/20/2010 after:11/1/2010 .
  4. Use zmmailbox with the postRestUrl option to IMPORT the RESTORE_USER account into the "USER" account now with appropriate options.
    • Examples:
      • $ /opt/zimbra/bin/zmmailbox -z -m USER@DOMAIN.com postRestURL “//?fmt=tgz&resolve=replace” /var/tmp/user-export.tgz
    • Note : A critical option in the above command is the &resolve=replace one. There are various ways you can handle the importing of data. Please review the following to determine what is best for you needs.

Restore Deleted Items - skipDeletes Option - ZCS6+

From the RFE comment:

        Added new option --skipDeletes to zmrestore.  If specified, skip 
        over delete operations during redo replay.  Delete ops are:

        DeleteItem (hard delete)
        DeleteMailbox
        EmptyFolder
        MoveItem, if moving item to Trash folder
        PurgeImapDeleted
        PurgeOldMessages

        Skipping these deletes can lead to other problems later, such as 
        conflicting paths, but it is assumed the priority is recovering as 
        much data as possible when using this option.

Restore Account Not Yet In Backups

Please see:

How To Restore To Events Within The Same Day

Work in progress and investigation.

Users Trash Items

User Ability To Recover Trash Purge

Please see RFE:

Retention Policy About Purges

Please see Mailbox_Purge

Can't Restore Or Find An Account That Was Renamed

When an account is "renamed", the old account name will no longer be "found" is your "default" type restore or backup queries. This can cause some confusions when one needs to restore to a time frame of when the account was under it's older name.

Restoring From CLI

First, identify what sessions labels hold the differing account names:

zmbackupquery -a USER@DOMAIN
zmbackupquery -a USER-RENAMED@DOMAIN

You should know have what you need to do follow the steps in the Ajcody-Backup-Restore-Issues#Restore_To_Time_Problems section.

Restoring From Admin Console GUI

I'll give a detail explanation of the situation when working around restores of renamed accounts in the admin console [web GUI]:

  1. If you in the GUI goto the "Restore" button, it first asks for an account rather than giving an option for date/time/session. I think you already stated, that "renamed" accounts don't show up in this query window. Therefore, one wouldn't really progress to the next window that would allow you to change the backup session label.
  2. They way you get around this is, you actually double click on the full session listing that you see on the backup page in the admin UI. This will bring you to another page, that is specific to that session. In there, you should see the old account name prior to the rename. You can then highlight that account listing and click on the "Restore" button. This will bring up the restore dialog, which will now have the date/time/session label auto-filled out.

Quota Is Stopping A Redirected Restore

Update

Reasoning for need, maybe the msg files coming from the restore are no longer "shared message blobs" and therefore increase the mailbox to a size it wasn't in the past. Changes to HSM maybe?

I think I'll need to create a RFE about adding an option to the zmrestore command to also include an option to set the COS value on a created account. Until then... Create a new COS and set it up to NOT have any quota. Once you kick off the backup and you see the account is created you can then apply the COS to that account. Call the cos something like no-quota. Here's the steps below.

To copy your cos (assuming your quota cause is the default one, change default to match your production cos your using).

To see all cos's

zmprov gac

Copies cos called default to cos named no-quota

zmprov cpc default no-quota

To remove quota to the new cos no-quota

zmprov mc no-quota zimbraMailQuota 0

To confirm

zmprov gc no-quota | grep -i quota

Now you would start your redirected restore and once you see the account is created, run the below in a separate shell. (example)

zmrestore -a USER@DOMAIN.com -restoreToTime 20080801011800 -lb full-20080726.050017.306 -br -ca -pre restore-

Once restore has kicked off - To apply the no-quota cos to the restored account:

zmprov sac restore-USER@DOMAIN.com no-quota

Restore Of Non-Account Items - Example - COS DL Etc

Cos and DL's are ldap entries basically.

From Backup and Restore

  • zmrestoreldap. This command restores the complete LDAP directory server, including accounts, domains, servers, COS and other data."

You'll see in a DR process, Network_Edition_Disaster_Recovery , that the zmrestoreldap is done before the zmrestoreoffline .

  • zmrestoreldap doesn't have options that allow specific items to be restored (COS, DL's, etc.). It only has option for named accounts (-a). One could try a ldapadd with a ldif of the COS or DL details. One could also take the information on the COS or DL within the ldap file in the backup session to at least have all the variables to manually add it back (via the zmprov command). Your looking at the backups on the LDAP master if your in a multi-server configuration.
/opt/zimbra/backup/sessions/full-xxxxxx/ldap/ldap.bak

Start of ldap entry example to search for:

  • Cos example
    • dn: cn=default,cn=cos,cn=zimbra
  • DL example
    • dn: uid=dl-group,ou=people,dc=mail,dc=domain,dc=com

To compare a current DL with past details, just save out the ldap entry from the backup to a txt file. And then do:

zmprov gdl maillist@domain.com

Make the necessary changes after comparing the two.

Restoring A Calendar (ics)

There seems to be a bug or odd expectation on how this command is currently working. If the appointment exists in the Calendar and the time is different with the same appointment in the ics file your importing - the time of the appointment will not change to the imported ics one. If you delete the event first, then the imported appointment will reflect the correct time.

Here's what I did to reproduce this situation. It seems this has been true for sometime, customer was on 4.5.11 and I was on 5.0.8

Created test account and made two appointments on friday - 9am and 4pm.
 Did a full backup.
 Restored test account to restore_user
 Ran :
     zmmailbox -z -m user@domain.com gru /Calendar > /tmp/calendarA.ics
     zmmailbox -z -m restore_user@domain.com gru /Calendar > /tmp/calendarB.ics
 And then
     diff /tmp/calendarA.ics /tmp/calendarB.ics [no differences]

 Now some tests.
 As user, I deleted the two appointments and then:
     zmmailbox -z -m user@domain.com pru /Calendar /tmp/calendarB.ics
 Refreshed Calendar as User in webclient.
     9am and 4pm appointment shows up.
 I then moved in the webclient the 9am appointment to 11am
 Did another restore:
     zmmailbox -z -m user@domain.com pru /Calendar /tmp/calendarB.ics
 Refreshed Calendar as User in webclient.
     11am and 4pm appointment shows up.
     ** The restore did not move the 11am appointment back to the 9am slot as in /tmp/calendarB.ics
     ** Assumption, this process will not over-write an appointment if it's there - it does not look to the time.
        Let's do a diff of the state of the calendar
        zmmailbox -z -m user@domain.com gru /Calendar > /tmp/calendarC.ics
        diff /tmp/calendarB.ics /tmp/calendarC.ics
          The DTSTAMP and SEQUENCE shows the difference in the time.
 If I delete the 11am appointment and then do the calendarB.ics restore the appointment shows up again at 9am.

I see this same behavior if I also use the web interface to export/import the calendar between the restored account and user one. Even when I import it into a NEW calendar, it even changes the two appointments to reflect the new calendar rather than the default one.

One Fix

One fix, if the situation allows, is to purge the current Calendar and then import the full ics file. This would be done like this:

zmmailbox -z -v -m ajcody@mail3.internal.homeunix.com ef /Calendar
* ef is for emptyFolder  zmmailbox help folder
*webclient shows all events gone but Calendars are still listed.
zmmailbox -z -v -m ajcody@mail3.internal.homeunix.com pru /Calendar /tmp/calendarB.ics
*webclient show all events with times as expected.

Second Fix

Haven't tried yet, but someone said you should be able to adjust the SEQUENCE number in the appointment and the import process will use that data (data/time) of the newer sequence number appointment.

Archiving User Accounts Out Of Production Use In Zimbra

Backup - Restore Method

To "archive" the user data with the ability of Zimbra later restoring if needed, one would rely on the backup/restore tools.

For example:

zmbackup -f -a user@domain

If you want to make this more "intelligent" later, one could do this:

mkdir /nfs-mount/archived-users/[user-name]
zmbackup -f -a userA@domain -t /nfs-mount/archived-users/userA-domain/

This would give some "intelligent" information for recovery later. Confirms it was an archived account, shows the "real" username, the date of the full backup, and so forth.

ls /nfs-mount/archived-users/userA-domain/
  full-20081104.131643.006
ls /nfs-mount/archived-users/userA-domain/full-20081104.131643.006
  accounts/ session.xml shared_blobs sys

If one would need to "restore" the account later, it would consume a license if the account was "deleted".

zmrestore -a userA@domain -ca -pre restored- -t /nfs-mount/archived-users/userA-domain/ -lb full-20081104.131643.006
Setup A Secondary Zimbra Box For Restores Of Archive Accounts

Your Zimbra license can be installed on multiple machines. One idea that might prove useful in handling these "archive" accounts for those situations when you need to investigate something is to setup a "archive" Zimbra box. You'll want to isolate this box from any "production" use. It will need to be configured to have the "domains" of the archive accounts. You can then use this box to restore the "archive" account and then use the administration tools to investigate the user data.

Use Of REST And Other Tools For Specific User Data

The following page, User_Migration , will shows numerous examples of how to export different types of data from a user account into a neutral file format that one could use for "archive" purposes.

Use Of The REST Command To Export ALL User Data - Version Dependant, 5.0.9+ [I believe]

Example of the commend syntax:

 /opt/zimbra/bin/zmmailbox -z -m user@domain.com getRestURL “//?fmt=tgz” > /tmp/account.tgz

Backup And HSM

Please see Bugs/RFE:

Using zmprov -l gaa To Create User Listing for zmbackup & zmrestore -a option

For example, see below. Note the use of the egrep -v option below, this would remove any matches from the grep. Useful if you want to only backup certain domains for example.

[zimbra@ldap2 sessions]$ zmbackup -f -a `zmprov -l gaa | egrep -v "ham|spam|virus|admin|galsync"`
full-20150130.210821.124

[zimbra@ldap2 sessions]$ zmbackup -f -a `zmprov -l gaa`
full-20150130.210922.637

[zimbra@ldap2 sessions]$ zmbackupquery -lb full-20150130.210821.124 -v
Label:   full-20150130.210821.124
Type:    full
Status:  completed
Started: Fri, 2015/01/30 16:08:21.124 EST
Ended:   Fri, 2015/01/30 16:08:50.181 EST
Redo log sequence range: 23 .. 23
Number of accounts: 3 out of 3 completed
Accounts:
  archive-search-results@ldap2.zimbra.DOMAIN.com: completed
  user1-20150115@ldap2.zimbra.DOMAIN.com.archive: completed
  user1@ldap2.zimbra.DOMAIN.com: completed

Total space: 14020MB
 Free space: 5977MB

[zimbra@ldap2 sessions]$ zmbackupquery -lb full-20150130.210922.637 -v
Label:   full-20150130.210922.637
Type:    full
Status:  completed
Started: Fri, 2015/01/30 16:09:22.637 EST
Ended:   Fri, 2015/01/30 16:09:33.083 EST
Redo log sequence range: 23 .. 23
Number of accounts: 9 out of 9 completed
Accounts:
  admin-20150122@ldap2.zimbra.DOMAIN.com.archive: completed
  admin@ldap2.zimbra.DOMAIN.com: completed
  archive-search-results@ldap2.zimbra.DOMAIN.com: completed
  galsync.tfnapjb9@ldap2.zimbra.DOMAIN.com: completed
  ham.yeygcogdd2@ldap2.zimbra.DOMAIN.com: completed
  spam.dpxyqjnm6t@ldap2.zimbra.DOMAIN.com: completed
  user1-20150115@ldap2.zimbra.DOMAIN.com.archive: completed
  user1@ldap2.zimbra.DOMAIN.com: completed
  virus-quarantine.wfkd4vvm1g@ldap2.zimbra.DOMAIN.com: completed

Total space: 14020MB
 Free space: 5977MB

Scripting Out Individual Backups Of Accounts

If you want to do individual backups of accounts using a for-loop, for example, you might want to include the -sync option from zmbackup. zmbackup without this will normal give an error as it passing the next zmbackup command stating that there's a current backup in progress.

Example command in some for-loop script is [without the -sync option]:

sudo -u zimbra /opt/zimbra/bin/zmbackup --fullBackup --account "$acct" --server "$host" --target "$dest"

zmbackup --help shows:

--sync  -sync Runs full backup synchronously. 

The admin guide mentions:

"Full backup is usually run asynchronously. When you begin the full backup, the label of the ongoing backup process is immediately displayed. The backup continues in the background. You can use the zmbackupquery command to check the status of the running backup at any time."

I couldn't find any other indication beyond that to explain in more details the purpose of that flag. But from what is stated above, it does look like the -sync flag will resolve the issues of "backup in progress" when scripting out multiple zmbackup commands.

If not, you could query for "Status: in progress" from the zmbackupquery command.

You can give the zmbackupquery command flags for date/time, label, account, -t target, and so forth [Do a zmbackupquery --help to see the options format].

Finding Message Blob In Users Backup

Get the zimbraId of the user, this is what is used in the backup session directory layout:

$ zmprov ga admin@`zmhostname` zimbraId
# name admin@zcs806.DOMAIN.com
zimbraId: e46a828b-cdda-4635-98ab-31b8ac0129b6

Get the mailboxId, this is what is used in the zmvolume primary message volume directory layout:

$ zmprov gmi admin@`zmhostname`
mailboxId: 1
quotaUsed: 77794874

Finding a message in the primary message volume:

$ cd /opt/zimbra/store/0/1/msg/9/

$ ls -la 40511-49376.msg
-rw-r----- 1 zimbra zimbra 1372 Mar 16 02:01 40511-49376.msg

Finding the users directory in a full backup session directory layout:

$ find /opt/zimbra/backup/sessions/full-20140426.080019.153/ -name e46a828b-cdda-4635-98ab-31b8ac0129b6 -print
/opt/zimbra/backup/sessions/full-20140426.080019.153/accounts/e46/a82/e46a828b-cdda-4635-98ab-31b8ac0129b6

Locating the message in the backup session - this example is where the backups are using the zip option:

$ cd /opt/zimbra/backup/sessions/full-20140426.080019.153/accounts/e46/a82/e46a828b-cdda-4635-98ab-31b8ac0129b6/blobs/

$ ls
blobs-1.zip  blobs-2.zip  blobs-3.zip  blobs-4.zip

$ for i in `ls /opt/zimbra/backup/sessions/full-20140426.080019.153/accounts/e46/a82/e46a828b-cdda-4635-98ab-31b8ac0129b6/blobs/`; do echo $i ; unzip -l $i | grep 40511-49376.msg; done

blobs-1.zip
     1372  04-26-2014 01:00   1/9/sha256_bKH8aeWJSh9bhzv5zsXrweA7+jZ88NUllkfV+7m9cJo=_40511-49376.msg1

Backing Up Backups - 3rd Party Tools And Software - Dealing With Directories With Hard Links

Description Of Hard Links

Zimbra uses hard links and special attention needs to be given to this fact. See hard links if your not familiar with hard links and their difference to symbolic links. Not all 3rd party backup software will handle or respect hard links. Many unix commands will need special flags to maintain hard links. When hard links are respected and also "copied" to the new location you could find your data usage become a large multiplication of the original size.

An good thread I found on the topic of preserving hard links for copy/move/backup operations is here:

I'll be summarizing the comments and including additional information I find on the topic below, based upon the command or software being used.

Zimbra and Single Instance Storage - Hard Links

If hard links are possible, we use them. The message must be identical and on to users on the same mailstore and stored on zmvolumes where hard linking is possible. Hard links exist only on the same partition. Postfix has a veriable, default_destination_recipient_limit , which will cause large recipient emails to be delivered in a way where they aren't identical [ see Ajcody-Hardlinks-And-Postfix-default_destination_recipient_limit for more details ].

Mike gives a good description of it here:

Redoing Hard Links RFE

For 8.0.2+ , see this option - zmdedupe :

Easy Way To SEE Hard Links In Use

I sent a test email with 5 accounts listed in the To field, that is shown below with the inode listing of 2133394. the first column of the ls [because of the i option] is listing the inode number of the file. I included the -type f because the . and .. will show directories using the same inode as the 'name' of the directory as listed.

# pwd
   /opt/zimbra/store2

# find . -links +1 -type f -exec ls -lai '{}' \; | sort
2133387 -rw-r----- 2 zimbra zimbra 1600 May  7 05:30 ./0/13/msg/0/281-4320.msg
2133387 -rw-r----- 2 zimbra zimbra 1600 May  7 05:30 ./0/27/msg/0/261-3021.msg
2133394 -rw-r----- 5 zimbra zimbra 1789 May  7 05:53 ./0/13/msg/0/282-4322.msg
2133394 -rw-r----- 5 zimbra zimbra 1789 May  7 05:53 ./0/14/msg/0/412-8476.msg
2133394 -rw-r----- 5 zimbra zimbra 1789 May  7 05:53 ./0/15/msg/0/284-9052.msg
2133394 -rw-r----- 5 zimbra zimbra 1789 May  7 05:53 ./0/25/msg/0/260-4021.msg
2133394 -rw-r----- 5 zimbra zimbra 1789 May  7 05:53 ./0/27/msg/0/268-3033.msg
2133404 -rw-r----- 2 zimbra zimbra 1518 May  7 05:22 ./0/13/msg/0/280-4318.msg
2133404 -rw-r----- 2 zimbra zimbra 1518 May  7 05:22 ./0/27/msg/0/260-3018.msg
Why Does A Message Blob With The Same Message-ID Have Multiple Inodes

Most likely, the message has a large recipient list and because of a variable in postfix, default_destination_recipient_limit , causes multiple deliveries of the message to have slight differences between them - for example, the times in the headers.

Please see the following:

Hard Links Used Within Zimbra Backup Directory - Sessions

Note, this describes an example when zimbraBackupMode equals Standard rather than Auto-Grouped


In regards to the backup sesions, hard links are only used for data that is in the shared_blobs directories from the various full backup sessions you have.

If you goto [below is an example used throughout]:

cd /opt/zimbra/backup/sessions/full-[some full dir]/shared_blobs/[some path]/t9RRjTIdwAZ3k,iJSWo0DxFKCbs=/

Then do a:

ls -li blob.dat
3869621 -rw-r----- 2 zimbra zimbra 5419 Jul 15 01:00 blob.dat

The first number is the inode of the file, which will be the same for all other hard links in use.

cd /opt/zimbra/backup/sessions
find . -inum 3869621 -print
  /opt/zimbra/backup/sessions/full-[some full dir #1]/shared_blobs/[some path]/t9RRjTIdwAZ3k,iJSWo0DxFKCbs=/blob.dat
  /opt/zimbra/backup/sessions/full-[some full dir #2]/shared_blobs/[some path]/t9RRjTIdwAZ3k,iJSWo0DxFKCbs=/blob.dat

Notice also, that the directory path naming scheme is also used for the user backup paths. The directory name that the blob.dat is in, will also be used in the user paths.

For example:

/opt/zimbra/backup/sessions/full-[some full dir]/shared_blobs/[some path]/t9RRjTIdwAZ3k,iJSWo0DxFKCbs=/blob.dat

if I then goto and do:

cd /opt/zimbra/backup/sessions/full-[some full dir]/
find . -name t9RRjTIdwAZ3k* print

I'll see matches in various user backups, with:

full-[some full dir]/accounts/c2e/23b/[zimbra ID of user]/blobs/3/0/t9RRjTIdwAZ3k,iJSWo0DxFKCbs=260-2041.msg4

Note though, those "user" files are not links [hard or soft] and they are of zero length in size.

Reference To Bugs Tied To Backups And Links:

Bacis Unix Commands

cp

cp [copy] Man page:

  • http://linux.die.net/man/1/cp
    • -a, --archive : same as -dpR
      • -d : same as --no-dereference --preserve=link
      • -p : same as --preserve=mode,ownership,timestamps
      • -R, -r, --recursive : copy directories recursively
  • -l, --link : link files instead of copying
  • -L, --dereference : always follow symbolic links
  • -P, --no-dereference : never follow symbolic links
  • -p : same as --preserve=mode,ownership,timestamps
    • --preserve[=ATTR_LIST] : preserve the specified attributes (default: mode,ownership,timestamps) and security contexts, if possible additional attributes: links, all
    • --no-preserve=ATTR_LIST : don't preserve the specified attributes
  • -s, --symbolic-link : make symbolic links instead of copying

No explicit mention of "hard links".

Formats from blog url above [I'll test these later and confirm/deny]:

  • cp -rpv
  • cp -av --preserve=all . /mnt/new
  • Simple `cp -a` using cp (GNU coreutils) 5.97 on my debian does the job quite nicely, I just checked. No need for the --preserve=all option, -a implies --preserve=link. It didn't seem to take too long either, but I would be surprised if it was very much better than rsync. Much easier to remember though.
mv

mv [move] Man page:

tar

tar [tape archive] Man page:

  • http://linux.die.net/man/1/tar
    • --check-links : warn if number of hard links to the file on the filesystem mismatch the number of links recorded in the archive
    • -h, --dereference : don't dump symlinks; dump the files they point to
      • No explicit reference to "hard links".

Please see the following on hard-links and GNU/Tar:

dd

dd [copy and covert - cc was reserved for the C complier] Man page:

No explicit mention of "hard links" in Man page.

rsync and nice

rsync and nice Man page:

A reasonable syntax to use for rsync is [ taken from Ajcody-Notes-Server-Move#The_Actual_Steps ]:

nice +19 rsync -avzH -e ssh --progress /opt/zimbra/ root@NEWHOSTIP:/opt/zimbra

or

nice -20 rsync -avzH -e ssh --delete --progress /opt/zimbra/ root@NEWHOSTIP:/opt/zimbra 
Notice the use of --delete in the last one.
find And cpio

find and cpio Man pages:

Possible syntax use to try [recursive copy of current working directory] :

mkdir /path/to/dest
cd /path/to/DATA
find . -print | cpio -Bpdumv /path/to/dest

Issue to note: "...cpio didn't properly preserve timestamps on directories."

dump And restore

dump and restore Man pages:

ditto - For Mac

ditto Man page:

xfs_copy

xfs_copy Man page:

LVM Tricks
LVM Snapshots

See Back Up (And Restore) LVM Partitions With LVM Snapshots

SAN Snapshot

Possible note of caution I've seen from someone, "When implementing snapshots for ZCS, you should do the snapshot across all ZCS LUNs for a single host at the same time using a consistency group (for netapp, I believe this means cg-start/cg-commit)."

Please see:

Cloud Backups
Amazon S3 , Amazon EC2 , SecoBackup And/Or Tar

I've not used Amazon S3 or SecoBackup. I have no idea about the pricing structure of Amazon S3 and how differing solutions might cause price differences. What I think would be a reasonable approach:

  • Adjust zimbra cron to:
    • run zmbackup as normally scheduled but then include:
      • tar and gzip "new" backup that was made to a "staging" partition.
      • Setup Secobackup [CLI method for cron] to then copy this tar'd/gzip'd file to the Amazon S3 cloud.
      • Remove local tar'd/gzip'd file from staging partition.

I purpose the tar'd/gzip'd step because I doubt there's a way to avoid the hard link issue with SecoBackup/Amazon S3. Why pay multiple times for the same data?

Some information a customer reported to me:

S3 does not work as a normal filesystem and you cannot mount it; hence 
it wouldn't normally work.

However, there are various projects out there which let you use S3 as 
a local POSIX-compliant file system.

Possible options:

s3fs-fuse
jungle disk
subcloud
persistentfs
Amazon EBS

To cut a long story short, PersistentFS came out on tops - it worked 
extremely well - however did not work with Zimbra at all once I set it 
up as the store (/opt/zimbra/store)

The problem is that the filesystem while it is POSIX compliant does not 
have support for hard linking (Which is what Zimbra does with tmp incoming 
messages to the store).  -- [bug 43019 below]

So, overall it's not really possible to do it right now with S3.

They should have hard link support soon.

References:

3rd Party Backup Tools And Software - Generally Not Apart Of The Basis OS

Amanda And Zmanda

Zmanda Home Page:

Amanda Home Page - Advanced Maryland Automatic Network Disk Archiver:

Found this, "Hard links. Maintains the integrity of hard links during backup."

Arkeia

Arkeia Home Page:

Found this, "The Arkeia solution accommodates full and incremental backups, scheduled or on demand, and preserves directory structure, registry, symbolic links and special attributes."

Also, a customer supplied what Arkeia's support sent them to the question. Here is that response:

When Arkeia encounters a hard link (a regular file with more than one reference to it):
  • if it is the first time we see this link, we backup the file and keep the inode number and the path to the file in a memory hash table.
  • if we have already seen the file, we backup the fact that it is a hard link and the target of the link. The files data are not backed up again.

Backup Exec - Symantec / Veritas

Backup Exec Home Page:

Unable to find any reference about the Linux/Unix Agent and the Backup Exec server being able to handle or not symbolic and hard links.

Backup Exec "server" is only available on Windows. One might inquire with Symantec if you can "swap out" your current investment in "Backup Exec" and use their NetBackup product. This supports hard links and the "server" can run on Windows or other *nixes. See Ajcody-Backup-Restore-Issues#NetBackup_-_Veritas.2FSymantec

BackupPC
Bacula

Bacula Home Page:

Found this:

hardlinks=yes|no
When enabled (default), this directive will cause hard links to be backed up. However, the File daemon keeps track of hard linked files and will backup the data only once. The process of keeping track of the hard links can be quite expensive if you have lots of them (tens of thousands or more). This doesn't occur on normal Unix systems, but if you use a program like BackupPC, it can create hundreds of thousands, or even millions of hard links. Backups become very long and the File daemon will consume a lot of CPU power checking hard links. In such a case, set hardlinks=no and hard links will not be backed up. Note, using this option will most likely backup more data and on a restore the file system will not be restored identically to the original.

Source:

BRU - TOLIS Group

BRU Home Page:

Found this in one of their manuals, you should confirm with them based upon the product version you'll be using:

Special Files - BRU will save and restore all types of filesystems and files with their proper ownership, access attributes, creation dates, and modification dates. BRU can be used to move an entire directory hierarchy from one system to another, with all files, including directories, block special files, character special files, fifos, hard links, and symbolic links reproduced with all attributes intact.
Lone-Tar - Lone Star Software Corp.

Lone-Tar Home Page:

Found the following references, though it's not explicit in stating "hard links" :

BACKUP FEATURES - Backs up everything including device files, empty directories, links, symbolic links, Virtual Files and NFS mounted file systems.
RESTORE FEATURES - Fast Seek File restore of files, sym-links, device files, and linked files.

Source:

NetBackup - Veritas/Symantec

NetBackup Home page:

Found the following:

Hard links to directories :
On most UNIX systems, only the root user can create a hard link to a directory. Some systems do not permit hard links and many vendors recommend that these links be avoided. NetBackup does not back up and restore hard-linked directories in the same manner as files:
  • During a backup, if NetBackup encounters hard-linked directories, the directories are backed up once for each hard link.
  • During a restore, NetBackup restores multiple copies of the hard-linked directory contents if the directories do not already exist on the disk. If the directories exist on disk, NetBackup restores the contents multiple times to the same disk location.
Hard links to files :
A hard link differs from a symbolic link in that a hard link is not a pointer to another file. A hard link is two directory entries that point to the same inode number.
If the backup selection list includes hard-linked files, the data is backed up only once during a backup. NetBackup uses the first file name reference that is found in the directory structure. If a subsequent file name reference is found, it is backed up as a link to the name of the first file. Backup up only the link means that only one backup copy of the data is created, regardless of the number of hard links. Any hard link to the data works.
For more information and examples, see “Hard links to files (NTFS volumes or UNIX)” on page 173.

Source:

NetVault - ORBiT

NetVault Home page:

Found this in their "NV Backup Administrators Guide" - pdf only:

[screen shot of GUI check box] The Attempt to Restore Hard Links’ option as revealed in the Restore Options tab on a Linux/UNIX-based version of the File System Plugin.
Attempt to Restore Hard Links (Linux/UNIX-based O/S, ONLY) - During a backup, when the first occurrence of a hard link is found, the complete data will be backed up. For all other occurrences, only the link is backed up. This data can only be restored when the first occurrence exists; trying to restore subsequent occurrences without the presence of the first causes the job to fail. Selecting this option will attempt to locate the full sequence so that all occurrences of the hard link will be restored.
rsnapshot

rsnapshot Home page:

Tivoli - IBM

Tivoli Home Page:

Found the following:

Understanding how hard links are handled
When you back up files that are hard-linked, Tivoli Storage Manager backs up each instance of the linked file. For example, if you back up two files that are hard-linked, Tivoli Storage Manager will back up the file data twice.
When you restore hard-linked files, Tivoli Storage Manager attempts to reestablish the links. For example, if you had a hard-linked pair of files, and only one of the hard-linked files is on your workstation, when you restore both files, they will be hard-linked. The files will also be hard-linked even if neither of the files exists at the time of restore, if both of the files are restored together in a single command. The one exception to this procedure occurs if you back up two files that are hard-linked and then break the connection between them on your workstation. If you restore the two files from the server, Tivoli Storage Manager will respect the current file system and not re-establish the hard link.
Attention: If you do not back up and restore all files that are hard-linked at the same time, problems will occur. To ensure that hard-linked files remain synchronized, back up all hard links at the same time and restore those same files together.


Source:

Other Related Items

freedup

freedup Man page:

freedup Homepage:

I've never used this tool, but from the description it seems it might come in handy for some circumstances.

"Establishes hard or symbolic links between identical files. Search all given file system trees for identical files and link them to the most frequently referenced inode or if equally referenced to the inode of the first file tree. If the devices differ a symbolic link is used instead of a hard link. Symbolic links will not replace files, when at least one of the directory trees is not starting with a '/'."
Tape Devices

Many times, "drivers" aren't needed for tape devices for linux, but many administrators are unaware of this and never give it a test. Instead, they just assume the device doesn't work and the tape vendor isn't supporting it because they "didn't publish" a driver for it.

Quantum

Resource:


Jump to: navigation, search