Difference between revisions of "Zimbra Next Generation Modules/DR/Broken or corrupted Volume (Missing BLOBs)"

 
(3 intermediate revisions by one other user not shown)
Line 1: Line 1:
HEADER
+
#REDIRECT [[Zimbra_NG_Modules/DR/Broken_or_corrupted_Volume_(Missing_BLOBs)]]
 
 
<div class="col-md-12"><br></div>
 
<div class="col-md-12"><br></div>
 
<ol class="breadcrumb">
 
  <li>[[Main Page|Zimbra Wiki]]</li>
 
  <li>[[Zimbra_Next_Generation_Modules]]</li>
 
  <li>[[Zimbra_NG_Backup]]</li>
 
  <li class="active">Disaster Recovery - Broken or corrupted Volume (Missing BLOBs)</li>
 
</ol>
 
__NOTOC__
 
<div class="col-md-12"><br /></div>
 
<div class="col-md-9">
 
    <h2 class="title-header" style="padding-bottom: 9px; border-bottom: 4px solid #0087c3;">Disaster Recovery - Broken or corrupted Volume (Missing BLOBs)</h2>
 
    <div class="col-md-12">
 
        <div class="ibox-content">
 
            <div class="post animated fadeInLeft animation-delay-8" style="padding-top:5px">
 
                <div class="panel panel-default">
 
                    <div class="panel-body">
 
                        <div class="row">
 
= Introduction =
 
This guide describes how to restore any missing BLOBs in a Zimbra volume.
 
 
 
BLOBs are one of the parts composing every Zimbra item, the other one being the item's metadata which is stored in Zimbra's Database. Specifically, a BLOB is a file within a volume containing the content of an item.
 
 
 
The most common causes for BLOB loss are:
 
* Storage corruption after a power outage
 
* Accidental file deletion
 
* Rollback of a volume to a previous state
 
 
 
== Scope of this Guide==
 
This guide applies to all the aforementioned scenarios.
 
 
 
The specific example is based on a target mailbox being checked for errors, but the process will restore any missing blob on all mailboxes on the server. If you wish to run your checks on all mailboxes, omit the <code>mailbox_ids 5</code> portion wherever it appears.
 
 
 
= Missing BLOBs=
 
== Searching for missing BLOBs ==
 
Searching for missing blobs on an account only requires a few easy steps.
 
 
 
* Get the user's mailboxID:
 
<pre>
 
zmprov gmi user001@example.com
 
</pre>
 
 
 
You will see output something like this<br>
 
<pre>
 
mailboxId: 5
 
quotaUsed: 64498843
 
</pre>
 
 
 
* Now we will check if there are missing blobs for the account
 
<pre>
 
zxsuite hsm docheckblobs start mailbox_ids 5
 
</pre>
 
 
 
The output will be something like this:
 
<pre>
 
operationId                              bdbd8112-8c2f-4f8f-bc7b-a82ef5388378
 
server                                    mailstore3.example.com
 
log path                                  /opt/zimbra/log/op_CheckBlobs_bdbd8112-8c2f-4f8f-bc7b-a82ef5388378.log
 
</pre>
 
<font size="3"> Use the<strong> --progress</strong> option if you want verbose output, <strong>''zxsuite --progress hsm docheckblobs start mailbox_ids 5''</strong></font>
 
 
 
* Now search for the word WARN in the CheckBlobs log
 
<pre>
 
grep WARN  /opt/zimbra/log/op_CheckBlobs_bdbd8112-8c2f-4f8f-bc7b-a82ef5388378.log
 
</pre>
 
 
 
If any BLOB is missing, you would see lines like the following in the output
 
<pre>
 
2017-08-22 15:16:53,877 WARN Missing Blob file: Mailbox id: 5 - Item id: 4610 - Revision: 8720 - Blob path: /opt/zimbra/store/0/5/msg/1/4610-8720.msg - Size: 1859 - Digest: dYV2+dXtH9AUAHH4au1jWquWgCeBCz7XpsYtHza0ofw=
 
</pre>
 
 
 
''All outputs and notifications will be identical both when checking a single mailbox or a whole volume''
 
 
 
= Restoring Missing BLOBs =
 
* To restore all BLOBs in a volume, the first information we need is the ID of the message volume itself:
 
<pre>
 
zxsuite hsm getAllVolumes
 
</pre>
 
 
 
Typically, you should see<br>
 
<pre>
 
primaries
 
                id                                                          1
 
                name                                                        message1
 
                path                                                        /opt/zimbra/store
 
                compressed                                                  false
 
                threshold                                                  4096
 
                storeType                                                  LOCAL
 
                isCurrent                                                  true
 
                volumeType                                                  primary
 
secondaries                           
 
        indexes                               
 
 
 
                id                                                          2
 
                name                                                        index1
 
                path                                                        /opt/zimbra/index
 
                compressed                                                  false
 
                threshold                                                  4096
 
                storeType                                                  LOCAL
 
                isCurrent                                                  true
 
                volumeType                                                  index
 
</pre>
 
 
 
In this case, the ID for /opt/zimbra/store is <strong>1</strong><br>
 
 
 
* The restore itself can be done using the <code>zxsuite backup doRestoreBlobs</code> command.
 
** By default, such command only does a dry run to allow the System Administrator to verify the amount of restorable BLOBs.
 
** The <code>--progress</code> option can be used to display the progress of the operation in the standard output.
 
<pre>
 
zxsuite --progress backup dorestoreblobs 1      <<== the --progress is optional
 
</pre>
 
 
 
You will see a lot of verbose output, and at the bottom, you will see results similar to this<br>
 
<pre>
 
Total blobs volume fixed: 0
 
Total blobs checked: 8156
 
Total broken blobs: 1
 
Total restorable blobs: 1
 
Total unrestorable blobs: 0
 
Total failed blob restores: 0
 
Total restored blob: 0
 
</pre>
 
Such results will be also delivered as a Notification in the dedicated section of the Next Generation Administration Zimlet and will be also emailed to all notification recipients as set in the Core section of the very same Zimlet.
 
 
 
<strong>Keep in mind, this is a dry run. Nothing is restored yet. When you are ready to restore, use the ''dryrun false'' option in the command.</strong>
 
 
 
* When you are ready to restore the blob, run the same dorestoreblobs command again, but this time add the <strong>dryrun false</strong> option
 
<strong>This is a high I/O operation, so it's better to run it during off-peak hours.</strong>
 
<pre>
 
zxsuite --progress backup dorestoreblobs 1 dryrun false
 
</pre>
 
 
 
You will see a lot of verbose output, and now you will see the number of restored blobs at the bottom of the output<br>
 
<pre>
 
Total blobs volume fixed: 0
 
Total blobs checked: 8156
 
Total broken blobs: 1
 
Total restorable blobs: 1
 
Total unrestorable blobs: 0
 
Total failed blob restores: 0
 
Total restored blob: 1
 
</pre>
 
 
 
= Volume Cleanup =
 
* Now that all BLOBs have been restored, the <code>doDeduplicate</code> operation of the HSM NG module can be used to consolidate the volume and re-create any missing file links.
 
<pre>
 
zxsuite --progress hsm dodeduplicate message1
 
</pre>
 
 
 
* Check one more time to make sure that no more missing BLOBs are detected:
 
<pre>
 
zxsuite hsm docheckblobs start mailbox_ids 5
 
</pre>
 
<font size="3"> Use the --progress option if you want verbose output, ''zxsuite --progress hsm docheckblobs start mailbox_ids 5''</font>
 
 
 
= Notable Request for Enhancements (RFEs) =
 
N/A
 
 
 
= Notable Bugs/Known Issues =
 
N/A
 
 
 
                        </div>
 
                    </div>
 
                    <div class="col-md-9">
 
                        <div class="panel-footer">
 
                            <p><i class="fa fa-clock-o"></i> Aug 30, 2017 - [https://www.zimbra.com/email-server-software/ Know more »]</p>
 
                        </div>
 
                    </div>
 
                </div>
 
            </div>
 
        </div>
 
    </div>
 
</div>
 
<div class="col-md-3"><br /></div>
 
<div class="col-md-3">
 
    <div class="panel panel-zimbrared-light-border">
 
        <div class="panel-heading">
 
            <h3 class="panel-title"><i class="fa fa-gear pull-left"></i> Zimbra NG Modules</h3>
 
        </div>
 
        <div class="panel-body">
 
            {{ZNG}}
 
        </div>
 
    </div>
 
</div>
 
<div class="col-md-3">
 
    <div class="panel panel-primary-light-border">
 
        <div class="panel-heading">
 
            <h3 class="panel-title"><i class="fa fa-info-circle pull-left"></i> Zimbra NG Modules Resources</h3>
 
        </div>
 
        <div class="panel-body">
 
            {{ZNGL}}
 
        </div>
 
    </div>
 
</div>
 
<div class="clearfix"></div>
 
<div class="col-md-12"><br></div>
 
{{FH}}
 

Latest revision as of 12:16, 29 November 2017

Jump to: navigation, search