Difference between revisions of "Zimbra Next Generation Modules/DR/Broken or corrupted Volume (Missing BLOBs)"

(Now we will check if there are missing blobs for the account)
Line 1: Line 1:
 
{{ZCS 8.8}}
 
{{ZCS 8.8}}
 
{{WIP}}
 
{{WIP}}
 +
= Introduction =
 +
This guide describes how to restore any missing BLOBs in a Zimbra volume.
  
<font size="6">To check (and repair) for missing blobs on an account, you can do the following</font>
+
BLOBs are one of the parts composing every Zimbra item, the other one being the item's metadata which is stored in Zimbra's Database. Specifically, a BLOB is a file within a volume containing the content of an item.
  
==<font size="5">Get the user's mailboxID</font>==
+
The most common causes for BLOB loss are:
 +
* Storage corruption after a power outage
 +
* Accidental file deletion
 +
* Rollback of a volume to a previous state
  
<code><pre>
+
== Scope of this Guide==
 +
This guide applies to all the aforementioned scenarios.
 +
 
 +
The specific example is based on a target mailbox being checked for errors, but the process will restore any missing blob on all mailboxes on the server. If you wish to run your checks on all mailboxes, omit the <code>mailbox_ids 5</code> portion wherever it appears.
 +
 
 +
= Missing BLOBs=
 +
== Searching for missing BLOBs ==
 +
Searching for missing blobs on an account only requires a few easy steps.
 +
 
 +
* Get the user's mailboxID:
 +
<pre>
 
zmprov gmi user001@example.com
 
zmprov gmi user001@example.com
</pre></code>
+
</pre>
  
 
You will see output something like this<br>
 
You will see output something like this<br>
<code><pre>
+
<pre>
 
mailboxId: 5
 
mailboxId: 5
 
quotaUsed: 64498843
 
quotaUsed: 64498843
</pre></code>
+
</pre>
  
==<font size="5">Now we will check if there are missing blobs for the account</font>==
+
* Now we will check if there are missing blobs for the account
 
+
<pre>
<code><pre>
 
 
zxsuite hsm docheckblobs start mailbox_ids 5
 
zxsuite hsm docheckblobs start mailbox_ids 5
</pre></code>
+
</pre>
  
You will see output something like this<br>
+
The output will be something like this:
<code><pre>
+
<pre>
 
operationId                              bdbd8112-8c2f-4f8f-bc7b-a82ef5388378
 
operationId                              bdbd8112-8c2f-4f8f-bc7b-a82ef5388378
 
server                                    mailstore3.example.com
 
server                                    mailstore3.example.com
 
log path                                  /opt/zimbra/log/op_CheckBlobs_bdbd8112-8c2f-4f8f-bc7b-a82ef5388378.log
 
log path                                  /opt/zimbra/log/op_CheckBlobs_bdbd8112-8c2f-4f8f-bc7b-a82ef5388378.log
</pre></code>
+
</pre>
 
<font size="3"> Use the<strong> --progress</strong> option if you want verbose output, <strong>''zxsuite --progress hsm docheckblobs start mailbox_ids 5''</strong></font>
 
<font size="3"> Use the<strong> --progress</strong> option if you want verbose output, <strong>''zxsuite --progress hsm docheckblobs start mailbox_ids 5''</strong></font>
  
==<font size="5">Now search for the word WARN in the CheckBlobs log</font>==
+
* Now search for the word WARN in the CheckBlobs log
<code><pre>
+
<pre>
 
grep WARN  /opt/zimbra/log/op_CheckBlobs_bdbd8112-8c2f-4f8f-bc7b-a82ef5388378.log
 
grep WARN  /opt/zimbra/log/op_CheckBlobs_bdbd8112-8c2f-4f8f-bc7b-a82ef5388378.log
</pre></code>
+
</pre>
  
if there was a missing BLOB, you would see similar output like this<br>
+
If any BLOB is missing, you would see lines like the following in the output
<code><pre>
+
<pre>
 
2017-08-22 15:16:53,877 WARN Missing Blob file: Mailbox id: 5 - Item id: 4610 - Revision: 8720 - Blob path: /opt/zimbra/store/0/5/msg/1/4610-8720.msg - Size: 1859 - Digest: dYV2+dXtH9AUAHH4au1jWquWgCeBCz7XpsYtHza0ofw=
 
2017-08-22 15:16:53,877 WARN Missing Blob file: Mailbox id: 5 - Item id: 4610 - Revision: 8720 - Blob path: /opt/zimbra/store/0/5/msg/1/4610-8720.msg - Size: 1859 - Digest: dYV2+dXtH9AUAHH4au1jWquWgCeBCz7XpsYtHza0ofw=
</pre></code>
+
</pre>
 +
 
 +
''All outputs and notifications will be identical both when checking a single mailbox or a whole volume''
  
==<font size="5">Now, lets restore the missing blob, but first we need the ID of the message volume</font>==
+
= Restoring Missing BLOBs =
<code><pre>
+
* To restore all BLOBs in a volume, the first information we need is the ID of the message volume itself:
 +
<pre>
 
zxsuite hsm getAllVolumes
 
zxsuite hsm getAllVolumes
</pre></code>
+
</pre>
  
 
Typically, you should see<br>
 
Typically, you should see<br>
<code><pre>
+
<pre>
 
primaries
 
primaries
 
                 id                                                          1
 
                 id                                                          1
Line 67: Line 84:
 
                 isCurrent                                                  true
 
                 isCurrent                                                  true
 
                 volumeType                                                  index
 
                 volumeType                                                  index
</pre></code>
+
</pre>
  
The ID for /opt/zimbra/store is <strong>1</strong><br>
+
In this case, the ID for /opt/zimbra/store is <strong>1</strong><br>
  
==<font size="5">Now lets see what will be restored</font>==
+
* The restore itself can be done using the <code>zxsuite backup doRestoreBlobs</code> command.
<code><pre>
+
** By default, such command only does a dry run to allow the System Administrator to verify the amount of restorable BLOBs.
 +
** The <code>--progress</code> option can be used to display the progress of the operation in the standard output.
 +
<pre>
 
zxsuite --progress backup dorestoreblobs 1      <<== the --progress is optional
 
zxsuite --progress backup dorestoreblobs 1      <<== the --progress is optional
</pre></code>
+
</pre>
  
 
You will see a lot of verbose output, and at the bottom, you will see results similar to this<br>
 
You will see a lot of verbose output, and at the bottom, you will see results similar to this<br>
<code><pre>
+
<pre>
 
Total blobs volume fixed: 0
 
Total blobs volume fixed: 0
 
Total blobs checked: 8156
 
Total blobs checked: 8156
Line 85: Line 104:
 
Total failed blob restores: 0
 
Total failed blob restores: 0
 
Total restored blob: 0
 
Total restored blob: 0
</pre></code>
+
</pre>
 +
Such results will be also delivered as a Notification in the dedicated section of the Next Generation Administration Zimlet and will be also emailed to all notification recipients as set in the Core section of the very same Zimlet.
 +
 
 
<strong>Keep in mind, this is a dry run. Nothing is restored yet. When you are ready to restore, use the ''dryrun false'' option in the command.</strong>
 
<strong>Keep in mind, this is a dry run. Nothing is restored yet. When you are ready to restore, use the ''dryrun false'' option in the command.</strong>
  
===<font size="5">When you are ready to restore the blob</font>===
+
* When you are ready to restore the blob, run the same dorestoreblobs command again, but this time add the <strong>dryrun false</strong> option
 
+
<strong>This is a high I/O operation, so it's better to run it during off-peak hours.</strong>
Run the same dorestoreblobs command again, but this time add the <strong>dryrun false</strong> option
+
<pre>
 
 
<code><pre>
 
 
zxsuite --progress backup dorestoreblobs 1 dryrun false
 
zxsuite --progress backup dorestoreblobs 1 dryrun false
</pre></code>
+
</pre>
  
 
You will see a lot of verbose output, and now you will see the number of restored blobs at the bottom of the output<br>  
 
You will see a lot of verbose output, and now you will see the number of restored blobs at the bottom of the output<br>  
 
+
<pre>
<code><pre>
 
 
Total blobs volume fixed: 0
 
Total blobs volume fixed: 0
 
Total blobs checked: 8156
 
Total blobs checked: 8156
Line 106: Line 124:
 
Total failed blob restores: 0
 
Total failed blob restores: 0
 
Total restored blob: 1
 
Total restored blob: 1
</pre></code>
+
</pre>
  
==<font size="5">Lastly, run the dodeduplicate command</font>==
+
= Volume Cleanup =
<code><pre>
+
* Now that all BLOBs have been restored, the <code>doDeduplicate</code> operation of the HSM NG module can be used to consolidate the volume and re-create any missing file links.
 +
<pre>
 
  zxsuite --progress hsm dodeduplicate message1
 
  zxsuite --progress hsm dodeduplicate message1
</pre></code>
+
</pre>
  
===<font size="5">Check one more time</font>===
+
* Check one more time to make sure that no more missing BLOBs are detected:
<code><pre>
+
<pre>
 
zxsuite hsm docheckblobs start mailbox_ids 5
 
zxsuite hsm docheckblobs start mailbox_ids 5
</pre></code>
+
</pre>
 
<font size="3"> Use the --progress option if you want verbose output, ''zxsuite --progress hsm docheckblobs start mailbox_ids 5''</font>
 
<font size="3"> Use the --progress option if you want verbose output, ''zxsuite --progress hsm docheckblobs start mailbox_ids 5''</font>
  
<font size="6">Notable Request for Enhancements (RFEs)</font>
+
= Notable Request for Enhancements (RFEs) =
 +
N/A
  
<font size="6">Notable Bugs</font>
+
= Notable Bugs/Known Issues =
 +
N/A

Revision as of 11:05, 31 August 2017

ZCS 8.8 Article ZCS 8.8

Introduction

This guide describes how to restore any missing BLOBs in a Zimbra volume.

BLOBs are one of the parts composing every Zimbra item, the other one being the item's metadata which is stored in Zimbra's Database. Specifically, a BLOB is a file within a volume containing the content of an item.

The most common causes for BLOB loss are:

  • Storage corruption after a power outage
  • Accidental file deletion
  • Rollback of a volume to a previous state

Scope of this Guide

This guide applies to all the aforementioned scenarios.

The specific example is based on a target mailbox being checked for errors, but the process will restore any missing blob on all mailboxes on the server. If you wish to run your checks on all mailboxes, omit the mailbox_ids 5 portion wherever it appears.

Missing BLOBs

Searching for missing BLOBs

Searching for missing blobs on an account only requires a few easy steps.

  • Get the user's mailboxID:
zmprov gmi user001@example.com

You will see output something like this

mailboxId: 5
quotaUsed: 64498843
  • Now we will check if there are missing blobs for the account
zxsuite hsm docheckblobs start mailbox_ids 5

The output will be something like this:

operationId                               bdbd8112-8c2f-4f8f-bc7b-a82ef5388378
server                                    mailstore3.example.com
log path                                  /opt/zimbra/log/op_CheckBlobs_bdbd8112-8c2f-4f8f-bc7b-a82ef5388378.log

Use the --progress option if you want verbose output, zxsuite --progress hsm docheckblobs start mailbox_ids 5

  • Now search for the word WARN in the CheckBlobs log
grep WARN  /opt/zimbra/log/op_CheckBlobs_bdbd8112-8c2f-4f8f-bc7b-a82ef5388378.log

If any BLOB is missing, you would see lines like the following in the output

2017-08-22 15:16:53,877 WARN Missing Blob file: Mailbox id: 5 - Item id: 4610 - Revision: 8720 - Blob path: /opt/zimbra/store/0/5/msg/1/4610-8720.msg - Size: 1859 - Digest: dYV2+dXtH9AUAHH4au1jWquWgCeBCz7XpsYtHza0ofw=

All outputs and notifications will be identical both when checking a single mailbox or a whole volume

Restoring Missing BLOBs

  • To restore all BLOBs in a volume, the first information we need is the ID of the message volume itself:
zxsuite hsm getAllVolumes

Typically, you should see

primaries
                id                                                          1
                name                                                        message1
                path                                                        /opt/zimbra/store
                compressed                                                  false
                threshold                                                   4096
                storeType                                                   LOCAL
                isCurrent                                                   true
                volumeType                                                  primary
secondaries                             
        indexes                                 

                id                                                          2
                name                                                        index1
                path                                                        /opt/zimbra/index
                compressed                                                  false
                threshold                                                   4096
                storeType                                                   LOCAL
                isCurrent                                                   true
                volumeType                                                  index

In this case, the ID for /opt/zimbra/store is 1

  • The restore itself can be done using the zxsuite backup doRestoreBlobs command.
    • By default, such command only does a dry run to allow the System Administrator to verify the amount of restorable BLOBs.
    • The --progress option can be used to display the progress of the operation in the standard output.
zxsuite --progress backup dorestoreblobs 1       <<== the --progress is optional

You will see a lot of verbose output, and at the bottom, you will see results similar to this

Total blobs volume fixed: 0
Total blobs checked: 8156
Total broken blobs: 1
Total restorable blobs: 1
Total unrestorable blobs: 0
Total failed blob restores: 0
Total restored blob: 0

Such results will be also delivered as a Notification in the dedicated section of the Next Generation Administration Zimlet and will be also emailed to all notification recipients as set in the Core section of the very same Zimlet.

Keep in mind, this is a dry run. Nothing is restored yet. When you are ready to restore, use the dryrun false option in the command.

  • When you are ready to restore the blob, run the same dorestoreblobs command again, but this time add the dryrun false option

This is a high I/O operation, so it's better to run it during off-peak hours.

zxsuite --progress backup dorestoreblobs 1 dryrun false

You will see a lot of verbose output, and now you will see the number of restored blobs at the bottom of the output

Total blobs volume fixed: 0
Total blobs checked: 8156
Total broken blobs: 1
Total restorable blobs: 1
Total unrestorable blobs: 0
Total failed blob restores: 0
Total restored blob: 1

Volume Cleanup

  • Now that all BLOBs have been restored, the doDeduplicate operation of the HSM NG module can be used to consolidate the volume and re-create any missing file links.
 zxsuite --progress hsm dodeduplicate message1
  • Check one more time to make sure that no more missing BLOBs are detected:
zxsuite hsm docheckblobs start mailbox_ids 5

Use the --progress option if you want verbose output, zxsuite --progress hsm docheckblobs start mailbox_ids 5

Notable Request for Enhancements (RFEs)

N/A

Notable Bugs/Known Issues

N/A

Jump to: navigation, search