Zimbra Next Generation Modules/DR/Broken or corrupted Volume (Missing BLOBs)

ZCS 8.8

 - This article is a Work in Progress, and may be unfinished or missing sections.

Introduction

This guide describes how to restore any missing BLOBs in a Zimbra volume.

BLOBs are one of the parts composing every Zimbra item, the other one being the item's metadata which is stored in Zimbra's Database. Specifically, a BLOB is a file within a volume containing the content of an item.

The most common causes for BLOB loss are:

• Storage corruption after a power outage
• Accidental file deletion
• Rollback of a volume to a previous state

Scope of this Guide

This guide applies to all the aforementioned scenarios.

The specific example is based on a target mailbox being checked for errors, but the process will restore any missing blob on all mailboxes on the server. If you wish to run your checks on all mailboxes, omit the mailbox_ids 5 portion wherever it appears.

Missing BLOBs

Searching for missing BLOBs

Searching for missing blobs on an account only requires a few easy steps.

• Get the user's mailboxID:

zmprov gmi user001@example.com

You will see output something like this

mailboxId: 5
quotaUsed: 64498843

• Now we will check if there are missing blobs for the account

zxsuite hsm docheckblobs start mailbox_ids 5

The output will be something like this:

operationId                               bdbd8112-8c2f-4f8f-bc7b-a82ef5388378
server                                    mailstore3.example.com
log path                                  /opt/zimbra/log/op_CheckBlobs_bdbd8112-8c2f-4f8f-bc7b-a82ef5388378.log

Use the --progress option if you want verbose output, zxsuite --progress hsm docheckblobs start mailbox_ids 5

• Now search for the word WARN in the CheckBlobs log

grep WARN  /opt/zimbra/log/op_CheckBlobs_bdbd8112-8c2f-4f8f-bc7b-a82ef5388378.log

If any BLOB is missing, you would see lines like the following in the output

2017-08-22 15:16:53,877 WARN Missing Blob file: Mailbox id: 5 - Item id: 4610 - Revision: 8720 - Blob path: /opt/zimbra/store/0/5/msg/1/4610-8720.msg - Size: 1859 - Digest: dYV2+dXtH9AUAHH4au1jWquWgCeBCz7XpsYtHza0ofw=

All outputs and notifications will be identical both when checking a single mailbox or a whole volume

Restoring Missing BLOBs

• To restore all BLOBs in a volume, the first information we need is the ID of the message volume itself:

zxsuite hsm getAllVolumes

Typically, you should see

primaries
                id                                                          1
                name                                                        message1
                path                                                        /opt/zimbra/store
                compressed                                                  false
                threshold                                                   4096
                storeType                                                   LOCAL
                isCurrent                                                   true
                volumeType                                                  primary
secondaries                             
        indexes                                 

                id                                                          2
                name                                                        index1
                path                                                        /opt/zimbra/index
                compressed                                                  false
                threshold                                                   4096
                storeType                                                   LOCAL
                isCurrent                                                   true
                volumeType                                                  index

In this case, the ID for /opt/zimbra/store is 1

• The restore itself can be done using the zxsuite backup doRestoreBlobs command.
  • By default, such command only does a dry run to allow the System Administrator to verify the amount of restorable BLOBs.
  • The --progress option can be used to display the progress of the operation in the standard output.

zxsuite --progress backup dorestoreblobs 1       <<== the --progress is optional

You will see a lot of verbose output, and at the bottom, you will see results similar to this

Total blobs volume fixed: 0
Total blobs checked: 8156
Total broken blobs: 1
Total restorable blobs: 1
Total unrestorable blobs: 0
Total failed blob restores: 0
Total restored blob: 0

Such results will be also delivered as a Notification in the dedicated section of the Next Generation Administration Zimlet and will be also emailed to all notification recipients as set in the Core section of the very same Zimlet.

Keep in mind, this is a dry run. Nothing is restored yet. When you are ready to restore, use the dryrun false option in the command.

• When you are ready to restore the blob, run the same dorestoreblobs command again, but this time add the dryrun false option

This is a high I/O operation, so it's better to run it during off-peak hours.

zxsuite --progress backup dorestoreblobs 1 dryrun false

You will see a lot of verbose output, and now you will see the number of restored blobs at the bottom of the output

Total blobs volume fixed: 0
Total blobs checked: 8156
Total broken blobs: 1
Total restorable blobs: 1
Total unrestorable blobs: 0
Total failed blob restores: 0
Total restored blob: 1

Volume Cleanup

• Now that all BLOBs have been restored, the doDeduplicate operation of the HSM NG module can be used to consolidate the volume and re-create any missing file links.

 zxsuite --progress hsm dodeduplicate message1

• Check one more time to make sure that no more missing BLOBs are detected:

zxsuite hsm docheckblobs start mailbox_ids 5

Use the --progress option if you want verbose output, zxsuite --progress hsm docheckblobs start mailbox_ids 5

Notable Request for Enhancements (RFEs)

N/A

Notable Bugs/Known Issues

N/A