Zimbra Next Generation Modules/DR/Broken or corrupted Volume (Missing BLOBs)

Missing BLOBs in ZCS 8.8

ZCS 8.8

 - This article is a Work in Progress, and may be unfinished or missing sections.

To check for missing blobs on an account, you can do the following

Get the user's mailboxID

zmprov gmi user001@example.com

You will see output something like this

mailboxId: 5
quotaUsed: 64498843

Now we will check if there are missing blobs for the account

zxsuite hsm docheckblobs start mailbox_ids 5

You will see output something like this

operationId                               bdbd8112-8c2f-4f8f-bc7b-a82ef5388378
server                                    mailstore3.example.com
log path                                 /opt/zimbra/log/op_CheckBlobs_bdbd8112-8c2f-4f8f-bc7b-a82ef5388378.log

Now search for the word WARN in the CheckBlobs log

grep WARN  /opt/zimbra/log/op_CheckBlobs_bdbd8112-8c2f-4f8f-bc7b-a82ef5388378.log

if there was a missing BLOB, you would see similar output like this

2017-08-22 15:16:53,877 WARN Missing Blob file: Mailbox id: 5 - Item id: 4610 - Revision: 8720 - Blob path: /opt/zimbra/store/0/5/msg/1/4610-8720.msg - Size: 1859 - Digest: dYV2+dXtH9AUAHH4au1jWquWgCeBCz7XpsYtHza0ofw=

Now, lets restore the missing blob, but first we need the ID of the message volume

zxsuite hsm getAllVolumes

Typically, you should see

primaries
                id                                                          1
                name                                                        message1
                path                                                        /opt/zimbra/store
                compressed                                                  false
                threshold                                                   4096
                storeType                                                   LOCAL
                isCurrent                                                   true
                volumeType                                                  primary
secondaries                             
        indexes                                 

                id                                                          2
                name                                                        index1
                path                                                        /opt/zimbra/index
                compressed                                                  false
                threshold                                                   4096
                storeType                                                   LOCAL
                isCurrent                                                   true
                volumeType                                                  index

The output for /opt/zimbra/store is 1

Now lets see what will restored. Keep in mind this is a dry run, nothing is restored yet!

zxsuite --progress backup dorestoreblobs 1

You will see a lot of verbose output, and at the bottom, you will see results similar to this

Total blobs volume fixed: 0
Total blobs checked: 8156
Total broken blobs: 1
Total restorable blobs: 1
Total unrestorable blobs: 0
Total failed blob restores: 0
Total restored blob: 0

When you are ready to restore the blob, run the same dorestoreblobs command again, but this time add the dryrun false option

zxsuite --progress backup dorestoreblobs 1 dryrun false

You will see a lot of verbose output, and now you will see the number of restored blobs

Total blobs volume fixed: 0
Total blobs checked: 8156
Total broken blobs: 1
Total restorable blobs: 1
Total unrestorable blobs: 0
Total failed blob restores: 0
Total restored blob: 1