Zimbra Next Generation Modules/Zimbra NG Backup/Unrestorable items

Zimbra NG Backup - Unrestorable items

How can I check if all of my items have been restored?

It's very easy: just check the appropriate "Operation Completed" notification you received as soon as the restore operation has been completed. It can be viewed in the "Notifications" section of the Zimbra Next Generation Modules Administration Zimlet and it's also emailed to the address you specified in the "Core" section of the Zimbra Next Generation Modules Administration Zimlet as the "Notification E-Mail recipient address".

The "skipped items" section contains a per-account list of unrestored items:

  [...]
  - stats -
  Restored Items: 15233
  Skipped Items:  125
  Unrestored Items: 10

  - unrestored items -
  account: account1@domain.com
  unrestored items: 1255,1369

  account: account2@domain.com
  unrestored items: 49965

  account: account14@domain.com
  unrestored items: 856,13339,45200, 45655
  [...]

Skipped Items vs. Unrestored Items

"Skipped" item: an item that has been already restored, either during the current restore or in a previous one.
"Unrestored" item: an item that has not been restored due to an issue in the restore process.

Why some of my items have not been restored?

There are different possible causes, the most common of which are:

Read Error: Either the raw item or the metadata file is not readable due to an I/O exception or a permission issue.
Broken item: Both the the raw item or the metadata file are readable by Zimbra NG Backup but their content is broken/corrupted.
Invalid item: Both the the raw item or the metadata file are readable and the content is correct, but Zimbra refuses to inject the item.

How can I identify unrestored items?

There are two ways to do so: via the Zimbra Next Generation Modules CLI and via the Zimbra Webclient. The first way can be used to search for the item within the backup/import path while the second way can be used to view the items in the source server.

Identifying Unrestorable items through the Zimbra Next Generation Modules CLI

The "getItem" command of the Zimbra Next Generation Modules CLI can display an item and the related metadata extracting all informations from a backup path/external backup.

The syntax of the command is the following:

   zxsuite backup getItem {account} {item} [attr1 value1 [attr2 value2...]]

PARAMETER LIST

NAME              TYPE               EXPECTED VALUES            DEFAULT
account(M)        Account Name/ID                               
item(M)           Integer                                       
backup_path(O)    Path                                          /opt/zimbra/backup/zextras/
dump_blob(O)      Boolean            true|false                 false
date(O)           Date               dd/mm/yyyy hh:mm:ss|all    last

(M) == mandatory parameter, (O) == optional parameter

Thus, to extract the raw data and metadata informations of the item whose itemID is 49965 belonging to account2@domain.com also including the full dump of the item's BLOB the command would be:

 zxsuite backup getItem account2@domain.com 49965 dump_blob true

Identifying Unrestorable items through the Zimbra WebClient

The comma separated list of unrestored items displayed in the "Operation Complete" notification can be used as a search argument in the Zimbra Webclient to perform an item search.

To do so:

Log into the Zimbra Administration Console in the source server.
Use the "View Mail" feature to access the account in which the unrestored items are.
In the search box, enter item: followed by the comma separated list of itemIDs.

 e.g.
 item: 856,13339,45200,45655

Template:WarningBoxxy

How can I restore unrestored items?

While an item not being restored is a clear sign of an issue, either with the item itself or with your current Zimbra + Zimbra Next Generation Modules setup, in some cases there are good chances of being able to restore an item even if it was not restored on the first import try.

In the following paragraphs you will find a collections of tips and tricks that can be helpful when dealing with different kinds of unrestorable items.

Items not restored because of a Read Error

A dutiful distinction must be done about the read errors that can cause items not to be restored:

hard errors: hardware failures and all other "destructive" errors that cause an unrecoverable data loss.
soft errors: "non-destructive" errors such as wrong permissions, filesystem errors, RAID issues (e.g.: broken RAID1 mirroring) etcetera.

While there is nothing much to do about hard errors, you can prevent or mitigate soft errors following these guidelines:

Run a filesystem check.
If using a RAID disk setup, check the array for possible issues (depending on RAID level).
Make sure that the 'zimbra' user has r/w access to the backup/import path, all its subfolders and all thereby contained files.
Carefully check the link quality of network-shared filesystems. If link quality is poor consider transferring the data with rsync.
If using SSHfs to remotely mount the backup/import path make sure to run the mount command as root using the "-o allow_other" option.

Items not restored because identified as Broken Items

Unfortunately, this is the worst category of unrestored items in terms of "salvageability".

Based on the degree of corruption of the item, it might be possible to recover either a previous state or the raw object (this is only valid for emails). To identify the degree of corruption, the "getItem" Zimbra Next Generation Modules CLI command comes handy:

   zxsuite backup getItem {account} {item} [attr1 value1 [attr2 value2...]]

PARAMETER LIST

NAME              TYPE               EXPECTED VALUES            DEFAULT
account(M)        Account Name/ID                               
item(M)           Integer                                       
backup_path(O)    Path                                          /opt/zimbra/backup/zextras/
dump_blob(O)      Boolean            true|false                 false
date(O)           Date               dd/mm/yyyy hh:mm:ss|all    last

(M) == mandatory parameter, (O) == optional parameter

Searching for the broken item setting the "backup_path" parameter to the import path and the "date" parameter to "all" will display all valid states for the item.

 zimbra@test:~$ zxsuite backup getItem admin@example.com 24700 backup_path /mnt/import/ date all
       itemStates                              
               start_date                                                  12/07/2013 16:35:44
               type                                                        message
               deleted                                                     true
               blob path /mnt/import/items/c0/c0,gUlvzQfE21z6YRXJnNkKL85PrRHw0KMQUqo,pMmQ=
               start_date                                                  12/07/2013 17:04:33
               type                                                        message
               deleted                                                     true
               blob path /mnt/import/items/c0/c0,gUlvzQfE21z6YRXJnNkKL85PrRHw0KMQUqo,pMmQ=
               start_date                                                  15/07/2013 10:03:26
               type                                                        message
               deleted                                                     true
               blob path /mnt/import/items/c0/c0,gUlvzQfE21z6YRXJnNkKL85PrRHw0KMQUqo,pMmQ=

If the item is an email you will be able to recover a standard .eml file through the following steps:

Identify the latest valid state

 /mnt/import/items/c0/c0,gUlvzQfE21z6YRXJnNkKL85PrRHw0KMQUqo,pMmQ=
               start_date                                                  15/07/2013 10:03:26
               type                                                        message
               deleted                                                     true
               blob path /mnt/import/items/c0/c0,gUlvzQfE21z6YRXJnNkKL85PrRHw0KMQUqo,pMmQ=

Identify the "blob path"

               blob path /mnt/import/items/c0/c0,gUlvzQfE21z6YRXJnNkKL85PrRHw0KMQUqo,pMmQ=

Use gzip to uncompress the BLOB file into an .eml file

 zimbra@test:~$ gunzip -c /mnt/import/items/c0/c0,gUlvzQfE21z6YRXJnNkKL85PrRHw0KMQUqo,pMmQ= > /tmp/restored.eml

 zimbra@test:~$ cat /tmp/restored.eml 
 
 Return-Path: zimbra@test.example.com
 
 Received: from test.example.com (LHLO test.example.com) (192.168.1.123)
  by test.example.com with LMTP; Fri, 12 Jul 2013 16:35:43 +0200 (CEST)
 
 Received: by test.example.com (Postfix, from userid 1001)
 	id 4F34A120CC4; Fri, 12 Jul 2013 16:35:43 +0200 (CEST)
 
 To: admin@example.com
 From: admin@example.com
 Subject: Service mailboxd started on test.example.com
 Message-Id: <20130712143543.4F34A120CC4@test.example.com>
 Date: Fri, 12 Jul 2013 16:35:43 +0200 (CEST)
 
 Jul 12 16:35:42 test zmconfigd[14198]: Service status change: test.example.com mailboxd changed from stopped to running

Done! You can now import the .eml file into the appropriate mailbox using your favorite client.

Template:WarningBoxxy

Items not restored because identified as Invalid Items

An item is identified as "Invalid" when, albeit being formally correct is discarded by Zimbra's LMTP Validator upon injection. This is common when importing items created on an older version of Zimbra to a newer one, as the validation rules are updated very often and because of this not all messages considered valid by a certain Zimbra version are still considered valid by a newer version.

If you experienced a lot of unrestored items during an import it might be a good idea to momentarily disable the LMTP validator and repeat the import:

To disable Zimbra's LMTP Validator run the following command as the Zimbra user:

 zmlocalconfig -e zimbra_lmtp_validate_messages=false

Once the import is completed you can enable the LMTP validator running

 zmlocalconfig -e zimbra_lmtp_validate_messages=true

warning

This is a "dirty" workaround, as items deemed invalid by the LMTP validator might cause display or mobile synchronization errors. Use at your own risk.

Aug 25, 2016 - Know more »