Desktop Sync Failures caused by message size mismatch

From Zimbra :: Wiki

Jump to: navigation, search
Admin Article

Article Information

This article applies to the following ZCS versions.
  ZCS 5.0 Article  ZCS 5.0

Contents

Problem Description

One is unable to desktop sync and a message similar to the following appears in the log. The following problem is caused by an inconsistency in server metadata specifying the size of BLOB-data and the on-disk file-sizes.

Please see bugs 31519[1], 31764[2] and 31937[3]

Currently, the workaround is to update to the latest TarFormatter (Zimbra Desktop) as worked around in 31937. In the future, we will have a fix script to migrate from incorrect to correct data.

2008-09-10 13:01:14,901 ERROR [...user@domain] [mid=20;] offline - sync failure: user@domain
com.zimbra.common.service.ServiceException: system failure: InitialSync.syncMessagesAsTgz
ExceptionId:...user@domain:1221076874901:a028898b78e232fd
Code:service.FAILURE
       at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:253)
       at com.zimbra.cs.mailbox.SyncExceptionHandler.checkRecoverableException(SyncExceptionHandler.java:38)
       at com.zimbra.cs.mailbox.InitialSync.syncMessagesAsTgz(InitialSync.java:882)
       at com.zimbra.cs.mailbox.InitialSync.syncMessages(InitialSync.java:840)
       at com.zimbra.cs.mailbox.InitialSync.syncMessagelikeItems(InitialSync.java:423)
       at com.zimbra.cs.mailbox.InitialSync.initialFolderSync(InitialSync.java:313)
       at com.zimbra.cs.mailbox.InitialSync.initialFolderSync(InitialSync.java:377)
       at com.zimbra.cs.mailbox.InitialSync.initialFolderSync(InitialSync.java:377)
       at com.zimbra.cs.mailbox.InitialSync.resume(InitialSync.java:188)
       at com.zimbra.cs.mailbox.InitialSync.resume(InitialSync.java:174)
       at com.zimbra.cs.mailbox.MailboxSync.sync(MailboxSync.java:192)
       at com.zimbra.cs.mailbox.OfflineMailbox.sync(OfflineMailbox.java:115)
       at com.zimbra.cs.mailbox.OfflineMailbox.syncOnTimer(OfflineMailbox.java:96)
       at com.zimbra.cs.mailbox.DesktopMailbox$1.run(DesktopMailbox.java:117)
       at java.util.TimerThread.mainLoop(Timer.java:512)
       at java.util.TimerThread.run(Timer.java:462)
Caused by: com.zimbra.common.service.ServiceException: system failure: Unable to read/write message id=443759
ExceptionId:...user@domain:1221076874901:a028898b78e232fd
Code:service.FAILURE
       at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:253)
       at com.zimbra.cs.mailbox.InitialSync.saveMessage(InitialSync.java:1014)
       at com.zimbra.cs.mailbox.InitialSync.syncMessagesAsTgz(InitialSync.java:879)
       ... 13 more
Caused by: java.io.EOFException: Unexpected end of ZLIB input stream
       at java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:216)
       at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:134)
       at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:87)
       at com.zimbra.common.util.tar.TarBuffer.readBlock(TarBuffer.java:257)
       at com.zimbra.common.util.tar.TarBuffer.readRecord(TarBuffer.java:223)
       at com.zimbra.common.util.tar.TarInputStream.read(TarInputStream.java:349)
       at com.zimbra.common.util.ByteUtil.readInput(ByteUtil.java:642)
       at com.zimbra.cs.mailbox.InitialSync.saveMessage(InitialSync.java:1003)
       ... 14 more

Investigation

Run the zmblobchk tool.

WARNING: This tool will produce large I/O loads as it will stat every single file in the Zimbra blob-store and may decompress any compressed blobs.

usage: zmblobchk
 -f,--file <arg>       save/load report to/from file
 -h,--help             this help message
 -k,--skip-fs          skip blob store reverse check (NEW IN 5.0.10)
 -l,--load <arg>       display report from file
 -p,--password <arg>   zimbra user mysql password
 -u,--user <arg>       check only the user/mbox-id (NEW IN 5.0.11)
 -z,--gzip             test compressed blobs (NEW IN 5.0.10)

In order to run a comprehensive check, run zmblobchk with the -z option, allowing a deep inspection of compressed blobs. If it is desired to reduce the run-time of the consistency check, one may specify the --user option to check the files owned by the particular user; this is particularly useful if only one user is experiencing problems and there is no need to verify the entire system.

zmblobchk output

The check may result in many inconsistencies found, but for the purpose of this problem, we are only interested in error reports that indicate wrong size. It may be helpful to redirect the output to a text file and grep the resulting text file for wrong size.

* MailboxGroup=2, mailbox=302, item=136109: wrong size: expected 7344, was 7342: 136109-204468.msg
* MailboxGroup=3, mailbox=203, item=102761: wrong size: expected 0, was 1490089: 102761-140546.msg

Fixes

For daring folks only

The courageous may examine the specified filenames, compare them to the metadata in the database and possibly update them manually.

WARNING: EXTREME DANGER, MANUALLY MANIPULATING DATABASE DATA CAN AND WILL RESULT IN AN UNUSABLE AND/OR CORRUPTED ZIMBRA STORE

For example, for the first record above, assuming that the blob size is correct and the metadata is incorrect, we can fix the problem by doing (as the zimbra user):

mysql mboxgroup2
     update mail_item set size = 7342 where id = 136109 and mailbox_id = 302

In the future (safe)

zmblobchk will be adding support for fixing various forms of blob inconsistencies (see bug 82333[4] , old bug was 27958[5]). Before that, we may create a data migration script ensure the inconsistent size metadata is made appropriate to the on-disk blob data.


Verified Against: ZCS 5.0.11 Date Created: 10/06/2008
Article ID: http://wiki.zimbra.com/index.php?title=Desktop_Sync_Failures_caused_by_message_size_mismatch Date Modified: 05/22/2013
Personal tools