Desktop Sync Failures caused by message size mismatch

Problem Description

One is unable to desktop sync and a message similar to the following appears in the log. The following problem is caused by an inconsistency in server metadata specifying the size of BLOB-data and the on-disk file-sizes.

Please see bugs 31519[1] and 31764[2]

2008-09-10 13:01:14,901 ERROR [...user@domain] [mid=20;] offline - sync failure: user@domain
com.zimbra.common.service.ServiceException: system failure: InitialSync.syncMessagesAsTgz
ExceptionId:...user@domain:1221076874901:a028898b78e232fd
Code:service.FAILURE
       at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:253)
       at com.zimbra.cs.mailbox.SyncExceptionHandler.checkRecoverableException(SyncExceptionHandler.java:38)
       at com.zimbra.cs.mailbox.InitialSync.syncMessagesAsTgz(InitialSync.java:882)
       at com.zimbra.cs.mailbox.InitialSync.syncMessages(InitialSync.java:840)
       at com.zimbra.cs.mailbox.InitialSync.syncMessagelikeItems(InitialSync.java:423)
       at com.zimbra.cs.mailbox.InitialSync.initialFolderSync(InitialSync.java:313)
       at com.zimbra.cs.mailbox.InitialSync.initialFolderSync(InitialSync.java:377)
       at com.zimbra.cs.mailbox.InitialSync.initialFolderSync(InitialSync.java:377)
       at com.zimbra.cs.mailbox.InitialSync.resume(InitialSync.java:188)
       at com.zimbra.cs.mailbox.InitialSync.resume(InitialSync.java:174)
       at com.zimbra.cs.mailbox.MailboxSync.sync(MailboxSync.java:192)
       at com.zimbra.cs.mailbox.OfflineMailbox.sync(OfflineMailbox.java:115)
       at com.zimbra.cs.mailbox.OfflineMailbox.syncOnTimer(OfflineMailbox.java:96)
       at com.zimbra.cs.mailbox.DesktopMailbox$1.run(DesktopMailbox.java:117)
       at java.util.TimerThread.mainLoop(Timer.java:512)
       at java.util.TimerThread.run(Timer.java:462)
Caused by: com.zimbra.common.service.ServiceException: system failure: Unable to read/write message id=443759
ExceptionId:...user@domain:1221076874901:a028898b78e232fd
Code:service.FAILURE
       at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:253)
       at com.zimbra.cs.mailbox.InitialSync.saveMessage(InitialSync.java:1014)
       at com.zimbra.cs.mailbox.InitialSync.syncMessagesAsTgz(InitialSync.java:879)
       ... 13 more
Caused by: java.io.EOFException: Unexpected end of ZLIB input stream
       at java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:216)
       at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:134)
       at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:87)
       at com.zimbra.common.util.tar.TarBuffer.readBlock(TarBuffer.java:257)
       at com.zimbra.common.util.tar.TarBuffer.readRecord(TarBuffer.java:223)
       at com.zimbra.common.util.tar.TarInputStream.read(TarInputStream.java:349)
       at com.zimbra.common.util.ByteUtil.readInput(ByteUtil.java:642)
       at com.zimbra.cs.mailbox.InitialSync.saveMessage(InitialSync.java:1003)
       ... 14 more

Investigation

Run the zmblobchk tool. WARNING: This tool will produce large I/O loads as it will stat every single file in the Zimbra blob-store and may decompress any compressed blobs.

usage: zmblobchk
 -f,--file <arg>       save/load report to/from file
 -h,--help             this help message
 -k,--skip-fs          skip blob store reverse check (NEW IN 5.0.10)
 -l,--load <arg>       display report from file
 -p,--password <arg>   zimbra user mysql password
 -u,--user <arg>       check only the user/mbox-id (NEW IN 5.0.11)
 -z,--gzip             test compressed blobs (NEW IN 5.0.10)

In order to run a comprehensive check, run zmblobchk with the -z option, allowing a deep inspection of compressed blobs. If it is desired to reduce the run-time of the consistency check, one may specify the --user option to check the files owned by the particular user; this is particularly useful if only one user is experiencing problems and there is no need to verify the entire system.

zmblobchk output

The check may result in many inconsistencies found, but for the purpose of this problem, we are only interested in error reports that indicate wrong size. It may be helpful to redirect the output to a text file and grep the resulting text file for wrong size.

* MailboxGroup=2, mailbox=302, item=136109: wrong size: expected 7344, was 7342: 136109-204468.msg
* MailboxGroup=3, mailbox=203, item=102761: wrong size: expected 0, was 1490089: 102761-140546.msg