Desktop Sync Failures caused by message size mismatch
Problem Description
One is unable to desktop sync and a message similar to the following appears in the log. The following problem is caused by an inconsistency in server metadata specifying the size of BLOB-data and the on-disk file-sizes.
Please see bugs 31519[1] and 31764[2]
2008-09-10 13:01:14,901 ERROR [...user@domain] [mid=20;] offline - sync failure: user@domain com.zimbra.common.service.ServiceException: system failure: InitialSync.syncMessagesAsTgz ExceptionId:...user@domain:1221076874901:a028898b78e232fd Code:service.FAILURE at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:253) at com.zimbra.cs.mailbox.SyncExceptionHandler.checkRecoverableException(SyncExceptionHandler.java:38) at com.zimbra.cs.mailbox.InitialSync.syncMessagesAsTgz(InitialSync.java:882) at com.zimbra.cs.mailbox.InitialSync.syncMessages(InitialSync.java:840) at com.zimbra.cs.mailbox.InitialSync.syncMessagelikeItems(InitialSync.java:423) at com.zimbra.cs.mailbox.InitialSync.initialFolderSync(InitialSync.java:313) at com.zimbra.cs.mailbox.InitialSync.initialFolderSync(InitialSync.java:377) at com.zimbra.cs.mailbox.InitialSync.initialFolderSync(InitialSync.java:377) at com.zimbra.cs.mailbox.InitialSync.resume(InitialSync.java:188) at com.zimbra.cs.mailbox.InitialSync.resume(InitialSync.java:174) at com.zimbra.cs.mailbox.MailboxSync.sync(MailboxSync.java:192) at com.zimbra.cs.mailbox.OfflineMailbox.sync(OfflineMailbox.java:115) at com.zimbra.cs.mailbox.OfflineMailbox.syncOnTimer(OfflineMailbox.java:96) at com.zimbra.cs.mailbox.DesktopMailbox$1.run(DesktopMailbox.java:117) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) Caused by: com.zimbra.common.service.ServiceException: system failure: Unable to read/write message id=443759 ExceptionId:...user@domain:1221076874901:a028898b78e232fd Code:service.FAILURE at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:253) at com.zimbra.cs.mailbox.InitialSync.saveMessage(InitialSync.java:1014) at com.zimbra.cs.mailbox.InitialSync.syncMessagesAsTgz(InitialSync.java:879) ... 13 more Caused by: java.io.EOFException: Unexpected end of ZLIB input stream at java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:216) at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:134) at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:87) at com.zimbra.common.util.tar.TarBuffer.readBlock(TarBuffer.java:257) at com.zimbra.common.util.tar.TarBuffer.readRecord(TarBuffer.java:223) at com.zimbra.common.util.tar.TarInputStream.read(TarInputStream.java:349) at com.zimbra.common.util.ByteUtil.readInput(ByteUtil.java:642) at com.zimbra.cs.mailbox.InitialSync.saveMessage(InitialSync.java:1003) ... 14 more
Investigation
Run the zmblobchk tool. WARNING: This tool will produce large I/O loads as it will stat every single file in the Zimbra blob-store and may decompress any compressed blobs.
usage: zmblobchk -f,--file <arg> save/load report to/from file -h,--help this help message -k,--skip-fs skip blob store reverse check (NEW IN 5.0.10) -l,--load <arg> display report from file -p,--password <arg> zimbra user mysql password -u,--user <arg> check only the user/mbox-id (NEW IN 5.0.11) -z,--gzip test compressed blobs (NEW IN 5.0.10)
In order to run a comprehensive check, run zmblobchk with the -z option, allowing a deep inspection of compressed blobs. If it is desired to reduce the run-time of the consistency check, one may specify the --user option to check the files owned by the particular user; this is particularly useful if only one user is experiencing problems and there is no need to verify the entire system.
zmblobchk output
The check may result in many inconsistencies found, but for the purpose of this problem, we are only interested in error reports that indicate wrong size. It may be helpful to redirect the output to a text file and grep the resulting text file for wrong size.
* MailboxGroup=2, mailbox=302, item=136109: wrong size: expected 7344, was 7342: 136109-204468.msg * MailboxGroup=3, mailbox=203, item=102761: wrong size: expected 0, was 1490089: 102761-140546.msg