Desktop Sync Failures caused by message size mismatch: Difference between revisions
No edit summary |
(adding article footer and category) |
||
Line 85: | Line 85: | ||
[[zmblobchk]] will be adding support for fixing various forms of blob inconsistencies (see bug 27958[http://bugzilla.zimbra.com/show_bug.cgi?id=27958]). Before that, we may create a data migration script ensure the inconsistent size metadata is made appropriate to the on-disk blob data. | [[zmblobchk]] will be adding support for fixing various forms of blob inconsistencies (see bug 27958[http://bugzilla.zimbra.com/show_bug.cgi?id=27958]). Before that, we may create a data migration script ensure the inconsistent size metadata is made appropriate to the on-disk blob data. | ||
{{Article Footer|Unknown|10/06/2008}} | |||
[[Category:Zimbra Desktop]] |
Revision as of 19:28, 13 October 2008
Problem Description
One is unable to desktop sync and a message similar to the following appears in the log. The following problem is caused by an inconsistency in server metadata specifying the size of BLOB-data and the on-disk file-sizes.
Please see bugs 31519[1], 31764[2] and 31937[3]
Currently, the workaround is to update to the latest TarFormatter (Zimbra Desktop) as worked around in 31937. In the future, we will have a fix script to migrate from incorrect to correct data.
2008-09-10 13:01:14,901 ERROR [...user@domain] [mid=20;] offline - sync failure: user@domain com.zimbra.common.service.ServiceException: system failure: InitialSync.syncMessagesAsTgz ExceptionId:...user@domain:1221076874901:a028898b78e232fd Code:service.FAILURE at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:253) at com.zimbra.cs.mailbox.SyncExceptionHandler.checkRecoverableException(SyncExceptionHandler.java:38) at com.zimbra.cs.mailbox.InitialSync.syncMessagesAsTgz(InitialSync.java:882) at com.zimbra.cs.mailbox.InitialSync.syncMessages(InitialSync.java:840) at com.zimbra.cs.mailbox.InitialSync.syncMessagelikeItems(InitialSync.java:423) at com.zimbra.cs.mailbox.InitialSync.initialFolderSync(InitialSync.java:313) at com.zimbra.cs.mailbox.InitialSync.initialFolderSync(InitialSync.java:377) at com.zimbra.cs.mailbox.InitialSync.initialFolderSync(InitialSync.java:377) at com.zimbra.cs.mailbox.InitialSync.resume(InitialSync.java:188) at com.zimbra.cs.mailbox.InitialSync.resume(InitialSync.java:174) at com.zimbra.cs.mailbox.MailboxSync.sync(MailboxSync.java:192) at com.zimbra.cs.mailbox.OfflineMailbox.sync(OfflineMailbox.java:115) at com.zimbra.cs.mailbox.OfflineMailbox.syncOnTimer(OfflineMailbox.java:96) at com.zimbra.cs.mailbox.DesktopMailbox$1.run(DesktopMailbox.java:117) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) Caused by: com.zimbra.common.service.ServiceException: system failure: Unable to read/write message id=443759 ExceptionId:...user@domain:1221076874901:a028898b78e232fd Code:service.FAILURE at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:253) at com.zimbra.cs.mailbox.InitialSync.saveMessage(InitialSync.java:1014) at com.zimbra.cs.mailbox.InitialSync.syncMessagesAsTgz(InitialSync.java:879) ... 13 more Caused by: java.io.EOFException: Unexpected end of ZLIB input stream at java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:216) at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:134) at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:87) at com.zimbra.common.util.tar.TarBuffer.readBlock(TarBuffer.java:257) at com.zimbra.common.util.tar.TarBuffer.readRecord(TarBuffer.java:223) at com.zimbra.common.util.tar.TarInputStream.read(TarInputStream.java:349) at com.zimbra.common.util.ByteUtil.readInput(ByteUtil.java:642) at com.zimbra.cs.mailbox.InitialSync.saveMessage(InitialSync.java:1003) ... 14 more
Investigation
Run the zmblobchk tool.
WARNING: This tool will produce large I/O loads as it will stat every single file in the Zimbra blob-store and may decompress any compressed blobs.
usage: zmblobchk -f,--file <arg> save/load report to/from file -h,--help this help message -k,--skip-fs skip blob store reverse check (NEW IN 5.0.10) -l,--load <arg> display report from file -p,--password <arg> zimbra user mysql password -u,--user <arg> check only the user/mbox-id (NEW IN 5.0.11) -z,--gzip test compressed blobs (NEW IN 5.0.10)
In order to run a comprehensive check, run zmblobchk with the -z option, allowing a deep inspection of compressed blobs. If it is desired to reduce the run-time of the consistency check, one may specify the --user option to check the files owned by the particular user; this is particularly useful if only one user is experiencing problems and there is no need to verify the entire system.
zmblobchk output
The check may result in many inconsistencies found, but for the purpose of this problem, we are only interested in error reports that indicate wrong size. It may be helpful to redirect the output to a text file and grep the resulting text file for wrong size.
* MailboxGroup=2, mailbox=302, item=136109: wrong size: expected 7344, was 7342: 136109-204468.msg * MailboxGroup=3, mailbox=203, item=102761: wrong size: expected 0, was 1490089: 102761-140546.msg
Fixes
For daring folks only
The courageous may examine the specified filenames, compare them to the metadata in the database and possibly update them manually.
WARNING: EXTREME DANGER, MANUALLY MANIPULATING DATABASE DATA CAN AND WILL RESULT IN AN UNUSABLE AND/OR CORRUPTED ZIMBRA STORE
For example, for the first record above, assuming that the blob size is correct and the metadata is incorrect, we can fix the problem by doing (as the zimbra user):
mysql mboxgroup2 update mail_item set size = 7342 where id = 136109 and mailbox_id = 302
In the future (safe)
zmblobchk will be adding support for fixing various forms of blob inconsistencies (see bug 27958[4]). Before that, we may create a data migration script ensure the inconsistent size metadata is made appropriate to the on-disk blob data.