Ajcody-Notes-No-Such-Blob: Difference between revisions

mNo edit summary
 
(29 intermediate revisions by 4 users not shown)
Line 1: Line 1:
{|  width="100%" border="0"
{{BC|Zeta Alliance}}                        <!-- Note, this will also add [[Category: Zeta Alliance]] to bottom of wiki page. -->
| bgcolor="orange" | [[Image:Attention.png]] - This article is NOT official Zimbra documentation. It is a user contribution and may include unsupported customizations, references, suggestions, or information.
__FORCETOC__                              <!-- Will force a TOC regards of size of article. __NOTOC__ if no TOC is wanted. -->
|}
<div class="col-md-12 ibox-content">
==NO_SUCH_BLOB Errors==            <!-- Normally will reflect page title. Is listed at very top of page. -->
{{KB|{{ZETA}}|{{ZCS 8.5}}|{{ZCS 8.0}}|{{ZCS 7.0}}|}}            <!-- Can only handle 3 ZCS versions. -->
{{WIP}}                                               <!-- For pages that are "work in progress". -->


==NO_SUCH_BLOB Errors==


===Actual No Such Blob Homepage===
===Actual No Such Blob Homepage===
Line 198: Line 200:
#!/usr/bin/perl
#!/usr/bin/perl


# This script compatible with Zimbra version 4.5.x only. Do not use with any other version.
# On 5.0.6+ there's a script, /opt/zimbra/bin/zmblobchk , that you can use instead of this.
# This script is compatible with Zimbra version 4.5.x and 5.x .
# OK, there's 2 MAILBOX_*_BITS values in the VOLUME table.
# OK, there's 2 MAILBOX_*_BITS values in the VOLUME table.
# Take the mailbox ID, right-shift it by MAILBOX_BITS, and take the lowest MAILBOX_GROUP_BITS of the result.
# Take the mailbox ID, right-shift it by MAILBOX_BITS, and take the lowest MAILBOX_GROUP_BITS of the result.
Line 501: Line 504:


* [[Account_mailbox_database_structure]]
* [[Account_mailbox_database_structure]]
* [[Ajcody-Mysql-Topics#mail_item_Types]]
* [[Ajcody-Mysql-Topics#Description_of_mboxgroup.5B.23.5D.mail_item_Type|Description of mboxgroup[#].mail_item Type]]
* [[Ajcody-Mysql-Topics]]


====Getting variables we need to query MySQL====
====Getting User Variables we need to query MySQL====


=====Getting the users mailboxId #=====
Please see [[Ajcody-Mysql-Topics#Getting_User_Variables_We_Need_To_Query_MySQL]]


For Mysql, this # returned below will become apart of mailbox_id=### = mailbox_id=3
====Mysql Query For A Users Specific Blob-Data ID====


<pre>
Please see [[Ajcody-Mysql-Topics#Useful_Mysql_Query_Examples|Useful Mysql Query Examples]]
[root@mail39 ~]# su - zimbra
[zimbra@mail39 ~]$ zmprov gmi ajcody@mail39.zimbra.DOMAIN.com
  mailboxId: 3
  quotaUsed: 169831
</pre>
 
=====Confirm zimbra_mailbox_groups Variable=====
 
Need to confirm zimbra_mailbox_groups variable to get then users mboxgroup since we need to run the expression below for it
 
<pre>
[zimbra@mail39 ~]$ zmlocalconfig zimbra_mailbox_groups
zimbra_mailbox_groups = 100
</pre>
 
=====Get Users mboxgroup #=====
 
Finally confirming what the mboxgroup is for the user by running the following below.
: 3 ='s the users mailboxId
: 100 ='s the variable from zimbra_mailbox_groups
For Mysql, this # returned below will become apart of mboxgroup###.mail_item = mboxgroup3.mail_item
 
<pre>
[zimbra@mail39 ~]$ expr 3 % 100
3
</pre>
 
====MySQL Query For A Users Specific Blob-Data ID====
 
Query of MySQL against users mboxgroup [mboxgroup#.mail_item] for a SPECIFIC message [id=#]
: '''''Note, in this case, the users mailbox_id isn't necessary'''''
 
<pre>
[zimbra@mail39 ~]$ mysql -e 'SELECT * FROM mboxgroup3.mail_item WHERE id=321\G'
*************************** 1. row ***************************
  mailbox_id: 3
          id: 321
        type: 5
  parent_id: NULL
  folder_id: 5
    index_id: 321
    imap_id: 321
        date: 1281984004
        size: 448
  volume_id: 1
blob_digest: 0pe,M6lHRy4KBXbIyEeR7AJzfkA=
      unread: 0
      flags: 8193
        tags: 0
      sender: Adam Cody
    subject: test w mobile sending
        name: NULL
    metadata: d1:f0:1:s45:Adam Cody <ajcody@mail39.zimbra.DOMAIN.com>1:t32:admin@mail38.zimbra.DOMAIN.com1:vi10ee
mod_metadata: 21394
change_date: 1281984004
mod_content: 21394
</pre>
 
====MySQL Query For ALL Message ID's Of A User====
 
As you'll see in the above query, it has type=5 . [[Ajcody-Mysql-Topics#mail_item_Types]] shows that type 5 is 'messages'.
 
<pre>
[zimbra@mail39 ~]$ mysql -e 'SELECT * FROM mboxgroup3.mail_item WHERE mailbox_id=3 AND type=5\G'
*************************** 1. row ***************************
  mailbox_id: 3
          id: 261
        type: 5
  parent_id: 263
  folder_id: 5
    index_id: 261
    imap_id: 261
        date: 1273609779
        size: 458
  volume_id: 3
blob_digest: baQatWGZSt5THO6tAGImpEOqVQY=
      unread: 0
      flags: 1
        tags: 0
      sender: Adam Cody
    subject: test test
        name: NULL
    metadata: d1:f9:test test1:s45:Adam Cody <ajcody@mail39.zimbra.DOMAIN.com>1:t45:Adam Cody <ajcody@mail39.zimbra.DOMAIN.com>1:vi10ee
mod_metadata: 6216
change_date: 1273609779
mod_content: 6215
*************************** 2. row ***************************
  mailbox_id: 3
          id: 262
        type: 5
  parent_id: 263
  folder_id: 2
    index_id: 262
    imap_id: 262
        date: 1273609779
        size: 1599
  volume_id: 3
blob_digest: s2BgZHas3+fgRhBYpIHv7uAEuCY=
      unread: 0
      flags: 0
        tags: 0
      sender: Adam Cody
    subject: test test
        name: NULL
    metadata: d1:f9:test test1:s45:Adam Cody <ajcody@mail39.zimbra.DOMAIN.com>1:vi10ee
mod_metadata: 6217
change_date: 1273609784
mod_content: 6216
*************************** 3. row ***************************
  mailbox_id: 3
          id: 264
        type: 5
  parent_id: NULL
  folder_id: 2
    index_id: 264
    imap_id: 264
        date: 1273610021
        size: 3467
  volume_id: 3
blob_digest: n,VKvXmfL,jiA6kC8zE2g3TkWuM=
      unread: 0
      flags: 0
        tags: 0
      sender: Test User
    subject: test html
        name: NULL
    metadata: d1:f14:BOLD Underline1:s43:Test User <test@mail39.zimbra.DOMAIN.com>1:vi10ee
mod_metadata: 6220
change_date: 1273610084
mod_content: 6219
*************************** 4. row ***************************
  mailbox_id: 3
          id: 281
        type: 5
  parent_id: 286
  folder_id: 5
    index_id: 281
    imap_id: 281
        date: 1277122331
        size: 463
  volume_id: 3
blob_digest: LifXXOMYCgjFsIj,+bsxi0cboOQ=
      unread: 0
      flags: 1
        tags: 0
      sender: Adam Cody
    subject: test mail39
        name: NULL
    metadata: d1:f0:1:s45:Adam Cody <ajcody@mail39.zimbra.DOMAIN.com>1:t33:ajcody@mail59.zimbra.DOMAIN.com1:vi10ee
mod_metadata: 13741
change_date: 1278082563
mod_content: 13701
*************************** 5. row ***************************
  mailbox_id: 3
          id: 283
        type: 5
  parent_id: NULL
  folder_id: 2
    index_id: 283
    imap_id: 283
        date: 1278081006
        size: 1773
  volume_id: 1
blob_digest: M06+uGgtQB8Ri3ZK8fWoIc+322Q=
      unread: 0
      flags: 0
        tags: 0
      sender: root
    subject: test
        name: NULL
    metadata: d1:f4:test1:s37:root <root@mail3.zimbra.DOMAIN.com>1:vi10ee
mod_metadata: 13717
change_date: 1278081017
mod_content: 13715
*************************** 6. row ***************************
  mailbox_id: 3
          id: 284
        type: 5
  parent_id: NULL
  folder_id: 2
    index_id: 284
    imap_id: 284
        date: 1278082511
        size: 1771
  volume_id: 1
blob_digest: K6aQpY9,LI65XsaQ,08qkBgwMMI=
      unread: 1
      flags: 0
        tags: 0
      sender: root
    subject: test
        name: NULL
    metadata: d1:f4:tets1:s37:root <root@mail3.zimbra.DOMAIN.com>1:vi10ee
mod_metadata: 13740
change_date: 1278082511
mod_content: 13740
</pre>
 
: '''cut --  it then continues for all the messages that exist for this account'''


====MySQL Query For ALL Message ID's In A Particular Folder Of A User====
----
 
As you'll see in the above query, it has type=5 . [[Ajcody-Mysql-Topics#mail_item_Types]] shows that type 5 is 'messages'. And we'll use zmmailbox against the user to find the id for folder_id that we want.
 
<pre>
[zimbra@mail39 ~]$ zmmailbox -z -m ajcody@mail39.zimbra.DOMAIN.com gaf
        Id  View      Unread  Msg Count  Path
----------  ----  ----------  ----------  ----------
        1  conv          0          0  / ()
        16  docu          0          1  /Briefcase ()
        10  appo          0          1  /Calendar ()
        14  mess          0          6  /Chats ()
        7  cont          0          1  /Contacts ()
        6  mess          0          1  /Drafts ()
        13  cont          0          2  /Emailed Contacts ()
        2  mess          10          19  /Inbox ()
        4  mess          0          0  /Junk ()
        12  wiki          0          0  /Notebook ()
        5  mess          0          6  /Sent ()
        15  task          0          0  /Tasks ()
        3  conv          0          0  /Trash ()
</pre>
 
Now we'll setup the query to also use the folder_id. In this example folder_id=5 for the users "Sent" folder.
 
<pre>
[zimbra@mail39 ~]$ mysql -e 'SELECT * FROM mboxgroup3.mail_item WHERE mailbox_id=3 AND type=5 AND folder_id=5\G'
*************************** 1. row ***************************
  mailbox_id: 3
          id: 261
        type: 5
  parent_id: 263
  folder_id: 5
    index_id: 261
    imap_id: 261
        date: 1273609779
        size: 458
  volume_id: 3
blob_digest: baQatWGZSt5THO6tAGImpEOqVQY=
      unread: 0
      flags: 1
        tags: 0
      sender: Adam Cody
    subject: test test
        name: NULL
    metadata: d1:f9:test test1:s45:Adam Cody <ajcody@mail39.zimbra.DOMAIN.com>1:t45:Adam Cody <ajcody@mail39.zimbra.DOMAIN.com>1:vi10ee
mod_metadata: 6216
change_date: 1273609779
mod_content: 6215
*************************** 2. row ***************************
  mailbox_id: 3
          id: 281
        type: 5
  parent_id: 286
  folder_id: 5
    index_id: 281
    imap_id: 281
        date: 1277122331
        size: 463
  volume_id: 3
blob_digest: LifXXOMYCgjFsIj,+bsxi0cboOQ=
      unread: 0
      flags: 1
        tags: 0
      sender: Adam Cody
    subject: test mail39
        name: NULL
    metadata: d1:f0:1:s45:Adam Cody <ajcody@mail39.zimbra.DOMAIN.com>1:t33:ajcody@mail59.zimbra.DOMAIN.com1:vi10ee
mod_metadata: 13741
change_date: 1278082563
mod_content: 13701
*************************** 3. row ***************************
  mailbox_id: 3
          id: 297
        type: 5
  parent_id: 286
  folder_id: 5
    index_id: 297
    imap_id: 297
        date: 1278084134
        size: 14896
  volume_id: 1
blob_digest: 66QUCgivALxwGk+p3hnQ4viVZXU=
      unread: 0
      flags: 8195
        tags: 0
      sender: Adam Cody
    subject: test mail39
        name: NULL
    metadata: d1:f22:-- Original Message --1:p4:Re: 1:s45:Adam Cody <ajcody@mail39.zimbra.DOMAIN.com>1:t45:Adam Cody <ajcody@mail59.zimbra.homeunix.com>1:vi10ee
mod_metadata: 13765
change_date: 1278084134
mod_content: 13765
*************************** 4. row ***************************
  mailbox_id: 3
          id: 299
        type: 5
  parent_id: NULL
  folder_id: 5
    index_id: 299
    imap_id: 299
        date: 1278084319
        size: 14452
  volume_id: 1
blob_digest: SGwGaRGJd7bJ735gD2SKJ5zKAlA=
      unread: 0
      flags: 8195
        tags: 0
      sender: Adam Cody
    subject: sent as new
        name: NULL
    metadata: d1:f0:1:s45:Adam Cody <ajcody@mail39.zimbra.DOMAIN.com>1:t42:ajcody <ajcody@mail59.zimbra.DOMAIN.com>1:vi10ee
mod_metadata: 13769
change_date: 1278084319
mod_content: 13769
*************************** 5. row ***************************
  mailbox_id: 3
          id: 305
        type: 5
  parent_id: NULL
  folder_id: 5
    index_id: 305
    imap_id: 305
        date: 1278085335
        size: 693
  volume_id: 1
blob_digest: s4ugNssQxHqqi+Azv8yB+evfSGA=
      unread: 0
      flags: 8193
        tags: 0
      sender: Adam Cody
    subject: content type
        name: NULL
    metadata: d1:f151:--=_Part_49_153544464.1278079895193 Content-Type: application/vnd.openxmlformats-officedocument.wordprocessingml.document; name=test-docm-file.docm ...1:s45:Adam Cody <ajcody@mail39.zimbra.DOMAIN.com>1:t17:ajcody@DOMAIN.com1:vi10ee
mod_metadata: 13784
change_date: 1278085335
mod_content: 13784
*************************** 6. row ***************************
  mailbox_id: 3
          id: 321
        type: 5
  parent_id: NULL
  folder_id: 5
    index_id: 321
    imap_id: 321
        date: 1281984004
        size: 448
  volume_id: 1
blob_digest: 0pe,M6lHRy4KBXbIyEeR7AJzfkA=
      unread: 0
      flags: 8193
        tags: 0
      sender: Adam Cody
    subject: test w mobile sending
        name: NULL
    metadata: d1:f0:1:s45:Adam Cody <ajcody@mail39.zimbra.DOMAIN.com>1:t32:admin@mail38.zimbra.DOMAIN.com1:vi10ee
mod_metadata: 21394
change_date: 1281984004
mod_content: 21394
</pre>


[[Category: Community Sandbox]]
[[Category: Community Sandbox]]
[[Category: Author:Ajcody]]
[[Category: Zeta Alliance]]

Latest revision as of 01:12, 21 June 2016

NO_SUCH_BLOB Errors

   KB 2497        Last updated on 2016-06-21  




0.00
(0 votes)
24px ‎  - This is Zeta Alliance Certified Documentation. The content has been tested by the Community.


Actual No Such Blob Homepage

Please see Ajcody-Notes-No-Such-Blob

Other Resource Overviews

It would be good to review this wiki page before you start any "changes":

http://wiki.zimbra.com/index.php?title=Account_mailbox_database_structure

And great forum posting:

http://www.zimbra.com/forums/administrators/19811-solved-missing-blob-errors-zimbra-4-5-10-a.html

Check Permissions First

If you recently did a server move or similar type of operation, make sure zimbra:zimbra permissions are applied correctly to store directory. You could run the following to double check permissions (as root):

/opt/zimbra/libexec/zmfixperms --verbose --extended

Perl To Find MailboxID's That Have Problem

And down and dirty way to see what mailboxes have this error:

grep -B2 NO_SUCH_BLOB /opt/zimbra/log/mailbox.lo* |grep mailbox= |sed -r 's/.*mailbox=([0-9]*).*$/\1/' |sort -u

Mapping MailboxId to User Account

Remember, mailboxId is unique to the mailstore - it's not a global user variable that's unique.

Stolen from forum post:

http://www.zimbra.com/forums/administrators/16885-mailbox_id-email-address.html

su - zimbra
mysql
use zimbra;
select comment from mailbox where id=257;
And you'll get back:
+----------------------+
| comment |
+----------------------+
| user@domain.com |
+----------------------+
1 row in set (0.00 sec)

If you wanted to see all the info for that account:
select * from mailbox where id=257;
And you'll get back:
+-----+----------+--------------------------------------+-----------------+--------------------+---------------+-----------------+-------------------+---------------+---------------+----------------+----------------------+------------------+--------------+--------------------+
| id | group_id | account_id | index_volume_id | item_id_checkpoint | contact_count | size_checkpoint | change_checkpoint | tracking_sync | tracking_imap | last_backup_at | comment | last_soap_access | new_messages | idx_deferred_count |
+-----+----------+--------------------------------------+-----------------+--------------------+---------------+-----------------+-------------------+---------------+---------------+----------------+----------------------+------------------+--------------+--------------------+
| 257 | 71 | xxxxxxxx-xxxx-xxxx-xxxxxxxxxxxxxxxxx | 2 | 257 | 0 | 1832 | 100 | 0 | 0 | NULL | user@domain.com | 1209652908 | 0 | 0 |
+-----+----------+--------------------------------------+-----------------+--------------------+---------------+-----------------+-------------------+---------------+---------------+----------------+----------------------+------------------+--------------+--------------------+
1 row in set (0.00 sec)

Zmblobchk for 5.0.6+ Systems

On 5.0.6+ there's a script in /opt/zimbra/bin called zmblobchk . This will list out information about the NO_SUCH_BLOB errors and other inconsistencies.

zmblobchk - RFE's

zmblobchk currently just lists "issues" - for the most part. Please see the following RFE's, some will give details about resolutions based upon what zmblobchk reported.

Example Output On A System With No Issues

su - zimbra
ls /opt/zimbra/zmblobchk.jar
  zmblobchk.jar
java -jar zmblobchk.jar
Retrieving volume information
Retrieving mboxgroup list
Spooling item list to /tmp/mailitems53650.lst
Retrieving items from mboxgroup1
Retrieving items from mboxgroup2
Retrieving items from mboxgroup3
Retrieving items from mboxgroup4
Retrieving items from mboxgroup5
Retrieving items from mboxgroup6
Retrieving items from mboxgroup7
Retrieving items from mboxgroup8
Retrieving items from mboxgroup9
Retrieving items from mboxgroup10
/
Processing BLOB store
-
/tmp/mailitems53650.lst: size 28141
Processed 329 items in 1223ms
No inconsistencies found

Example Output On A System With Issues

Last login: Thu Oct 30 12:03:12 on ttys000
COLDMILE-LM-3:~ ajcody$ ssh root@mail3.internal.homeunix.com
root@mail3.internal.homeunix.com's password: 
Last login: Sat Nov  1 14:25:50 2008 from 192.168.0.13
[root@mail3 ~]# su - zimbra
[zimbra@mail3 ~]$ zmblobchk 
Retrieving volume information
Retrieving mboxgroup list
Spooling item list to /tmp/mailitems39690.lst
Retrieving items from mboxgroup1
Retrieving items from mboxgroup2
Retrieving items from mboxgroup3
Retrieving items from mboxgroup4
Retrieving items from mboxgroup5
Retrieving items from mboxgroup6
Retrieving items from mboxgroup7
Retrieving items from mboxgroup8
Retrieving items from mboxgroup9
Retrieving items from mboxgroup10
Retrieving items from mboxgroup11
Retrieving items from mboxgroup12
Retrieving items from mboxgroup13
Retrieving items from mboxgroup14
Retrieving items from mboxgroup15
Retrieving items from mboxgroup16
Retrieving items from mboxgroup17
Retrieving items from mboxgroup18
Retrieving items from mboxgroup19
Retrieving items from mboxgroup20
-
Processing BLOB store
/
/tmp/mailitems39690.lst: size 470853
Processed 5873 items in 18462ms
11844 inconsistencies found
 * MailboxGroup=1, mailbox=1, item=303: wrong volume, expected at /vol1/0/1/msg/0/303-2062.msg, found at /opt/zimbra/store/0/1/msg/0/303-2062.msg (move to correct volume)
 * MailboxGroup=1, mailbox=1, item=304: wrong volume, expected at /vol1/0/1/msg/0/304-2064.msg, found at /opt/zimbra/store/0/1/msg/0/304-2064.msg (move to correct volume)
 * MailboxGroup=1, mailbox=1, item=305: wrong volume, expected at /vol1/0/1/msg/0/305-2077.msg, found at /opt/zimbra/store/0/1/msg/0/305-2077.msg (move to correct volume)
 * MailboxGroup=1, mailbox=1, item=306: wrong volume, expected at /vol1/0/1/msg/0/306-2091.msg, found at /opt/zimbra/store/0/1/msg/0/306-2091.msg (move to correct volume)

## CUT OF 1000's OF SIMILAR LINES ##

 * MailboxGroup=1, mailbox=1, item=6480: file not found: /vol1/0/1/msg/1/6480-20100.msg (delete associated metadata)
 * MailboxGroup=3, mailbox=3, item=400: wrong volume, expected at /vol1/0/3/msg/0/400-11287.msg, found at /opt/zimbra/store/0/3/msg/0/400-11287.msg (move to correct volume)
 * MailboxGroup=3, mailbox=3, item=425: wrong volume, expected at /vol1/0/3/msg/0/425-12939.msg, found at /opt/zimbra/store/0/3/msg/0/425-12939.msg (move to correct volume)
 * MailboxGroup=3, mailbox=3, item=427: wrong volume, expected at /vol1/0/3/msg/0/427-12943.msg, found at /opt/zimbra/store/0/3/msg/0/427-12943.msg (move to correct volume)
 * MailboxGroup=3, mailbox=3, item=440: file not found: /vol1/0/3/msg/0/440-13400.msg (delete associated metadata)
 * /opt/zimbra/store/0/3/msg/0/363-6194.msg: no associated metadata (delete blob)
 * /opt/zimbra/store/0/3/msg/0/360-5000.msg: no associated metadata (delete blob)
 * /opt/zimbra/store/0/3/msg/0/350-3129.msg: no associated metadata (delete blob)
 * /opt/zimbra/store/0/3/msg/0/320-2272.msg: no associated metadata (delete blob)
 
## CUT OF 1000's OF SIMILAR LINES ##

 * /opt/zimbra/store/0/2/msg/0/304-6907.msg: no associated metadata (delete blob)
 * /opt/zimbra/store/0/2/msg/0/315-6918.msg: no associated metadata (delete blob)
 * /opt/zimbra/store/0/2/msg/0/312-6915.msg: no associated metadata (delete blob)
 * /opt/zimbra/store/0/2/msg/0/308-6911.msg: no associated metadata (delete blob)
 * /opt/zimbra/store/0/2/msg/0/303-6906.msg: no associated metadata (delete blob)
 * /opt/zimbra/store/0/2/msg/0/301-6904.msg: no associated metadata (delete blob)
 *** 11844 faults found
Report saved to: /tmp/zmblobc39689.rpt
[zimbra@mail3 ~]$ 

To see the reported output again:

zmblobchk -l /tmp/zmblobc39689.rpt


Error - Not Enough Memory To Run zmblobchk

If the "Processing BLOB store" part errors out with Java out of memory, you could try adjusting the variable for it. The default is set to 30%. This would require a mailstore/jetty restart though [zmmailboxdctl restart]. If the mailstore doesn't come back up, check /opt/zimbra/log/zmmailbox.log - you most likely exceeded the threshold for your box. Remember, this is a percentage and a 32bit machine with more than 4GB can cause issues with this setting since it could try to allocate more than it actually can to the thread.

# ZCS 5.0 and later
$ zmlocalconfig -e mailboxd_java_heap_memory_percent=40

And zmmailboxdctl restart when it's appropriate.

BLOB Issue Script

I've commented out the two delete lines and also the "$msghash &= $fmask;" one.

If you're using HSM, this script isn't smart enough to detect it. It will flag and remove entries that are in the HSM store because it doesn't see them in the primary store. You can modify the output though to use the actual path to the HSM store. I had one case where this was the case and copying the missing blobs to the HSM path worked.

Comment from other support staff member on a case that used this.

Once you've gotten a list of the 'NOT OK' files generated by this tool, you'll want to check the store directories to see if the message blobs exist with an incorrect change number. That's the second number in the filename. If the first number matches the ID it's looking for but the second does not, you can rename the file to the new change number and get the message blob back without losing data. If not, then the blob is completely missing. If that's the case, we should try to figure out what happened to it by going through the mailbox.log files. You should be able to grep for the missing message ID and the user's account ID to find a log entry showing something happening to it. It's possible it was deleted and the deletion was never written to mysql for some reason.

#!/usr/bin/perl

# On 5.0.6+ there's a script, /opt/zimbra/bin/zmblobchk , that you can use instead of this.
# This script is compatible with Zimbra version 4.5.x and 5.x .
# OK, there's 2 MAILBOX_*_BITS values in the VOLUME table.
# Take the mailbox ID, right-shift it by MAILBOX_BITS, and take the lowest MAILBOX_GROUP_BITS of the result.
# That's your mailbox hash.
# Take the message ID, right-shift it by FILE_BITS, and take the lowest FILE_GROUP_BITS of the result.  That's your msgid hash.
# I think.
# <mbx-hash>/<mbx-id>/msg/<msgid-hash>/<msgid>-<mod_content>.msg

my ($fbits, $fgbits, $mbits, $mgbits, $basepath) = split (' ',`echo "select file_bits,file_group_bits,mailbox_bits,mailbox_group_bits,path from volume where type='1'" | mysql -N zimbra`);
my $ARGV = shift;

chomp $basepath;

my $mbmask = "1" x $mgbits;
my $fmask = "1" x $fgbits;

foreach (`echo "select id, group_id, account_id, comment from mailbox where comment = '$ARGV'" | mysql -N zimbra`) {
	chomp;
	my $path = "$basepath/";
	my ($id, $grid, $aid, $nm) = split;
	my $mbhash = $id >> $mbits;
	$mbhash &= $mbmask;
	$path .= $mbhash."/".$id."/msg/";
	foreach my $msgstuff (`echo "select id, mod_content, type from mail_item where blob_digest is not null and mailbox_id=${id};" | mysql -N mboxgroup${grid}`) {
		chomp $msgstuff;
		next if $msgstuff eq "";
		my ($msgid, $modContent, $type) = split (' ',$msgstuff);
		my $msghash = $msgid >> $fbits;
#		$msghash &= $fmask;
		my $nm = $msgid;
		if ($modContent) {$nm .= "-$modContent";}
		my $npath = $path.$msghash."/".$nm.".msg";
		if (-e $npath) {print $npath."\t\tOK\n";} 
		else {
		   print $npath."\t\tNOT OK\n";
		   # not ok, remove the entry from the database so it is not a nuisance
		   print "Delete from mail_item where MSGID is ${msgid} and MAILBOXID is ${id} \n";
		   print "Uncomment line below me in script to have me delete. \n";
#		   `echo "delete from mail_item where id=${msgid} and mailbox_id=${id}" | mysql -N mboxgroup${grid}`;
		   if ($type eq "11") {
	       print "Delete from appointment where MSGID is ${msgid} and MAILBOX_ID is ${id} \n";
	       print "Uncomment line below me in script to have me delete. \n";
#              `echo "delete from appointment where item_id=${msgid} and mailbox_id=${id}" | mysql -N mboxgroup${grid}`;
		   }
		}
#		print $npath."\n";
	}
}

If this doesn't run, make sure perl is installed and in the correct path.

You run this as the zimbra user.

What I did was the following.

Copied the script to /opt/zimbra/bin/ as root and called it blob-check.pl . chmod +x /opt/zimbra/bin/blob-check.pl

su - zimbra
mkdir /tmp/BLOB-CHECKS
for i in `zmprov gaa`; do blob-check.pl $i | grep "NOT OK" >> /tmp/BLOB-CHECKS/$i.txt; done
ls -la /tmp/BLOB-CHECKS

The ls will help identify the more troubled accounts. You'll end up with something like this:

[zimbra@mail3 ~]$ ls -la /tmp/BLOB-CHECKS/
total 16
drwxr-x--- 2 zimbra zimbra 4096 Jul 10 23:40 .
drwxrwxrwt 8 root   root   4096 Jul 10 23:46 ..
-rw-r----- 1 zimbra zimbra    0 Jul 10 23:40 admin@mail3.internal.homeunix.com.txt
-rw-r----- 1 zimbra zimbra    0 Jul 10 23:40 ajcody2@mail3.internal.homeunix.com.txt
-rw-r----- 1 zimbra zimbra   49 Jul 10 23:40 ajcody@mail3.internal.homeunix.com.txt
-rw-r----- 1 zimbra zimbra    0 Jul 10 23:40 forward@mail3.internal.homeunix.com.txt
-rw-r----- 1 zimbra zimbra    0 Jul 10 23:40 ham.bidiob2mm@mail3.internal.homeunix.com.txt
-rw-r----- 1 zimbra zimbra    0 Jul 10 23:40 secondary@secondary.internal.homeunix.com.txt
-rw-r----- 1 zimbra zimbra    0 Jul 10 23:40 spam.rormmtcyy@mail3.internal.homeunix.com.txt
-rw-r----- 1 zimbra zimbra    0 Jul 10 23:40 wiki@mail3.internal.homeunix.com.txt
-rw-r----- 1 zimbra zimbra    0 Jul 10 23:40 wiki@secondary.internal.homeunix.com.txt

And the output of one that shows some size:

$ cat /tmp/BLOB-CHECKS/ajcody@mail3.internal.homeunix.com.txt
/opt/zimbra/store/0/3/msg/0/360-6070.msg		NOT OK

Remember this from above, "you'll want to check the store directories to see if the message blobs exist with an incorrect change number. That's the second number in the filename. If the first number matches the ID it's looking for but the second does not, you can rename the file to the new change number and get the message blob back without losing data."

One can uncomment the 2 lines in the script with delete and run the script again. This will remove the reference to the blob.

To re-index the users mailbox [this can take start, cancel, status]:

zmprov rim user@domainname start

Running the script again shouldn't show "NOT OK" lines.

FYI - I still need to gather more information about this situation and how to guide one to make a decision to delete, re-index, and so forth.

BLOB Script To Copy From Restore

You might need to use these variables with the restore command. I need to find a way for someone to figure out what restore to goto for a particular blob:

-restoreToTime <arg> - Replay the redo logs until the time specified.

-restoreToIncreLabel <arg> - Replay redo logs up to and including this incremental backup.

-restoreToRedoSeq <arg> - Replay up to and including this redo log sequence

-br - Replays the redo logs in backup only, which excludes archived and current redo logs of the system.

-rf - Restores to the full backup only, does not include any incremental backups since that backup

This is a walk through to test the proceedure.

$ zmrestore -ca -a ajcody@mail3.internal.homeunix.com -pre restored-
$ zmprov gmi restored-ajcody@mail3.internal.homeunix.com
mailboxId: 12
quotaUsed: 251513
$ zmprov gmi ajcody@mail3.internal.homeunix.com
mailboxId: 3
quotaUsed: 251512
$ diff /opt/zimbra/store/0/12/msg/0/257-25.msg /opt/zimbra/store/0/3/msg/0/257-25.msg
$ rm /opt/zimbra/store/0/3/msg/0/257-25.msg
rm: remove regular file `/opt/zimbra/store/0/3/msg/0/257-25.msg'? y
$ cp /opt/zimbra/store/0/12/msg/0/257-25.msg /opt/zimbra/store/0/3/msg/0/257-25.msg
$ grep Subject /opt/zimbra/store/0/3/msg/0/257-25.msg
Log into main account and confirm I can see/use that mail.

The script to look at the blob-check.pl output and do the copies after a redirected restore is here:

#!/bin/bash

#START OF VARIABLES#
#CHANGEME to existing user account
USER=ajcody@mail3.internal.homeunix.com

#CHANGEME to the restored account name
# zmrestore -ca -a ajcody@mail3.internal.homeunix.com -pre restored-
RESTOREDUSER=restored-ajcody@mail3.internal.homeunix.com

# To get FILE, blob-check.pl ajcody@mail3.internal.homeunix.com > /tmp/BLOB-CHECKS/ajcody.txt
# blob-check.pl located at http://wiki.zimbra.com/index.php?title=Ajcody-Notes#BLOB_Issue_Script
FILE=/tmp/BLOB-CHECKS/ajcody.txt #CHANGEME

#CHANGEME, if needed, to the mailstore. /opt/zimbra/store is default
STOREDIR=/opt/zimbra/store

#END VARIABLES#

USERUID=`zmprov gmi $USER | grep mailboxId | cut -f2 -d: | cut -c 2-10`
USERGID=`perl -e 'print $USERUID >> 12 ; print "\n"'`
USERPATH=`echo $USERGID/\$USERUID`

RESTOREDUID=`zmprov gmi $RESTOREDUSER | grep mailboxId | cut -f2 -d: | cut -c 2-10`
RESTOREDGID=`perl -e 'print $RESTOREDUID >> 12 ; print "\n"'`
RESTOREDPATH=`echo $RESTOREDGID/\$RESTOREDUID`

echo USERUID $USERUID
echo USERGID $USERGID
echo USERPATH $USERPATH
echo RESTOREDUID $RESTOREDUID
echo RESTOREDGID $RESTOREDGID
echo RESTOREDPATH $RESTOREDPATH

cd $STOREDIR
echo "Run ECHO first to confirm copies look right and then uncomment COPY in script"
for ROOTBLOBPATH in `grep "NOT OK" $FILE | cut -f7-9 -d/ | cut -f1`
do
# Comment out echo and uncomment copy after dry run
echo $RESTOREDPATH/$ROOTBLOBPATH $USERPATH/$ROOTBLOBPATH
#cp -uv $RESTOREDPATH/$ROOTBLOBPATH $USERPATH/$ROOTBLOBPATH
done

BLOB Script To Copy From Restore (HSM Issue)

One customer ran into an issue that the blob references were really expecting them in the HSM mailstore path.

Example:

Default mailstore for PROD : /opt/zimbra/mail/

HSM mailstore path : /opt/zimbra/mail0 which was another partition mount, compared to /opt/zimbra/mail

So, what this script did was used the output from the blob-check.pl script which has the missing blobs formatted for the PROD path rather than the HSM path. The blob-check.pl script doesn't handle HSM data. The script then finds the blobs in the full backup directory of the user and copies them into the HSM path that the WEBCLIENT is expecting. This whole situation with this customer might of been a fluke. So don't consider these steps as applying to your situation unless you do your research. This case took 9 days to resolve and figure out.

Notice, if you use this script below you really need to review it in details and adjust it for your situation. There's more to adjust beyond the base variables.

#!/bin/bash

# Steps
# Make sure directory path exists for the copy command
# Get mailboxId - zmprov ga USER@DOMAIN | grep zimbraId:
# mkdir /opt/zimbra/mail0/0/[mailboxId #]/msg/{1..15}  or how what ever # you need.
# Change USEREMAIL
# Get path for BACKUPFILES and uncomment for use
# Run find command with echo and confirm. Then rerun using the find with cp rather than echo

# Put in user's email for USEREMAIL
USEREMAIL="USER@DOMAIN"
ZIMBRAID=`zmprov ga $USEREMAIL | grep zimbraId: | awk '{print $2}'`
# See below
# ROOTBACKUPPATH="/opt/zimbra/mail0/BACKUP/sessions/full-20080707.004336.789"

#The line below takes to long, see instructions underneath
#BACKUPFILES=`find $ROOTBACKUPPATH -name *$ZIMBRAID*`
# To run manually, which will increase speed of script if you then paste full path in for BACKUPFILES var
# zmprov ga USER@DOMAIN | grep zimbraId: | awk '{print $2}' 
# Then place the out in this find command, leave the wildcards on each side of it
# find /opt/zimbra/mail0/BACKUP/sessions/full-20080707.004336.789 -name *OUTPUTHERE* 
# This should give you path like the exampe below, paste it between the quotes and uncomment the variable BACKUPFILES
#EXAMPLE FOR BACKUPFILES="/opt/zimbra/mail0/BACKUP/sessions/full-20080707.004336.789/accounts/a07/1b5/a071b5ad-b341-4de4-b6ee-4463f322873b"
# BACKUPFILES="PASTE_IN_PATH_HER"

for BLOBPATH in `blob-check.pl $USEREMAIL | grep "NOT OK" | awk '{print $1}' | sed 's/store/mail0/'`
do
MSGID=`echo $BLOBPATH | awk -F/ '{print $NF}'`
# Run echo first to make sure copy path seems correct, ctrl-c to cancel. And then uncomment the other find command with the cp. Comment the echo one.
find $BACKUPFILES -name *$MSGID* -exec  echo {} $BLOBPATH \;
#find $BACKUPFILES -name *$MSGID* -exec cp -uv {} $BLOBPATH \;
done

The Really Bad Situation, Blob References Gone & Mail Still Missing

For now, this is all I have to offer.

Method One - zmlmtpinject

See forum post:

http://www.zimbra.com/forums/installation/12617-recover-data-mbox-folders.html#post64962

Method Two - Use IMAPSYNC

I haven't tested this and would just be guessing, but I believe IMAPSYNC could be used.

From the IMAPSYNC notes, "imapsync is the adequate tool because it reduces the amount of data transferred by not transferring a given message if it is already on both sides. Same headers, same message size and the transfer is done only once. All flags are preserved, unread will stay unread, read will stay read, deleted will stay deleted."

This would involve having imap setup on ZCS and then doing a redirected user restore (which would consume a license).

See the following on imapsync details:

Imapsync During A Certain Time Frame

You might want to check out the imapsync options of --maxage and --minage

Proposed Steps
  • Confirm these steps are necessary
  • Install imapsync
  • Do a redirected restore of the account
    • This is an example format of the redirected restore:
      • zmrestore -ca -a prod-user@servername.com -pre restored- -rf -lb full-'date of full'
      • The "new" account would be restored-userPROD@servername.com . The -rf flags should look only at the backup data in regards to the restore and not catch the "production" references to missing blobs and what not.
  • Confirm in webclient that msg's are there
  • Do a IMAPSYNC
    • Two ways, unsure what is the best method
      • From PROD to NEW
        • imapsync --buffersize 8192000 --nosyncacls --subscribe --syncinternaldates --host1 servername.com --user1 prod-user --password1 test123 -user2 restored-prod-user --host2 servername.com --password2 test123 --authmech1 PLAIN --authmech2 PLAIN
      • From NEW to PROD
        • imapsync --buffersize 8192000 --nosyncacls --subscribe --syncinternaldates --host1 servername.com --user1 restored-prod-user --password1 test123 -user2 prod-user --host2 servername.com --password2 test123 --authmech1 PLAIN --authmech2 PLAIN
  • Reindex restored-prod-user
  • Rename prod-user to prod-user-old
    • zmprov ra prod-user@servername.com prod-user-old@servername.com
  • Rename restored-prod-user to prod-user
    • zmprov ra restored-prod-user@servername.com prod-user@servername.com
  • Do a full backup of each account
    • zmbackup -f - <servername.com> -a <newaccountname@servername.com>

Notes On - No Associated Metadata

Tentative Research At This Point - Haven't Proven Out Yet

This is for the errors you see from the blobchk like this:

"* /opt/zimbra/store/0/1303/msg/0/1059-1726.msg: no associated metadata  (delete blob) " 

You could use the following command to bring the blob back in and regenerate the metadata -- zmmailbox am along -d . The am flag details are : addMessage(am) [opts] {dest-folder-path} {filename-or-dir} [{filename-or-dir} ...] . But the -d flag states : -d/--date <arg> received date (msecs since epoch) . To get the epoch time in seconds using the same receive data, for example :

ls -l --time-style=+%s /opt/zimbra/store/0/1303/msg/0/1059-1726.msg

And then remember the -d requires milliseconds [a thousandth (1/1,000) of a second] so you'll need to convert it. Or, you could just use something like this below. This with the cut would work well for a for loop script against a number of msg files.

ls -l --time-style=+%s000 --si /opt/zimbra/store/0/1303/msg/0/1059-1726.msg | cut -d" " -f6
  1272642001000
ls -l --time-style=+%s000 --si /opt/zimbra/store/0/1303/msg/0/1059-1726.msg
  -rw-r--r-- 1 zimbra zimbra 0 1272642001000 

Another suggestion for scripting was:

date=$(stat -c +%Y /opt/zimbra/store/0/1303/msg/0/1059-1726.msg); echo $(($date*1000));

Mysql Queries To See What Is Shown For Missing Message ID

References

See also :

Getting User Variables we need to query MySQL

Please see Ajcody-Mysql-Topics#Getting_User_Variables_We_Need_To_Query_MySQL

Mysql Query For A Users Specific Blob-Data ID

Please see Useful Mysql Query Examples


Jump to: navigation, search