Difference between revisions of "Steps to extract blobs from redologs using zmredodump"

(Created page with "===<h1>Steps to extract blobs from redologs using zmredodump</h1>=== <hr> <br> <h2>Problem:</h2> 1st Scenario :- Extract blobs from redologs when we don't have a valid back...")
 
 
(7 intermediate revisions by 3 users not shown)
Line 1: Line 1:
===<h1>Steps to extract blobs from redologs using zmredodump</h1>===
+
{{BC|Certified}}
 +
__FORCETOC__
 +
<div class="col-md-12 ibox-content">
 +
=Extract blobs from redologs using zmredodump=  
 
<hr>
 
<hr>
<br>
+
{{KB|{{ZC}}|{{ZCS 8.8}}|{{ZCS 8.7}}|{{ZCS 8.6}}|{{ZCS 8.5}}|}}
  
<h2>Problem:</h2>
+
====Problem====
 +
Extracting blobs from redologs. Possible scenarios where this may be needed:
 +
* Extract blobs from redologs when we don't have a valid backup to perform the restore.
 +
* Extract blobs from redologs when full-backup is missing and only incremental backups are available.
 +
* Also, we can use this procedure when the redolog sequence is broken and we can't use zmredoplay.
  
1st Scenario :- Extract blobs from redologs when we don't have a valid backup to perform the restore. 
+
====Solution====
OR
+
This should only be used when there is no valid backup to perform the restore, as this is just a workaround to recover as much data as possible.
2nd Scenario :- Extract blobs from redologs when fullbackup is missing and only incremental backups are available. 
 
OR
 
3rd Scenario :- Also, We can use this procedure when the redolog sequence is broken and we can't use zmredoplay.
 
 
 
 
 
<h2>.Solution:</h2>
 
 
 
This is a workaround to recover maximum available data, and should only be done when there is no valid backup to perform the restore.
 
With the help of "zmredodump" tool we can extract the blob content in plain text format. See the help and details of zmredodump using "zmredodump --help". 
 
 
 
Zmredodump with option "--show-blob" will print blobs and related operations (like move, delete, etc), where the actual blob will be in between <START OF BLOB> and <END OF BLOB> markers.
 
 
 
This is the syntax of "zmredodump" command:-
 
  
 +
With the help of ''zmredodump''  we can extract the blob content in plain text format.
 +
The "--show-blob" option will print blobs and related operations (like move, delete, etc): 
 
  zmredodump -m <mailbox id> /path/of/redolog/file/redo.log --show-blob
 
  zmredodump -m <mailbox id> /path/of/redolog/file/redo.log --show-blob
 +
The actual blob is present between the '''<START OF BLOB>''' and '''<END OF BLOB>''' markers
  
 +
We then start the process to extract blobs for a user.
  
Here we start the process.
+
====Extract a single blob====
 
 
===Single Blob===
 
 
 
To extract a blob for a user from the redolog file using zmredodump command :
 
  
====Step1 :====
+
* '''Process blob details from redolog file'''
  
The user has mailbox-id "2".
+
In this example we are assuming user has mailbox-id "2".
  
  zmredodump -m 2 /opt/zimbra/redolog/archive/redo-20160223.193045.991-seq63.log --show-blob | less
+
  zmredodump -m 2 /opt/zimbra/redolog/archive/redo-20160223.193045.991-seq63.log --show-blob  
  
<pre>
+
VERIFYING DIRECTORY: /opt/zimbra/redolog/archive
========================================================================================================
+
VERIFYING: /opt/zimbra/redolog/archive/redo-20160224.193055.837-seq65.log
VERIFYING DIRECTORY: /opt/zimbra/redolog/archive
+
HEADER
VERIFYING: /opt/zimbra/redolog/archive/redo-20160224.193055.837-seq65.log
+
------
HEADER
+
sequence: 65
------
+
open:    0
sequence: 65
+
filesize: 43948
open:    0
+
serverId: c8e2f007-a1da-4660-ae89-15d3b2a8e200
filesize: 43948
+
created:  Wed, 2016/02/24 23:51:39.487 IST (1456338099487)
serverId: c8e2f007-a1da-4660-ae89-15d3b2a8e200
+
first op: Wed, 2016/02/24 23:58:01.933 IST (1456338481933)
created:  Wed, 2016/02/24 23:51:39.487 IST (1456338099487)
+
last op:  Thu, 2016/02/25 01:00:43.108 IST (1456342243108)
first op: Wed, 2016/02/24 23:58:01.933 IST (1456338481933)
+
version:  1.42
last op:  Thu, 2016/02/25 01:00:43.108 IST (1456342243108)
+
------
version:  1.42
+
[00000200 - 0000083f: 1600 bytes; tstamp: 2016/02/24 23:58:01.933 IST] txn 1456338154.1 [CreateMessage] ver=1.42, tstamp=1456338481933, change=20600, mailbox=1, id=2260, rcpt=admin@hcent7.lab.zimbra.com, rcvDate=1456338481835, shared=false, blobDigest="m3biKcUSkugQxOXnEHVbi341JxT2OyeTnYUTlY6Gcyw=", size=1358, dataLen=1358, folder=2, conv=-1, convFirstMsgId=-1, calItemPartStat=NE, noICal=false, flags=512, tags=[], bodyType=1, path=/opt/zimbra/store/incoming/1456338152510-0.msg
------
+
'''<START OF BLOB>'''
[00000200 - 0000083f: 1600 bytes; tstamp: 2016/02/24 23:58:01.933 IST] txn 1456338154.1 [CreateMessage] ver=1.42, tstamp=1456338481933, change=20600, mailbox=1, id=2260, rcpt=admin@hcent7.lab.zimbra.com, rcvDate=1456338481835, shared=false, blobDigest="m3biKcUSkugQxOXnEHVbi341JxT2OyeTnYUTlY6Gcyw=", size=1358, dataLen=1358, folder=2, conv=-1, convFirstMsgId=-1, calItemPartStat=NE, noICal=false, flags=512, tags=[], bodyType=1, path=/opt/zimbra/store/incoming/1456338152510-0.msg
+
Return-Path: admin@hcent7.lab.zimbra.com
<START OF BLOB>
+
Received: from hcent7.lab.zimbra.com (LHLO hcent7.lab.zimbra.com)
Return-Path: admin@hcent7.lab.zimbra.com
 
Received: from hcent7.lab.zimbra.com (LHLO hcent7.lab.zimbra.com)
 
 
  (10.15.8.19) by hcent7.lab.zimbra.com with LMTP; Wed, 24 Feb 2016 23:57:59
 
  (10.15.8.19) by hcent7.lab.zimbra.com with LMTP; Wed, 24 Feb 2016 23:57:59
 
  +0530 (IST)
 
  +0530 (IST)
Received: from localhost (localhost [127.0.0.1])
+
Received: from localhost (localhost [127.0.0.1])
 
         by hcent7.lab.zimbra.com (Postfix) with ESMTP id 3D37AC5AAA0F
 
         by hcent7.lab.zimbra.com (Postfix) with ESMTP id 3D37AC5AAA0F
 
         for <admin@hcent7.lab.zimbra.com>; Wed, 24 Feb 2016 23:52:06 +0530 (IST)
 
         for <admin@hcent7.lab.zimbra.com>; Wed, 24 Feb 2016 23:52:06 +0530 (IST)
X-Virus-Scanned: amavisd-new at hcent7.lab.zimbra.com
+
X-Virus-Scanned: amavisd-new at hcent7.lab.zimbra.com
X-Spam-Flag: NO
+
X-Spam-Flag: NO
X-Spam-Score: -1.901
+
X-Spam-Score: -1.901
X-Spam-Level:
+
X-Spam-Level:
X-Spam-Status: No, score=-1.901 tagged_above=-10 required=6.6
+
X-Spam-Status: No, score=-1.901 tagged_above=-10 required=6.6
 
         tests=[BAYES_00=-1.9, NO_RELAYS=-0.001]
 
         tests=[BAYES_00=-1.9, NO_RELAYS=-0.001]
 
         autolearn=ham autolearn_force=no
 
         autolearn=ham autolearn_force=no
Received: from hcent7.lab.zimbra.com ([127.0.0.1])
+
Received: from hcent7.lab.zimbra.com ([127.0.0.1])
 
         by localhost (hcent7.lab.zimbra.com [127.0.0.1]) (amavisd-new, port 10024)
 
         by localhost (hcent7.lab.zimbra.com [127.0.0.1]) (amavisd-new, port 10024)
 
         with ESMTP id a7Vc0nHRYA7e for <admin@hcent7.lab.zimbra.com>;
 
         with ESMTP id a7Vc0nHRYA7e for <admin@hcent7.lab.zimbra.com>;
 
         Wed, 24 Feb 2016 23:51:56 +0530 (IST)
 
         Wed, 24 Feb 2016 23:51:56 +0530 (IST)
Received: by hcent7.lab.zimbra.com (Postfix, from userid 995)
+
Received: by hcent7.lab.zimbra.com (Postfix, from userid 995)
 
         id E2413C5A120D; Wed, 24 Feb 2016 23:51:55 +0530 (IST)
 
         id E2413C5A120D; Wed, 24 Feb 2016 23:51:55 +0530 (IST)
To: admin@hcent7.lab.zimbra.com
+
To: admin@hcent7.lab.zimbra.com
From: admin@hcent7.lab.zimbra.com
+
From: admin@hcent7.lab.zimbra.com
Subject: Service mailboxd stopped on hcent7.lab.zimbra.com
+
Subject: Service mailboxd stopped on hcent7.lab.zimbra.com
Message-Id: <20160224182155.E2413C5A120D@hcent7.lab.zimbra.com>
+
Message-Id: <20160224182155.E2413C5A120D@hcent7.lab.zimbra.com>
Date: Wed, 24 Feb 2016 23:51:55 +0530 (IST)
+
Date: Wed, 24 Feb 2016 23:51:55 +0530 (IST)
 +
 +
Feb 24 23:51:53 hcent7 zmconfigd[4430]: Service status change: hcent7.lab.zimbra.com mailboxd changed from running to stopped
 +
 +
'''<END OF BLOB>'''
 +
[00000840 - 00000871: 50 bytes; tstamp: 2016/02/24 23:58:07.775 IST] txn 1456338154.1 [CommitTxn] ver=1.42, tstamp=1456338487775, mailbox=1, txnType=CreateMessage
 +
[00000872 - 00000ead: 1596 bytes; tstamp: 2016/02/24 23:58:08.561 IST] txn 1456338154.2 [CreateMessage] ver=1.42, tstamp=1456338488561, change=20601, mailbox=1, id=2261, rcpt=admin@hcent7.lab.zimbra.com, rcvDate=1456338488561, shared=false, blobDigest="sIk30SXTvBw8lt+UjjBuVmoQcJJl6agNI+wHzBOJH8I=", size=1354, dataLen=1354, folder=2, conv=-1, convFirstMsgId=-1, calItemPartStat=NE, noICal=false, flags=512, tags=[], bodyType=1, path=/opt/zimbra/store/incoming/1456338152510-2.msg
  
Feb 24 23:51:53 hcent7 zmconfigd[4430]: Service status change: hcent7.lab.zimbra.com mailboxd changed from running to stopped
+
* '''Create a file that only has the content between ''<START OF BLOB>'' and ''<END OF BLOB>''. This is our actual blob file.'''
  
<END OF BLOB>
+
In our example, we create "''/tmp/message1_recover.msg''" with the blob content.
[00000840 - 00000871: 50 bytes; tstamp: 2016/02/24 23:58:07.775 IST] txn 1456338154.1 [CommitTxn] ver=1.42, tstamp=1456338487775, mailbox=1, txnType=CreateMessage
 
[00000872 - 00000ead: 1596 bytes; tstamp: 2016/02/24 23:58:08.561 IST] txn 1456338154.2 [CreateMessage] ver=1.42, tstamp=1456338488561, change=20601, mailbox=1, id=2261, rcpt=admin@hcent7.lab.zimbra.com, rcvDate=1456338488561, shared=false, blobDigest="sIk30SXTvBw8lt+UjjBuVmoQcJJl6agNI+wHzBOJH8I=", size=1354, dataLen=1354, folder=2, conv=-1, convFirstMsgId=-1, calItemPartStat=NE, noICal=false, flags=512, tags=[], bodyType=1, path=/opt/zimbra/store/incoming/1456338152510-2.msg
 
========================================================================================================
 
</pre>
 
  
 
+
Return-Path: admin@hcent7.lab.zimbra.com
====Step2 :====
+
Received: from hcent7.lab.zimbra.com (LHLO hcent7.lab.zimbra.com)
 
 
Now create a file that only has the content between <START OF BLOB> and <END OF BLOB>. This is our actual blob file.
 
 
 
Here we are creating a file "/tmp/message1_recover.msg" with the blob content (without dotted lines).
 
 
 
<pre>
 
----------------------------------------------------------------------------------------------
 
Return-Path: admin@hcent7.lab.zimbra.com
 
Received: from hcent7.lab.zimbra.com (LHLO hcent7.lab.zimbra.com)
 
 
  (10.15.8.19) by hcent7.lab.zimbra.com with LMTP; Wed, 24 Feb 2016 23:57:59
 
  (10.15.8.19) by hcent7.lab.zimbra.com with LMTP; Wed, 24 Feb 2016 23:57:59
 
  +0530 (IST)
 
  +0530 (IST)
Received: from localhost (localhost [127.0.0.1])
+
Received: from localhost (localhost [127.0.0.1])
 
         by hcent7.lab.zimbra.com (Postfix) with ESMTP id 3D37AC5AAA0F
 
         by hcent7.lab.zimbra.com (Postfix) with ESMTP id 3D37AC5AAA0F
        for <admin@hcent7.lab.zimbra.com>; Wed, 24 Feb 2016 23:52:06 +0530 (IST)
+
        for <admin@hcent7.lab.zimbra.com>; Wed, 24 Feb 2016 23:52:06 +0530 (IST)
X-Virus-Scanned: amavisd-new at hcent7.lab.zimbra.com
+
X-Virus-Scanned: amavisd-new at hcent7.lab.zimbra.com
X-Spam-Flag: NO
+
X-Spam-Flag: NO
X-Spam-Score: -1.901
+
X-Spam-Score: -1.901
X-Spam-Level:
+
X-Spam-Level:
X-Spam-Status: No, score=-1.901 tagged_above=-10 required=6.6
+
X-Spam-Status: No, score=-1.901 tagged_above=-10 required=6.6
 
         tests=[BAYES_00=-1.9, NO_RELAYS=-0.001]
 
         tests=[BAYES_00=-1.9, NO_RELAYS=-0.001]
 
         autolearn=ham autolearn_force=no
 
         autolearn=ham autolearn_force=no
Received: from hcent7.lab.zimbra.com ([127.0.0.1])
+
Received: from hcent7.lab.zimbra.com ([127.0.0.1])
 
         by localhost (hcent7.lab.zimbra.com [127.0.0.1]) (amavisd-new, port 10024)
 
         by localhost (hcent7.lab.zimbra.com [127.0.0.1]) (amavisd-new, port 10024)
 
         with ESMTP id a7Vc0nHRYA7e for <admin@hcent7.lab.zimbra.com>;
 
         with ESMTP id a7Vc0nHRYA7e for <admin@hcent7.lab.zimbra.com>;
 
         Wed, 24 Feb 2016 23:51:56 +0530 (IST)
 
         Wed, 24 Feb 2016 23:51:56 +0530 (IST)
Received: by hcent7.lab.zimbra.com (Postfix, from userid 995)
+
Received: by hcent7.lab.zimbra.com (Postfix, from userid 995)
 
         id E2413C5A120D; Wed, 24 Feb 2016 23:51:55 +0530 (IST)
 
         id E2413C5A120D; Wed, 24 Feb 2016 23:51:55 +0530 (IST)
To: admin@hcent7.lab.zimbra.com
+
To: admin@hcent7.lab.zimbra.com
From: admin@hcent7.lab.zimbra.com
+
From: admin@hcent7.lab.zimbra.com
Subject: Service mailboxd stopped on hcent7.lab.zimbra.com
+
Subject: Service mailboxd stopped on hcent7.lab.zimbra.com
Message-Id: <20160224182155.E2413C5A120D@hcent7.lab.zimbra.com>
+
Message-Id: <20160224182155.E2413C5A120D@hcent7.lab.zimbra.com>
Date: Wed, 24 Feb 2016 23:51:55 +0530 (IST)
+
Date: Wed, 24 Feb 2016 23:51:55 +0530 (IST)
 
 
Feb 24 23:51:53 hcent7 zmconfigd[4430]: Service status change: hcent7.lab.zimbra.com mailboxd changed from running to stopped
 
 
 
----------------------------------------------------------------------------------------------
 
</pre>
 
 
 
 
 
====Step3 :====
 
  
Now we will use "zmmailbox addMessage" to add blob "/tmp/message1_recover.msg" to a folder in user's mailbox.
+
Feb 24 23:51:53 hcent7 zmconfigd[4430]: Service status change: hcent7.lab.zimbra.com mailboxd changed from running to stopped
  
It's a better approach to create a separate folder to add extracted blob.
+
* '''Use "zmmailbox addMessage" to add blob "/tmp/message1_recover.msg" to a folder in user's mailbox'''
  
 +
The following commands add the blob to a folder (called 'Recovery') in the user's mailbox. The folder needs to be created before this:
 
  zmmailbox -z -m USER@DOMAIN.COM createFolder --view message /Recovery   
 
  zmmailbox -z -m USER@DOMAIN.COM createFolder --view message /Recovery   
 
  zmmailbox -z -m USER@DOMAIN.COM addMessage /Recovery /tmp/message1_recover.msg
 
  zmmailbox -z -m USER@DOMAIN.COM addMessage /Recovery /tmp/message1_recover.msg
  
 +
Check the newly added messages using the '''zmmailbox''' command or in the webclient.
  
====Step4 :====
+
====Extract Multiple Blobs====
 
 
Now check newly added messages using zmmailbox command or in the webclient.
 
 
 
 
 
Above procedure shows how we can extract single blob from a redolog file.
 
 
 
Now here we will show how to extract multiple blobs for a user from single redolog file or multiple redolog files/folder.
 
 
 
===Multiple Blob===
 
 
 
====Step1 :====
 
  
Create a folder to save all extracted blobs and :
+
The above procedure shows how we can extract single blob from a redolog file.
 +
We will now build on this, and show how to extract multiple blobs for a user from single redolog file or multiple redolog files/folder.
  
 +
*  Create a folder to save all extracted blobs
 
  mkdir /tmp/redodump_blobs
 
  mkdir /tmp/redodump_blobs
  
 
+
Now switch to the created folder and run following commands
====Step2 :====
 
 
 
Now switch to created folder in step1 and run following commands.
 
 
 
 
  cd /tmp/redodump_blobs
 
  cd /tmp/redodump_blobs
 +
zmredodump -m 2 /opt/zimbra/redolog/archive/ --show-blob | sed -n '/<START OF BLOB>/,/<END OF BLOB>/p' | awk -v n=0 '/^<START OF BLOB>/{close("blob"n);n++;next} {print > "blob"n".msg"}'
 +
perl -pi -e 's/<END OF BLOB>//g' ./blob*.msg
  
zmredodump -m 2 /opt/zimbra/redolog/archive/ --show-blob | sed -n '/<START OF BLOB>/,/<END OF BLOB>/p' | awk -v n=0 '/^<START OF BLOB>/{close("blob"n);n++;next} {print > "blob"n".msg"}' perl -pi -e 's/<END OF BLOB>//g' ./blob*.msg
+
The above commands will extract blobs and create separate files for each blob like the following:
 
 
Above commands will extract blobs and create separate files for each blob like the following:
 
  
<pre>
+
blob1.msg
blob1.msg
+
blob2.msg
blob2.msg
+
blob3.msg
blob3.msg
+
blob4.msg  
blob4.msg  
+
.....
.....
+
.....
.....
+
blobN.msg  
blobN.msg  
 
</pre>
 
 
 
====Step3 :====
 
 
 
Create a new folder in user's mailbox and add extracted blobs in bulk :
 
  
 +
*  Create a new folder in the user's mailbox and add the extracted blobs in bulk :
 
  zmmailbox -z -m USER@DOMAIN.COM createFolder --view message /Recovery
 
  zmmailbox -z -m USER@DOMAIN.COM createFolder --view message /Recovery
 
  zmmailbox -z -m USER@DOMAIN.COM addMessage /Recovery /tmp/redodump_blobs/
 
  zmmailbox -z -m USER@DOMAIN.COM addMessage /Recovery /tmp/redodump_blobs/
  
 
+
Check the newly added messages using 'the ''zmmailbox''' command or in the webclient.
====Step4 :====
 
 
 
Check newly added messages using zmmailbox command or in the webclient.
 
  
  
Heera Singh Koranga
 
  
"Heera Singh" <hsingh@zimbra.com>
+
{| class="wikitable" style="background-color:#d0f0c0;" cellpadding="10"
 +
|'''Submitted by''':  Heera Singh Koranga
 +
|}{{Article Footer|ZCS 8.8, 8.7, 8.6|2017-09-12}}

Latest revision as of 21:50, 8 August 2018

Extract blobs from redologs using zmredodump


   KB 23378        Last updated on 2018-08-8  




0.00
(0 votes)

Problem

Extracting blobs from redologs. Possible scenarios where this may be needed:

  • Extract blobs from redologs when we don't have a valid backup to perform the restore.
  • Extract blobs from redologs when full-backup is missing and only incremental backups are available.
  • Also, we can use this procedure when the redolog sequence is broken and we can't use zmredoplay.

Solution

This should only be used when there is no valid backup to perform the restore, as this is just a workaround to recover as much data as possible.

With the help of zmredodump we can extract the blob content in plain text format. The "--show-blob" option will print blobs and related operations (like move, delete, etc):

zmredodump -m <mailbox id> /path/of/redolog/file/redo.log --show-blob

The actual blob is present between the <START OF BLOB> and <END OF BLOB> markers

We then start the process to extract blobs for a user.

Extract a single blob

  • Process blob details from redolog file

In this example we are assuming user has mailbox-id "2".

zmredodump -m 2 /opt/zimbra/redolog/archive/redo-20160223.193045.991-seq63.log --show-blob 
VERIFYING DIRECTORY: /opt/zimbra/redolog/archive
VERIFYING: /opt/zimbra/redolog/archive/redo-20160224.193055.837-seq65.log
HEADER
------
sequence: 65
open:     0
filesize: 43948
serverId: c8e2f007-a1da-4660-ae89-15d3b2a8e200
created:  Wed, 2016/02/24 23:51:39.487 IST (1456338099487)
first op: Wed, 2016/02/24 23:58:01.933 IST (1456338481933)
last op:  Thu, 2016/02/25 01:00:43.108 IST (1456342243108)
version:  1.42
------
[00000200 - 0000083f: 1600 bytes; tstamp: 2016/02/24 23:58:01.933 IST] txn 1456338154.1 [CreateMessage] ver=1.42, tstamp=1456338481933, change=20600, mailbox=1, id=2260, rcpt=admin@hcent7.lab.zimbra.com, rcvDate=1456338481835, shared=false, blobDigest="m3biKcUSkugQxOXnEHVbi341JxT2OyeTnYUTlY6Gcyw=", size=1358, dataLen=1358, folder=2, conv=-1, convFirstMsgId=-1, calItemPartStat=NE, noICal=false, flags=512, tags=[], bodyType=1, path=/opt/zimbra/store/incoming/1456338152510-0.msg
<START OF BLOB>
Return-Path: admin@hcent7.lab.zimbra.com
Received: from hcent7.lab.zimbra.com (LHLO hcent7.lab.zimbra.com)
(10.15.8.19) by hcent7.lab.zimbra.com with LMTP; Wed, 24 Feb 2016 23:57:59
+0530 (IST)
Received: from localhost (localhost [127.0.0.1])
       by hcent7.lab.zimbra.com (Postfix) with ESMTP id 3D37AC5AAA0F
       for <admin@hcent7.lab.zimbra.com>; Wed, 24 Feb 2016 23:52:06 +0530 (IST)
X-Virus-Scanned: amavisd-new at hcent7.lab.zimbra.com
X-Spam-Flag: NO
X-Spam-Score: -1.901
X-Spam-Level:
X-Spam-Status: No, score=-1.901 tagged_above=-10 required=6.6
       tests=[BAYES_00=-1.9, NO_RELAYS=-0.001]
       autolearn=ham autolearn_force=no
Received: from hcent7.lab.zimbra.com ([127.0.0.1])
       by localhost (hcent7.lab.zimbra.com [127.0.0.1]) (amavisd-new, port 10024)
       with ESMTP id a7Vc0nHRYA7e for <admin@hcent7.lab.zimbra.com>;
       Wed, 24 Feb 2016 23:51:56 +0530 (IST)
Received: by hcent7.lab.zimbra.com (Postfix, from userid 995)
       id E2413C5A120D; Wed, 24 Feb 2016 23:51:55 +0530 (IST)
To: admin@hcent7.lab.zimbra.com
From: admin@hcent7.lab.zimbra.com
Subject: Service mailboxd stopped on hcent7.lab.zimbra.com
Message-Id: <20160224182155.E2413C5A120D@hcent7.lab.zimbra.com>
Date: Wed, 24 Feb 2016 23:51:55 +0530 (IST)

Feb 24 23:51:53 hcent7 zmconfigd[4430]: Service status change: hcent7.lab.zimbra.com mailboxd changed from running to stopped

<END OF BLOB>
[00000840 - 00000871: 50 bytes; tstamp: 2016/02/24 23:58:07.775 IST] txn 1456338154.1 [CommitTxn] ver=1.42, tstamp=1456338487775, mailbox=1, txnType=CreateMessage
[00000872 - 00000ead: 1596 bytes; tstamp: 2016/02/24 23:58:08.561 IST] txn 1456338154.2 [CreateMessage] ver=1.42, tstamp=1456338488561, change=20601, mailbox=1, id=2261, rcpt=admin@hcent7.lab.zimbra.com, rcvDate=1456338488561, shared=false, blobDigest="sIk30SXTvBw8lt+UjjBuVmoQcJJl6agNI+wHzBOJH8I=", size=1354, dataLen=1354, folder=2, conv=-1, convFirstMsgId=-1, calItemPartStat=NE, noICal=false, flags=512, tags=[], bodyType=1, path=/opt/zimbra/store/incoming/1456338152510-2.msg
  • Create a file that only has the content between <START OF BLOB> and <END OF BLOB>. This is our actual blob file.

In our example, we create "/tmp/message1_recover.msg" with the blob content.

Return-Path: admin@hcent7.lab.zimbra.com
Received: from hcent7.lab.zimbra.com (LHLO hcent7.lab.zimbra.com)
(10.15.8.19) by hcent7.lab.zimbra.com with LMTP; Wed, 24 Feb 2016 23:57:59
+0530 (IST)
Received: from localhost (localhost [127.0.0.1])
       by hcent7.lab.zimbra.com (Postfix) with ESMTP id 3D37AC5AAA0F
        for <admin@hcent7.lab.zimbra.com>; Wed, 24 Feb 2016 23:52:06 +0530 (IST)
X-Virus-Scanned: amavisd-new at hcent7.lab.zimbra.com
X-Spam-Flag: NO
X-Spam-Score: -1.901
X-Spam-Level:
X-Spam-Status: No, score=-1.901 tagged_above=-10 required=6.6
       tests=[BAYES_00=-1.9, NO_RELAYS=-0.001]
       autolearn=ham autolearn_force=no
Received: from hcent7.lab.zimbra.com ([127.0.0.1])
       by localhost (hcent7.lab.zimbra.com [127.0.0.1]) (amavisd-new, port 10024)
       with ESMTP id a7Vc0nHRYA7e for <admin@hcent7.lab.zimbra.com>;
       Wed, 24 Feb 2016 23:51:56 +0530 (IST)
Received: by hcent7.lab.zimbra.com (Postfix, from userid 995)
       id E2413C5A120D; Wed, 24 Feb 2016 23:51:55 +0530 (IST)
To: admin@hcent7.lab.zimbra.com
From: admin@hcent7.lab.zimbra.com
Subject: Service mailboxd stopped on hcent7.lab.zimbra.com
Message-Id: <20160224182155.E2413C5A120D@hcent7.lab.zimbra.com>
Date: Wed, 24 Feb 2016 23:51:55 +0530 (IST)
Feb 24 23:51:53 hcent7 zmconfigd[4430]: Service status change: hcent7.lab.zimbra.com mailboxd changed from running to stopped
  • Use "zmmailbox addMessage" to add blob "/tmp/message1_recover.msg" to a folder in user's mailbox

The following commands add the blob to a folder (called 'Recovery') in the user's mailbox. The folder needs to be created before this:

zmmailbox -z -m USER@DOMAIN.COM createFolder --view message /Recovery   
zmmailbox -z -m USER@DOMAIN.COM addMessage /Recovery /tmp/message1_recover.msg

Check the newly added messages using the zmmailbox command or in the webclient.

Extract Multiple Blobs

The above procedure shows how we can extract single blob from a redolog file. We will now build on this, and show how to extract multiple blobs for a user from single redolog file or multiple redolog files/folder.

  • Create a folder to save all extracted blobs
mkdir /tmp/redodump_blobs
  • Now switch to the created folder and run following commands
cd /tmp/redodump_blobs
zmredodump -m 2 /opt/zimbra/redolog/archive/ --show-blob | sed -n '/<START OF BLOB>/,/<END OF BLOB>/p' | awk -v n=0 '/^<START OF BLOB>/{close("blob"n);n++;next} {print > "blob"n".msg"}' 
perl -pi -e 's/<END OF BLOB>//g' ./blob*.msg

The above commands will extract blobs and create separate files for each blob like the following:

blob1.msg
blob2.msg
blob3.msg
blob4.msg 
.....
.....
blobN.msg 
  • Create a new folder in the user's mailbox and add the extracted blobs in bulk :
zmmailbox -z -m USER@DOMAIN.COM createFolder --view message /Recovery
zmmailbox -z -m USER@DOMAIN.COM addMessage /Recovery /tmp/redodump_blobs/
  • Check the newly added messages using 'the zmmailbox' command or in the webclient.


Submitted by: Heera Singh Koranga
Verified Against: ZCS 8.8, 8.7, 8.6 Date Created: 2017-09-12
Article ID: https://wiki.zimbra.com/index.php?title=Steps_to_extract_blobs_from_redologs_using_zmredodump Date Modified: 2018-08-08



Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »


Jump to: navigation, search