Steps to extract blobs from redologs using zmredodump: Difference between revisions
(Created page with "===<h1>Steps to extract blobs from redologs using zmredodump</h1>=== <hr> <br> <h2>Problem:</h2> 1st Scenario :- Extract blobs from redologs when we don't have a valid back...") |
No edit summary |
||
| Line 1: | Line 1: | ||
===<h1> | ===<h1>Extract blobs from redologs using zmredodump</h1>=== | ||
<hr> | <hr> | ||
<br> | <br> | ||
| Line 5: | Line 5: | ||
<h2>Problem:</h2> | <h2>Problem:</h2> | ||
1st Scenario :- Extract blobs from redologs when we don't have a valid backup to perform the restore.<br> | |||
OR | OR<br> | ||
2nd Scenario :- Extract blobs from redologs when full-backup is missing and only incremental backups are available.<br> | |||
OR | OR<br> | ||
3rd Scenario :- Also, we can use this procedure when the redolog sequence is broken and we can't use zmredoplay.<br> | |||
<h2> | <h2>Solution:</h2> | ||
This is a workaround to recover maximum available data, and should only be done when there is no valid backup to perform the restore. | This is a workaround to recover maximum available data, and should only be done when there is no valid backup to perform the restore.<br> | ||
With the help of "zmredodump" tool we can extract the blob content in plain text format. See the help and details of zmredodump using "zmredodump --help". | With the help of "zmredodump" tool we can extract the blob content in plain text format. See the help and details of zmredodump using "zmredodump --help". | ||
Zmredodump with option "--show-blob" will print blobs and related operations (like move, delete, etc), where the actual blob will be in between <START OF BLOB> and <END OF BLOB> markers. | Zmredodump with option "--show-blob" will print blobs and related operations (like move, delete, etc), where the actual blob will be in between '''<START OF BLOB>''' and '''<END OF BLOB>''' markers. | ||
This is the syntax of "zmredodump" command:- | This is the syntax of "zmredodump" command:- | ||
zmredodump -m <mailbox id> /path/of/redolog/file/redo.log --show-blob | zmredodump -m <mailbox id> /path/of/redolog/file/redo.log --show-blob | ||
Here we start the process | Here we start the process to extract blobs for a user from the redolog file using zmredodump command : | ||
===Single Blob=== | ===Extract A Single Blob=== | ||
'''1).''' Process blob details from redolog file. | |||
In this example we are assuming user has mailbox-id "2". | |||
zmredodump -m 2 /opt/zimbra/redolog/archive/redo-20160223.193045.991-seq63.log --show-blob | less | zmredodump -m 2 /opt/zimbra/redolog/archive/redo-20160223.193045.991-seq63.log --show-blob | less | ||
<pre> | <pre> | ||
VERIFYING DIRECTORY: /opt/zimbra/redolog/archive | VERIFYING DIRECTORY: /opt/zimbra/redolog/archive | ||
VERIFYING: /opt/zimbra/redolog/archive/redo-20160224.193055.837-seq65.log | VERIFYING: /opt/zimbra/redolog/archive/redo-20160224.193055.837-seq65.log | ||
| Line 84: | Line 79: | ||
[00000840 - 00000871: 50 bytes; tstamp: 2016/02/24 23:58:07.775 IST] txn 1456338154.1 [CommitTxn] ver=1.42, tstamp=1456338487775, mailbox=1, txnType=CreateMessage | [00000840 - 00000871: 50 bytes; tstamp: 2016/02/24 23:58:07.775 IST] txn 1456338154.1 [CommitTxn] ver=1.42, tstamp=1456338487775, mailbox=1, txnType=CreateMessage | ||
[00000872 - 00000ead: 1596 bytes; tstamp: 2016/02/24 23:58:08.561 IST] txn 1456338154.2 [CreateMessage] ver=1.42, tstamp=1456338488561, change=20601, mailbox=1, id=2261, rcpt=admin@hcent7.lab.zimbra.com, rcvDate=1456338488561, shared=false, blobDigest="sIk30SXTvBw8lt+UjjBuVmoQcJJl6agNI+wHzBOJH8I=", size=1354, dataLen=1354, folder=2, conv=-1, convFirstMsgId=-1, calItemPartStat=NE, noICal=false, flags=512, tags=[], bodyType=1, path=/opt/zimbra/store/incoming/1456338152510-2.msg | [00000872 - 00000ead: 1596 bytes; tstamp: 2016/02/24 23:58:08.561 IST] txn 1456338154.2 [CreateMessage] ver=1.42, tstamp=1456338488561, change=20601, mailbox=1, id=2261, rcpt=admin@hcent7.lab.zimbra.com, rcvDate=1456338488561, shared=false, blobDigest="sIk30SXTvBw8lt+UjjBuVmoQcJJl6agNI+wHzBOJH8I=", size=1354, dataLen=1354, folder=2, conv=-1, convFirstMsgId=-1, calItemPartStat=NE, noICal=false, flags=512, tags=[], bodyType=1, path=/opt/zimbra/store/incoming/1456338152510-2.msg | ||
</pre> | </pre> | ||
'''2).''' Now create a file that only has the content between '''<START OF BLOB>''' and '''<END OF BLOB>'''. This is our actual blob file. | |||
Here we are creating a file "''/tmp/message1_recover.msg''" with the blob content. | |||
Here we are creating a file "/tmp/message1_recover.msg" with the blob content | |||
<pre> | <pre> | ||
Return-Path: admin@hcent7.lab.zimbra.com | Return-Path: admin@hcent7.lab.zimbra.com | ||
Received: from hcent7.lab.zimbra.com (LHLO hcent7.lab.zimbra.com) | Received: from hcent7.lab.zimbra.com (LHLO hcent7.lab.zimbra.com) | ||
| Line 124: | Line 115: | ||
Feb 24 23:51:53 hcent7 zmconfigd[4430]: Service status change: hcent7.lab.zimbra.com mailboxd changed from running to stopped | Feb 24 23:51:53 hcent7 zmconfigd[4430]: Service status change: hcent7.lab.zimbra.com mailboxd changed from running to stopped | ||
</pre> | </pre> | ||
Now we will use "zmmailbox addMessage" to add blob "/tmp/message1_recover.msg" to a folder in user's mailbox. | '''3).''' Now we will use "zmmailbox addMessage" to add blob "/tmp/message1_recover.msg" to a folder in user's mailbox. | ||
It's a better approach to create a separate folder to add extracted blob. | It's a better approach to create a separate folder to add extracted blob. | ||
zmmailbox -z -m USER@DOMAIN.COM createFolder --view message /Recovery | zmmailbox -z -m USER@DOMAIN.COM createFolder --view message /Recovery | ||
zmmailbox -z -m USER@DOMAIN.COM addMessage /Recovery /tmp/message1_recover.msg | zmmailbox -z -m USER@DOMAIN.COM addMessage /Recovery /tmp/message1_recover.msg | ||
Now check newly added messages using zmmailbox command or in the webclient. | '''4).''' Now check newly added messages using '''zmmailbox''' command or in the webclient. | ||
| Line 147: | Line 135: | ||
Now here we will show how to extract multiple blobs for a user from single redolog file or multiple redolog files/folder. | Now here we will show how to extract multiple blobs for a user from single redolog file or multiple redolog files/folder. | ||
=== | <br> | ||
===Extract Multiple Blobs=== | |||
'''1).''' Create a folder to save all extracted blobs. | |||
mkdir /tmp/redodump_blobs | mkdir /tmp/redodump_blobs | ||
'''2).''' Now switch to created folder in step1 and run following commands. | |||
Now switch to created folder in step1 and run following commands. | |||
cd /tmp/redodump_blobs | cd /tmp/redodump_blobs | ||
zmredodump -m 2 /opt/zimbra/redolog/archive/ --show-blob | sed -n '/<START OF BLOB>/,/<END OF BLOB>/p' | awk -v n=0 '/^<START OF BLOB>/{close("blob"n);n++;next} {print > "blob"n".msg"}' | |||
zmredodump -m 2 /opt/zimbra/redolog/archive/ --show-blob | sed -n '/<START OF BLOB>/,/<END OF BLOB>/p' | awk -v n=0 '/^<START OF BLOB>/{close("blob"n);n++;next} {print > "blob"n".msg"}' perl -pi -e 's/<END OF BLOB>//g' ./blob*.msg | perl -pi -e 's/<END OF BLOB>//g' ./blob*.msg | ||
Above commands will extract blobs and create separate files for each blob like the following: | Above commands will extract blobs and create separate files for each blob like the following: | ||
| Line 176: | Line 161: | ||
</pre> | </pre> | ||
'''3).''' Create a new folder in user's mailbox and add extracted blobs in bulk : | |||
zmmailbox -z -m USER@DOMAIN.COM createFolder --view message /Recovery | zmmailbox -z -m USER@DOMAIN.COM createFolder --view message /Recovery | ||
zmmailbox -z -m USER@DOMAIN.COM addMessage /Recovery /tmp/redodump_blobs/ | zmmailbox -z -m USER@DOMAIN.COM addMessage /Recovery /tmp/redodump_blobs/ | ||
Check newly added messages using zmmailbox command or in the webclient. | '''4).''' Check newly added messages using '''zmmailbox''' command or in the webclient. | ||
Submitted by: Heera Singh Koranga | |||
Revision as of 05:51, 21 August 2017
Extract blobs from redologs using zmredodump
Problem:
1st Scenario :- Extract blobs from redologs when we don't have a valid backup to perform the restore.
OR
2nd Scenario :- Extract blobs from redologs when full-backup is missing and only incremental backups are available.
OR
3rd Scenario :- Also, we can use this procedure when the redolog sequence is broken and we can't use zmredoplay.
Solution:
This is a workaround to recover maximum available data, and should only be done when there is no valid backup to perform the restore.
With the help of "zmredodump" tool we can extract the blob content in plain text format. See the help and details of zmredodump using "zmredodump --help".
Zmredodump with option "--show-blob" will print blobs and related operations (like move, delete, etc), where the actual blob will be in between <START OF BLOB> and <END OF BLOB> markers.
This is the syntax of "zmredodump" command:-
zmredodump -m <mailbox id> /path/of/redolog/file/redo.log --show-blob
Here we start the process to extract blobs for a user from the redolog file using zmredodump command :
Extract A Single Blob
1). Process blob details from redolog file. In this example we are assuming user has mailbox-id "2".
zmredodump -m 2 /opt/zimbra/redolog/archive/redo-20160223.193045.991-seq63.log --show-blob | less
VERIFYING DIRECTORY: /opt/zimbra/redolog/archive
VERIFYING: /opt/zimbra/redolog/archive/redo-20160224.193055.837-seq65.log
HEADER
------
sequence: 65
open: 0
filesize: 43948
serverId: c8e2f007-a1da-4660-ae89-15d3b2a8e200
created: Wed, 2016/02/24 23:51:39.487 IST (1456338099487)
first op: Wed, 2016/02/24 23:58:01.933 IST (1456338481933)
last op: Thu, 2016/02/25 01:00:43.108 IST (1456342243108)
version: 1.42
------
[00000200 - 0000083f: 1600 bytes; tstamp: 2016/02/24 23:58:01.933 IST] txn 1456338154.1 [CreateMessage] ver=1.42, tstamp=1456338481933, change=20600, mailbox=1, id=2260, rcpt=admin@hcent7.lab.zimbra.com, rcvDate=1456338481835, shared=false, blobDigest="m3biKcUSkugQxOXnEHVbi341JxT2OyeTnYUTlY6Gcyw=", size=1358, dataLen=1358, folder=2, conv=-1, convFirstMsgId=-1, calItemPartStat=NE, noICal=false, flags=512, tags=[], bodyType=1, path=/opt/zimbra/store/incoming/1456338152510-0.msg
<START OF BLOB>
Return-Path: admin@hcent7.lab.zimbra.com
Received: from hcent7.lab.zimbra.com (LHLO hcent7.lab.zimbra.com)
(10.15.8.19) by hcent7.lab.zimbra.com with LMTP; Wed, 24 Feb 2016 23:57:59
+0530 (IST)
Received: from localhost (localhost [127.0.0.1])
by hcent7.lab.zimbra.com (Postfix) with ESMTP id 3D37AC5AAA0F
for <admin@hcent7.lab.zimbra.com>; Wed, 24 Feb 2016 23:52:06 +0530 (IST)
X-Virus-Scanned: amavisd-new at hcent7.lab.zimbra.com
X-Spam-Flag: NO
X-Spam-Score: -1.901
X-Spam-Level:
X-Spam-Status: No, score=-1.901 tagged_above=-10 required=6.6
tests=[BAYES_00=-1.9, NO_RELAYS=-0.001]
autolearn=ham autolearn_force=no
Received: from hcent7.lab.zimbra.com ([127.0.0.1])
by localhost (hcent7.lab.zimbra.com [127.0.0.1]) (amavisd-new, port 10024)
with ESMTP id a7Vc0nHRYA7e for <admin@hcent7.lab.zimbra.com>;
Wed, 24 Feb 2016 23:51:56 +0530 (IST)
Received: by hcent7.lab.zimbra.com (Postfix, from userid 995)
id E2413C5A120D; Wed, 24 Feb 2016 23:51:55 +0530 (IST)
To: admin@hcent7.lab.zimbra.com
From: admin@hcent7.lab.zimbra.com
Subject: Service mailboxd stopped on hcent7.lab.zimbra.com
Message-Id: <20160224182155.E2413C5A120D@hcent7.lab.zimbra.com>
Date: Wed, 24 Feb 2016 23:51:55 +0530 (IST)
Feb 24 23:51:53 hcent7 zmconfigd[4430]: Service status change: hcent7.lab.zimbra.com mailboxd changed from running to stopped
<END OF BLOB>
[00000840 - 00000871: 50 bytes; tstamp: 2016/02/24 23:58:07.775 IST] txn 1456338154.1 [CommitTxn] ver=1.42, tstamp=1456338487775, mailbox=1, txnType=CreateMessage
[00000872 - 00000ead: 1596 bytes; tstamp: 2016/02/24 23:58:08.561 IST] txn 1456338154.2 [CreateMessage] ver=1.42, tstamp=1456338488561, change=20601, mailbox=1, id=2261, rcpt=admin@hcent7.lab.zimbra.com, rcvDate=1456338488561, shared=false, blobDigest="sIk30SXTvBw8lt+UjjBuVmoQcJJl6agNI+wHzBOJH8I=", size=1354, dataLen=1354, folder=2, conv=-1, convFirstMsgId=-1, calItemPartStat=NE, noICal=false, flags=512, tags=[], bodyType=1, path=/opt/zimbra/store/incoming/1456338152510-2.msg
2). Now create a file that only has the content between <START OF BLOB> and <END OF BLOB>. This is our actual blob file.
Here we are creating a file "/tmp/message1_recover.msg" with the blob content.
Return-Path: admin@hcent7.lab.zimbra.com
Received: from hcent7.lab.zimbra.com (LHLO hcent7.lab.zimbra.com)
(10.15.8.19) by hcent7.lab.zimbra.com with LMTP; Wed, 24 Feb 2016 23:57:59
+0530 (IST)
Received: from localhost (localhost [127.0.0.1])
by hcent7.lab.zimbra.com (Postfix) with ESMTP id 3D37AC5AAA0F
for <admin@hcent7.lab.zimbra.com>; Wed, 24 Feb 2016 23:52:06 +0530 (IST)
X-Virus-Scanned: amavisd-new at hcent7.lab.zimbra.com
X-Spam-Flag: NO
X-Spam-Score: -1.901
X-Spam-Level:
X-Spam-Status: No, score=-1.901 tagged_above=-10 required=6.6
tests=[BAYES_00=-1.9, NO_RELAYS=-0.001]
autolearn=ham autolearn_force=no
Received: from hcent7.lab.zimbra.com ([127.0.0.1])
by localhost (hcent7.lab.zimbra.com [127.0.0.1]) (amavisd-new, port 10024)
with ESMTP id a7Vc0nHRYA7e for <admin@hcent7.lab.zimbra.com>;
Wed, 24 Feb 2016 23:51:56 +0530 (IST)
Received: by hcent7.lab.zimbra.com (Postfix, from userid 995)
id E2413C5A120D; Wed, 24 Feb 2016 23:51:55 +0530 (IST)
To: admin@hcent7.lab.zimbra.com
From: admin@hcent7.lab.zimbra.com
Subject: Service mailboxd stopped on hcent7.lab.zimbra.com
Message-Id: <20160224182155.E2413C5A120D@hcent7.lab.zimbra.com>
Date: Wed, 24 Feb 2016 23:51:55 +0530 (IST)
Feb 24 23:51:53 hcent7 zmconfigd[4430]: Service status change: hcent7.lab.zimbra.com mailboxd changed from running to stopped
3). Now we will use "zmmailbox addMessage" to add blob "/tmp/message1_recover.msg" to a folder in user's mailbox.
It's a better approach to create a separate folder to add extracted blob.
zmmailbox -z -m USER@DOMAIN.COM createFolder --view message /Recovery zmmailbox -z -m USER@DOMAIN.COM addMessage /Recovery /tmp/message1_recover.msg
4). Now check newly added messages using zmmailbox command or in the webclient.
Above procedure shows how we can extract single blob from a redolog file.
Now here we will show how to extract multiple blobs for a user from single redolog file or multiple redolog files/folder.
Extract Multiple Blobs
1). Create a folder to save all extracted blobs.
mkdir /tmp/redodump_blobs
2). Now switch to created folder in step1 and run following commands.
cd /tmp/redodump_blobs
zmredodump -m 2 /opt/zimbra/redolog/archive/ --show-blob | sed -n '/<START OF BLOB>/,/<END OF BLOB>/p' | awk -v n=0 '/^<START OF BLOB>/{close("blob"n);n++;next} {print > "blob"n".msg"}'
perl -pi -e 's/<END OF BLOB>//g' ./blob*.msg
Above commands will extract blobs and create separate files for each blob like the following:
blob1.msg blob2.msg blob3.msg blob4.msg ..... ..... blobN.msg
3). Create a new folder in user's mailbox and add extracted blobs in bulk :
zmmailbox -z -m USER@DOMAIN.COM createFolder --view message /Recovery zmmailbox -z -m USER@DOMAIN.COM addMessage /Recovery /tmp/redodump_blobs/
4). Check newly added messages using zmmailbox command or in the webclient.
Submitted by: Heera Singh Koranga
