Zimbra Desktop 2 Storage Migration: Difference between revisions

(Adding article footer and categories)
(Added remainder of scripts)
Line 20: Line 20:


== Scripts ==
== Scripts ==
Requirements: Python, Java, Bash
These scripts should be saved to a common directory. Run <pre>make</pre> to compile the Java code. Run <pre>upgradezd remote_host</pre> to do the upgrade... This will require manually adjustment of the scripts...


=== Java gzip implementation ===
=== Java gzip implementation ===


Save this as <pre>gzip.java</pre>:
<pre>
<pre>
import java.util.zip.GZIPOutputStream;
import java.util.zip.GZIPOutputStream;
Line 65: Line 70:
=== Java gunzip implementation ===
=== Java gunzip implementation ===


Save this as <pre>gunzip.java</pre>:
<pre>
<pre>
import java.util.zip.GZIPInputStream;
import java.util.zip.GZIPInputStream;
Line 107: Line 113:
=== Makefile ===
=== Makefile ===


Save this as <pre>Makefile</pre>:
<pre>
<pre>
%.class: %.java
%.class: %.java
Line 120: Line 127:
</pre>
</pre>


=== Python renaming code ===
Save this as <pre>zd2rename.py</pre>:
<pre>
#!/usr/bin/env python
import sys, os, re, logging
ZD2_STORE = sys.argv[1]
logging.getLogger().setLevel(logging.INFO)
FNAME_RE = re.compile("([0-9]+)-([0-9]+)[.]msg")
local_maps = {}
for line in sys.stdin:
    line = line.strip()
    if not line:
        continue
    dirname, fname = os.path.dirname(line), os.path.basename(line)
    fmatch = FNAME_RE.match(fname)
    if not fmatch:
        logging.warning("Unexpected filename: %s", filename)
        continue
    prefix, suffix = fmatch.group(1), fmatch.group(2)
    # print "mv %s/%s-*.msg %s/%s" % (dirname, prefix, dirname, fname)
    zd2_dirname = os.path.join(ZD2_STORE, dirname)
    if dirname not in local_maps:
        if not os.path.isdir(zd2_dirname):
            logging.warning("Directory %s not found locally", dirname)
            local_maps[dirname] = None
            continue
        local_map = {}
        for local_fname in os.listdir(zd2_dirname):
            local_fmatch = FNAME_RE.match(local_fname)
            if local_fmatch:
                local_map.setdefault(local_fmatch.group(1), []).append(local_fname)
        local_maps[dirname] = local_map
    else:
        local_map = local_maps[dirname]
        if local_map is None:
            continue
    local_matches = local_map.get(prefix, [])
    if len(local_matches) == 1:
        local_src, local_target = os.path.join(zd2_dirname, local_matches[0]), os.path.join(zd2_dirname, fname)
        if local_src != local_target:
            logging.info("Renaming %s/%s to %s/%s", dirname, local_matches[0], dirname, fname)
            os.rename(local_src, local_target)
    elif len(local_matches) > 1:
        # cat all the matches onto the file
        logging.warning("Unexpected multiple match for %s/%s: %s", dirname, " ".join(local_matches))
        open(os.path.join(zd2_dirname, fname), "wb").writelines(open(os.path.join(zd2_dirname, local_fname), "rb").read() for local_fname in local_matches)
    else:
        logging.info("File not found locally: %s/%s", dirname, fname)
</pre>
=== Upgrade controller script ===
Save this as <pre>upgradezd</pre>, then <pre>chmod a+x upgradezd</pre>. Manually edit as required (see especially <pre>TODO</pre>):
<pre>
#!/bin/bash
zd1_dir=$HOME/zimbra/zdesktop
zd2_dir=$HOME/zdesktop
src_dir="`dirname "$0"`"
src_dir="`cd "$src_dir" ; pwd`"
zd1_store=$zd1_dir/store
zd2_store=$zd2_dir/store
mkdir -p $zd2_dir
# initial transfer of store
rsync -a $zd1_store/ $zd2_store/
# gzip of all large blobs
(
    cd $zd2_store
    gzh=$'\x1f\x8b'
    find . -type f -name "*.msg" -size "+140k" -print0 | xargs -0 -n 1 -I % bash -c "gzh='$gzh' ; src_dir='$src_dir' ; "'{ h="`head -c 2 %`" ; [[ "$h" != "$gzh" ]] && echo % && java -jar $src_dir/gzip.jar % %.gz && mv %.gz % && ls -l % ; }'
)
# adjust store ids to match
(
    cd $zd2_store
    # TODO: manually adjust these instructions so that your store names match
    mv 0/4 0/3
)
# now rename to match names on the other side...
remote_host="$1"
[[ "$remote_host" == "" ]] && { echo syntax "$0" remote_host >&2 ; exit 1 ; }
remote_dir=zdesktop
remote_store=$remote_dir/store
ssh $remote_host "cd $remote_store ; find . -type f -name '*.msg'" | tee $src_dir/remote-listing | python zd2rename.py "$zd2_store"
rsync -avzP $remote_host:$remote_store/ $zd2_store/
</pre>


{{Article Footer|Zimbra Desktop 2|2/25/2010}}
{{Article Footer|Zimbra Desktop 2|2/25/2010}}

Revision as of 08:21, 26 February 2010

Zimbra Desktop 2 Storage Migration

These are unofficial notes on how the store for items in Zimbra Desktop 2 differs from that in Zimbra Desktop 1, designed to aid in migration.

The rationale behind this is:

  • Zimbra Desktop 2 requires a totally fresh setup; no migration path from ZD1 is provided
  • I am living in a country with slow and expensive bandwidth
  • Our mail server is in a country with fast and cheap bandwidth

So my plan is:

  • Install Zimbra Desktop 2 somewhere with fast and cheap bandwidth to our mail server
  • Set up the same accounts as on Zimbra Desktop 1
  • Use rsync to efficiently transfer the Zimbra Desktop 2 setup without having to retransmit all messages

Complications that mean the installations don't exactly match:

  • The file store uses different names for the same mail items
  • Zimbra Desktop 2 uses gzip for all mail items over (approximately) 150kb; Zimbra Desktop 1 doesn't
  • The gzip compression doesn't seem to exactly match that produced by the command-line gzip tool at any setting, so we need to use the Java gzip classes to do compression
  • Zimbra Desktop 1 seems to store some strange items

Scripts

Requirements: Python, Java, Bash

These scripts should be saved to a common directory. Run

make

to compile the Java code. Run

upgradezd remote_host

to do the upgrade... This will require manually adjustment of the scripts...

Java gzip implementation

Save this as

gzip.java

:

import java.util.zip.GZIPOutputStream;
import java.io.FileOutputStream;
import java.io.FileInputStream;
import java.io.IOException;

public class gzip {
    public static void main(String args[]) {
        if(args.length<=0)
        {  
            System.out.println("Please enter the valid file name");
        }
        else
        {  
            try
            {  
                String inFilename = args[0];
                String outFilename = args[1];
                GZIPOutputStream out = new GZIPOutputStream(new FileOutputStream(outFilename));
                FileInputStream in = new FileInputStream(inFilename);
                // Transfer bytes from the input file to the gzip output stream
                byte[] buf = new byte[1024];
                int len;
                while ((len = in.read(buf)) > 0)
                {  
                    out.write(buf, 0, len);
                }
                in.close();
                out.finish();
                out.close();
            }
            catch(IOException e)
            {
              System.out.println("Exception has been thrown" + e);
            }
        }
    }
}

Java gunzip implementation

Save this as

gunzip.java

:

import java.util.zip.GZIPInputStream;
import java.io.FileOutputStream;
import java.io.FileInputStream;
import java.io.IOException;

public class gunzip {
    public static void main(String args[]) {
        if(args.length<=0)
        {  
            System.out.println("Please enter the valid file name");
        }
        else
        {  
            try
            {  
                String inFilename = args[0];
                String outFilename = args[1];
                GZIPInputStream gzipInputStream =
                new GZIPInputStream(new FileInputStream(inFilename));
                FileOutputStream out = new FileOutputStream(outFilename);
                // Transfer bytes from the compressed file to the output file
                byte[] buf = new byte[1024];
                int len;
                while ((len = gzipInputStream.read(buf)) > 0)
                {  
                    out.write(buf, 0, len);
                }
                gzipInputStream.close();
                out.close();
            }
            catch(IOException e)
            {
              System.out.println("Exception has been thrown" + e);
            }
        }
    }
}

Makefile

Save this as

Makefile

:

%.class: %.java
        javac $<

%.manifest: %.java
        echo "Main-Class: $(shell basename $<)" > $@

%.jar: %.manifest %.class
        jar -cfm $@ $+

all: gunzip.jar gzip.jar

Python renaming code

Save this as

zd2rename.py

:

#!/usr/bin/env python
import sys, os, re, logging
ZD2_STORE = sys.argv[1]
logging.getLogger().setLevel(logging.INFO)
FNAME_RE = re.compile("([0-9]+)-([0-9]+)[.]msg")
local_maps = {}
for line in sys.stdin:
    line = line.strip()
    if not line:
        continue
    dirname, fname = os.path.dirname(line), os.path.basename(line)
    fmatch = FNAME_RE.match(fname)
    if not fmatch:
        logging.warning("Unexpected filename: %s", filename)
        continue
    prefix, suffix = fmatch.group(1), fmatch.group(2)
    # print "mv %s/%s-*.msg %s/%s" % (dirname, prefix, dirname, fname)
    zd2_dirname = os.path.join(ZD2_STORE, dirname)
    if dirname not in local_maps:
        if not os.path.isdir(zd2_dirname):
            logging.warning("Directory %s not found locally", dirname)
            local_maps[dirname] = None
            continue
        local_map = {}
        for local_fname in os.listdir(zd2_dirname):
            local_fmatch = FNAME_RE.match(local_fname)
            if local_fmatch:
                local_map.setdefault(local_fmatch.group(1), []).append(local_fname)
        local_maps[dirname] = local_map
    else:
        local_map = local_maps[dirname]
        if local_map is None:
            continue
    local_matches = local_map.get(prefix, [])
    if len(local_matches) == 1:
        local_src, local_target = os.path.join(zd2_dirname, local_matches[0]), os.path.join(zd2_dirname, fname)
        if local_src != local_target:
            logging.info("Renaming %s/%s to %s/%s", dirname, local_matches[0], dirname, fname)
            os.rename(local_src, local_target)
    elif len(local_matches) > 1:
        # cat all the matches onto the file
        logging.warning("Unexpected multiple match for %s/%s: %s", dirname, " ".join(local_matches))
        open(os.path.join(zd2_dirname, fname), "wb").writelines(open(os.path.join(zd2_dirname, local_fname), "rb").read() for local_fname in local_matches)
    else:
        logging.info("File not found locally: %s/%s", dirname, fname)

Upgrade controller script

Save this as

upgradezd

, then

chmod a+x upgradezd

. Manually edit as required (see especially

TODO

):

#!/bin/bash

zd1_dir=$HOME/zimbra/zdesktop
zd2_dir=$HOME/zdesktop
src_dir="`dirname "$0"`"
src_dir="`cd "$src_dir" ; pwd`"

zd1_store=$zd1_dir/store
zd2_store=$zd2_dir/store

mkdir -p $zd2_dir

# initial transfer of store
rsync -a $zd1_store/ $zd2_store/

# gzip of all large blobs
(
    cd $zd2_store
    gzh=$'\x1f\x8b'
    find . -type f -name "*.msg" -size "+140k" -print0 | xargs -0 -n 1 -I % bash -c "gzh='$gzh' ; src_dir='$src_dir' ; "'{ h="`head -c 2 %`" ; [[ "$h" != "$gzh" ]] && echo % && java -jar $src_dir/gzip.jar % %.gz && mv %.gz % && ls -l % ; }'
)

# adjust store ids to match
(
    cd $zd2_store
    # TODO: manually adjust these instructions so that your store names match
    mv 0/4 0/3
)

# now rename to match names on the other side...
remote_host="$1"
[[ "$remote_host" == "" ]] && { echo syntax "$0" remote_host >&2 ; exit 1 ; }
remote_dir=zdesktop
remote_store=$remote_dir/store

ssh $remote_host "cd $remote_store ; find . -type f -name '*.msg'" | tee $src_dir/remote-listing | python zd2rename.py "$zd2_store"

rsync -avzP $remote_host:$remote_store/ $zd2_store/
Verified Against: Zimbra Desktop 2 Date Created: 2/25/2010
Article ID: https://wiki.zimbra.com/index.php?title=Zimbra_Desktop_2_Storage_Migration Date Modified: 2010-02-26



Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »

Jump to: navigation, search