Zimbra Desktop 2 Storage Migration

Revision as of 08:21, 26 February 2010 by Davidfraser (talk | contribs) (Added remainder of scripts)
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

Zimbra Desktop 2 Storage Migration

These are unofficial notes on how the store for items in Zimbra Desktop 2 differs from that in Zimbra Desktop 1, designed to aid in migration.

The rationale behind this is:

  • Zimbra Desktop 2 requires a totally fresh setup; no migration path from ZD1 is provided
  • I am living in a country with slow and expensive bandwidth
  • Our mail server is in a country with fast and cheap bandwidth

So my plan is:

  • Install Zimbra Desktop 2 somewhere with fast and cheap bandwidth to our mail server
  • Set up the same accounts as on Zimbra Desktop 1
  • Use rsync to efficiently transfer the Zimbra Desktop 2 setup without having to retransmit all messages

Complications that mean the installations don't exactly match:

  • The file store uses different names for the same mail items
  • Zimbra Desktop 2 uses gzip for all mail items over (approximately) 150kb; Zimbra Desktop 1 doesn't
  • The gzip compression doesn't seem to exactly match that produced by the command-line gzip tool at any setting, so we need to use the Java gzip classes to do compression
  • Zimbra Desktop 1 seems to store some strange items

Scripts

Requirements: Python, Java, Bash

These scripts should be saved to a common directory. Run

make

to compile the Java code. Run

upgradezd remote_host

to do the upgrade... This will require manually adjustment of the scripts...

Java gzip implementation

Save this as

gzip.java

:

import java.util.zip.GZIPOutputStream;
import java.io.FileOutputStream;
import java.io.FileInputStream;
import java.io.IOException;

public class gzip {
    public static void main(String args[]) {
        if(args.length<=0)
        {  
            System.out.println("Please enter the valid file name");
        }
        else
        {  
            try
            {  
                String inFilename = args[0];
                String outFilename = args[1];
                GZIPOutputStream out = new GZIPOutputStream(new FileOutputStream(outFilename));
                FileInputStream in = new FileInputStream(inFilename);
                // Transfer bytes from the input file to the gzip output stream
                byte[] buf = new byte[1024];
                int len;
                while ((len = in.read(buf)) > 0)
                {  
                    out.write(buf, 0, len);
                }
                in.close();
                out.finish();
                out.close();
            }
            catch(IOException e)
            {
              System.out.println("Exception has been thrown" + e);
            }
        }
    }
}

Java gunzip implementation

Save this as

gunzip.java

:

import java.util.zip.GZIPInputStream;
import java.io.FileOutputStream;
import java.io.FileInputStream;
import java.io.IOException;

public class gunzip {
    public static void main(String args[]) {
        if(args.length<=0)
        {  
            System.out.println("Please enter the valid file name");
        }
        else
        {  
            try
            {  
                String inFilename = args[0];
                String outFilename = args[1];
                GZIPInputStream gzipInputStream =
                new GZIPInputStream(new FileInputStream(inFilename));
                FileOutputStream out = new FileOutputStream(outFilename);
                // Transfer bytes from the compressed file to the output file
                byte[] buf = new byte[1024];
                int len;
                while ((len = gzipInputStream.read(buf)) > 0)
                {  
                    out.write(buf, 0, len);
                }
                gzipInputStream.close();
                out.close();
            }
            catch(IOException e)
            {
              System.out.println("Exception has been thrown" + e);
            }
        }
    }
}

Makefile

Save this as

Makefile

:

%.class: %.java
        javac $<

%.manifest: %.java
        echo "Main-Class: $(shell basename $<)" > $@

%.jar: %.manifest %.class
        jar -cfm $@ $+

all: gunzip.jar gzip.jar

Python renaming code

Save this as

zd2rename.py

:

#!/usr/bin/env python
import sys, os, re, logging
ZD2_STORE = sys.argv[1]
logging.getLogger().setLevel(logging.INFO)
FNAME_RE = re.compile("([0-9]+)-([0-9]+)[.]msg")
local_maps = {}
for line in sys.stdin:
    line = line.strip()
    if not line:
        continue
    dirname, fname = os.path.dirname(line), os.path.basename(line)
    fmatch = FNAME_RE.match(fname)
    if not fmatch:
        logging.warning("Unexpected filename: %s", filename)
        continue
    prefix, suffix = fmatch.group(1), fmatch.group(2)
    # print "mv %s/%s-*.msg %s/%s" % (dirname, prefix, dirname, fname)
    zd2_dirname = os.path.join(ZD2_STORE, dirname)
    if dirname not in local_maps:
        if not os.path.isdir(zd2_dirname):
            logging.warning("Directory %s not found locally", dirname)
            local_maps[dirname] = None
            continue
        local_map = {}
        for local_fname in os.listdir(zd2_dirname):
            local_fmatch = FNAME_RE.match(local_fname)
            if local_fmatch:
                local_map.setdefault(local_fmatch.group(1), []).append(local_fname)
        local_maps[dirname] = local_map
    else:
        local_map = local_maps[dirname]
        if local_map is None:
            continue
    local_matches = local_map.get(prefix, [])
    if len(local_matches) == 1:
        local_src, local_target = os.path.join(zd2_dirname, local_matches[0]), os.path.join(zd2_dirname, fname)
        if local_src != local_target:
            logging.info("Renaming %s/%s to %s/%s", dirname, local_matches[0], dirname, fname)
            os.rename(local_src, local_target)
    elif len(local_matches) > 1:
        # cat all the matches onto the file
        logging.warning("Unexpected multiple match for %s/%s: %s", dirname, " ".join(local_matches))
        open(os.path.join(zd2_dirname, fname), "wb").writelines(open(os.path.join(zd2_dirname, local_fname), "rb").read() for local_fname in local_matches)
    else:
        logging.info("File not found locally: %s/%s", dirname, fname)

Upgrade controller script

Save this as

upgradezd

, then

chmod a+x upgradezd

. Manually edit as required (see especially

TODO

):

#!/bin/bash

zd1_dir=$HOME/zimbra/zdesktop
zd2_dir=$HOME/zdesktop
src_dir="`dirname "$0"`"
src_dir="`cd "$src_dir" ; pwd`"

zd1_store=$zd1_dir/store
zd2_store=$zd2_dir/store

mkdir -p $zd2_dir

# initial transfer of store
rsync -a $zd1_store/ $zd2_store/

# gzip of all large blobs
(
    cd $zd2_store
    gzh=$'\x1f\x8b'
    find . -type f -name "*.msg" -size "+140k" -print0 | xargs -0 -n 1 -I % bash -c "gzh='$gzh' ; src_dir='$src_dir' ; "'{ h="`head -c 2 %`" ; [[ "$h" != "$gzh" ]] && echo % && java -jar $src_dir/gzip.jar % %.gz && mv %.gz % && ls -l % ; }'
)

# adjust store ids to match
(
    cd $zd2_store
    # TODO: manually adjust these instructions so that your store names match
    mv 0/4 0/3
)

# now rename to match names on the other side...
remote_host="$1"
[[ "$remote_host" == "" ]] && { echo syntax "$0" remote_host >&2 ; exit 1 ; }
remote_dir=zdesktop
remote_store=$remote_dir/store

ssh $remote_host "cd $remote_store ; find . -type f -name '*.msg'" | tee $src_dir/remote-listing | python zd2rename.py "$zd2_store"

rsync -avzP $remote_host:$remote_store/ $zd2_store/
Verified Against: Zimbra Desktop 2 Date Created: 2/25/2010
Article ID: https://wiki.zimbra.com/index.php?title=Zimbra_Desktop_2_Storage_Migration Date Modified: 2010-02-26



Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »

Jump to: navigation, search