Zimbra Desktop 2 Storage Migration

1. Zimbra Tech Center
2. Community Sandbox
3. Zimbra Desktop 2 Storage Migration

Zimbra Desktop 2 Storage Migration

 - This is archive documentation, which means it is not supported or valid for recent versions of Zimbra Collaboration.

 - This article is a Work in Progress, and may be unfinished or missing sections.

Zimbra Desktop 2 Storage Migration

Rationale and Plan

These are unofficial notes on how the store for items in Zimbra Desktop 2 differs from that in Zimbra Desktop 1, designed to aid in migration.

The rationale behind this is:

• Zimbra Desktop 2 requires a totally fresh setup; no migration path from ZD1 is provided
• I am living in a country with slow and expensive bandwidth
• Our mail server is in a country with fast and cheap bandwidth

So my plan is:

• Install Zimbra Desktop 2 somewhere with fast and cheap bandwidth to our mail server
• Set up the same accounts as on Zimbra Desktop 1
• Use rsync to efficiently transfer the Zimbra Desktop 2 setup without having to retransmit all messages
• Regenerate indexes instead of transferring them

Differences in Storage scheme

• Zimbra Desktop 2 uses sqlite instead of derby for the local database
• Zimbra Desktop 2 uses gzip for all mail items over (approximately) 140kb; Zimbra Desktop 1 doesn't

Complications that mean the installations don't exactly match:

• The file store uses different names for the same mail items
• The indexes produced (for text searching) will be different (and can be fairly large)
• The gzip compression doesn't seem to exactly match that produced by the command-line gzip tool at any setting, so we need to use the Java gzip classes to do compression
• Zimbra Desktop 1 seems to have stored some strange items on my machine

Scripts

Requirements: Python, Java, Bash

These scripts should be saved to a common directory. Run

make

to compile the Java code. Run

./upgradezd remote_host

to do the upgrade (where $remote_host is the ssh connectin string for the remote host and user)... This will require manually adjustment of the scripts...

Java gzip implementation

Save this as gzip.java:

import java.util.zip.GZIPOutputStream;
import java.io.FileOutputStream;
import java.io.FileInputStream;
import java.io.IOException;

public class gzip {
    public static void main(String args[]) {
        if(args.length<=0)
        {  
            System.out.println("Please enter the valid file name");
        }
        else
        {  
            try
            {  
                String inFilename = args[0];
                String outFilename = args[1];
                GZIPOutputStream out = new GZIPOutputStream(new FileOutputStream(outFilename));
                FileInputStream in = new FileInputStream(inFilename);
                // Transfer bytes from the input file to the gzip output stream
                byte[] buf = new byte[1024];
                int len;
                while ((len = in.read(buf)) > 0)
                {  
                    out.write(buf, 0, len);
                }
                in.close();
                out.finish();
                out.close();
            }
            catch(IOException e)
            {
              System.out.println("Exception has been thrown" + e);
            }
        }
    }
}

Java gunzip implementation

Save this as gunzip.java:

import java.util.zip.GZIPInputStream;
import java.io.FileOutputStream;
import java.io.FileInputStream;
import java.io.IOException;

public class gunzip {
    public static void main(String args[]) {
        if(args.length<=0)
        {  
            System.out.println("Please enter the valid file name");
        }
        else
        {  
            try
            {  
                String inFilename = args[0];
                String outFilename = args[1];
                GZIPInputStream gzipInputStream =
                new GZIPInputStream(new FileInputStream(inFilename));
                FileOutputStream out = new FileOutputStream(outFilename);
                // Transfer bytes from the compressed file to the output file
                byte[] buf = new byte[1024];
                int len;
                while ((len = gzipInputStream.read(buf)) > 0)
                {  
                    out.write(buf, 0, len);
                }
                gzipInputStream.close();
                out.close();
            }
            catch(IOException e)
            {
              System.out.println("Exception has been thrown" + e);
            }
        }
    }
}

Makefile

Save this as Makefile:

%.class: %.java
        javac $<

%.manifest: %.java
        echo "Main-Class: $(shell basename $<)" > $@

%.jar: %.manifest %.class
        jar -cfm $@ $+

all: gunzip.jar gzip.jar

Python renaming code

Save this as zd2rename.py:

#!/usr/bin/env python
import sys, os, re, logging
ZD2_STORE = sys.argv[1]
logging.getLogger().setLevel(logging.INFO)
FNAME_RE = re.compile("([0-9]+)-([0-9]+)[.]msg")
local_maps = {}
for line in sys.stdin:
    line = line.strip()
    if not line:
        continue
    dirname, fname = os.path.dirname(line), os.path.basename(line)
    fmatch = FNAME_RE.match(fname)
    if not fmatch:
        logging.warning("Unexpected filename: %s", filename)
        continue
    prefix, suffix = fmatch.group(1), fmatch.group(2)
    # print "mv %s/%s-*.msg %s/%s" % (dirname, prefix, dirname, fname)
    zd2_dirname = os.path.join(ZD2_STORE, dirname)
    if dirname not in local_maps:
        if not os.path.isdir(zd2_dirname):
            logging.warning("Directory %s not found locally", dirname)
            local_maps[dirname] = None
            continue
        local_map = {}
        for local_fname in os.listdir(zd2_dirname):
            local_fmatch = FNAME_RE.match(local_fname)
            if local_fmatch:
                local_map.setdefault(local_fmatch.group(1), []).append(local_fname)
        local_maps[dirname] = local_map
    else:
        local_map = local_maps[dirname]
        if local_map is None:
            continue
    local_matches = local_map.get(prefix, [])
    if len(local_matches) == 1:
        local_src, local_target = os.path.join(zd2_dirname, local_matches[0]), os.path.join(zd2_dirname, fname)
        if local_src != local_target:
            logging.info("Renaming %s/%s to %s/%s", dirname, local_matches[0], dirname, fname)
            os.rename(local_src, local_target)
    elif len(local_matches) > 1:
        # cat all the matches onto the file
        logging.warning("Unexpected multiple match for %s/%s: %s", dirname, " ".join(local_matches))
        open(os.path.join(zd2_dirname, fname), "wb").writelines(open(os.path.join(zd2_dirname, local_fname), "rb").read() for local_fname in local_matches)
    else:
        logging.info("File not found locally: %s/%s", dirname, fname)

Upgrade controller script

Save this as upgradezd, then run:

chmod a+x upgradezd

Manually edit as required (see especially the TODO about store numbers). Stop Zimbra desktop on both sides before running the upgrade.

Run with:

./upgradezd $remote_host

where $remote_host is the ssh connection string for the remote machine and user.

#!/bin/bash

zd1_dir=$HOME/zimbra/zdesktop
zd2_dir=$HOME/zdesktop
src_dir="`dirname "$0"`"
src_dir="`cd "$src_dir" ; pwd`"

zd1_store=$zd1_dir/store
zd2_store=$zd2_dir/store

mkdir -p $zd2_dir

# initial transfer of store
rsync -a $zd1_store/ $zd2_store/

# gzip of all large blobs
(
    cd $zd2_store
    gzh=$'\x1f\x8b'
    find . -type f -name "*.msg" -size "+140k" -print0 | xargs -0 -n 1 -I % bash -c "gzh='$gzh' ; src_dir='$src_dir' ; "'{ h="`head -c 2 %`" ; [[ "$h" != "$gzh" ]] && echo % && java -jar $src_dir/gzip.jar % %.gz && mv %.gz % && ls -l % ; }'
)

# adjust store ids to match
(
    cd $zd2_store
    # TODO: manually adjust these instructions so that your store names match
    mv 0/4 0/3
)

# now rename to match names on the other side...
remote_host="$1"
[[ "$remote_host" == "" ]] && { echo syntax "$0" remote_host >&2 ; exit 1 ; }
remote_dir=zdesktop
remote_store=$remote_dir/store

ssh $remote_host "cd $remote_store ; find . -type f -name '*.msg'" | tee $src_dir/remote-listing | python zd2rename.py "$zd2_store"

rsync -avzP $remote_host:$remote_store/ $zd2_store/

# after having completed install
rsync -avzP --exclude index $remote_host:$remote_store/ $zd2_dir/
rm -fr $zd2_dir/index/*

# now start zimbra desktop 2 and regenerate indexes