Mail Migration

From Zimbra :: Wiki

Jump to: navigation, search

Contents

from IMAP

using imapsync (Recommended Method)

Currently, the recommended method for migrating users to Zimbra from an existing IMAP server is with the imapsync tool written by Gilles Lamiral. When in doubt, please refer to the documentation for your version of imapsync for up-to-date options and settings.

Ubuntu 8.x will have this already in the repository so:

sudo aptitude install imapsync

GertThiel originally posted this Guide_to_imapsync in the zimbra forums.

related forum links:


Kerio

Only mail can be migrated at the moment. Using the following imapsync command to migrate emails from a single user to zimbra

   perl imapsync --buffersize 8192000 --nosyncacls --subscribe --syncinternaldates \
   --host1 kerio.host.com --user1 user@host.com --password1 keriopasswd --sep1 "/" --prefix1 "" \
   --host2 zimbra.host.com --user2 user@host.com --password2 zimbrapasswd

Dovecot

See Migrating from Dovecot with External LDAP

Cyrus

http://www.zimbra.com/forums/migration/6223-migrating-cyrus-zimbra-imapsync.html

Commercial Methods

YippieMove

Note: this was probably originally added by someone from YippieMove (User:Gettyless)

YippieMove is a SaaS for email migration. The service allows you move between any two email providers that supports IMAP, including Zimbra. No download is required and all transfers are performed on their servers. YippieMove charges $14.95 per account transfered (volume discount available). A screencast is available here.

MigrationWiz

MigrationWiz is a cloud-based email migration solution which can migrate between Zimbra and other systems (including POP, IMAP, Exchange, Google Apps, etc.). MigrationWiz's current price is $9.99 per mailbox, with volume discounts available.

MoveMyMail

MoveMyMail is a new service which focuses on IMAP email migrations. It is currently free while in beta. They do a good job of moving to/from Zimbra, Google Apps/Gmail, Yahoo, etc. It is online-only, and does not require any downloading. They have a bulk option available for free as well.

POP3

pop2imap

pop2imap is a tool used to get a pop inbox and syncronise it to an IMAP folder. You can get it from [1]

It is very similar to imapsync above.

   pop2imap  \
   --host1 some.pop.com.au --user1 yourAccount --passfile1 yourPasswordFile \
   --host2 zimbra.imap.com.au --user2 yourZimbraAccount --passfile2 yourZimbraPasswordFile \
   --folder MyOldPOPMail

Will copy the INBOX on your pop account to MyOldPOPMail folder.

There are some problems with pop2imap though.

  • It will not create the folder, you must have that manually created
  • It does not support SSL, however this is easy to do with "stunnel" - providing the SSL tunnel to the server
  • It downloads all of the messages first to get headers, although some pop servers do this well, many do not and this is a very time consuming operation. Modifications to the script are simple to skip this step and force a copy of the mail (technically no longer a sync, but a straight copy).

imap_tools

imap_tools (http://www.athensfbc.com/imap_tools/) has has batch mode with lists built in for all activities...POP3-IMAP, IMAP-IMAP...

Commercial Tools

Both MoveMyMail and MigrationWiz support POP3 -> IMAP migrations.

from Exchange

There are at least two ways to migrate the content of an individual MS Exchange user. The first is to use the Zimbra Migration Wizard. The second is to export the contents of the MS Exchange folders to a .pst file and import them using the Zimbra PST Import Wizard. A third option is to export the MS Exchange data to a .pst file and restore it once the Zimbra profile is created for a corresponding Zimbra user either through the Zimbra Connector for Outlook or through IMAP.

See forums: Migration from Exchange

Information on using the Zimbra Exchange Migration Wizard can be found in the Migration Wizard for Exchange Installation Guide.

separate wiki page is here -- Migrating from Exchange

from Outlook

If you are using Outlook without Exchange, you can import the local mail folders directly into Zimbra with the Microsoft Outlook PST Import Tool from the Download page.

from Lotus Notes/Domino

Lotus Notes/Domino to ZCS can be found in the Migration Wizard for Domino Installation Guide

from iMail

The first step is to export the users information including the password. I used this utility for CommuniGate Pro which creates a fixed length file. I then coverted it to a Tab-Delimited file with this utility so that I could use the Bulk Create process. After you create your Domains and provision the accounts, turn on IMAP4 for iMail and use the IMAP migration process.


from mbox files

The mbox file format is commonly used by many programs, most notably sendmail. The advantage of migrating from mbox files is that no passwords or special accounts are needed. There are several scripts that people have submitted to the wiki:


mbox > Maildir > Zimbra hash-dirs

If the mbox file can be converted to maildir format, you can use the zmmailbox utility to add messages to a user's mailbox. This will preserve the original time and date information, and will allow administrators to sort the mail into folders giong through IMAP.

OR zmlmtpinject

Usage for both outlined here: http://www.zimbra.com/forums/installation/12617-recover-data-store-folders.html#post64962

#!/usr/local/bin/perl
#
# mbox2maildir: coverts mbox file to maildir directory - the reverse of
# maildir2mbox from the qmail distribution.
#
# Usage: mbox2maildir uses the same environment variables as maildir2mbox:
# MAILDIR is the name of your maildir directory; MAIL is the name of your
# mbox file; MAILTMP is ignored.  MAIL is deleted after the conversion.
#
# WARNING: there is no locking; don't run more than one of these!  you
# have been warned.
#
# based on convert-and-create by Russell Nelson <nelson@qmail.org>
# kludged into this by Ivan Kohler <ivan@voicenet.com> 97-sep-17
 
require 'stat.pl';
 
local $SIG{HUP} = 'IGNORE';
local $SIG{INT} = 'IGNORE';
local $SIG{QUIT} = 'IGNORE'; 
local $SIG{TERM} = 'IGNORE';
local $SIG{TSTP} = 'IGNORE';
 
($name, $passwd, $uid, $gid, $quota, $comment, $gcos, $dir, $shell) =
  getpwuid($<);
 
die "fatal: home dir $dir doesn't exist\n" unless -e $dir;
&Stat($dir);
die "fatal: $name is $uid, but $dir is owned by $st_uid\n" if $uid != $st_uid;
 
chdir($dir) or die "fatal: unable to chdir to $dir\n";
$spoolname = "$ENV{MAILDIR}";
-d $spoolname or mkdir $spoolname,0700
  or die("fatal: $spoolname doesn't exist and can't be created.\n");
 
chdir($spoolname) or die("fatal: unable to chdir to $spoolname.\n");
-d "tmp" or mkdir("tmp",0700) or die("fatal: unable to make tmp/ subdir\n");
-d "new" or mkdir("new",0700) or die("fatal: unable to make new/ subdir\n");
-d "cur" or mkdir("cur",0700) or die("fatal: unable to make cur/ subdir\n");
 
open(SPOOL, "<$ENV{MAIL}")
  or die "Unable to open $ENV{$MAIL}\n";
$i = time;
while(<SPOOL>) {
  if (/^From /) {
    $fn = sprintf("new/%d.$$.mbox", $i);
    open(OUT, ">$fn") or die("fatal: unable to create new message");
    $i++;
    next;
  }
  s/^>From /From /;
  print OUT or die("fatal: unable to write to new message");
}
close(SPOOL);
close(OUT);
unlink("$ENV{MAIL}");

Here's another method which expands on the previous example. This method specifically targets mbox-style mail files in user home directories and the mbox in /var/spool/mail. Additionally, it grabs the Address Books from OpenWebmail. It then takes all this and sends it to Zimbra.

This method is now available on github: git://github.com/s1037989/owm2zcs.git

# cat /usr/local/bin/owm2zcs 

#!/usr/bin/perl

# Creates Zimbra account
# Adds an alias for First.Last@example.com
# Imports all mbox-style mail folders
# Imports all openwebmail-stored vCard address books

# <USAGE>

# Need an import.csv with exactly this format (tab-delimited):
# uid	password	displayName	sn	givenName

# On owmserver:
# Make sure there's enough diskspace on owmserver:/tmp or replace /tmp in this example
# with something else.  This could be 10, 50, 100 or more GB.
# export DOMAIN=example.com ; mkdir -p /tmp/owm2zcs ; cd /tmp/owm2zcs
# 1) Usage: /usr/local/bin/owm2zcs import.csv
# 2) Usage: prog | /usr/local/biz/owm2zcs

# find ./ -size +10M -exec cp -al --parents '{}' ../owm2zcs-large/ \;
# find ./ -size +10M -exec rm -f '{}' \;
# tar cvzpf - * | ssh zimbra "mkdir -p $(pwd) ; cd $(pwd) ; tar xvzpf - -C $(pwd) ; /opt/zimbra/bin/zmprov < $(pwd)/owm2zcs-11475.prov"
# - or - figure out how to increase max message upload size
# http://wiki.zimbra.com/wiki/Configuring_maxmessagesize

# On zimbraserver:
# Make sure there's enough diskspace on zimbraserver:/tmp or replace /tmp in this example
# with something else.  This could be 10, 50, 100 or more GB.
# Transfer the file structure from OWM to ZCS with SSH and TAR (Or probably more ideally with Rsync):
# 1) zimbraserver:/# ssh owmserver "tar cvzpf - /tmp/owm2zcs" | tar xvzpf - -C /
# 2) zimbraserver:/# export RSYNC_PASSWORD=password rsync -vazi rsync://owmserver/root/tmp/owm2zcs /tmp
# /opt/zimbra/bin/zmprov < owm2zcs.prov | tee 2>owm2zcs-err.log owm2zcs-out.log

# </USAGE>

# There's a few hard-coded items in here yet, so be careful.
# I used this script to migrate 60GB of mail files for 50 accounts today.
# Will be doing the remaining 450 tomorrow.

# Script by Stefan Adams <stefan@cogentinnovators.com> 2010-08-13
# based on the script above from wiki.zimbra.com's Mail_Migration page and...
# based on convert-and-create by Russell Nelson <nelson@qmail.org>
# kludged into this by Ivan Kohler <ivan@voicenet.com> 97-sep-17

# Convert mbox2Maildir
# find /tmp/Maildir/ -type d -name cur -o -name new -o -name tmp | perl -pi -e 's/\/(cur|new|tmp)$//' | sort -u | perl -MFile::Basename -pi -e 'chomp; $Maildir = $_; ($mbox) = /Maildir$/ ? ('mbox') : (/Maildir\/\.(.*?)$/); s/Maildir$/mail\/mbox/; s/Maildir\/\./mail\//; $root = dirname $_; $_ = "echo $_ ; mkdir -p \"$root\" ; env MAILDIR=\"$Maildir\" MAIL=\"$_\" MAILTMP=\"$root/tmp\" /var/qmail/bin/maildir2mbox\n"' | sh

$|=1;
#use strict;
#use warnings;
use File::Path;
use File::Basename;
use Text::vCard::Addressbook;

#local $SIG{HUP} = 'IGNORE';
#local $SIG{INT} = 'IGNORE';
#local $SIG{QUIT} = 'IGNORE'; 
#local $SIG{TERM} = 'IGNORE';
#local $SIG{TSTP} = 'IGNORE';

die "DOMAIN environment variable not set.\n" unless $ENV{DOMAIN};
$ENV{CONVMBOX} |= 'OWM';
$ENV{CONVABOOK} |= 'OWM Contacts';

my @import = ();
if ( $ARGV[0] ) {
	open IMPORT, $ARGV[0] or die "Cannot read $ARGV[0]: $!\n";
	@import = <IMPORT>;
	close IMPORT;
} else {
	@import = <STDIN>;
}

open PROV, ">owm2zcs-$$.prov" or die "Cannot create provisioning script: $!\n";
open CSV, ">owm2zcs-$$.csv" or die "Cannot create report-book: $!\n";
foreach ( @import ) {
	my $selectMailbox = 0;
	my @mailbox = ();
	my @abook = ();

	# Get details of user to import
	chomp;
	my ($uid, $password, $displayName, $sn, $givenName, $homeDirectory) = map { s/"//g; $_ } split /\t/;
	unless ( $homeDirectory ) {
		($uid, undef, undef, undef, undef, undef, undef, $homeDirectory) = getpwnam($uid) or do { print STDERR "$uid not found: $!\n"; next; };
	}
	do { print STDERR "$homeDirectory problem\n"; next; } unless -d $homeDirectory;
	$password = 'password1' if !$password || $password =~ /^\{\w+\}.+$/;
	$displayName ||= $uid;
	my ($f, $l);
	if ( $displayName =~ /^(\w+)\s+(\w+)$/ ) {
		($f, $l) = ($1, $2);
	} elsif ( $displayName =~ /^(\w+),\s*(\w+)$/ ) {
		($f, $l) = ($2, $1);
	} elsif ( $displayName =~ /^(\w+)\s+(\w+)\s+(\w+)$/ ) {
		($f, $l) = ($1, $3);
	} elsif ( $displayName =~ /^(\w+)\s+(\w+)\s*,/ ) {
		($f, $l) = ($1, $2);
	} else {
		my ($f) = ($displayName =~ /^(\w+)/);
		my ($l) = ($displayName =~ /(\w+)$/);
	}
	$givenName ||= $f;
	$sn ||= $l || $uid;
	
	my $alias = "$givenName.$sn";
	$alias =~ s/\s+//g;

	print "Converting for $uid ($homeDirectory).\n";
	
	unless ( $ENV{NOCREATE} ) {
		# Create Account and possibly an alias
		if ( $givenName ) {
			print PROV "createAccount $uid\@$ENV{DOMAIN} \"$password\" displayName \"$displayName\" sn \"$sn\" givenName \"$givenName\" zimbraPrefFromAddress $alias\@$ENV{DOMAIN} zimbraPrefReplyToAddress $alias\@$ENV{DOMAIN}\n";
			print PROV "addAccountAlias $uid\@$ENV{DOMAIN} $alias\@$ENV{DOMAIN}\n";
		} else {
			print PROV "createAccount $uid\@$ENV{DOMAIN} \"$password\" displayName \"$displayName\" sn \"$sn\"\n";
		}
	}
	
	unless ( $ENV{NOMBOX} ) {
		# Import mbox(es)
		foreach my $spool ( grep { -f $_ && -s _ && -r _ && `file "$_"` =~ /text/ } map { s/\"/\\"/g; $_ } glob("$homeDirectory/mail/*"), "/var/spool/mail/$uid" ) {
			my $oldspool = $spool;
			$spool = basename $oldspool;
			$spool = $spool eq 'mbox' || $oldspool eq "/var/spool/mail/$uid" ? 'Inbox' : "$spool";
			print "mbox | $oldspool -> $spool... ";
			-e $uid || mkdir "$uid" or do { print STDERR "Skipped: $!"; last; };
			do { print STDERR "Skipped."; next; } if -d "$uid/$spool";
			-e "$uid/$spool" || mkdir "$uid/$spool" or do { print STDERR "Skipped: $!"; last; };
			open SPOOL, "$oldspool" or do { print STDERR "Skipped: $!"; next; };
			$i = time;
			while ( <SPOOL> ) {
				if ( /^From / ) {
					$fn = sprintf("$uid/$spool/%d.$$.mbox", $i);
					open OUT, ">$fn" or do { print STDERR "Skipped: $!"; next; };
					$i++;
					next;
				}
				s/^>From /From /;
				print OUT;
			}
			close SPOOL;
			close OUT;
			print "Done.";
		} continue {
			print "\n";
			if ( -d "$uid/$spool" ) {
				push @mailbox, "createFolder \"/$ENV{CONVMBOX}/$spool\"\n";
				push @mailbox, "addMessage \"/$ENV{CONVMBOX}/$spool\" \"$uid/$spool\"\n";
				$selectMailbox = 1;
			}
		}
	}

	unless ( $ENV{NOABOOK} ) {
		# Import vcard(s)
		foreach my $abook ( grep { -f $_ && -s _ && -r _ && `file "$_"` =~ /text|vcard/i } map { s/\"/\\"/g; $_ } glob("$homeDirectory/.openwebmail/webaddr/*") ) {
			print "book | $abook... ";
			$newabook = basename $abook;
			my $address_book = Text::vCard::Addressbook->new({'source_file' => $abook}) or do { print STDERR "Skipped: $!"; next; };
	
			do { print STDERR "Skipped."; next; } if -e "$uid-$newabook.csv";
			open CSV, ">$uid-$newabook.csv" or do { print STDERR "Skipped: $!"; next; };
			print CSV "fullName,email,firstName,lastName\n";
			foreach my $vcard ( $address_book->vcards() ) {
				next unless $vcard->fullname() && $vcard->email();
				my @fullname = $vcard->fullname() =~ /,/ ? reverse split /\s+/, $vcard->fullname(), 2 : split /\s+/, $vcard->fullname(), 2;
				print CSV join ',', $vcard->fullname()||'', $vcard->email()||'', $fullname[0]||'', $fullname[1]||'';
				print CSV "\n";
			}
			close CSV;
			print "Done.";
		} continue {
			print "\n";
			if ( -e "$uid-$newabook.csv" && -s _ ) {
				push @abook, "createFolder -V contact \"/$ENV{CONVABOOK}/$newabook\"\n";
				push @abook, "postRestURL \"/$ENV{CONVABOOK}/$newabook\" \"$uid-$newabook.csv\"\n";
				$selectMailbox = 1;
			}
		}
	}

	# User actually has something to import
	if ( $selectMailbox ) {
		print PROV "selectMailbox $uid\@$ENV{DOMAIN}\n";
		print PROV "createFolder \"/$ENV{CONVMBOX}\"\n" if $#mailbox >= 0;
		print PROV "createFolder -V contact \"/$ENV{CONVABOOK}\"\n" if $#abook >= 0;
		print PROV @mailbox, @abook;
		print PROV "exit\n";
	}

	# Sign the report-book.
	print CSV "$uid\t$ENV{DOMAIN}\t$password\t$displayName\t$sn\t$givenName\n";
}
close CSV;
close PROV;

print "Inspect: owm2zcs-$$.{prov,csv}\n";
print "And then import:\n";
print "tar cvzpf - owm2zcs-$$.prov | ssh zimbra \"mkdir -p \$(pwd) ; cd \$(pwd) ; tar xvzpf - -C \$(pwd) ; /opt/zimbra/bin/zmprov < \$(pwd)/owm2zcs-$$.prov\"\n";

from Maildir

Merak

for migrating emails from Merak Mail server, use the following script This script worked with imap and pop3 maildirs

#!/usr/bin/python

# -*- coding: utf-8 -*-
"""
Created on Wed May  4 13:35:07 2011

@author: uchti (Denis Chumachenko)
"""

ZMMBOX="/opt/zimbra/bin/zmmailbox"
BFOLDER="oldbox"

import sys,os   



if len(sys.argv)>1 :
    os.path.realpath
    if sys.argv[1][-1]=='/':
        basedir=sys.argv[1]
    else:
        basedir=sys.argv[1]+'/'
    if os.path.exists(basedir):    
        n = raw_input("WARNING: Program will remove all empty files and *. dat files from the directory !"+"\nDo you agree? (y / n)")
        if n=='y' or n=='Y':
            findempty=os.popen('find '+basedir+' -size 0').readlines()  
            for fi in findempty:
                print 'Removeing :'+fi.replace('\n','')
                os.remove(fi.replace('\n',''))
            finddat=os.popen('find '+basedir+' -name *.dat').readlines()
            for fi in finddat:
                print 'Removeing :'+fi.replace('\n','')
                os.remove(fi.replace('\n',''))            
            for PATH, dirs, files in os.walk(basedir):            
                if PATH!=basedir:
                    DOMAIN=PATH.replace(basedir,'').split('/')[0]
                    USER=PATH.replace(basedir+DOMAIN+'/','').split('/')[0]
                    DIR=PATH.replace(basedir+DOMAIN+'/'+USER+'/','')
                    if DIR == PATH: DIR='';
                    if USER!='':
                        k=0
                        print '==============================='
                        print 'PATH = '+PATH
                        print "Search and modify imap files.." 
                        for f in files:                                               
                            fileExt=os.path.splitext(f)[-1]
                            if fileExt=='.imap':
                                fhash=f.replace(fileExt,'')[-8:]
                                fr=open(PATH+'/'+f,'r')
                                tmp=fr.readline()
                                tmp=tmp.replace('\n','').replace('\r','')
                                if tmp[:1]=='0' and tmp[-8:]==fhash:
                                    content=fr.read()      
                                    fw=open(PATH+'/'+f,'w+')
                                    fw.write(content)
                                    fw.close()
                                    k+=1
                                fr.close()
                        print "Modify : "+str(k)+" files"
                        print '-------------------------'
                        print 'Start import emails'                    
                        print 'DOMAIN = '+DOMAIN
                        print 'USER = '+USER
                        print 'DIR = '+DIR
                        createcmd=ZMMBOX+' -z -m '+USER+'@'+DOMAIN+' createFolder "/'+BFOLDER+'/'+DIR+'"' 
                        if DIR!='':
                            addmailcmd=ZMMBOX+' -z -m '+USER+'@'+DOMAIN+' addMessage "/'+BFOLDER+'/'+DIR+'/" "'+PATH+'"'
                        else:
                            addmailcmd=ZMMBOX+' -z -m '+USER+'@'+DOMAIN+' addMessage "/'+BFOLDER+'/" "'+PATH+'"'
                        os.system(createcmd)
                        os.system(addmailcmd)
    else:
        print "Path not exist"
        
else:
    print "Example: merak2zimbra_emails.py path_to_dir_with_domains"

related forum posts

http://www.zimbra.com/forums/migration/24675-maildir-zimbra.html

https://www.zimbra.com/forums/administrators/10760-help-zmmailbox-addmessage.html

from Thunderbird Local Folders

lots of local folders => mbox => dovecot => imapsync

The easiest way I found for importing a lot of Thunderbird local folders is to set up a temporary dovecot server, convert the thunderbird mail stores to linux mbox mailstores and then imapsync it from the temp server to the zimbra server:

  • Install dovecot on a temporary server. On Debian etch: apt-get install dovecot-imapd, and then edit /etc/dovecot.conf and set the protocols to include imap. Restart dovecot. That is enough for our purposes.
  • Download the nstouwimap script here: http://www.cs.rice.edu/~dwallach/nstouwimap/. It will be used to convert Thunderbird mailboxes to mbox files that can be read by dovecot.
  • Copy All the local folders from all users to the server.
  • Create a transfer user on the dovecot server.
  • In the user's home directory, create a mail folder: /home/transferuser/mail.
  • For each mailbox, do the following (this can be scripted):
    • Clear the old mail folder on the server: rm -rf /home/transferuser/mail/*
    • Run nstouwimap /path/to/user/LocalFolders /home/transferuser/mail to convert the Local Folders to the transfer user's dovecot mailbox.
    • Run imapsync to transfer the account over to zimbra.

From Tobit David Zehn

(Might apply to other versions of Tobit David)

Note: thorough scripting experience required! Author suggests learning Perl :)

Step 1 - TAS → IMAP → Maildir

  • Set remote access credentials on the Archive folder in the Infocenter GUI. Here, we'll assume "export" for the user name, "zimbra" for the password.
  • Copy the archive.ini file from the main archive folder to every other folder. After this you may have to restart David. I'm not sure.
find /path/to/david/archive -type d -exec cp /path/to/david/archive/archive.ini {} \;
  • Start maserver if it isn't running, to provide IMAP access
/path/to/david/util/linux/DvISEctl start maserver
  • Install imapproxy to avoid that maserver crashes on disconnection, and configure imapproxy.conf to use a huge timeout. We'll assume port 9143 is available.
server_hostname localhost
cache_size 3072
listen_port 9143
server_port 143
cache_expiration_time 3000
proc_username nobody
proc_groupname nogroup
stat_filename /var/run/pimpstats
protocol_log_filename /var/log/imapproxy_protocol.log
syslog_facility LOG_MAIL
send_tcp_keepalives no
enable_select_cache no
foreground_mode no
force_tls no
chroot_directory /var/lib/imapproxy/chroot
enable_admin_commands no
  • Find or create something that maps the TAS (Tobit Archive System) by reading archive.dir files which list the subdirectories (aka subarchives, subfolders) per folder.
When scripting this, read 430 bytes per subarchive, ignore the first 113 bytes, then read three information elements per subarchive. (Perl unpack template: "x113 aZ* aZ* aZ*".) Each element consists of one identification byte (1 = internal name, 3 = path, 4 = display name, 38 = user) and a null-terminated string, and you'll need the displayed name to use as the IMAP folder name, and the path to know where to find the next archive.dir to traverse.
  • Find a way to cope with slashes in Tobit archive names; Zimbra has a pure distinction between path separators and allowed characters, whereas Tobit allows "/" in an archive name but also uses it as the path separator. For example, we chose to export "/Users/example/friends/family" as "/Users/example/friends~family".
  • Also, rename archives that have an & or " in them. You can use these in Zimbra, but you can't export these folders via IMAP. As far as I know, the safest way to rename these entries is to use the Infocenter GUI and do it manually.
  • Export the e-mail via IMAP. I found imapsync to be insufficiently capable of coping with Tobit's broken IMAP implementation, whereas mbsync did quite well. We successfully used the following .mbsyncrc per folder where $REMOTE is the IMAP folder name, and $LOCAL is the local Maildir name.
Expunge none
Create Both
Maildirstore local
 Path ~/Mail/
IMAPStore tobit
 Host 127.0.0.1
 Port 9143
 User export
 Pass zimbra
 RequireSSL no
Channel foo
 Master ":tobit:$REMOTE"
 Slave ":local:$LOCAL"
  • You'll note that this mbsync configuration exports only one folder. This is because mbsync doesn't retry if something went wrong, while Tobit's maserver gave errors a lot. Whenever that happens (mbsync doesn't exit 0), restart maserver and try the folder again. I'd suggest limiting the number of retries to 15 per folder, logging folders that didn't work out at all.
  • Exporting the mail is SLOW. It took five full days to export 33 GB of mail (1764 folders, 199020 messages) on a P4 2.4 GHz from SATA disks. You'll note that as mailboxes are bigger, the time needed to export them grows exponentially; as the number of messages in a folder increases, exporting also takes longer per message. (It appears (strace, hurrah!) that it reads every message of the folder just to expose a single one.)) I suggest keeping folders smaller than 2000 messages before you migrate. top(1) shows that maserver is very active while mbsync and imapproxy remain unnoticeable.

Step 2 - Maildir → Zimbra

This is surprisingly easy, compared to the complex task of pulling the mail out of Tobit David. For each user: create a text file with three zmmailbox commands per Maildir and execute zmmailbox -z -m username@example.com < $TEXTFILE

createFolder "/$FOLDERNAME"
addMessage "/$FOLDERNAME" "/path/to/$FOLDERNAME/cur"
addMessage "/$FOLDERNAME" "/path/to/$FOLDERNAME/new"

Unfortunately, the read/unread status is lost, and every message appears read in Zimbra. It'd be nice if addMessage had a switch to mark the messages as unread. I think this is not a big deal. You or your users may disagree.

It took two hours to import this on a Core2 Quad with SATA disks in RAID1.

Good luck! --Juerd 18:15, 26 May 2009 (UTC)

from Desknow

There are the following that could be migrated over to Zimbra:

  1. Desknow mail accounts, mail folders and the mails themselves
  2. Desknow Files
  3. Desknow Message Boards
  4. Desknow Calendar
  5. Desknow Tasks
  6. Desknow Contacts

The focus here is on point 1.

Desknow Version

The study is based on:

  • Desknow v3.2.10 Lite Mode. (Note: This is why IMAPsync cannot be used. Desknow disables IMAP in Lite Mode.)
  • The Zimbra version is 5.0.16 Community Edition

Desknow : How does it store emails

The following is a typical directory layout of Desknow to store emails for each domain, mail account and mail folder:

desknowdata\
    usermail\
        {domain}\
            {user}\
                Inbox\
                    11a8ccfd307_N93M_34-0.eml
                    11a8ccfd307_N93M_50-0.eml
                    11a9e179bdc_49LX_8-0.eml
                    :
                    ... more mails ...
                    :
                Drafts\
                Sent Items\
                Deleted Items\
                Spam\
                {folder1}\
                    {sub-folder1}\
                    {sub-folder2}\
                    :
                    ... more sub-folders ...
                    :
                {folder2}\
                :
                ... more folders ...
                :

Desknow stores mails under the desknowdata\usermail directory.

On the next level, there are sub-directories for each domain in Desknow. That is the {domain} in the above.

Under each {domain}, there are sub-directories for each mail account. That is the {user} in the above.

Under each {user}, there are sub-directories for each mail folder. There are a number of preset folders as follows:

  • Inbox (same as Inbox in Zimbra)
  • Drafts (same as Drafts in Zimbra)
  • Sent Items (same as Sent in Zimbra)
  • Deleted Items (Same as Trash in Zimbra)
  • Spam (Same as Junk in Zimbra)

Shell Script to Automate the Migration

The Unix shell script dknw2zmbr (source code in later section that follows) will generate another shell script (name and location are configurable). When the generated shell script is executed, it will accomplish the mail accounts, mail folders and mail migration over from Desknow to Zimbra.

Customisation

The shell script has a number of variables that will allow customization to your Desknow installation and Zimbra installation. Please check the beginning of the shell script source code below.

Limitations

  • The password of the new mail accounts in Zimbra is always {user}1a2b3c E.g. password for 'chantaiman' is 'chantaiman1a2b3c'
  • The migration is now on one single domain. If there are multiple domains, the migration has to be executed multiple number of time, each with different customisation.
  • Some mails might not have the received date such as "Date :" or the date are not in recognisable format. In that case, the receive date in Zimbra will be set to the migration date/time.

To Do

  • Handle errors better. Eg. if a mail folder cannot be created, then skip the migration of the mails in the folder etc
  • Retain user password. Looks like 'zmprov cm' can be used. Yet, it seems that the encrypted passwords of the users might need to be obtained from Desknow beforehand. Need more study on how to obtain the Desknow encrypted passwords.

Source code of the shell script

#!/bin/bash
# ##############################################################################
#
# dknw2zmbr - migrate desknow mails to Zimbra
# 
# Usage : > sudo dknw2zmbr
#
# ######################## C O N F I G U R A T I O N S #########################
#---+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+
#
# Output Configurations
# ---------------------
# BATCH_FILE       : The generated shell script that actually execute the
#                    migration
# ERROR_CMMD_FILE  : For future use 
# FINISH_CMMD_FILE : For future use
# Z_SERVER         : Server name of the Zimbra installation
# Z_DOMAIN         : Destination domain in Zimbra
# Z_ADMIN_PASSWORD : Password for 'admin' in Zimbra
#
BATCH_FILE=/tmp/desknow2zimbra
ERROR_CMMD_FILE=/tmp/desknow2zimbra_error_commands
FINISH_CMMD_FILE=/tmp/desknow2zimbra_finish_commands
#
Z_SERVER=mail.xxxxx.com
Z_DOMAIN=xxxxx.com
Z_ADMIN_PASSWORD=PpAaSsWwRrDd123   # change this to your one
#
# Input Configurations
# --------------------
# DESKNOWDATA      : Location of the 'desknowdata\usermail\{domain}' directory
#
DESKNOWDATA=~/desknowmail/xxxxx.com
# ##############################################################################
#
export BATCH_FILE
export ERROR_CMMD_FILE
export FINISH_CMMD_FILE
export DESKNOWDATA 
export Z_SERVER
export Z_DOMAIN
export Z_ADMIN_PASSWORD

function exec_n_log () 
# 
# Descriptions
# ------------
# Execute the command passed as parameter $1.  Log errors to $ERROR_CMMD_FILE,
# and log successful commands to $FINISH_CMMD_FILE
#
# Parameters
# ----------
# $1 : the command to be executed
#
{
    echo "$1" >> $BATCH_FILE
#       #
#   $1  # I can't seem to make this work. I intended to do this to execute the
#       # command passed in as $1.  Yet, error occurs with my 'sed' command.
#       # Other simple command like 'ls -F' works.  Any idea?
#       #
    if [ $? -gt 0 ]
    then
        echo "$1" >> $ERROR_CMMD_FILE
        echo "ERROR :" "$1"
    else
        echo "$1" >> $FINISH_CMMD_FILE
    fi
}

export -f exec_n_log     # export this to make 'find ... -exec ...' work

function process_folders () {
#
# Descriptions
# ------------
# Called by 'find ... -exec ...'.  'find ... -exec ...' pass the directories/
# files to this fundtion one-by-one.  It then determines what to do based on
# whether it is a directory or a file.
#
# Parameters
# ----------
# $0 : the full path passed in from 'find ... -exec ...'
# $1 : the mail account
# 
    if [ -d "$0" ]
    # If it is a directory, use zmmailbox to create Zimbra folders
    then
        echo "DIR   :" $1 ":" "${0}"
        if [ "$0" != "." -a "$0" != "./Inbox" -a "$0" != "./Sent Items" -a "$0" != "./Drafts" -a "$0" != "./Deleted Items" -a "$0" != "./Spam" ]
        # Do not create those predefined Desknow folders which has Zimbra counterparts ...
        then
            exec_n_log "/opt/zimbra/bin/zmmailbox -a admin@${Z_DOMAIN} -p ${Z_ADMIN_PASSWORD} -m ${1}@${Z_DOMAIN} cf '${0#\.}'"
        fi
    elif [ -f "$0" ]
    # If it is a file, use curl and REST interface to import the mail ...
    then
        # get folder for hierachy from full path
        fpath=${0%/*}
        ufolder=${fpath#\.}

        # translate predefined Desknow folders into Zimbra counterparts
        ufolder=${ufolder/#\/Sent Items/\/Sent}
        ufolder=${ufolder/#\/Deleted Items/\Trash}
        ufolder=${ufolder/#\/Spam/\Junk}

        # encode special characters in folder name as the folder name will
        # become URL in REST interface.  *** ORDER MATTERS ***
        ufolder=${ufolder//\%/\%25}
        ufolder=${ufolder// /\%20}
        ufolder=${ufolder//\!/\%21}
        ufolder=${ufolder//\"/\%22}
        ufolder=${ufolder//\#/\%23}
        ufolder=${ufolder//\$/\%24}
        ufolder=${ufolder//\&/\%26}
        ufolder=${ufolder//\'/\%27}
        ufolder=${ufolder//\(/\%28}
        ufolder=${ufolder//\)/\%29}
        ufolder=${ufolder//\*/\%2A}
        ufolder=${ufolder//\+/\%2B}
        ufolder=${ufolder//\,/\%2C}
        ufolder=${ufolder//\-/\%2D}
        ufolder=${ufolder//\./\%2E}
        ufolder=${ufolder//\:/\%3A}
        ufolder=${ufolder//\;/\%3B}
        ufolder=${ufolder//\</\%3C}
        ufolder=${ufolder//\=/\%3D}
        ufolder=${ufolder//\>/\%3E}
        ufolder=${ufolder//\?/\%3F}
        ufolder=${ufolder//\@/\%40}
        ufolder=${ufolder//\[/\%5B}
        ufolder=${ufolder//\\/\%5C}
        ufolder=${ufolder//\]/\%5D}
        ufolder=${ufolder//\^/\%5E}
        ufolder=${ufolder//\_/\%5F}
        ufolder=${ufolder//\`/\%60}
        ufolder=${ufolder//\{/\%7B}
        ufolder=${ufolder//\|/\%7C}
        ufolder=${ufolder//\}/\%7D}
        ufolder=${ufolder//\~/\%7E}

        # get filename of the desknow mail file from full path
        ffile=${0##*/}

        exec_n_log "sed 's/\(^Date\:\)\(.*\)/X-Zimbra-Received\:\2\n\1\2/' \"`pwd`${0#\.}\" | curl -u ${1}@${Z_DOMAIN}:${1}1a2b3c -T - http://${Z_SERVER}/home/${1}@${Z_DOMAIN}${ufolder}"
  
        echo "FILE  :" $1 ":" "$0"
    else
        # What is it?
        echo "UKNWN :" $1 ":" "$0"
    fi
}

export -f process_folders     # export this to make 'find ... -exec ...' work

#
# Set up the output files
#
echo '#!/bin/bash' > $BATCH_FILE
echo "# `date`" >> $BATCH_FILE
cp $BATCH_FILE $ERROR_CMMD_FILE
cp $BATCH_FILE $FINISH_CMMD_FILE

#
# change to the desknowdata directory
#
pushd $DESKNOWDATA > /dev/null

#
# MAIN
# ----
# Loop through 'desknowdata/usermail/{domain}'.  Each entry should be
# a sub-directory with name being the mail account user.
#
for i in *
do
    if [ "$i" != "admin" ]
    # Do not create the 'admin' account, it is aready there...

    then

        # create the mail user account
        exec_n_log "/opt/zimbra/bin/zmprov ca ${i}@${Z_DOMAIN} ${i}1a2b3c displayName ${i}"
    fi

    # change directory into the mail user account...
    pushd $i > /dev/null

    # create mail folder and import mail one-by-one
    find . -exec /bin/bash -c 'process_folders "$1"' '{}' $i  \;

    # jump out from the user mial account
    popd > /dev/null  
done
popd > /dev/null

# make the generated shell script executable
chmod 755 $BATCH_FILE

Verified Against: unknown Date Created: 4/26/2009
Article ID: http://wiki.zimbra.com/index.php?title=Mail_Migration Date Modified: 07/14/2012
Personal tools