Mail Migration

Revision as of 20:23, 18 August 2009 by Msallee (talk | contribs) (using imapsync (Recommended Method))

from IMAP

using imapsync (Recommended Method)

Currently, the recommended method for migrating users to Zimbra from an existing IMAP server is with the imapsync tool written by Gilles Lamiral. When in doubt, please refer to the documentation for your version of imapsync for up-to-date options and settings.

Ubuntu 8.x will have this already in the repository so:

sudo aptitude install imapsync

GertThiel originally posted this Guide_to_imapsync in the zimbra forums.

related forum links:

Kerio

Only mail can be migrated at the moment. Using the following imapsync command to migrate emails from a single user to zimbra

   perl imapsync --buffersize 8192000 --nosyncacls --subscribe --syncinternaldates \
   --host1 kerio.host.com --user1 user@host.com --password1 keriopasswd --sep1 "/" --prefix1 "" \
   --host2 zimbra.host.com --user2 user@host.com --password2 zimbrapasswd

Dovecot

See User Migration From Dovecot With External LDAP

using YippieMove (Commercial Method)

Note: this was probably originally added by someone from YippieMove (User:Gettyless)

YippieMove is a SaaS for email migration. The service allows you move between any two email providers that supports IMAP, including Zimbra. No download is required and all transfers are performed on their servers. YippieMove charges $14.95 per account transfered (volume discount available). A screencast is available here.

POP3

pop2imap

pop2imap is a tool used to get a pop inbox and syncronise it to an IMAP folder. You can get it from [1]

It is very similar to imapsync above.

   pop2imap  \
   --host1 some.pop.com.au --user1 yourAccount --passfile1 yourPasswordFile \
   --host2 zimbra.imap.com.au --user2 yourZimbraAccount --passfile2 yourZimbraPasswordFile \
   --folder MyOldPOPMail

Will copy the INBOX on your pop account to MyOldPOPMail folder.

There are some problems with pop2imap though.

  • It will not create the folder, you must have that manually created
  • It does not support SSL, however this is easy to do with "stunnel" - providing the SSL tunnel to the server
  • It downloads all of the messages first to get headers, although some pop servers do this well, many do not and this is a very time consuming operation. Modifications to the script are simple to skip this step and force a copy of the mail (technically no longer a sync, but a straight copy).

imap_tools

imap_tools (http://www.athensfbc.com/imap_tools/) has has batch mode with lists built in for all activities...POP3-IMAP, IMAP-IMAP...

from Exchange

There are a least two ways to migrate the content of the individual MS Exchange user. First, is to use the Zimbra Migration Wizard. Second, is to export the contents of the MS Exchange folders to a .pst file and import them using the Zimbra PST Import Wizard. A third option is to export the MS Exchange data to a .pst file and restore it once the Zimbra profile is created for a corresponding Zimbra user either through the Zimbra Connector for Outlook or through IMAP.

See forums: Migration from Exchange

Information on using the Zimbra Exchange Migration Wizard can be found in the Migration Wizard for Exchange Installation Guide.

separate wiki page is here -- Migrating from Exchange

from Outlook

If you are using Outlook without Exchange, you can import the local mail folders directly into Zimbra with the PST Import Wizard.

from Lotus Notes/Domino

Lotus Notes/Domino to ZCS can be found in the Migration Wizard for Domino Installation Guide

from iMail

The first step is to export the users information including the password. I used this utility for CommuniGate Pro which creates a fixed length file. I then coverted it to a Tab-Delimited file with this utility so that I could use the Bulk Create process. After you create your Domains and provision the accounts, turn on IMAP4 for iMail and use the IMAP migration process.

from mbox files

The mbox file format is commonly used by many programs, most notably sendmail. The advantage of migrating from mbox files is that no passwords or special accounts are needed. There are several scripts that people have submitted to the wiki:


mbox > Maildir > Zimbra hash-dirs

If the mbox file can be converted to maildir format, you can use the zmmailbox utility to add messages to a user's mailbox. This will preserve the original time and date information, and will allow administrators to sort the mail into folders giong through IMAP.

OR zmlmtpinject

Usage for both outlined here: http://www.zimbra.com/forums/installation/12617-recover-data-store-folders.html#post64962

#!/usr/local/bin/perl
#
# mbox2maildir: coverts mbox file to maildir directory - the reverse of
# maildir2mbox from the qmail distribution.
#
# Usage: mbox2maildir uses the same environment variables as maildir2mbox:
# MAILDIR is the name of your maildir directory; MAIL is the name of your
# mbox file; MAILTMP is ignored.  MAIL is deleted after the conversion.
#
# WARNING: there is no locking; don't run more than one of these!  you
# have been warned.
#
# based on convert-and-create by Russell Nelson <nelson@qmail.org>
# kludged into this by Ivan Kohler <ivan@voicenet.com> 97-sep-17
 
require 'stat.pl';
 
local $SIG{HUP} = 'IGNORE';
local $SIG{INT} = 'IGNORE';
local $SIG{QUIT} = 'IGNORE'; 
local $SIG{TERM} = 'IGNORE';
local $SIG{TSTP} = 'IGNORE';
 
($name, $passwd, $uid, $gid, $quota, $comment, $gcos, $dir, $shell) =
  getpwuid($<);
 
die "fatal: home dir $dir doesn't exist\n" unless -e $dir;
&Stat($dir);
die "fatal: $name is $uid, but $dir is owned by $st_uid\n" if $uid != $st_uid;
 
chdir($dir) or die "fatal: unable to chdir to $dir\n";
$spoolname = "$ENV{MAILDIR}";
-d $spoolname or mkdir $spoolname,0700
  or die("fatal: $spoolname doesn't exist and can't be created.\n");
 
chdir($spoolname) or die("fatal: unable to chdir to $spoolname.\n");
-d "tmp" or mkdir("tmp",0700) or die("fatal: unable to make tmp/ subdir\n");
-d "new" or mkdir("new",0700) or die("fatal: unable to make new/ subdir\n");
-d "cur" or mkdir("cur",0700) or die("fatal: unable to make cur/ subdir\n");
 
open(SPOOL, "<$ENV{MAIL}")
  or die "Unable to open $ENV{$MAIL}\n";
$i = time;
while(<SPOOL>) {
  if (/^From /) {
    $fn = sprintf("new/%d.$$.mbox", $i);
    open(OUT, ">$fn") or die("fatal: unable to create new message");
    $i++;
    next;
  }
  s/^>From /From /;
  print OUT or die("fatal: unable to write to new message");
}
close(SPOOL);
close(OUT);
unlink("$ENV{MAIL}");

from Maildir

Note: If your wondering if these scripts work properly for subdirectories of subdirectories (and so on), imap stores it's mail on the filesystem by separating subdirectories by periods:

drwx------  6 temp eng     4096 Feb 16 17:12 .Test
drwx------  6 temp eng     4096 Apr 30 17:42 .Test.Test
drwx------  5 temp eng     4096 Apr 30 17:42 .Test.Test.Test

... that is why, for example, Jarosław Czarniak's bash script can just do a find on cur and new directories.

related forum posts

http://www.zimbra.com/forums/migration/24675-maildir-zimbra.html

https://www.zimbra.com/forums/administrators/10760-help-zmmailbox-addmessage.html

from Thunderbird Local Folders

lots of local folders => mbox => dovecot => imapsync

The easiest way I found for importing a lot of Thunderbird local folders is to set up a temporary dovecot server, convert the thunderbird mail stores to linux mbox mailstores and then imapsync it from the temp server to the zimbra server:

  • Install dovecot on a temporary server. On Debian etch: apt-get install dovecot-imapd, and then edit /etc/dovecot.conf and set the protocols to include imap. Restart dovecot. That is enough for our purposes.
  • Download the nstouwimap script here: http://www.cs.rice.edu/~dwallach/nstouwimap/. It will be used to convert Thunderbird mailboxes to mbox files that can be read by dovecot.
  • Copy All the local folders from all users to the server.
  • Create a transfer user on the dovecot server.
  • In the user's home directory, create a mail folder: /home/transferuser/mail.
  • For each mailbox, do the following (this can be scripted):
    • Clear the old mail folder on the server: rm -rf /home/transferuser/mail/*
    • Run nstouwimap /path/to/user/LocalFolders /home/transferuser/mail to convert the Local Folders to the transfer user's dovecot mailbox.
    • Run imapsync to transfer the account over to zimbra.

From Tobit David Zehn

(Might apply to other versions of Tobit David)

Note: thorough scripting experience required! Author suggests learning Perl :)

Step 1 - TAS → IMAP → Maildir

  • Set remote access credentials on the Archive folder in the Infocenter GUI. Here, we'll assume "export" for the user name, "zimbra" for the password.
  • Copy the archive.ini file from the main archive folder to every other folder. After this you may have to restart David. I'm not sure.
find /path/to/david/archive -type d -exec cp /path/to/david/archive/archive.ini {} \;
  • Start maserver if it isn't running, to provide IMAP access
/path/to/david/util/linux/DvISEctl start maserver
  • Install imapproxy to avoid that maserver crashes on disconnection, and configure imapproxy.conf to use a huge timeout. We'll assume port 9143 is available.
server_hostname localhost
cache_size 3072
listen_port 9143
server_port 143
cache_expiration_time 3000
proc_username nobody
proc_groupname nogroup
stat_filename /var/run/pimpstats
protocol_log_filename /var/log/imapproxy_protocol.log
syslog_facility LOG_MAIL
send_tcp_keepalives no
enable_select_cache no
foreground_mode no
force_tls no
chroot_directory /var/lib/imapproxy/chroot
enable_admin_commands no
  • Find or create something that maps the TAS (Tobit Archive System) by reading archive.dir files which list the subdirectories (aka subarchives, subfolders) per folder.
When scripting this, read 430 bytes per subarchive, ignore the first 113 bytes, then read three information elements per subarchive. (Perl unpack template: "x113 aZ* aZ* aZ*".) Each element consists of one identification byte (1 = internal name, 3 = path, 4 = display name, 38 = user) and a null-terminated string, and you'll need the displayed name to use as the IMAP folder name, and the path to know where to find the next archive.dir to traverse.
  • Find a way to cope with slashes in Tobit archive names; Zimbra has a pure distinction between path separators and allowed characters, whereas Tobit allows "/" in an archive name but also uses it as the path separator. For example, we chose to export "/Users/example/friends/family" as "/Users/example/friends~family".
  • Also, rename archives that have an & or " in them. You can use these in Zimbra, but you can't export these folders via IMAP. As far as I know, the safest way to rename these entries is to use the Infocenter GUI and do it manually.
  • Export the e-mail via IMAP. I found imapsync to be insufficiently capable of coping with Tobit's broken IMAP implementation, whereas mbsync did quite well. We successfully used the following .mbsyncrc per folder where $REMOTE is the IMAP folder name, and $LOCAL is the local Maildir name.
Expunge none
Create Both
Maildirstore local
 Path ~/Mail/
IMAPStore tobit
 Host 127.0.0.1
 Port 9143
 User export
 Pass zimbra
 RequireSSL no
Channel foo
 Master ":tobit:$REMOTE"
 Slave ":local:$LOCAL"
  • You'll note that this mbsync configuration exports only one folder. This is because mbsync doesn't retry if something went wrong, while Tobit's maserver gave errors a lot. Whenever that happens (mbsync doesn't exit 0), restart maserver and try the folder again. I'd suggest limiting the number of retries to 15 per folder, logging folders that didn't work out at all.
  • Exporting the mail is SLOW. It took five full days to export 33 GB of mail (1764 folders, 199020 messages) on a P4 2.4 GHz from SATA disks. You'll note that as mailboxes are bigger, the time needed to export them grows exponentially; as the number of messages in a folder increases, exporting also takes longer per message. (It appears (strace, hurrah!) that it reads every message of the folder just to expose a single one.)) I suggest keeping folders smaller than 2000 messages before you migrate. top(1) shows that maserver is very active while mbsync and imapproxy remain unnoticeable.

Step 2 - Maildir → Zimbra

This is surprisingly easy, compared to the complex task of pulling the mail out of Tobit David. For each user: create a text file with three zmmailbox commands per Maildir and execute zmmailbox -z -m username@example.com < $TEXTFILE

createFolder "/$FOLDERNAME"
addMessage "/$FOLDERNAME" "/path/to/$FOLDERNAME/cur"
addMessage "/$FOLDERNAME" "/path/to/$FOLDERNAME/new"

Unfortunately, the read/unread status is lost, and every message appears read in Zimbra. It'd be nice if addMessage had a switch to mark the messages as unread. I think this is not a big deal. You or your users may disagree.

It took two hours to import this on a Core2 Quad with SATA disks in RAID1.

Good luck! --Juerd 18:15, 26 May 2009 (UTC)

Jump to: navigation, search