Guide to imapsync
the guide
Before you can use imapsync you must have both the source IMAP message store and ZCS up running and accessible to user accounts via IMAP. You can check that using an email client before starting the migration. You will need the login names (i.e. email addresses) and passwords for the users to be migrated. In addition, imapsync will not be able to authenticate to the Zimbra server until you enable clear text login for the IMAP service. You can set that option under the IMAP tab of the Global Settings or individual Server settings in the Zimbra Admin Console UI. There are also a few Perl module dependances, including; Digest::MD5, IO::Socket::SSL,Term::ReadKey, Digest:HMAC, libmail:imapclient (Mail::IMAPClient? requires exactly version 2.2.9, as of this writing), and Date::Manip. You can install these with your favorite package manager, from RPM's or with cpan. Finally, consider that imapsync will be a heavy load on your CPU and memory; the system running the migration will be less responsive.
Imapsync will run faster and consume less CPU with larger I/O buffers. The default is only 4KB; increasing it by one or two orders of magnitude is recommended for typical Zimbra-ready configurations. Examples below use 8MB buffer size.
For this example my existing IMAP server is running on server.gtds.lan and I set up Zimbra on a new machine named zimbra.gtds.lan.
imapsync --buffersize 8192000 --nosyncacls --subscribe --syncinternaldates \ --host1 server.gtds.lan --user1 yourAccount --password1 yourPassword \ --host2 zimbra.gtds.lan --user2 yourZimbraAccount --password2 yourZimbraPassword
Of course the complete command belongs on one line (signified by the backslashes: \).
A slightly more secure method is to write each password into a separate file, and then use the --passfile{1|2} options intead of the --password{1|2} options:
imapsync --buffersize 8192000 --nosyncacls --subscribe --syncinternaldates \ --host1 server.gtds.lan --user1 yourAccount --passfile1 yourPasswordFile \ --host2 zimbra.gtds.lan --user2 yourZimbraAccount --passfile2 yourZimbraPasswordFile
If your old IMAP server doesen't support NAMESPACE you have to also add appropriate options for example on iMail 6.0 --sep1 . --prefix1 INBOX. are required.
If you receive this error (and using the above command line you are pretty much guaranteed to): NO CAPABILITY for AUTHENTICATE LOGIN from your Zimbra server please add the switch --noauthmd5 to the end of your very long command line.
Your destination account on the Zimbra server should also have Clear Text Login enabled. You can enable this under the administration GUI in the Configuration menu, under Servers, in the IMAP tab of your server. Click the checkbox. It is assumed your source server also has Clear Text login enabled, but to do that please read the documentation of your respective server.
You may interrupt imapsync at any time with CTRL-C. Simply restart the same command again to resume the migration. You can also run it more than once to sync changes for a staged migration. Imapsync will not copy the same data twice, so you do not have to be concerned about duplicate messages.
I used imapsync to migrate from a Cyrus IMAPd to Zimbra. The ReadMe lists a number of other IMAP servers compatible with imapsync.
script to download and build imapsync and all of the required packages
#!/bin/bash # #CopyLeft 2006 Steve Fink #stevef-at-ublug.org # #This script will get all of #the necessary packages to #build imapsync 1.219 except OpenSSL #OpenSSL & it's development libraries are #required and OpenSSL is usually installed #already so check and get the matching libraries # #you have to run this script as root #or under sudo so it can do all the installs # #I don't recommend doing the extended tests #they mostly fail anyway but imapsync still works mkdir imapsync cd imapsync # wget http://search.cpan.org/CPAN/authors/id/D/DC/DCONWAY/Parse-RecDescent-1.94.tar.gz tar xvfpz Parse-RecDescent-1.94.tar.gz rm Parse-RecDescent-1.94.tar.gz cd Parse-RecDescent-1.94/ perl Makefile.PL make make install # cd .. wget http://search.cpan.org/CPAN/authors/id/G/GA/GAAS/Digest-MD5-2.33.tar.gz tar xvfpz Digest-MD5-2.33.tar.gz rm Digest-MD5-2.33.tar.gz cd Digest-MD5-2.33/ perl Makefile.PL make make install # cd .. wget http://search.cpan.org/CPAN/authors/id/J/JS/JSTOWE/TermReadKey-2.30.tar.gz tar xvfpz TermReadKey-2.30.tar.gz rm TermReadKey-2.30.tar.gz cd TermReadKey-2.30/ perl Makefile.PL make make install # cd .. wget http://search.cpan.org/CPAN/authors/id/A/AS/ASPA/IO-Socket-SSL-0.81.tar.gz tar xvfpz IO-Socket-SSL-0.81.tar.gz rm IO-Socket-SSL-0.81.tar.gz cd IO-Socket-SSL-0.81/ perl Makefile.PL make make install # cd .. wget http://search.cpan.org/CPAN/authors/id/F/FL/FLORA/Net_SSLeay.pm-1.30.tar.gz tar xvfpz Net_SSLeay.pm-1.30.tar.gz rm Net_SSLeay.pm-1.30.tar.gz cd Net_SSLeay.pm-1.30/ ./Makefile.PL -t make install # cd .. wget http://search.cpan.org/CPAN/authors/id/D/DJ/DJKERNEN/Mail-IMAPClient-2.2.9.tar.gz tar xvfpz Mail-IMAPClient-2.2.9.tar.gz rm Mail-IMAPClient-2.2.9.tar.gz cd Mail-IMAPClient-2.2.9/ perl Makefile.PL make make test make install # cd .. wget http://www.linux-france.org/prj/imapsync/dist/imapsync-1.219.tgz tar xvfpz imapsync-1.219.tgz rm imapsync-1.219.tgz cd imapsync-1.219/ perl -c imapsync make install # cd .. cd .. rm -Rf imapsync/
Save this script as buildimapsync, chmod 755 buildimapsync, sudo buildimapsync. Note that common build tools, such as make and a C compiler must be available.
Note: see Talk:User_Migration if you're having problems with the imapsync source.
Note: The current version (as of 2007 05 22) of imapsync is 1.219. If that last wget command returns 404 Not Found, check the parent folder for a newer version.
Note: The linux-france.org server has been unreachable for a very long time. Try a mirror site (v1.217) or (v1.223) if the link above does not work.
If you use version 1.223, once installed change line 1163 of /usr/bin/imapsync to this:
$d = UnixDate(ParseDate($d), "%d-%b-%Y %H:%M:%S %z");
(Thanks to Spremeau in the message boards for this tip)
Enabling Emails With Large File Attachments (Zimbra IMAP Max Size is 10mb)
The default Zimbra IMAP maximum email size is 10mb. This will cause errors to be raised during the imapsync for emails which are larger than 10mb. Errors such as the following will be displayed in the imapsync log.
+ Copying msg #131:11690588 to folder Folders/Projects/Consulting flags from : [\Seen]["16-Jul-2007 14:57:35 +1000"] Couldn't append msg #131 (Subject:[RE: Consultancy Approval]) to folder Folders/Projects/Consulting: 5160 BAD parse error: request too long
OR
+ Copying msg #87598:54935762 to folder INBOX flags from : [\Seen]["08-Feb-2008 14:09:05 -0500"] Couldn't append msg #87598 (Subject:[Re: photos]) to folder INBOX: 894 NO [TOOBIG] request too long
The Zimbra IMAP server uses zimbraFileUploadMaxSize to set the max size. The default is 10mb if it's not set. At this time the only way to set it is via the command line.
Use the following command as the zimbra user to increase to 50mb:
zmprov mcf zimbraFileUploadMaxSize 50000000
Then restart the mailboxd server process.
For more details see this thread, http://www.zimbra.com/forums/developers/1461-max-imap-message-size.html
UPDATE: Since Zimbra 5.0.6 the parameter has changed to zimbraMtaMaxMessageSize:
zmprov mcf zimbraMtaMaxMessageSize 50000000
See this bug report http://bugzilla.zimbra.com/show_bug.cgi?id=27610
imapsync over SSL
You can use SSL connections with imapsync providing "--ssl1" and/or "--ssl2" (and the "--port1 and/or --port2" if defaults are different) for each of the accounts you wish to migrate.
For example:
[root@freebsd ~]# imapsync \ --buffersize 8192000 --nosyncacls --subscribe --syncinternaldates \ --host1 oldmail.mydomain.com --user1 user1 --password1 secret1 --ssl1 \ --port1 993 \ --host2 zimbra.mydomain.com --user2 user2 --password2 secret2 --ssl2 \ --port2 993 \ --noauthmd5
Installing imapsync on FreeBSD 6.2:
1. Installing the binary:
[root@freebsd ~]# pkg_add -r imapsync
2. Installing the port (from latest sources):
[root@freebsd ~]# cd /usr/ports/mail/imapsync [root@freebsd ~]# make install clean
Make sure you have updated your ports with portsnap. If you are updating your ports for the first time:
[root@freebsd ~]# portsnap fetch [root@freebsd ~]# portsnap extract
More details on how to use portsnap:
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/ports-using.html)
The following version worked for me using option 2 above:
Mail::IMAPClient version is 2.2.9 and imapsync version 1.223 2007/10/30
Installing imapsync on Ubuntu 7.10:
[user1@ubuntu:~]$ sudo apt-get install imapsync
The command above will automatically build imapsync. You should be able to use the command above to accomplish an imapsync over SSL migration.
The following version worked for me on Ubuntu 7.10:
Mail::IMAPClient version is 2.2.9 and imapsync version 1.219 2007/04/04
NOTE: imapsync will not be able to authenticate to ZCS until you "Enable clear text login" for the IMAP service.
You can set that option under Global Settings or under Servers under IMAP in the ZCS Administration Console.
Batch Proccessing
You can automate multiple users by using this script.
#!/bin/bash logfile="sinklog.txt" host1=123.123.123.123 #host1 is Source host2=321.321.321.321 #host2 is Dest domain=xyz.com #domain is where email account is #everything after @ symbol ###### Do not modify past here ####################################### date=`date +%X_-_%x` echo "" >> $logfile echo "------------------------------------" >> $logfile echo "IMAPSync started.. $date" >> $logfile echo "" >> $logfile { while IFS=';' read u1 p1; do user=$u1"@"$domain echo "Syncing User $user" date=`date +%X_-_%x` echo "Start Syncing User $u1" echo "Starting $u1 $date" >> $logfile imapsync --nosyncacls --syncinternaldates --host1 $host1 --user1 "$user" --password1 \ "$p1"--host2 $host2 --user2 "$user" --password2 "$p1" date=`date +%X_-_%x` echo "User $user done" echo "Finished $user $date" >> $logfile echo "" >> $logfile done ; } < userlist.txt date=`date +%X_-_%x` echo "" >> $logfile echo "IMAPSync Finished.. $date" >> $logfile echo "------------------------------------" >> $logfile
Now create a CSV file with the user names and passwords in the following format.
user;password user2;password2
Here is a second batch script that will allow different usernames on the Source and Destination servers.
Name this file imapsyncbatch then chmod to 755 execute with ./imapsyncbatch can be used with the
--dry option for a dry run.
#!/bin/bash ##Modified by Steve Fink stevef-at-ublug.org ##This IMAPSync Batch Script is used when you have different ##usernames on the Source and Destination servers ##(kinda what IMAPSync was intended for) ##the format for the user-list.csv file is ##sourceusername,sourcepassword,destinationusername,destinationpassword ## Get the info while [ -z $infile ]; do echo "What is the path to the input file?" read infile done while [ -z $host1 ]; do echo "What is the Source Host? (mail1.domain.com)" read host1 done while [ -z $host2 ]; do echo "What is the Destination Host? (mail2.domain.com)" read host2 done while [ -z $domain ]; do echo "What is the Domain? (domain.com)" read domain done while [ -z $logfile ]; do echo "Where would you like the log? (synclog.txt)" read logfile done if [ ! -f $infile ] then { echo "The input file does not exist!" echo "" echo "What is the path to the input file?" read infile } fi INPUTFILE=$infile clear echo "" echo "" echo "IMAPSync is about to begin using:" echo "Input File $INPUTFILE" echo "Source Host $host1" echo "Destination Host $host2" echo "Domain $domain" echo "Log File $logfile" echo "" echo "" echo "Is this information correct?" echo "Press Enter to continue or" echo "Hit CTRL+C to start over" read wait ## Begin IMAPSync date=`date +%X_-_%x` echo "IMAPSync Logfile started @ $logfile" echo "" >> $logfile echo "------------------------------------" >> $logfile echo "IMAPSync started.. $date" >> $logfile echo "" >> $logfile #Get rid of the commas tr "," " " <$INPUTFILE | while read u1 p1 u2 p2 do user=$u1"@"$domain user2=$u2"@"$domain echo "Syncing User $user to $user2" date=`date +%X_-_%x` echo "Start Syncing User $user to $user2" echo "Starting $u1 $date" >> $logfile imapsync $1 --nosyncacls --syncinternaldates \ --exclude "#KnownSpam" --exclude "#FalsePositives" \ --exclude "Trash" --exclude "Deleted Items" \ --exclude "Deleted Messages" --exclude "Deleted" \ --exclude "Sent" --exclude "Sent Items" \ --exclude "Sent Messages" \ --host1 $host1 --user1 "$user" --password1 "$p1" \ --host2 $host2 --user2 "$user2" --password2 "$p2" date=`date +%X_-_%x` echo "User $user to $user2 done" echo "Finished $user to $user2 $date" >> $logfile echo "" >> $logfile done date=`date +%X_-_%x` echo "" >> $logfile echo "IMAPSync Finished.. $date" >> $logfile echo "------------------------------------" >> $logfile
The format for the .csv file for this script is: sourceusername,sourcepassword,destinationusername,destinationpassword
Batch processing without knowing the passwords
In the case your authentication backend stores user password in encrypted format and you don't know them, you may have the possibility to add a second authentication backend with the same users but a password of your choice.
During a cyrus imap to zimbra migration, here's how it worked :
- temporary openldap installed on third server, provisioned with the list of accounts to be migrated
- zimbra's external authentication configured with this ldap server (you can use 2 different ldap urls, only bind account must be the same)
- cyrus imap was already set up to use pam_ldap, which did not manage well the presence of 2 different ldap servers for authentication (if the official ldap server denied authentication, the second wasn't used)
- if cyrus imap relies on pam_ldap, you can switch to pam_mysql for example. Create a database, table with login and password, and feed...
With that in place, you can keep your cyrus and zimbra server running with no change for the users, but synchronize with the scripts showed above.