Guide to imapsync

From Zimbra :: Wiki

Jump to: navigation, search
Admin Article

Article Information

This article applies to the following ZCS versions.
  ZCS 5.0 Article  ZCS 5.0

Contents

The Guide

Before you can use imapsync you must have both the source IMAP message store and ZCS up running and accessible to user accounts via IMAP. You can check that using an email client before starting the migration. You will need the login names (i.e. email addresses) and passwords for the users to be migrated. In addition, imapsync will not be able to authenticate to the Zimbra server until you enable clear text login for the IMAP service. You can set that option under the IMAP tab of the Global Settings or individual Server settings in the Zimbra Admin Console UI. There are also a few Perl module dependances, including; Digest::MD5, IO::Socket::SSL,Term::ReadKey, Digest:HMAC, libmail:imapclient (Mail::IMAPClient? requires exactly version 2.2.9, as of this writing), and Date::Manip. You can install these with your favorite package manager, from RPM's or with cpan. Finally, consider that imapsync will be a heavy load on your CPU and memory; the system running the migration will be less responsive.

Imapsync will run faster and consume less CPU with larger I/O buffers. The default is only 4KB; increasing it by one or two orders of magnitude is recommended for typical Zimbra-ready configurations. Examples below use 8MB buffer size.

For this example my existing IMAP server is running on server.gtds.lan and I set up Zimbra on a new machine named zimbra.gtds.lan.

   imapsync --buffersize 8192000 --nosyncacls --subscribe --syncinternaldates \
   --host1 server.gtds.lan --user1 yourAccount --password1 yourPassword \
   --host2 zimbra.gtds.lan --user2 yourZimbraAccount --password2 yourZimbraPassword

Of course the complete command belongs on one line (signified by the backslashes: \).

A slightly more secure method is to write each password into a separate file, and then use the --passfile{1|2} options intead of the --password{1|2} options:

   imapsync --buffersize 8192000 --nosyncacls --subscribe --syncinternaldates \
   --host1 server.gtds.lan --user1 yourAccount --passfile1 yourPasswordFile \
   --host2 zimbra.gtds.lan --user2 yourZimbraAccount --passfile2 yourZimbraPasswordFile

If your old IMAP server doesen't support NAMESPACE you have to also add appropriate options for example on iMail 6.0 --sep1 . --prefix1 INBOX. are required.

If you receive this error (and using the above command line you are pretty much guaranteed to): NO CAPABILITY for AUTHENTICATE LOGIN from your Zimbra server please add the switch --noauthmd5 to the end of your very long command line.

Your destination account on the Zimbra server should also have Clear Text Login enabled. You can enable this under the administration GUI in the Configuration menu, under Servers, in the IMAP tab of your server. Click the checkbox. It is assumed your source server also has Clear Text login enabled, but to do that please read the documentation of your respective server.

You may interrupt imapsync at any time with CTRL-C. Simply restart the same command again to resume the migration. You can also run it more than once to sync changes for a staged migration. Imapsync will not copy the same data twice, so you do not have to be concerned about duplicate messages.

I used imapsync to migrate from a Cyrus IMAPd to Zimbra. The ReadMe lists a number of other IMAP servers compatible with imapsync.

script to download and build imapsync and all of the required packages

#!/bin/bash
#
#CopyLeft 2006 Steve Fink
#stevef-at-ublug.org
#
#This script will get all of
#the necessary packages to
#build imapsync 1.219 except OpenSSL
#OpenSSL & it's development libraries are
#required and OpenSSL is usually installed
#already so check and get the matching libraries
#
#you have to run this script as root
#or under sudo so it can do all the installs
#
#I don't recommend doing the extended tests
#they mostly fail anyway but imapsync still works


mkdir imapsync
cd imapsync
#
wget http://search.cpan.org/CPAN/authors/id/D/DC/DCONWAY/Parse-RecDescent-1.94.tar.gz
tar xvfpz Parse-RecDescent-1.94.tar.gz
rm Parse-RecDescent-1.94.tar.gz
cd Parse-RecDescent-1.94/
perl Makefile.PL
make
make install
#
cd ..
wget http://search.cpan.org/CPAN/authors/id/G/GA/GAAS/Digest-MD5-2.33.tar.gz
tar xvfpz Digest-MD5-2.33.tar.gz
rm Digest-MD5-2.33.tar.gz
cd Digest-MD5-2.33/
perl Makefile.PL
make
make install
#
cd ..
wget http://search.cpan.org/CPAN/authors/id/J/JS/JSTOWE/TermReadKey-2.30.tar.gz
tar xvfpz TermReadKey-2.30.tar.gz
rm TermReadKey-2.30.tar.gz
cd TermReadKey-2.30/
perl Makefile.PL
make
make install
#
cd ..
wget http://search.cpan.org/CPAN/authors/id/A/AS/ASPA/IO-Socket-SSL-0.81.tar.gz
tar xvfpz IO-Socket-SSL-0.81.tar.gz
rm IO-Socket-SSL-0.81.tar.gz
cd IO-Socket-SSL-0.81/
perl Makefile.PL
make
make install
#
cd ..
wget http://search.cpan.org/CPAN/authors/id/F/FL/FLORA/Net_SSLeay.pm-1.30.tar.gz
tar xvfpz Net_SSLeay.pm-1.30.tar.gz
rm Net_SSLeay.pm-1.30.tar.gz
cd Net_SSLeay.pm-1.30/
./Makefile.PL -t
make install
#
cd ..
wget http://search.cpan.org/CPAN/authors/id/D/DJ/DJKERNEN/Mail-IMAPClient-2.2.9.tar.gz
tar xvfpz Mail-IMAPClient-2.2.9.tar.gz
rm Mail-IMAPClient-2.2.9.tar.gz
cd Mail-IMAPClient-2.2.9/
perl Makefile.PL
make
make test
make install
#
cd ..
wget http://www.linux-france.org/prj/imapsync/dist/imapsync-1.219.tgz
tar xvfpz imapsync-1.219.tgz
rm imapsync-1.219.tgz
cd imapsync-1.219/
perl -c imapsync
make install
#
cd ..
cd ..
rm -Rf imapsync/

Save this script as buildimapsync, chmod 755 buildimapsync, sudo buildimapsync. Note that common build tools, such as make and a C compiler must be available.

Note: see Talk:User_Migration if you're having problems with the imapsync source.

Note: The current version (as of 2007 05 22) of imapsync is 1.219. If that last wget command returns 404 Not Found, check the parent folder for a newer version.

Note: The linux-france.org server has been unreachable for a very long time. Try a mirror site (v1.217) or (v1.223) if the link above does not work.

If you use version 1.223, once installed change line 1163 of /usr/bin/imapsync to this:

$d = UnixDate(ParseDate($d), "%d-%b-%Y %H:%M:%S %z");

(Thanks to Spremeau in the message boards for this tip)

IMAPSync Examples with Zimbra 7.x, RHEL 6.x / CentOS 6.0 and TLS Connections

On CentOS 6.x / RHEL 6.x you may install imapsync 1.488 which supports both SSL and TLS Auth. Since Zimbra default settings prohibit plaintext logins we use the following method to just test if IMAPSync would actually work

  imapsync --buffersize 8192000 --nosyncacls --subscribe --syncinternaldates --fast --dry \
  --host1 zimbra.firshost.com --user1 user@firsthost.com --password1 TotallySecret123 --tls1 \
  --host2 zimbra.secondhost.com --user2 user@secondhost.com --password2 TotallySecret456 --tls2

This will do nothing nothing but show if it would work. If you really want to migrate the data just remove the "--dry" option from above example.

Tweaking Zimbra performance

During large migrations via imapsync Zimbra performance can be a bottleneck. Change this setting to improve migration speed. It will startup only one indexing process per 20 emails, instead of one process per email. This setting is an "official" recommendation for large installs.

 zmprov mc <cos name> zimbraBatchedIndexingSize 20


Enabling Emails With Large File Attachments (Zimbra IMAP Max Size is 10mb)

The default Zimbra IMAP maximum email size is 10mb. This will cause errors to be raised during the imapsync for emails which are larger than 10mb. Errors such as the following will be displayed in the imapsync log.

+ Copying msg #131:11690588 to folder Folders/Projects/Consulting
flags from : [\Seen]["16-Jul-2007 14:57:35 +1000"]
Couldn't append msg #131 (Subject:[RE: Consultancy Approval]) to folder Folders/Projects/Consulting: 5160 BAD parse error: request too long

OR

+ Copying msg #87598:54935762 to folder INBOX
flags from : [\Seen]["08-Feb-2008 14:09:05 -0500"]
Couldn't append msg #87598 (Subject:[Re: photos]) to folder INBOX: 894 NO [TOOBIG] request too long

The Zimbra IMAP server uses zimbraFileUploadMaxSize to set the max size. The default is 10mb if it's not set. At this time the only way to set it is via the command line.

Use the following command as the zimbra user to increase to 50mb:

zmprov mcf zimbraFileUploadMaxSize 50000000

Then restart the mailboxd server process.

For more details see this thread, http://www.zimbra.com/forums/developers/1461-max-imap-message-size.html

UPDATE: Since Zimbra 5.0.6 the parameter has changed to zimbraMtaMaxMessageSize:

zmprov mcf zimbraMtaMaxMessageSize 50000000

See this bug report http://bugzilla.zimbra.com/show_bug.cgi?id=27610

imapsync over SSL

You can use SSL connections with imapsync providing "--ssl1" and/or "--ssl2" (and the "--port1 and/or --port2" if defaults are different) for each of the accounts you wish to migrate.

For example:

[root@freebsd ~]# imapsync \
--buffersize 8192000 --nosyncacls --subscribe --syncinternaldates \
--host1 oldmail.mydomain.com --user1 user1 --password1 secret1 --ssl1 \
--port1 993 \
--host2 zimbra.mydomain.com --user2 user2 --password2 secret2 --ssl2 \
--port2 993 \
--noauthmd5

Installing imapsync on FreeBSD 6.2:

1. Installing the binary:

[root@freebsd ~]# pkg_add -r imapsync

2. Installing the port (from latest sources):

[root@freebsd ~]# cd /usr/ports/mail/imapsync
[root@freebsd ~]# make install clean

Make sure you have updated your ports with portsnap. If you are updating your ports for the first time:

[root@freebsd ~]# portsnap fetch
[root@freebsd ~]# portsnap extract

More details on how to use portsnap:

http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/ports-using.html)

The following version worked for me using option 2 above:

Mail::IMAPClient version is 2.2.9 and imapsync version 1.223 2007/10/30


Installing imapsync on Ubuntu 7.10:

[user1@ubuntu:~]$ sudo apt-get install imapsync

The command above will automatically build imapsync. You should be able to use the command above to accomplish an imapsync over SSL migration.

The following version worked for me on Ubuntu 7.10:

Mail::IMAPClient version is 2.2.9 and imapsync version 1.219 2007/04/04


NOTE: imapsync will not be able to authenticate to ZCS until you "Enable clear text login" for the IMAP service. You can set that option under Global Settings or under Servers under IMAP in the ZCS Administration Console.

Batch Processing

You can automate multiple users by using this script.

    #!/bin/bash
    
    logfile="sinklog.txt"
    
    host1=123.123.123.123
    #host1 is Source
    
    host2=321.321.321.321
    #host2 is Dest
    
    domain=xyz.com
    #domain is where email account is
    #everything after @ symbol
    
    ###### Do not modify past here
    #######################################
    
    date=`date +%X_-_%x`
    
    echo "" >> $logfile
    echo "------------------------------------" >> $logfile
    echo "IMAPSync started..  $date" >> $logfile
    echo "" >> $logfile
    
    { while IFS=';' read  u1 p1; do 
            user=$u1"@"$domain    
    	     echo "Syncing User $user"
            date=`date +%X_-_%x`
            echo "Start Syncing User $u1"
            echo "Starting $u1 $date" >> $logfile
    imapsync --nosyncacls --syncinternaldates --host1 $host1 --user1 "$user" --password1 \ 
    "$p1"--host2 $host2 --user2 "$user" --password2 "$p1"
            date=`date +%X_-_%x`
            echo "User $user done"
            echo "Finished $user $date" >> $logfile
            echo "" >> $logfile
    
            done ; } < userlist.txt
    
    date=`date +%X_-_%x`
    
    echo "" >> $logfile
    echo "IMAPSync Finished..  $date" >> $logfile
    echo "------------------------------------" >> $logfile

Create a file userlist.txt with the user names and passwords in the following format:

    user;password
    user2;password2


Here is a second batch script that will allow different usernames on the Source and Destination servers. Name this file imapsyncbatch then chmod to 755 execute with ./imapsyncbatch can be used with the --dry option for a dry run.

        #!/bin/bash
        ##Modified by Steve Fink stevef-at-ublug.org
	##This IMAPSync Batch Script is used when you have different
	##usernames on the Source and Destination servers
	##(kinda what IMAPSync was intended for)
	##the format for the user-list.csv file is
	##sourceusername,sourcepassword,destinationusername,destinationpassword 
	
	## Get the info
		while [ -z $infile ]; do
			echo "What is the path to the input file?"
			read infile
		done
		while [ -z $host1 ]; do
			echo "What is the Source Host? (mail1.domain.com)"
			read host1
		done
		while [ -z $host2 ]; do
			echo "What is the Destination Host? (mail2.domain.com)"
			read host2
		done
		while [ -z $domain ]; do
			echo "What is the Domain? (domain.com)"
			read domain
		done
		while [ -z $logfile ]; do
			echo "Where would you like the log? (synclog.txt)"
			read logfile
		done

		if [ ! -f $infile ]
        	then
                	{
				echo "The input file does not exist!"
				echo ""
				echo "What is the path to the input file?"
				read infile
			}
		fi
		
		INPUTFILE=$infile
		
		clear
		echo ""
		echo ""
		echo "IMAPSync is about to begin using:"
		echo "Input File $INPUTFILE"
		echo "Source Host $host1"
		echo "Destination Host $host2"
		echo "Domain $domain"
		echo "Log File $logfile"
		echo ""
		echo ""
		echo "Is this information correct?"
                echo "Press Enter to continue or"
		echo "Hit CTRL+C to start over"
		read wait

	## Begin IMAPSync
		date=`date +%X_-_%x`
		echo "IMAPSync Logfile started @ $logfile"
		echo "" >> $logfile
		echo "------------------------------------" >> $logfile
    		echo "IMAPSync started..  $date" >> $logfile
    		echo "" >> $logfile

	#Get rid of the commas
		tr "," " " <$INPUTFILE | while read u1 p1 u2 p2
			do

        			user=$u1"@"$domain
           			user2=$u2"@"$domain
             			echo "Syncing User $user to $user2"
            			date=`date +%X_-_%x`
            			echo "Start Syncing User $user to $user2"
            			echo "Starting $u1 $date" >> $logfile
    				imapsync $1 --nosyncacls --syncinternaldates \
				--exclude "#KnownSpam" --exclude "#FalsePositives" \
				--exclude "Trash" --exclude "Deleted Items" \
				--exclude "Deleted Messages" --exclude "Deleted" \
				--exclude "Sent" --exclude "Sent Items" \
				--exclude "Sent Messages" \
				--host1 $host1 --user1 "$user" --password1 "$p1" \
				--host2 $host2 --user2 "$user2" --password2 "$p2"
            			date=`date +%X_-_%x`
            			echo "User $user to $user2 done"
            			echo "Finished $user to $user2 $date" >> $logfile
            			echo "" >> $logfile

            		done

    		date=`date +%X_-_%x`

    		echo "" >> $logfile
    		echo "IMAPSync Finished..  $date" >> $logfile
    		echo "------------------------------------" >> $logfile

The format for the .csv file for this script is: sourceusername,sourcepassword,destinationusername,destinationpassword

Batch processing without knowing the passwords

In the case your authentication backend stores user password in encrypted format and you don't know them, you may have the possibility to add a second authentication backend with the same users but a password of your choice.

During a cyrus imap to zimbra migration, here's how it worked :

  • temporary openldap installed on third server, provisioned with the list of accounts to be migrated
  • zimbra's external authentication configured with this ldap server (you can use 2 different ldap urls, only bind account must be the same)
  • cyrus imap was already set up to use pam_ldap, which did not manage well the presence of 2 different ldap servers for authentication (if the official ldap server denied authentication, the second wasn't used)
  • if cyrus imap relies on pam_ldap, you can switch to pam_mysql for example. Create a database, table with login and password, and feed...

With that in place, you can keep your cyrus and zimbra server running with no change for the users, but synchronize with the scripts showed above.

Migrating large Mailboxes and avoid timeouts

If you want to migrate large mailboxes (hundreds of folders and sub-folders containing several thousand messages) the default behaviour of imapsync may cause timeouts on the ZCS side. By default imapsync computes a lot of numbers for the sake of statistics only. By switching this off using

   –-nofoldersizes --skipsize –-fast

in addition to above options the sync will be much faster and timeouts due to the time needed for computation of statistics will not accour anymore.

Info on migrating from courier-imap and imapsync: http://wiki.zimbra.com/index.php?title=Imapsync_from_courier-imap_without_passwords[1]

Examples

$ imapsync \
  # imapsync with options that may help to optimize and modify disallowed folder names 
  --maxsize 52428800 --buffersize 52428800 \
  --fastio1 --fastio2 \
  --nofoldersizes --nosyncacls --subscribe --syncinternaldates \
  --noauthmd5 --authmech2 PLAIN \
  # do not migrate folders that looks like (ignore case) junk|spam|trash
  --exclude '(?i)\b(Junk|Spam|Trash)\b' \
  # avoid all X- headers when comparing messages
  '--skipheader 'X-*' \
  # avoid all nonstandard system flags (system flags start with '\')
  # RFC 3501: \Answered, \Flagged, \Deleted, \Seen, \Recent, \Draft
  # NOTE: need \\\\ because we want s/\\(?...) and Perl makes \\ => \
  --regexflag 's/\\\\(?!Answered|Flagged|Deleted|Seen|Recent|Draft)[^\s]*\s*//ig' \
  # translate folder names for characters ZCS does not allow
  # * colon (:) to hyphen (-)
  --regextrans2 's,:,-,g' \
  # double quote (") to single quote (')
  --regextrans2 's,\",'\'',g'
  # Remove spaces before separators and at the end of the folder name
  --regextrans2 's,\s+(?=/|$),,g'
  # change reserved folder names to something that is allowed
  --regextrans2 's,^(Briefcase|Calendar|Contacts|Emailed Contacts|Notebook|Tasks)(?=/|$), $1 Folder,ig' \
  ### site-variable options
  --host1 "$host1" --host2 "$host2" \
  --user1 "$username" --user2 "$username"
  --password1 "$pass1" --authuser2 "$auth2" --password2 "$pass2" \
  --regextrans2 's,\",-,g' \ # change quotes to dashes
  --regextrans2 's,&AAo-|&AA0ACg-|&AA0ACgANAAo-(?=/|$),,g' \
  --ssl1 --authmech1 PLAIN --maxcommandlength1 16384 \
  # --delete2 \ # be careful - this can be destructible if not used carefully
  # --dry --debug --debugimap \ # debug options
Verified Against: ZCS 5.0.6 & imapsync 1.219 Date Created: 4/24/2009
Article ID: http://wiki.zimbra.com/index.php?title=Guide_to_imapsync Date Modified: 08/8/2012
Personal tools