Guide to imapsync

From Zimbra :: Wiki

Jump to: navigation, search
Admin Article

Article Information

This article applies to the following ZCS versions.
  ZCS 5.0 Article  ZCS 5.0

Contents

The Guide

Before you can use imapsync you must have both the source IMAP message store and ZCS up running and accessible to user accounts via IMAP. You can check that using an email client before starting the migration. You will need the login names (i.e. email addresses) and passwords for the users to be migrated. In addition, imapsync will not be able to authenticate to the Zimbra server until you enable clear text login for the IMAP service. You can set that option under the IMAP tab of the Global Settings or individual Server settings in the Zimbra Admin Console UI. There are also a few Perl module dependances, including; Digest::MD5, IO::Socket::SSL,Term::ReadKey, Digest:HMAC, libmail:imapclient (Mail::IMAPClient? requires exactly version 2.2.9, as of this writing), and Date::Manip. You can install these with your favorite package manager, from RPM's or with cpan. Finally, consider that imapsync will be a heavy load on your CPU and memory; the system running the migration will be less responsive.

Imapsync may run faster and consume less CPU with larger I/O buffers, but no benchmarking of this has been performed. The default buffer size is 4KB; increasing it by one or two orders of magnitude is not uncommon for typical Zimbra-ready configurations. To increase the buffersize to 8MB, add the arguments "--buffersize 8192000" when running imapsync.

For this example my existing IMAP server is running on server.gtds.lan and I set up Zimbra on a new machine named zimbra.gtds.lan.

   imapsync --nosyncacls --subscribe --syncinternaldates \
   --host1 server.gtds.lan --user1 yourAccount --password1 yourPassword \
   --host2 zimbra.gtds.lan --user2 yourZimbraAccount --password2 yourZimbraPassword

Of course the complete command belongs on one line (signified by the backslashes: \).

A slightly more secure method is to write each password into a separate file, and then use the --passfile{1|2} options intead of the --password{1|2} options:

   imapsync --nosyncacls --subscribe --syncinternaldates \
   --host1 server.gtds.lan --user1 yourAccount --passfile1 yourPasswordFile \
   --host2 zimbra.gtds.lan --user2 yourZimbraAccount --passfile2 yourZimbraPasswordFile

If your old IMAP server doesen't support NAMESPACE you have to also add appropriate options for example on iMail 6.0 the following are required:

 --sep1 . --prefix1 INBOX

Your destination account on the Zimbra server should also have Clear Text Login enabled. You can enable this under the administration GUI in the Configuration menu, under Servers, in the IMAP tab of your server. Click the checkbox. It is assumed your source server also has Clear Text login enabled, but to do that please read the documentation of your respective server.

You may interrupt imapsync at any time with CTRL-C. Simply restart the same command again to resume the migration. You can also run it more than once to sync changes for a staged migration. Imapsync will not copy the same data twice, so you do not have to be concerned about duplicate messages.

I used imapsync to migrate from a Cyrus IMAPd to Zimbra. The ReadMe lists a number of other IMAP servers compatible with imapsync.

See http://imapsync.lamiral.info/INSTALL for more information on installing imapsync.

Note: see Talk:User_Migration if you're having problems with the imapsync source.

IMAPSync with TLS Connections (STARTTLS)

Since Zimbra default settings prohibit plaintext logins we use the following method to just test if IMAPSync would actually work:

$ imapsync --nosyncacls --subscribe --syncinternaldates --fast --dry \
--host1 zimbra.firshost.com --user1 user@firsthost.com --password1 TotallySecret123 --tls1 \
--host2 zimbra.secondhost.com --user2 user@secondhost.com --password2 TotallySecret456 --tls2

This will do nothing nothing but show if it would work. If you really want to migrate the data just remove the "--dry" option from above example.

Enabling Emails With Large File Attachments (Zimbra IMAP Max Size is 10MB)

The default Zimbra IMAP maximum email size is 10MB. This will cause errors to be raised during the imapsync for emails which are larger than 10MB. Errors such as the following will be displayed in the imapsync log.

+ Copying msg #131:11690588 to folder Folders/Projects/Consulting
flags from : [\Seen]["16-Jul-2007 14:57:35 +1000"]
Couldn't append msg #131 (Subject:[RE: Consultancy Approval]) to folder Folders/Projects/Consulting: 5160 BAD parse error: request too long

OR

+ Copying msg #87598:54935762 to folder INBOX
flags from : [\Seen]["08-Feb-2008 14:09:05 -0500"]
Couldn't append msg #87598 (Subject:[Re: photos]) to folder INBOX: 894 NO [TOOBIG] request too long

The Zimbra IMAP server uses zimbraFileUploadMaxSize to set the max size. The default is 10MB if it's not set. At this time the only way to set it is via the command line.

Use the following command as the zimbra user to increase to 50MB:

zmprov mcf zimbraMtaMaxMessageSize 52428800

imapsync over SSL

You can use SSL connections with imapsync providing "--ssl1" and/or "--ssl2" (and the "--port1 and/or --port2" if using nonstandard ports) for each of the accounts you wish to migrate.

For example:

$ imapsync \
--nosyncacls --subscribe --syncinternaldates \
--host1 oldmail.mydomain.com --user1 user1 --password1 secret1 --ssl1 \
--host2 zimbra.mydomain.com --user2 user2 --password2 secret2 --ssl2

NOTE: imapsync will not be able to authenticate to ZCS until you "Enable clear text login" for the IMAP service. You can set that option under Global Settings or under Servers under IMAP in the ZCS Administration Console.

Example

Remove the comments (lines starting with '#') in order to use this command:

$ imapsync \
  # imapsync with options that may help to optimize and modify disallowed folder names 
  --maxsize 52428800 --buffersize 8192000 \
  --nofoldersizes --nosyncacls --subscribe --syncinternaldates \
  --authmech2 PLAIN \
  # do not migrate folders that looks like (ignore case) junk|spam|trash
  --exclude '(?i)\b(Junk|Spam|Trash)\b' \
  # avoid all X- headers when comparing messages
  '--skipheader 'X-*' \
  # avoid all nonstandard system flags (system flags start with '\')
  # RFC 3501: \Answered, \Flagged, \Deleted, \Seen, \Recent, \Draft
  # NOTE: need \\\\ because we want s/\\(?...) and Perl makes \\ => \
  --regexflag 's/\\\\(?!Answered|Flagged|Deleted|Seen|Recent|Draft)[^\s]*\s*//ig' \
  # translate folder names for characters ZCS does not allow
  # * colon (:) to hyphen (-)
  --regextrans2 's,:,-,g' \
  # double quote (") to single quote (')
  --regextrans2 's,\",'\'',g'
  # Remove spaces before separators and at the end of the folder name
  --regextrans2 's,\s+(?=/|$),,g'
  # change reserved folder names to something that is allowed
  --regextrans2 's,^(Briefcase|Calendar|Contacts|Emailed Contacts|Notebook|Tasks)(?=/|$), $1 Folder,ig' \
  ### site-variable options
  --host1 "$host1" --host2 "$host2" \
  --user1 "$username" --user2 "$username"
  --password1 "$pass1" --authuser2 "$auth2" --password2 "$pass2" \
  --regextrans2 's,\",-,g' \ # change quotes to dashes
  --regextrans2 's,&AAo-|&AA0ACg-|&AA0ACgANAAo-(?=/|$),,g' \
  --ssl1 --authmech1 PLAIN --maxcommandlength1 16384 \
  # --delete2 \ # be careful - this can be destructible if not used carefully
  # --dry --debug --debugimap \ # debug options

Migrating large Mailboxes and avoid timeouts

If you want to migrate large mailboxes (hundreds of folders and sub-folders containing several thousand messages) the default behaviour of imapsync may cause timeouts on the ZCS side. By default imapsync computes a lot of numbers for the sake of statistics only. By switching this off using

   --nofoldersizes --skipsize --fast

in addition to above options the sync will be much faster and timeouts due to the time needed for computation of statistics will not accour anymore.

Info on migrating from courier-imap and imapsync: http://wiki.zimbra.com/index.php?title=Imapsync_from_courier-imap_without_passwords[1]

Migrating without knowing the passwords

In the case your authentication backend stores user password in encrypted format and you don't know them, you may have the possibility to add a second authentication backend with the same users but a password of your choice.

During a cyrus imap to zimbra migration, here's how it worked :

  • temporary openldap installed on third server, provisioned with the list of accounts to be migrated
  • zimbra's external authentication configured with this ldap server (you can use 2 different ldap urls, only bind account must be the same)
  • cyrus imap was already set up to use pam_ldap, which did not manage well the presence of 2 different ldap servers for authentication (if the official ldap server denied authentication, the second wasn't used)
  • if cyrus imap relies on pam_ldap, you can switch to pam_mysql for example. Create a database, table with login and password, and feed...

With that in place, you can keep your cyrus and zimbra server running with no change for the users, but synchronize with the scripts shown below.

Batch Processing

You can automate migrating multiple users by using a script like the following:

    #!/bin/bash
    
    logfile="sinklog.txt"
    host1=123.123.123.123    # host1 is Source
    host2=321.321.321.321    # host2 is Dest
    domain=xyz.com           # domain is where email account is
                             # everything after @ symbol
    
    ###### Do not modify past here ######
    date=`date +%X_-_%x`
    echo "" >> $logfile
    echo "------------------------------------" >> $logfile
    echo "IMAPSync started..  $date" >> $logfile
    echo "" >> $logfile
    
    { while IFS=';' read  u1 p1; do 
            user=$u1"@"$domain    
    	     echo "Syncing User $user"
            date=`date +%X_-_%x`
            echo "Start Syncing User $u1"
            echo "Starting $u1 $date" >> $logfile
    imapsync --nosyncacls --syncinternaldates --host1 $host1 --user1 "$user" --password1 \ 
    "$p1"--host2 $host2 --user2 "$user" --password2 "$p1"
            date=`date +%X_-_%x`
            echo "User $user done"
            echo "Finished $user $date" >> $logfile
            echo "" >> $logfile
            done ; } < userlist.txt
    
    date=`date +%X_-_%x`
    echo "" >> $logfile
    echo "IMAPSync Finished..  $date" >> $logfile
    echo "------------------------------------" >> $logfile

Create a file userlist.txt with the user names and passwords in the following format:

    user;password
    user2;password2

Here is a second batch script that will allow different usernames on the Source and Destination servers. Name this file imapsyncbatch then chmod to 755 execute with ./imapsyncbatch can be used with the --dry option for a dry run.

        #!/bin/bash
        ##Modified by Steve Fink stevef-at-ublug.org
	##This IMAPSync Batch Script is used when you have different
	##usernames on the Source and Destination servers
	##(kinda what IMAPSync was intended for)
	##the format for the user-list.csv file is
	##sourceusername,sourcepassword,destinationusername,destinationpassword 
	
	## Get the info
		while [ -z $infile ]; do
			echo "What is the path to the input file?"
			read infile
		done
		while [ -z $host1 ]; do
			echo "What is the Source Host? (mail1.domain.com)"
			read host1
		done
		while [ -z $host2 ]; do
			echo "What is the Destination Host? (mail2.domain.com)"
			read host2
		done
		while [ -z $domain ]; do
			echo "What is the Domain? (domain.com)"
			read domain
		done
		while [ -z $logfile ]; do
			echo "Where would you like the log? (synclog.txt)"
			read logfile
		done

		if [ ! -f $infile ]; then
			echo "The input file does not exist!"
			echo ""
			echo "What is the path to the input file?"
			read infile
		fi
		INPUTFILE=$infile
		
		clear
		echo "IMAPSync is about to begin using:"
		echo "Input File $INPUTFILE"
		echo "Source Host $host1"
		echo "Destination Host $host2"
		echo "Domain $domain"
		echo "Log File $logfile"
		echo ""
		echo "Is this information correct?"
                echo "Press Enter to continue or"
		echo "Hit CTRL+C to start over"
		read wait

	## Begin IMAPSync
		date=`date +%X_-_%x`
		echo "IMAPSync Logfile started @ $logfile"
		echo "" >> $logfile
		echo "------------------------------------" >> $logfile
    		echo "IMAPSync started..  $date" >> $logfile
    		echo "" >> $logfile

	#Get rid of the commas
		tr "," " " <$INPUTFILE | while read u1 p1 u2 p2
			do
        			user=$u1"@"$domain
           			user2=$u2"@"$domain
             			echo "Syncing User $user to $user2"
            			date=`date +%X_-_%x`
            			echo "Start Syncing User $user to $user2"
            			echo "Starting $u1 $date" >> $logfile
    				imapsync $1 --nosyncacls --syncinternaldates \
				--exclude "#KnownSpam" --exclude "#FalsePositives" \
				--exclude "Trash" --exclude "Deleted Items" \
				--exclude "Deleted Messages" --exclude "Deleted" \
				--exclude "Sent" --exclude "Sent Items" \
				--exclude "Sent Messages" \
				--host1 $host1 --user1 "$user" --password1 "$p1" \
				--host2 $host2 --user2 "$user2" --password2 "$p2"
            			date=`date +%X_-_%x`
            			echo "User $user to $user2 done"
            			echo "Finished $user to $user2 $date" >> $logfile
            			echo "" >> $logfile
            		done

    		date=`date +%X_-_%x`
    		echo "" >> $logfile
    		echo "IMAPSync Finished..  $date" >> $logfile
    		echo "------------------------------------" >> $logfile

The format for the .csv file for this script is: sourceusername,sourcepassword,destinationusername,destinationpassword

Verified Against: ZCS & imapsync Date Created: 4/24/2009
Article ID: http://wiki.zimbra.com/index.php?title=Guide_to_imapsync Date Modified: 03/6/2015
Personal tools