Difference between revisions of "Guide to imapsync"

(reorganized, removed info on installing old versions of imapsync, added reference to imapsync home page and INSTALL instructions instead)
Line 3: Line 3:
 
Before you can use imapsync you must have both the source IMAP message store and ZCS up running and accessible to user accounts via IMAP. You can check that using an email client before starting the migration. You will need the login names (i.e. email addresses) and passwords for the users to be migrated. In addition, imapsync will not be able to authenticate to the Zimbra server until you enable clear text login for the IMAP service. You can set that option under the IMAP tab of the Global Settings or individual Server settings in the Zimbra Admin Console UI.  There are also a few Perl module dependances, including; Digest::MD5, IO::Socket::SSL,Term::ReadKey, Digest:HMAC, libmail:imapclient (Mail::IMAPClient? requires ''exactly'' version 2.2.9, as of this writing), and Date::Manip. You can install these with your favorite package manager, from RPM's or with cpan. Finally, consider that imapsync will be a heavy load on your CPU and memory; the system running the migration will be less responsive.   
 
Before you can use imapsync you must have both the source IMAP message store and ZCS up running and accessible to user accounts via IMAP. You can check that using an email client before starting the migration. You will need the login names (i.e. email addresses) and passwords for the users to be migrated. In addition, imapsync will not be able to authenticate to the Zimbra server until you enable clear text login for the IMAP service. You can set that option under the IMAP tab of the Global Settings or individual Server settings in the Zimbra Admin Console UI.  There are also a few Perl module dependances, including; Digest::MD5, IO::Socket::SSL,Term::ReadKey, Digest:HMAC, libmail:imapclient (Mail::IMAPClient? requires ''exactly'' version 2.2.9, as of this writing), and Date::Manip. You can install these with your favorite package manager, from RPM's or with cpan. Finally, consider that imapsync will be a heavy load on your CPU and memory; the system running the migration will be less responsive.   
  
Imapsync will run faster and consume less CPU with larger I/O buffers.  The default is only 4KB; increasing it by one or two orders of magnitude is recommended for typical Zimbra-ready configurations.  Examples below use 8MB buffer size.  
+
Imapsync may run faster and consume less CPU with larger I/O buffers, but no benchmarking of this has been performed.  The default buffer size is 4KB; increasing it by one or two orders of magnitude is not uncommon for typical Zimbra-ready configurations.  To increase the buffersize to 8MB, add the arguments "--buffersize 8192000" when running imapsync.
  
 
For this example my existing IMAP server is running on server.gtds.lan and I set up Zimbra on a new machine named zimbra.gtds.lan.
 
For this example my existing IMAP server is running on server.gtds.lan and I set up Zimbra on a new machine named zimbra.gtds.lan.
  
     imapsync --buffersize 8192000 --nosyncacls --subscribe --syncinternaldates \
+
     imapsync --nosyncacls --subscribe --syncinternaldates \
 
     --host1 server.gtds.lan --user1 yourAccount --password1 yourPassword \
 
     --host1 server.gtds.lan --user1 yourAccount --password1 yourPassword \
 
     --host2 zimbra.gtds.lan --user2 yourZimbraAccount --password2 yourZimbraPassword
 
     --host2 zimbra.gtds.lan --user2 yourZimbraAccount --password2 yourZimbraPassword
Line 15: Line 15:
 
A slightly more secure method is to write each password into a separate file, and then use the --passfile{1|2} options intead of the --password{1|2} options:
 
A slightly more secure method is to write each password into a separate file, and then use the --passfile{1|2} options intead of the --password{1|2} options:
  
     imapsync --buffersize 8192000 --nosyncacls --subscribe --syncinternaldates \
+
     imapsync --nosyncacls --subscribe --syncinternaldates \
 
     --host1 server.gtds.lan --user1 yourAccount --passfile1 yourPasswordFile \
 
     --host1 server.gtds.lan --user1 yourAccount --passfile1 yourPasswordFile \
 
     --host2 zimbra.gtds.lan --user2 yourZimbraAccount --passfile2 yourZimbraPasswordFile
 
     --host2 zimbra.gtds.lan --user2 yourZimbraAccount --passfile2 yourZimbraPasswordFile
  
If your old IMAP server doesen't support NAMESPACE you have to also add appropriate options for example on iMail 6.0 --sep1 . --prefix1 INBOX. are required
+
If your old IMAP server doesen't support NAMESPACE you have to also add appropriate options for example on iMail 6.0 the following are required:
  
If you receive this error (and using the above command line you are pretty much guaranteed to):  NO CAPABILITY for AUTHENTICATE LOGIN from your Zimbra server please add the switch --noauthmd5 to the end of your very long command line.
+
  --sep1 . --prefix1 INBOX
  
 
Your destination account on the Zimbra server should also have Clear Text Login enabled. You can enable this under the administration GUI in the Configuration menu, under Servers, in the IMAP tab of your server. Click the checkbox. It is assumed your source server also has Clear Text login enabled, but to do that please read the documentation of your respective server.
 
Your destination account on the Zimbra server should also have Clear Text Login enabled. You can enable this under the administration GUI in the Configuration menu, under Servers, in the IMAP tab of your server. Click the checkbox. It is assumed your source server also has Clear Text login enabled, but to do that please read the documentation of your respective server.
Line 29: Line 29:
 
I used imapsync to migrate from a Cyrus IMAPd to Zimbra. The [http://www.linux-france.org/prj/imapsync/README ReadMe] lists a number of other IMAP servers compatible with imapsync.
 
I used imapsync to migrate from a Cyrus IMAPd to Zimbra. The [http://www.linux-france.org/prj/imapsync/README ReadMe] lists a number of other IMAP servers compatible with imapsync.
  
=== script to download and build imapsync and all of the required packages ===
+
See http://imapsync.lamiral.info/INSTALL for more information on installing imapsync.
<pre>
 
#!/bin/bash
 
#
 
#CopyLeft 2006 Steve Fink
 
#stevef-at-ublug.org
 
#
 
#This script will get all of
 
#the necessary packages to
 
#build imapsync 1.219 except OpenSSL
 
#OpenSSL & it's development libraries are
 
#required and OpenSSL is usually installed
 
#already so check and get the matching libraries
 
#
 
#you have to run this script as root
 
#or under sudo so it can do all the installs
 
#
 
#I don't recommend doing the extended tests
 
#they mostly fail anyway but imapsync still works
 
 
 
 
 
mkdir imapsync
 
cd imapsync
 
#
 
wget http://search.cpan.org/CPAN/authors/id/D/DC/DCONWAY/Parse-RecDescent-1.94.tar.gz
 
tar xvfpz Parse-RecDescent-1.94.tar.gz
 
rm Parse-RecDescent-1.94.tar.gz
 
cd Parse-RecDescent-1.94/
 
perl Makefile.PL
 
make
 
make install
 
#
 
cd ..
 
wget http://search.cpan.org/CPAN/authors/id/G/GA/GAAS/Digest-MD5-2.33.tar.gz
 
tar xvfpz Digest-MD5-2.33.tar.gz
 
rm Digest-MD5-2.33.tar.gz
 
cd Digest-MD5-2.33/
 
perl Makefile.PL
 
make
 
make install
 
#
 
cd ..
 
wget http://search.cpan.org/CPAN/authors/id/J/JS/JSTOWE/TermReadKey-2.30.tar.gz
 
tar xvfpz TermReadKey-2.30.tar.gz
 
rm TermReadKey-2.30.tar.gz
 
cd TermReadKey-2.30/
 
perl Makefile.PL
 
make
 
make install
 
#
 
cd ..
 
wget http://search.cpan.org/CPAN/authors/id/A/AS/ASPA/IO-Socket-SSL-0.81.tar.gz
 
tar xvfpz IO-Socket-SSL-0.81.tar.gz
 
rm IO-Socket-SSL-0.81.tar.gz
 
cd IO-Socket-SSL-0.81/
 
perl Makefile.PL
 
make
 
make install
 
#
 
cd ..
 
wget http://search.cpan.org/CPAN/authors/id/F/FL/FLORA/Net_SSLeay.pm-1.30.tar.gz
 
tar xvfpz Net_SSLeay.pm-1.30.tar.gz
 
rm Net_SSLeay.pm-1.30.tar.gz
 
cd Net_SSLeay.pm-1.30/
 
./Makefile.PL -t
 
make install
 
#
 
cd ..
 
wget http://search.cpan.org/CPAN/authors/id/D/DJ/DJKERNEN/Mail-IMAPClient-2.2.9.tar.gz
 
tar xvfpz Mail-IMAPClient-2.2.9.tar.gz
 
rm Mail-IMAPClient-2.2.9.tar.gz
 
cd Mail-IMAPClient-2.2.9/
 
perl Makefile.PL
 
make
 
make test
 
make install
 
#
 
cd ..
 
wget http://www.linux-france.org/prj/imapsync/dist/imapsync-1.219.tgz
 
tar xvfpz imapsync-1.219.tgz
 
rm imapsync-1.219.tgz
 
cd imapsync-1.219/
 
perl -c imapsync
 
make install
 
#
 
cd ..
 
cd ..
 
rm -Rf imapsync/
 
 
 
</pre>
 
 
 
Save this script as buildimapsync, chmod 755 buildimapsync, sudo buildimapsync.  Note that common build tools, such as make and a C compiler must be available.
 
  
 
Note: see  [[Talk:User_Migration]] if you're having problems with the imapsync source.
 
Note: see  [[Talk:User_Migration]] if you're having problems with the imapsync source.
  
Note: The current version (as of 2007 05 22) of imapsync is 1.219. If that last wget command returns 404 Not Found, check [http://www.linux-france.org/prj/imapsync/dist/ the parent folder] for a newer version.
+
=== IMAPSync with TLS Connections (STARTTLS) ===
 
 
Note: The linux-france.org server has been unreachable for a very long time.  Try a mirror site [http://www.filewatcher.com/m/imapsync-1.217.tgz.117424.0.0.html (v1.217)] or [http://www.filewatcher.com/m/imapsync-1.223.tgz.140393.0.0.html (v1.223)] if the link above does not work.
 
  
If you use version 1.223, once installed change line 1163 of /usr/bin/imapsync to this:
+
Since Zimbra default settings prohibit plaintext logins we use the following method to just test if IMAPSync would actually work:
  
<pre>
+
$ imapsync --nosyncacls --subscribe --syncinternaldates --fast --dry \
$d = UnixDate(ParseDate($d), "%d-%b-%Y %H:%M:%S %z");
+
--host1 zimbra.firshost.com --user1 user@firsthost.com --password1 TotallySecret123 --tls1 \
</pre>
+
--host2 zimbra.secondhost.com --user2 user@secondhost.com --password2 TotallySecret456 --tls2
 
 
[http://www.zimbra.com/forums/migration/11932-solved-imapsync-invalid-date-format.html (Thanks to Spremeau in the message boards for this tip)]
 
 
 
=== IMAPSync Examples with Zimbra 7.x, RHEL 6.x / CentOS 6.0 and TLS Connections ===
 
 
 
On CentOS 6.x / RHEL 6.x you may install imapsync 1.488 which supports both SSL and TLS Auth.
 
Since Zimbra default settings prohibit plaintext logins we use the following method to just test if IMAPSync would actually work
 
 
 
  imapsync --buffersize 8192000 --nosyncacls --subscribe --syncinternaldates --fast --dry \
 
  --host1 zimbra.firshost.com --user1 user@firsthost.com --password1 TotallySecret123 --tls1 \
 
  --host2 zimbra.secondhost.com --user2 user@secondhost.com --password2 TotallySecret456 --tls2
 
  
 
This will do nothing nothing but show if it would work. If you really want to migrate the data just remove the "--dry" option from above example.
 
This will do nothing nothing but show if it would work. If you really want to migrate the data just remove the "--dry" option from above example.
  
=== Tweaking Zimbra performance ===
+
=== Enabling Emails With Large File Attachments (Zimbra IMAP Max Size is 10MB) ===
During large migrations via imapsync Zimbra performance can be a bottleneck. Change this setting to improve migration speed. It will startup only one indexing process per 20 emails, instead of one process per email. This setting is an "official" recommendation for large installs.
+
The default Zimbra IMAP maximum email size is 10MB. This will cause errors to be raised during the imapsync for emails which are larger than 10MB. Errors such as the following will be displayed in the imapsync log.
  zmprov mc <cos name> zimbraBatchedIndexingSize 20
 
 
 
 
 
=== Enabling Emails With Large File Attachments (Zimbra IMAP Max Size is 10mb) ===
 
The default Zimbra IMAP maximum email size is 10mb. This will cause errors to be raised during the imapsync for emails which are larger than 10mb. Errors such as the following will be displayed in the imapsync log.
 
  
 
<pre>
 
<pre>
Line 169: Line 60:
 
</pre>
 
</pre>
  
The Zimbra IMAP server uses zimbraFileUploadMaxSize to set the max size. The default is 10mb if it's not set.  At this time the only way to set it is via the command line.
+
The Zimbra IMAP server uses zimbraFileUploadMaxSize to set the max size. The default is 10MB if it's not set.  At this time the only way to set it is via the command line.
 
 
Use the following command as the zimbra user to increase to 50mb:
 
<pre>zmprov mcf zimbraFileUploadMaxSize 50000000</pre>
 
 
 
Then '''restart the mailboxd''' server process.
 
 
 
For more details see this thread, http://www.zimbra.com/forums/developers/1461-max-imap-message-size.html
 
  
'''UPDATE:'''  Since Zimbra 5.0.6 the parameter has changed to zimbraMtaMaxMessageSize:
+
Use the following command as the zimbra user to increase to 50MB:
  
<pre>zmprov mcf zimbraMtaMaxMessageSize 50000000</pre>
+
<pre>zmprov mcf zimbraMtaMaxMessageSize 52428800</pre>
 
 
See this bug report http://bugzilla.zimbra.com/show_bug.cgi?id=27610
 
  
 
=== imapsync over SSL ===
 
=== imapsync over SSL ===
  
You can use SSL connections with imapsync providing "--ssl1" and/or "--ssl2" (and the "--port1 and/or --port2" if defaults are different) for each of the accounts you wish to migrate.
+
You can use SSL connections with imapsync providing "--ssl1" and/or "--ssl2" (and the "--port1 and/or --port2" if using nonstandard ports) for each of the accounts you wish to migrate.
  
 
For example:
 
For example:
  
  [root@freebsd ~]# imapsync \
+
  $ imapsync \
  --buffersize 8192000 --nosyncacls --subscribe --syncinternaldates \
+
  --nosyncacls --subscribe --syncinternaldates \
 
  --host1 oldmail.mydomain.com --user1 user1 --password1 secret1 --ssl1 \
 
  --host1 oldmail.mydomain.com --user1 user1 --password1 secret1 --ssl1 \
--port1 993 \
+
  --host2 zimbra.mydomain.com --user2 user2 --password2 secret2 --ssl2
  --host2 zimbra.mydomain.com --user2 user2 --password2 secret2 --ssl2 \
 
--port2 993 \
 
--noauthmd5
 
  
Installing imapsync on FreeBSD 6.2:
+
NOTE: imapsync will not be able to authenticate to ZCS until you "Enable clear text login" for the IMAP service.
 +
You can set that option under Global Settings or under Servers under IMAP in the ZCS Administration Console.
  
1. Installing the binary:
+
=== Example ===
  
[root@freebsd ~]# pkg_add -r imapsync
+
Remove the comments (lines starting with '#') in order to use this command:
  
2. Installing the port (from latest sources):
+
<pre>
 +
$ imapsync \
 +
  # imapsync with options that may help to optimize and modify disallowed folder names
 +
  --maxsize 52428800 --buffersize 8192000 \
 +
  --nofoldersizes --nosyncacls --subscribe --syncinternaldates \
 +
  --authmech2 PLAIN \
 +
  # do not migrate folders that looks like (ignore case) junk|spam|trash
 +
  --exclude '(?i)\b(Junk|Spam|Trash)\b' \
 +
  # avoid all X- headers when comparing messages
 +
  '--skipheader 'X-*' \
 +
  # avoid all nonstandard system flags (system flags start with '\')
 +
  # RFC 3501: \Answered, \Flagged, \Deleted, \Seen, \Recent, \Draft
 +
  # NOTE: need \\\\ because we want s/\\(?...) and Perl makes \\ => \
 +
  --regexflag 's/\\\\(?!Answered|Flagged|Deleted|Seen|Recent|Draft)[^\s]*\s*//ig' \
 +
  # translate folder names for characters ZCS does not allow
 +
  # * colon (:) to hyphen (-)
 +
  --regextrans2 's,:,-,g' \
 +
  # double quote (") to single quote (')
 +
  --regextrans2 's,\",'\'',g'
 +
  # Remove spaces before separators and at the end of the folder name
 +
  --regextrans2 's,\s+(?=/|$),,g'
 +
  # change reserved folder names to something that is allowed
 +
  --regextrans2 's,^(Briefcase|Calendar|Contacts|Emailed Contacts|Notebook|Tasks)(?=/|$), $1 Folder,ig' \
 +
  ### site-variable options
 +
  --host1 "$host1" --host2 "$host2" \
 +
  --user1 "$username" --user2 "$username"
 +
  --password1 "$pass1" --authuser2 "$auth2" --password2 "$pass2" \
 +
  --regextrans2 's,\",-,g' \ # change quotes to dashes
 +
  --regextrans2 's,&AAo-|&AA0ACg-|&AA0ACgANAAo-(?=/|$),,g' \
 +
  --ssl1 --authmech1 PLAIN --maxcommandlength1 16384 \
 +
  # --delete2 \ # be careful - this can be destructible if not used carefully
 +
  # --dry --debug --debugimap \ # debug options
 +
</pre>
  
[root@freebsd ~]# cd /usr/ports/mail/imapsync
+
=== Migrating large Mailboxes and avoid timeouts ===
[root@freebsd ~]# make install clean
 
  
Make sure you have updated your ports with portsnap. If you are updating your ports for the first time:
+
If you want to migrate large mailboxes (hundreds of folders and sub-folders containing several thousand messages) the default behaviour of imapsync may cause timeouts on the ZCS side. By default imapsync computes a lot of numbers for the sake of statistics only. By switching this off using
 
[root@freebsd ~]# portsnap fetch
 
[root@freebsd ~]# portsnap extract
 
 
More details on how to use portsnap:
 
 
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/ports-using.html)
 
  
The following version worked for me using option 2 above:
+
    --nofoldersizes --skipsize --fast
  
Mail::IMAPClient version is 2.2.9 and imapsync version 1.223 2007/10/30
+
in addition to above options the sync will be much faster and timeouts due to the time needed for computation of statistics will not accour anymore.
  
 +
Info on migrating from courier-imap and imapsync:
 +
http://wiki.zimbra.com/index.php?title=Imapsync_from_courier-imap_without_passwords[http://wiki.zimbra.com/index.php?title=Imapsync_from_courier-imap_without_passwords]
  
Installing imapsync on Ubuntu 7.10:
+
=== Migrating without knowing the passwords ===
  
[user1@ubuntu:~]$ sudo apt-get install imapsync
+
In the case your authentication backend stores user password in encrypted format and you don't know them, you may have the possibility to add a second authentication backend with the same users but a password of your choice.
 
 
The command above will automatically build imapsync. You should be able to use the command above to accomplish an imapsync over SSL migration.
 
 
 
The following version worked for me on Ubuntu 7.10:
 
 
 
Mail::IMAPClient version is 2.2.9 and imapsync version 1.219 2007/04/04
 
  
 +
During a cyrus imap to zimbra migration, here's how it worked :
 +
* temporary openldap installed on third server, provisioned with the list of accounts to be migrated
 +
* zimbra's external authentication configured with this ldap server (you can use 2 different ldap urls, only bind account must be the same)
 +
* cyrus imap was already set up to use pam_ldap, which did not manage well the presence of 2 different ldap servers for authentication (if the official ldap server denied authentication, the second wasn't used)
 +
* if cyrus imap relies on pam_ldap, you can switch to pam_mysql for example. Create a database, table with login and password, and feed...
  
NOTE: imapsync will not be able to authenticate to ZCS until you "Enable clear text login" for the IMAP service.
+
With that in place, you can keep your cyrus and zimbra server running with no change for the users, but synchronize with the scripts shown below.
You can set that option under Global Settings or under Servers under IMAP in the ZCS Administration Console.
 
  
=== Batch Processing ===
+
== Batch Processing ==
  
You can automate multiple users by using this script.
+
You can automate migrating multiple users by using a script like the following:
  
 
     #!/bin/bash
 
     #!/bin/bash
 
      
 
      
 
     logfile="sinklog.txt"
 
     logfile="sinklog.txt"
 +
    host1=123.123.123.123    # host1 is Source
 +
    host2=321.321.321.321    # host2 is Dest
 +
    domain=xyz.com          # domain is where email account is
 +
                              # everything after @ symbol
 
      
 
      
    host1=123.123.123.123
+
     ###### Do not modify past here ######
    #host1 is Source
 
   
 
    host2=321.321.321.321
 
    #host2 is Dest
 
   
 
    domain=xyz.com
 
    #domain is where email account is
 
    #everything after @ symbol
 
   
 
     ###### Do not modify past here
 
    #######################################
 
   
 
 
     date=`date +%X_-_%x`
 
     date=`date +%X_-_%x`
   
 
 
     echo "" >> $logfile
 
     echo "" >> $logfile
 
     echo "------------------------------------" >> $logfile
 
     echo "------------------------------------" >> $logfile
Line 277: Line 172:
 
             echo "Finished $user $date" >> $logfile
 
             echo "Finished $user $date" >> $logfile
 
             echo "" >> $logfile
 
             echo "" >> $logfile
   
 
 
             done ; } < userlist.txt
 
             done ; } < userlist.txt
 
      
 
      
 
     date=`date +%X_-_%x`
 
     date=`date +%X_-_%x`
   
 
 
     echo "" >> $logfile
 
     echo "" >> $logfile
 
     echo "IMAPSync Finished..  $date" >> $logfile
 
     echo "IMAPSync Finished..  $date" >> $logfile
Line 290: Line 183:
 
     user;password
 
     user;password
 
     user2;password2
 
     user2;password2
 
  
 
Here is a second batch script that will allow different usernames on the Source and Destination servers.
 
Here is a second batch script that will allow different usernames on the Source and Destination servers.
Line 327: Line 219:
 
done
 
done
  
if [ ! -f $infile ]
+
if [ ! -f $infile ]; then
        then
+
echo "The input file does not exist!"
                {
+
echo ""
echo "The input file does not exist!"
+
echo "What is the path to the input file?"
echo ""
+
read infile
echo "What is the path to the input file?"
 
read infile
 
}
 
 
fi
 
fi
 
 
INPUTFILE=$infile
 
INPUTFILE=$infile
 
 
 
clear
 
clear
echo ""
 
echo ""
 
 
echo "IMAPSync is about to begin using:"
 
echo "IMAPSync is about to begin using:"
 
echo "Input File $INPUTFILE"
 
echo "Input File $INPUTFILE"
Line 348: Line 234:
 
echo "Domain $domain"
 
echo "Domain $domain"
 
echo "Log File $logfile"
 
echo "Log File $logfile"
echo ""
 
 
echo ""
 
echo ""
 
echo "Is this information correct?"
 
echo "Is this information correct?"
Line 366: Line 251:
 
tr "," " " <$INPUTFILE | while read u1 p1 u2 p2
 
tr "," " " <$INPUTFILE | while read u1 p1 u2 p2
 
do
 
do
 
 
         user=$u1"@"$domain
 
         user=$u1"@"$domain
 
           user2=$u2"@"$domain
 
           user2=$u2"@"$domain
Line 385: Line 269:
 
             echo "Finished $user to $user2 $date" >> $logfile
 
             echo "Finished $user to $user2 $date" >> $logfile
 
             echo "" >> $logfile
 
             echo "" >> $logfile
 
 
             done
 
             done
  
 
     date=`date +%X_-_%x`
 
     date=`date +%X_-_%x`
 
 
     echo "" >> $logfile
 
     echo "" >> $logfile
 
     echo "IMAPSync Finished..  $date" >> $logfile
 
     echo "IMAPSync Finished..  $date" >> $logfile
Line 398: Line 280:
 
sourceusername,sourcepassword,destinationusername,destinationpassword
 
sourceusername,sourcepassword,destinationusername,destinationpassword
  
=== Batch processing without knowing the passwords ===
+
{{Article Footer|ZCS & imapsync|4/24/2009}}
 
 
In the case your authentication backend stores user password in encrypted format and you don't know them, you may have the possibility to add a second authentication backend with the same users but a password of your choice.
 
 
 
During a cyrus imap to zimbra migration, here's how it worked :
 
* temporary openldap installed on third server, provisioned with the list of accounts to be migrated
 
* zimbra's external authentication configured with this ldap server (you can use 2 different ldap urls, only bind account must be the same)
 
* cyrus imap was already set up to use pam_ldap, which did not manage well the presence of 2 different ldap servers for authentication (if the official ldap server denied authentication, the second wasn't used)
 
* if cyrus imap relies on pam_ldap, you can switch to pam_mysql for example. Create a database, table with login and password, and feed...
 
 
 
With that in place, you can keep your cyrus and zimbra server running with no change for the users, but synchronize with the scripts showed above.
 
 
 
== Migrating large Mailboxes and avoid timeouts ==
 
 
 
If you want to migrate large mailboxes (hundreds of folders and sub-folders containing several thousand messages) the default behaviour of imapsync may cause timeouts on the ZCS side. By default imapsync computes a lot of numbers for the sake of statistics only. By switching this off using
 
 
 
    –-nofoldersizes --skipsize –-fast
 
 
 
in addition to above options the sync will be much faster and timeouts due to the time needed for computation of statistics will not accour anymore.
 
 
 
Info on migrating from courier-imap and imapsync:
 
http://wiki.zimbra.com/index.php?title=Imapsync_from_courier-imap_without_passwords[http://wiki.zimbra.com/index.php?title=Imapsync_from_courier-imap_without_passwords]
 
 
 
== Examples ==
 
 
 
<pre>
 
$ imapsync \
 
  # imapsync with options that may help to optimize and modify disallowed folder names
 
  --maxsize 52428800 --buffersize 52428800 \
 
  --fastio1 --fastio2 \
 
  --nofoldersizes --nosyncacls --subscribe --syncinternaldates \
 
  --noauthmd5 --authmech2 PLAIN \
 
  # do not migrate folders that looks like (ignore case) junk|spam|trash
 
  --exclude '(?i)\b(Junk|Spam|Trash)\b' \
 
  # avoid all X- headers when comparing messages
 
  '--skipheader 'X-*' \
 
  # avoid all nonstandard system flags (system flags start with '\')
 
  # RFC 3501: \Answered, \Flagged, \Deleted, \Seen, \Recent, \Draft
 
  # NOTE: need \\\\ because we want s/\\(?...) and Perl makes \\ => \
 
  --regexflag 's/\\\\(?!Answered|Flagged|Deleted|Seen|Recent|Draft)[^\s]*\s*//ig' \
 
  # translate folder names for characters ZCS does not allow
 
  # * colon (:) to hyphen (-)
 
  --regextrans2 's,:,-,g' \
 
  # double quote (") to single quote (')
 
  --regextrans2 's,\",'\'',g'
 
  # Remove spaces before separators and at the end of the folder name
 
  --regextrans2 's,\s+(?=/|$),,g'
 
  # change reserved folder names to something that is allowed
 
  --regextrans2 's,^(Briefcase|Calendar|Contacts|Emailed Contacts|Notebook|Tasks)(?=/|$), $1 Folder,ig' \
 
  ### site-variable options
 
  --host1 "$host1" --host2 "$host2" \
 
  --user1 "$username" --user2 "$username"
 
  --password1 "$pass1" --authuser2 "$auth2" --password2 "$pass2" \
 
  --regextrans2 's,\",-,g' \ # change quotes to dashes
 
  --regextrans2 's,&AAo-|&AA0ACg-|&AA0ACgANAAo-(?=/|$),,g' \
 
  --ssl1 --authmech1 PLAIN --maxcommandlength1 16384 \
 
  # --delete2 \ # be careful - this can be destructible if not used carefully
 
  # --dry --debug --debugimap \ # debug options
 
</pre>
 
 
 
{{Article Footer|ZCS 5.0.6 & imapsync 1.219|4/24/2009}}
 
  
 
[[Category:Migration]]  
 
[[Category:Migration]]  

Revision as of 15:16, 6 March 2015

Admin Article

Article Information

This article applies to the following ZCS versions.

ZCS 5.0 Article ZCS 5.0

The Guide

Before you can use imapsync you must have both the source IMAP message store and ZCS up running and accessible to user accounts via IMAP. You can check that using an email client before starting the migration. You will need the login names (i.e. email addresses) and passwords for the users to be migrated. In addition, imapsync will not be able to authenticate to the Zimbra server until you enable clear text login for the IMAP service. You can set that option under the IMAP tab of the Global Settings or individual Server settings in the Zimbra Admin Console UI. There are also a few Perl module dependances, including; Digest::MD5, IO::Socket::SSL,Term::ReadKey, Digest:HMAC, libmail:imapclient (Mail::IMAPClient? requires exactly version 2.2.9, as of this writing), and Date::Manip. You can install these with your favorite package manager, from RPM's or with cpan. Finally, consider that imapsync will be a heavy load on your CPU and memory; the system running the migration will be less responsive.

Imapsync may run faster and consume less CPU with larger I/O buffers, but no benchmarking of this has been performed. The default buffer size is 4KB; increasing it by one or two orders of magnitude is not uncommon for typical Zimbra-ready configurations. To increase the buffersize to 8MB, add the arguments "--buffersize 8192000" when running imapsync.

For this example my existing IMAP server is running on server.gtds.lan and I set up Zimbra on a new machine named zimbra.gtds.lan.

   imapsync --nosyncacls --subscribe --syncinternaldates \
   --host1 server.gtds.lan --user1 yourAccount --password1 yourPassword \
   --host2 zimbra.gtds.lan --user2 yourZimbraAccount --password2 yourZimbraPassword

Of course the complete command belongs on one line (signified by the backslashes: \).

A slightly more secure method is to write each password into a separate file, and then use the --passfile{1|2} options intead of the --password{1|2} options:

   imapsync --nosyncacls --subscribe --syncinternaldates \
   --host1 server.gtds.lan --user1 yourAccount --passfile1 yourPasswordFile \
   --host2 zimbra.gtds.lan --user2 yourZimbraAccount --passfile2 yourZimbraPasswordFile

If your old IMAP server doesen't support NAMESPACE you have to also add appropriate options for example on iMail 6.0 the following are required:

 --sep1 . --prefix1 INBOX

Your destination account on the Zimbra server should also have Clear Text Login enabled. You can enable this under the administration GUI in the Configuration menu, under Servers, in the IMAP tab of your server. Click the checkbox. It is assumed your source server also has Clear Text login enabled, but to do that please read the documentation of your respective server.

You may interrupt imapsync at any time with CTRL-C. Simply restart the same command again to resume the migration. You can also run it more than once to sync changes for a staged migration. Imapsync will not copy the same data twice, so you do not have to be concerned about duplicate messages.

I used imapsync to migrate from a Cyrus IMAPd to Zimbra. The ReadMe lists a number of other IMAP servers compatible with imapsync.

See http://imapsync.lamiral.info/INSTALL for more information on installing imapsync.

Note: see Talk:User_Migration if you're having problems with the imapsync source.

IMAPSync with TLS Connections (STARTTLS)

Since Zimbra default settings prohibit plaintext logins we use the following method to just test if IMAPSync would actually work:

$ imapsync --nosyncacls --subscribe --syncinternaldates --fast --dry \
--host1 zimbra.firshost.com --user1 user@firsthost.com --password1 TotallySecret123 --tls1 \
--host2 zimbra.secondhost.com --user2 user@secondhost.com --password2 TotallySecret456 --tls2

This will do nothing nothing but show if it would work. If you really want to migrate the data just remove the "--dry" option from above example.

Enabling Emails With Large File Attachments (Zimbra IMAP Max Size is 10MB)

The default Zimbra IMAP maximum email size is 10MB. This will cause errors to be raised during the imapsync for emails which are larger than 10MB. Errors such as the following will be displayed in the imapsync log.

+ Copying msg #131:11690588 to folder Folders/Projects/Consulting
flags from : [\Seen]["16-Jul-2007 14:57:35 +1000"]
Couldn't append msg #131 (Subject:[RE: Consultancy Approval]) to folder Folders/Projects/Consulting: 5160 BAD parse error: request too long

OR

+ Copying msg #87598:54935762 to folder INBOX
flags from : [\Seen]["08-Feb-2008 14:09:05 -0500"]
Couldn't append msg #87598 (Subject:[Re: photos]) to folder INBOX: 894 NO [TOOBIG] request too long

The Zimbra IMAP server uses zimbraFileUploadMaxSize to set the max size. The default is 10MB if it's not set. At this time the only way to set it is via the command line.

Use the following command as the zimbra user to increase to 50MB:

zmprov mcf zimbraMtaMaxMessageSize 52428800

imapsync over SSL

You can use SSL connections with imapsync providing "--ssl1" and/or "--ssl2" (and the "--port1 and/or --port2" if using nonstandard ports) for each of the accounts you wish to migrate.

For example:

$ imapsync \
--nosyncacls --subscribe --syncinternaldates \
--host1 oldmail.mydomain.com --user1 user1 --password1 secret1 --ssl1 \
--host2 zimbra.mydomain.com --user2 user2 --password2 secret2 --ssl2

NOTE: imapsync will not be able to authenticate to ZCS until you "Enable clear text login" for the IMAP service. You can set that option under Global Settings or under Servers under IMAP in the ZCS Administration Console.

Example

Remove the comments (lines starting with '#') in order to use this command:

$ imapsync \
  # imapsync with options that may help to optimize and modify disallowed folder names 
  --maxsize 52428800 --buffersize 8192000 \
  --nofoldersizes --nosyncacls --subscribe --syncinternaldates \
  --authmech2 PLAIN \
  # do not migrate folders that looks like (ignore case) junk|spam|trash
  --exclude '(?i)\b(Junk|Spam|Trash)\b' \
  # avoid all X- headers when comparing messages
  '--skipheader 'X-*' \
  # avoid all nonstandard system flags (system flags start with '\')
  # RFC 3501: \Answered, \Flagged, \Deleted, \Seen, \Recent, \Draft
  # NOTE: need \\\\ because we want s/\\(?...) and Perl makes \\ => \
  --regexflag 's/\\\\(?!Answered|Flagged|Deleted|Seen|Recent|Draft)[^\s]*\s*//ig' \
  # translate folder names for characters ZCS does not allow
  # * colon (:) to hyphen (-)
  --regextrans2 's,:,-,g' \
  # double quote (") to single quote (')
  --regextrans2 's,\",'\'',g'
  # Remove spaces before separators and at the end of the folder name
  --regextrans2 's,\s+(?=/|$),,g'
  # change reserved folder names to something that is allowed
  --regextrans2 's,^(Briefcase|Calendar|Contacts|Emailed Contacts|Notebook|Tasks)(?=/|$), $1 Folder,ig' \
  ### site-variable options
  --host1 "$host1" --host2 "$host2" \
  --user1 "$username" --user2 "$username"
  --password1 "$pass1" --authuser2 "$auth2" --password2 "$pass2" \
  --regextrans2 's,\",-,g' \ # change quotes to dashes
  --regextrans2 's,&AAo-|&AA0ACg-|&AA0ACgANAAo-(?=/|$),,g' \
  --ssl1 --authmech1 PLAIN --maxcommandlength1 16384 \
  # --delete2 \ # be careful - this can be destructible if not used carefully
  # --dry --debug --debugimap \ # debug options

Migrating large Mailboxes and avoid timeouts

If you want to migrate large mailboxes (hundreds of folders and sub-folders containing several thousand messages) the default behaviour of imapsync may cause timeouts on the ZCS side. By default imapsync computes a lot of numbers for the sake of statistics only. By switching this off using

   --nofoldersizes --skipsize --fast

in addition to above options the sync will be much faster and timeouts due to the time needed for computation of statistics will not accour anymore.

Info on migrating from courier-imap and imapsync: http://wiki.zimbra.com/index.php?title=Imapsync_from_courier-imap_without_passwords[1]

Migrating without knowing the passwords

In the case your authentication backend stores user password in encrypted format and you don't know them, you may have the possibility to add a second authentication backend with the same users but a password of your choice.

During a cyrus imap to zimbra migration, here's how it worked :

  • temporary openldap installed on third server, provisioned with the list of accounts to be migrated
  • zimbra's external authentication configured with this ldap server (you can use 2 different ldap urls, only bind account must be the same)
  • cyrus imap was already set up to use pam_ldap, which did not manage well the presence of 2 different ldap servers for authentication (if the official ldap server denied authentication, the second wasn't used)
  • if cyrus imap relies on pam_ldap, you can switch to pam_mysql for example. Create a database, table with login and password, and feed...

With that in place, you can keep your cyrus and zimbra server running with no change for the users, but synchronize with the scripts shown below.

Batch Processing

You can automate migrating multiple users by using a script like the following:

    #!/bin/bash
    
    logfile="sinklog.txt"
    host1=123.123.123.123    # host1 is Source
    host2=321.321.321.321    # host2 is Dest
    domain=xyz.com           # domain is where email account is
                             # everything after @ symbol
    
    ###### Do not modify past here ######
    date=`date +%X_-_%x`
    echo "" >> $logfile
    echo "------------------------------------" >> $logfile
    echo "IMAPSync started..  $date" >> $logfile
    echo "" >> $logfile
    
    { while IFS=';' read  u1 p1; do 
            user=$u1"@"$domain    
    	     echo "Syncing User $user"
            date=`date +%X_-_%x`
            echo "Start Syncing User $u1"
            echo "Starting $u1 $date" >> $logfile
    imapsync --nosyncacls --syncinternaldates --host1 $host1 --user1 "$user" --password1 \ 
    "$p1"--host2 $host2 --user2 "$user" --password2 "$p1"
            date=`date +%X_-_%x`
            echo "User $user done"
            echo "Finished $user $date" >> $logfile
            echo "" >> $logfile
            done ; } < userlist.txt
    
    date=`date +%X_-_%x`
    echo "" >> $logfile
    echo "IMAPSync Finished..  $date" >> $logfile
    echo "------------------------------------" >> $logfile

Create a file userlist.txt with the user names and passwords in the following format:

    user;password
    user2;password2

Here is a second batch script that will allow different usernames on the Source and Destination servers. Name this file imapsyncbatch then chmod to 755 execute with ./imapsyncbatch can be used with the --dry option for a dry run.

        #!/bin/bash
        ##Modified by Steve Fink stevef-at-ublug.org
	##This IMAPSync Batch Script is used when you have different
	##usernames on the Source and Destination servers
	##(kinda what IMAPSync was intended for)
	##the format for the user-list.csv file is
	##sourceusername,sourcepassword,destinationusername,destinationpassword 
	
	## Get the info
		while [ -z $infile ]; do
			echo "What is the path to the input file?"
			read infile
		done
		while [ -z $host1 ]; do
			echo "What is the Source Host? (mail1.domain.com)"
			read host1
		done
		while [ -z $host2 ]; do
			echo "What is the Destination Host? (mail2.domain.com)"
			read host2
		done
		while [ -z $domain ]; do
			echo "What is the Domain? (domain.com)"
			read domain
		done
		while [ -z $logfile ]; do
			echo "Where would you like the log? (synclog.txt)"
			read logfile
		done

		if [ ! -f $infile ]; then
			echo "The input file does not exist!"
			echo ""
			echo "What is the path to the input file?"
			read infile
		fi
		INPUTFILE=$infile
		
		clear
		echo "IMAPSync is about to begin using:"
		echo "Input File $INPUTFILE"
		echo "Source Host $host1"
		echo "Destination Host $host2"
		echo "Domain $domain"
		echo "Log File $logfile"
		echo ""
		echo "Is this information correct?"
                echo "Press Enter to continue or"
		echo "Hit CTRL+C to start over"
		read wait

	## Begin IMAPSync
		date=`date +%X_-_%x`
		echo "IMAPSync Logfile started @ $logfile"
		echo "" >> $logfile
		echo "------------------------------------" >> $logfile
    		echo "IMAPSync started..  $date" >> $logfile
    		echo "" >> $logfile

	#Get rid of the commas
		tr "," " " <$INPUTFILE | while read u1 p1 u2 p2
			do
        			user=$u1"@"$domain
           			user2=$u2"@"$domain
             			echo "Syncing User $user to $user2"
            			date=`date +%X_-_%x`
            			echo "Start Syncing User $user to $user2"
            			echo "Starting $u1 $date" >> $logfile
    				imapsync $1 --nosyncacls --syncinternaldates \
				--exclude "#KnownSpam" --exclude "#FalsePositives" \
				--exclude "Trash" --exclude "Deleted Items" \
				--exclude "Deleted Messages" --exclude "Deleted" \
				--exclude "Sent" --exclude "Sent Items" \
				--exclude "Sent Messages" \
				--host1 $host1 --user1 "$user" --password1 "$p1" \
				--host2 $host2 --user2 "$user2" --password2 "$p2"
            			date=`date +%X_-_%x`
            			echo "User $user to $user2 done"
            			echo "Finished $user to $user2 $date" >> $logfile
            			echo "" >> $logfile
            		done

    		date=`date +%X_-_%x`
    		echo "" >> $logfile
    		echo "IMAPSync Finished..  $date" >> $logfile
    		echo "------------------------------------" >> $logfile

The format for the .csv file for this script is: sourceusername,sourcepassword,destinationusername,destinationpassword

Verified Against: ZCS & imapsync Date Created: 4/24/2009
Article ID: https://wiki.zimbra.com/index.php?title=Guide_to_imapsync Date Modified: 2015-03-06



Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »

Jump to: navigation, search