ZCS to ZCS With-Different-LDAP-Servers Migration Example
ZCS to ZCS with different LDAP Servers Migration examples
- This is archive documentation, which means it is not supported or valid for recent versions of Zimbra Collaboration.
- This article is a Work in Progress, and may be unfinished or missing sections.
Scripts Provided as an EXAMPLE only
Be sure to carefully read/review/modify these scripts as necessary for your situation
What these scripts were used Originally used for
ZCS 6.0.4NE - ZCS 6.0.7NE Migration from one provider to another
A global admin account WAS available for the remote server (old provider)
The (new server) was provisioned and 100% under our control.
This *should* work for just about any zimbra version, from one server to another (between seperate LDAP's)
none of this is necessary if you can do a NE backup and restore.. but if you are (say) moving from a dedicated Zimbra server in someone's office, to a NE cluster (with other domains on it) you couldn't do that, or if you are moving a domain from a cluster with others, to a dedicated server... these scripts might help (although, if you control both LDAP servers 100%, you may be able to use other scripts to also export and import the user passwords through raw ldap, to reduce customer impact)
Scripts/Files used
00_README.txt
These scripts assume you have a global admin on the remote server (at least I did for this example) They *may* work for a single domain with a domain admin, or at least some of them will If/when I have to do a migration with only a domain admin privlidge, I will adjust the scripts appropriately to solve the issue, and update this page (or feel free to add some comments to this page) **Your mileage may vary, use at your own risk **these scripts are provided for concept, and may need to be customized for your particular situation It is NOT recommended to use /tmp as your migration folder Be sure your migration folder is properly secured, and owned by zimbra.zimbra, scripts are chmod 700 to zimbra so they can be executed.
config.sh
#!/bin/bash oldhost="youroldhostnameorIPhere" oldadmin="globaladminaccount@oldhost.com" oldadminpass="globaladminpassword" newTmpPass="123DomainTemporary@pass321" migrationfolder=/tmp newcosID="60668d2f-5200-5c56-828a-201378e53a90" ## THIS IS THE GUID OF YOUR NEW COS TO PUT USERS IN
01_GetOldDetails.sh
#!/bin/bash ## Read the config script when this script starts to set all the variables to the shell source config.sh cd $migrationfolder ## note, some of what is obtained here is unnecessary, but I wanted to get all that I could, and keep it in case I need it later on due to a quirk ##First, get the list of mailboxes from the old server ## note: zmprov gaa won't work without ldap access, so we have to use zmsoap to get all the accounts :( zmsoap -z GetAllAccountsRequest -u https://$oldhost:7071/service/admin/soap -a $oldadmin -p $oldadminpass > $migrationfolder/oldgetaccountdetails cat $migrationfolder/oldgetaccountdetails | grep '<a n="mail"' |cut -f2 -d">"|cut -f1 -d"<" > $migrationfolder/mailboxlist.txt ## Get other details we may want.... just in case mkdir $migrationfolder/accounts mkdir $migrationfolder/cos mkdir $migrationfolder/grants mkdir $migrationfolder/domains mkdir $migrationfolder/resources ######### Domain details zmprov -s $oldhost:7071 -a $oldadmin -p $oldadminpass gad > $migrationfolder/domainslist.txt for i in `cat $migrationfolder/domainslist.txt` do echo Getting domain Details for $i zmprov -s $oldhost:7071 -a $oldadmin -p $oldadminpass gd $i > $migrationfolder/domains/$i.details done ######### COS details zmprov -s $oldhost:7071 -a $oldadmin -p $oldadminpass gac > $migrationfolder/coslist.txt for i in `cat $migrationfolder/coslist.txt` do echo Getting COS Details for $i zmprov -s $oldhost:7071 -a $oldadmin -p $oldadminpass gc $i > $migrationfolder/cos/$i.cosdetails done ######### Grant details (These can't easily be re-imported, but I wanted to keep a list of what used to be there) for i in `cat $migrationfolder/mailboxlist.txt` do echo Getting Grant Details for $i zmprov -s $oldhost:7071 -a $oldadmin -p $oldadminpass gsi $i > $migrationfolder/grants/$i.grants done ######### Distribution list details zmprov -s $oldhost:7071 -a $oldadmin -p $oldadminpass gadl > $migrationfolder/distlist.txt mkdir $migrationfolder/distlist for i in `cat $migrationfolder/distlist.txt` do echo Getting Dist list $i zmprov -s $oldhost:7071 -a $oldadmin -p $oldadminpass gdl $i > $migrationfolder/distlist/$i.distlistdetails zmprov -s $oldhost:7071 -a $oldadmin -p $oldadminpass gdlm $i > $migrationfolder/distlist/$i.distlistmembership done ######### mailbox details for i in `cat $migrationfolder/mailboxlist.txt` do echo Getting Mailbox Details and account membership for $i zmprov -s $oldhost:7071 -a $oldadmin -p $oldadminpass ga $i > $migrationfolder/accounts/$i.accountdetails zmprov -s $oldhost:7071 -a $oldadmin -p $oldadminpass gam $i > $migrationfolder/accounts/$i.accountmembership zmprov -s $oldhost:7071 -a $oldadmin -p $oldadminpass gsig $i > $migrationfolder/accounts/$i.accountsignature done ######### Calendar Resource details echo Getting All calendar Resources zmprov -s $oldhost:7071 -a $oldadmin -p $oldadminpass gacr > $migrationfolder/resources.txt for i in `cat $migrationfolder/resources.txt` do echo Getting resource $i zmprov -s $oldhost:7071 -a $oldadmin -p $oldadminpass gcr $i > $migrationfolder/resources/$i.resourcedetails done ######### Global Configuration details echo getting global Configuration Details zmprov -s $oldhost:7071 -a $oldadmin -p $oldadminpass gacf > $migrationfolder/gacf.txt
02_MakeMigrationScripts.sh
#!/bin/bash
## Run the config script when this script starts to set all the variables to the shell
source config.sh
######### THIS SCRIPT DOES NOT ACTUALLY CREATE ANYTHING, IT JUST MAKES SCRIPTS WHICH ARE TO BE RUN TO DO THE ACTUAL WORK (zmprov.* files)
## Note, the zmprov.* files DO have passwords in them, so be sure they are kept safe
cd $migrationfolder
########### Make Domains creation script (this assumes you are migrating all domains.. adjust appropriately)
rm -f $migrationfolder/zmprov.createdomains
echo "#It is suggested to create these manually to set all the COS right, etc" > $migrationfolder/zmprov.createdomains
echo "#This script is just for concept" > $migrationfolder/zmprov.createdomains
echo "#Use only with good judgement" > $migrationfolder/zmprov.createdomains
echo "#" > $migrationfolder/zmprov.createdomains
for i in `cat $migrationfolder/domainslist.txt`
do
echo "# You should create domain $i manually, to set all the right settings"
echo "#zmprov cd $i" >> $migrationfolder/zmprov.createdomains
done
########### Make Distribution list creation script
rm -f $migrationfolder/zmprov.createdistlists
for i in `cat $migrationfolder/distlist.txt`
do
echo zmprov cdl $i >> $migrationfolder/zmprov.createdistlists
members=""
for j in `cat $migrationfolder/distlist/$i.distlistdetails | grep zimbraMailForwardingAddress | cut -f2 -d " "`
do
members="$members$j "
done
displayName=`cat $migrationfolder/distlist/$i.distlistdetails | grep displayName | cut -f2 -d ":"|awk '{gsub(/^ +| +$/,"")}1'`
cn=`cat $migrationfolder/distlist/$i.distlistdetails | grep cn: | cut -f2 -d ":"|awk '{gsub(/^ +| +$/,"")}1'`
echo zmprov modifyDistributionList $i cn \"$cn\" displayName \"$displayName\" >> $migrationfolder/zmprov.createdistlists
echo zmprov adlm $i $members >> $migrationfolder/zmprov.createdistlists
for k in `cat $migrationfolder/distlist/$i.distlistdetails | grep zimbraMailAlias | cut -f2 -d " "`
do
echo zmprov adla $i $k >> $migrationfolder/zmprov.createdistlists
done
done
########### Make Alias creation script
rm -f $migrationfolder/zmprov.createaliases
for i in `cat $migrationfolder/mailboxlist.txt`
do
for j in `cat $migrationfolder/accounts/$i.accountdetails | grep -i zimbraMailAlias | cut -f2 -d " "`
do
echo zmprov aaa $i $j >> $migrationfolder/zmprov.createaliases
done
done
########### Make Calendar Resource creation script (Not 100% sure these imports are usable/correlatable to a calender after importing
########## My migration had them in there, but they weren't using them - usage may vary)
rm -f $migrationfolder/zmprov.createcalres
for i in `cat $migrationfolder/resources.txt`
do
displayname=`cat resources/$i.resourcedetails| grep displayName| cut -f2 -d ":"|awk '{gsub(/^ +| +$/,"")}1'`
zimbraCalResType=`cat resources/$i.resourcedetails| grep zimbraCalResType| cut -f2 -d ":"|awk '{gsub(/^ +| +$/,"")}1'`
echo zmprov createCalendarResource $i $newTmpPass displayName \"$displayname\" zimbraCalResType \"$zimbraCalResType\">> $migrationfolder/zmprov.createcalres
done
########### Make zimbraMailSieveScript (webmail filters)
rm -f /tmp/tmpsieve
for i in `cat $migrationfolder/mailboxlist.txt`
do
echo -n "zmprov ma $i zimbraMailSieveScript '" >> /tmp/tmpsieve
cat accounts/$i.accountdetails | sed -n '/zimbraMailSieveScript: /,/zimbraMailSignatureMaxLength/p' | sed 's/zimbraMailSieveScript: //g'|grep -v zimbraMailSignatureMaxLength >> /tmp/tmpsieve
echo "'" >> /tmp/tmpsieve
done
cat /tmp/tmpsieve | grep -v "zimbraMailSieveScript '$" > zmprov.recreatesieve
## NOTE: IF SOMEONE HAS A ' in their sieve script, this may not re-import right.. update the name of the filter manually in this script if that errors, and re-run
########### Force password Change script
for i in `cat $migrationfolder/mailboxlist.txt`
do
echo "zmprov ma $i zimbraPasswordMustChange TRUE " >> $migrationfolder/zmprov.zzz_forcepasswordchange
done
########### Make user creation script
rm -f $migrationfolder/zmprov.createusers
rm -f $migrationfolder/zmprov.createsigs
echo "#Note, signature creation script may die if there is a ' in it." > $migrationfolder/zmprov.createsigs
for i in `cat $migrationfolder/mailboxlist.txt`
do
# For debugging
echo Working on $i
#### ************************* FOR DOING THIS YOU NEED TO READ HERE !!******************************
## FOR THIS MIGRATION, I WAS PROVIDED A TXT FILE userpass.txt THAT CONTAINED username|password
## IF YOU DO NOT HAVE SUCH A FILE, OR THE USER DOESN'T EXIST IN THAT FILE, IT WILL USE THE TMP PASS CONFIGURED ABOVE
## If you have access to the old LDAP, and are experienced with modifying LDAP, you may be able to migrate the passwords..
##
touch $migrationfolder/userpass.txt
user=`echo $i|cut -f1 -d "@"`
passline=`cat $migrationfolder/userpass.txt | grep -i "$user|"`
password=`echo $passline|cut -f2 -d"|"`
if [ "x$password" == "x" ]; then
password=$newTmpPass
fi
displaynamea=`cat $migrationfolder/accounts/$i.accountdetails|grep displayName|cut -f2 -d ":"|awk '{gsub(/^ +| +$/,"")}1'`
givenname=`cat $migrationfolder/accounts/$i.accountdetails|grep givenName|cut -f2 -d ":"|awk '{gsub(/^ +| +$/,"")}1'`
lastname=`cat $migrationfolder/accounts/$i.accountdetails|grep sn:|cut -f2 -d ":"|awk '{gsub(/^ +| +$/,"")}1'`
zimbraMailQuota=`cat $migrationfolder/accounts/$i.accountdetails|grep zimbraMailQuota|cut -f2 -d ":"|awk '{gsub(/^ +| +$/,"")}1'`
signature=`cat $migrationfolder/accounts/$i.accountsignature | sed -n '/zimbraPrefMailSignature: /,/zimbraSignatureId/p' | sed 's/zimbraPrefMailSignature: //g'|grep -v zimbraSignatureId|while read line; do echo -n "$line\n"; done`
cos="zimbraCOSId $newcosID"
echo zmprov ca $i $password displayname \"$displaynamea\" givenName \"$givenname\" sn \"$lastname\" $cos zimbraMailQuota $zimbraMailQuota >> $migrationfolder/zmprov.createusers
if [ "x$signature" != "x" ]; then
signame=`cat $migrationfolder/accounts/$i.accountsignature|grep zimbraSignatureName|sed 's/zimbraSignatureName: //g'`
echo zmprov csig $i \"$signame\" zimbraPrefMailSignature \' >> $migrationfolder/zmprov.createsigs
cat $migrationfolder/accounts/$i.accountsignature | sed -n '/zimbraPrefMailSignature: /,/zimbraSignatureId/p' | sed 's/zimbraPrefMailSignature: //g'|grep -v zimbraSignatureId |while read line; do
echo "$line" >> $migrationfolder/zmprov.createsigs;
done
echo "'" >> $migrationfolder/zmprov.createsigs
echo zmprov ma $i zimbraSignatureName \"$signame\" >> $migrationfolder/zmprov.createsigs
fi
done
########### Make the Actual Migration Scripts to be used @ cutover time
##
mkdir $migrationfolder/migrationscripts
mkdir $migrationfolder/migrationmailboxes
migratescriptmaster="$migrationfolder/migratescriptmaster"
rm -f $migratescriptmaster
rm -f $migratescriptmaster.restore
touch $migratescriptmaster
touch $migratescriptmaster.restore
counter=1
for i in `cat $migrationfolder/mailboxlist.txt`
do
echo "echo $counter of 25" >> $migratescriptmaster
echo "echo working on Downloading $i" >> $migratescriptmaster
echo "zmmailbox -u https://$oldhost:7071 -a $oldadmin -p $oldadminpass -m $i getRestURL \"//?fmt=tgz\" > $migrationfolder/migrationmailboxes/$i.tgz" >> $migratescriptmaster
echo "echo $counter of 25" >> $migratescriptmaster.restore
echo "echo Working on Restoring $i" >> $migratescriptmaster.restore
echo "zmmailbox -z -m $i postRestURL \"//?fmt=tgz&resolve=replace\" $migrationfolder/migrationmailboxes/$i.tgz" >> $migratescriptmaster.restore
counter=`expr $counter + 1`
done
rm -f migrationscripts/*_*
# splitting by 75, as I wanted 25 users in a batch, and each one is 3 lines long in the script file
# Then I started a number of screen sessions, and started each one in a window, so I could run 15 or 20 scripts at a time
## See my Wiki page on using "screen"
split -l 75 $migratescriptmaster $migrationfolder/migrationscripts/getdata_
split -l 75 $migratescriptmaster.restore $migrationfolder/migrationscripts/restorelocally_
# Note: This displays the numbering system slightly odd when running it, as it shows "304 of 25" but I wanted it that way, so I could also see the total process point
Process / What to do
Here is basically what I did:
Planning
Ensured I was able to get all the data I would need for the migration (Ran 01_getolddetails.sh)
Pre-Setup
- Once the local Zimbra was installed, I manually created the LDAP, security, settings, and COS/domains that I wanted
- Then I ran the 02_MakeMigrationScripts.sh and got my setup scripts read
- Run the appropriate scripts (zmprov.*) scripts, I ran them in this order
- zmprov.createusers
- zmprov.createaliases
- zmprov.createcalres
- zmprov.createdistlists
- zmprov.createdistlistaliasesonly
- zmprov.createsigs
- zmprov.recreatesievescript (I actually re-ran this at the end too, so it would hopefully link up to the right folders, after they exist- i.e. post import)
- zmprov.zzz_forcepasswordchange is optional, if you were not provided passwords, however, it is highly suggested - it sets all accounts to be forced to change password on first login
Post-Setup testing
yes, you should test for mail flow, firewall access, etc, to be sure the server works, setup antispam, etc, etc.. once everything looks good, you should be ready.
Prep for Go-Live
Make sure you control DNS for this domain
- Change TTL to say 5 minutes, to prepare for this migration
- Be sure all users know when it is going to happen, and that right after it happens, they will NOT see any of their old data in their accounts immediately, but that it will slowly start re-appearing after a few hours. New email will not get lost.
Migration Day
- At the point of "go-live", change the MX records, give it about 10 mins to ensure new mail is flowing to new server
- Then Start running the migrationscripts/* scripts using screen (see link to screen page at end) (you can probably run a number of them at one time, depending on the size of the source mailboxes. Restoring will take 3x longer than downloading, so calculate entire time appropriately. (obviously, run the getdata_** scripts, before the restorelocally_** scripts.. I ran the getdata_** scripts in seperate screens, then as each one finished, I started the restorelocally_** script for that "batch" and recorded the times as I went, so I knew my entire process timeframe.
- Watch the process.
- watch the logs the entire time too for errors
Process
In a nutshell, you just change the MX records, after they have moved, start migrating old data, and start watching logs for anything odd.. after migration, take a long nap.
Don't Forget
More Info may be coming... The "$migrationfolder" obviously will get big, as it has a copy of everyone's old mailbox, so just keep it around as long as you need it. Or: delete the "migrationmailboxes" subfolder to save space (wait at least a day or two if possible), and keep the rest of the data for a few weeks (or longer), in case some detail was missed, and you need to see how it was on the old server in the config files that were saved.
Things that do not migrate (or do not migrate well or automatically anyway)
- User Grants (folders that were shared with other users)
- locked accounts (you have to run a script to unlock them remotely, then migrate, then re-local locally, if you need them - see script generated named zmprov.movelockedaccounts
- Not 100% sure if calendar resources are going to properly be connected, I have not had a scenario where I could test this.. yet
- ZCO users will NEED to uninstall and re-install, due to the internal configurations of the ZDB
- Global Signatures
- Custom (or additional) zimlets
- Domain customizations (we downloaded all configs, so if we missed something we could easily find it in the configs)
Conclusion
Not much to say here.. Good luck!
I hope they help others
Created and run originally by user:Richardteachout
Notes
You will definitely need: Using_Screen_for_Session_Mgmt_(Never_have_your_scripts_die_because_you_lost_your_connection) to assist you, just in case.
Comments by other users
Section saved, so in this wiki others can add comments here of their results of using these script, and the scenarios that worked for them
