Difference between revisions of "ZCS to ZCS With-Different-LDAP-Servers Migration Example"

m (Don't Forget: section revision)
m (added unoffical link @ top)
Line 1: Line 1:
 +
{|  width="100%" border="0"
 +
|  bgcolor="orange" | [[Image:Attention.png]] - This article is NOT official Zimbra documentation. It is a user contribution and may include unsupported customizations, references, suggestions, or information.
 +
|}
 
{{WIP}}
 
{{WIP}}
 
<h1>Scripts Provided as an EXAMPLE only</h1>
 
<h1>Scripts Provided as an EXAMPLE only</h1>

Revision as of 20:11, 16 August 2010

Attention.png - This article is NOT official Zimbra documentation. It is a user contribution and may include unsupported customizations, references, suggestions, or information.

Scripts Provided as an EXAMPLE only

Be sure to carefully read/review/modify these scripts as necessary for your situation

What these scripts were used Originally used for

ZCS 6.0.4NE - ZCS 6.0.7NE Migration from one provider to another
A global admin account WAS available for the remote server (old provider)
The (new server) was provisioned and 100% under our control.
This *should* work for just about any zimbra version, from one server to another (between seperate LDAP's)
none of this is necessary if you can do a NE backup and restore.. but if you are (say) moving from a dedicated Zimbra server in someone's office, to a NE cluster (with other domains on it) you couldn't do that, or if you are moving a domain from a cluster with others, to a dedicated server... these scripts might help (although, if you control both LDAP servers 100%, you may be able to use other scripts to also export and import the user passwords through raw ldap, to reduce customer impact)

Scripts/Files used

00_README.txt

These scripts assume you have a global admin on the remote server (at least I did for this example)
They *may* work for a single domain with a domain admin, or at least some of them will
If/when I have to do a migration with only a domain admin privlidge, I will adjust the scripts appropriately to solve the issue, and update this page (or feel free to add some comments to this page)

**Your mileage may vary, use at your own risk
**these scripts are provided for concept, and may need to be customized for your particular situation

It is NOT recommended to use /tmp as your migration folder
Be sure your migration folder is properly secured, and owned by zimbra.zimbra, scripts are chmod 700 to zimbra so they can be executed.

config.sh

#!/bin/bash
oldhost="youroldhostnameorIPhere"
oldadmin="globaladminaccount@oldhost.com"
oldadminpass="globaladminpassword"
newTmpPass="123DomainTemporary@pass321"
migrationfolder=/tmp
newcosid="60668d2f-5200-5c56-828a-201378e53a90" ## THIS IS THE GUID OF YOUR NEW COS TO PUT USERS IN 

01_GetOldDetails.sh

#!/bin/bash

## Run the config script when this script starts to set all the variables to the shell
bash config.sh

cd $migrationfolder

## note, some of what is obtained here is unnecessary, but I wanted to get all that I could, and keep it in case I need it later on due to a quirk

##First, get the list of mailboxes from the old server
## note: zmprov gaa won't work without ldap access, so we have to use zmsoap to get all the accounts :(
zmsoap -z GetAllAccountsRequest -u https://$oldhost:7071/service/admin/soap -a $oldadmin -p $oldadminpass > $migrationfolder/oldgetaccountdetails
cat $migrationfolder/oldgetaccountdetails  | grep '<a n="mail"' |cut -f2 -d">"|cut -f1 -d"<" > $migrationfolder/mailboxlist.txt

## Get other details we may want.... just in case
mkdir $migrationfolder/accounts
mkdir $migrationfolder/cos
mkdir $migrationfolder/grants
mkdir $migrationfolder/domains
mkdir $migrationfolder/resources


######### Domain details
zmprov -s $oldhost:7071 -a $oldadmin -p $oldadminpass gad > $migrationfolder/domainslist.txt
for i in `cat $migrationfolder/domainslist.txt`
do
echo Getting domain Details for $i
zmprov -s $oldhost:7071 -a $oldadmin -p $oldadminpass gd $i > $migrationfolder/domains/$i.details
done

######### COS details
zmprov -s $oldhost:7071 -a $oldadmin -p $oldadminpass gac > $migrationfolder/coslist.txt
for i in `$migrationfolder/coslist.txt`
do
echo Getting COS Details for $i
zmprov -s $oldhost:7071 -a $oldadmin -p $oldadminpass gc $i > $migrationfolder/cos/$i.cosdetails
done

######### Grant details (These can't easily be re-imported, but I wanted to keep a list of what used to be there)
for i in `cat $migrationfolder/mailboxlist.txt`
do
echo Getting Grant Details for $i
zmprov -s $oldhost:7071 -a $oldadmin -p $oldadminpass gsi $i > $migrationfolder/grants/$i.grants
done

######### Distribution list details
zmprov -s $oldhost:7071 -a $oldadmin -p $oldadminpass gadl > $migrationfolder/distlist.txt
mkdir $migrationfolder/distlist
for i in `cat $migrationfolder/distlist.txt`
do
echo Getting Dist list $i
zmprov -s $oldhost:7071 -a $oldadmin -p $oldadminpass gdl $i > $migrationfolder/distlist/$i.distlistdetails
zmprov -s $oldhost:7071 -a $oldadmin -p $oldadminpass gdlm $i > $migrationfolder/distlist/$i.distlistmembership
done

######### mailbox details
for i in `cat $migrationfolder/mailboxlist.txt`
do
echo Getting Mailbox Details and account membership for $i
zmprov -s $oldhost:7071 -a $oldadmin -p $oldadminpass ga $i > $migrationfolder/accounts/$i.accountdetails
zmprov -s $oldhost:7071 -a $oldadmin -p $oldadminpass gam $i > $migrationfolder/accounts/$i.accountmembership
zmprov -s $oldhost:7071 -a $oldadmin -p $oldadminpass gsig $i > $migrationfolder/accounts/$i.accountsignature
done

######### Calendar Resource details
echo Getting All calendar Resources
zmprov -s $oldhost:7071 -a $oldadmin -p $oldadminpass gcr > $migrationfolder/resources.txt
for i in `cat $migrationfolder/resources.txt`
do
echo Getting resource $i
zmprov -s $oldhost:7071 -a $oldadmin -p $oldadminpass gcr $i > $migrationfolder/resources/$i.resourcedetails
done

######### Global Configuration details
echo getting global Configuration Details
zmprov -s $oldhost:7071 -a $oldadmin -p $oldadminpass gacf > $migrationfolder/gacf.txt

02_MakeMigrationScripts.sh

#!/bin/bash

## Run the config script when this script starts to set all the variables to the shell
bash config.sh

#########   THIS SCRIPT DOES NOT ACTUALLY CREATE ANYTHING, IT JUST MAKES SCRIPTS WHICH ARE TO BE RUN TO DO THE ACTUAL WORK (zmprov.* files)
## Note, the zmprov.* files DO have passwords in them, so be sure they are kept safe

cd $migrationfolder

########### Make Domains creation script  (this assumes you are migrating all domains.. adjust appropriately)
rm -f $migrationfolder/zmprov.createdomains
echo "#It is suggested to create these manually to set all the COS right, etc" > $migrationfolder/zmprov.createdomains
echo "#This script is just for concept" > $migrationfolder/zmprov.createdomains
echo "#Use only with good judgement" > $migrationfolder/zmprov.createdomains
echo "#" > $migrationfolder/zmprov.createdomains
for i in `cat $migrationfolder/domainslist.txt`
do
echo "# You should create domain $i manually, to set all the right settings"
echo "#zmprov cd $i" >> $migrationfolder/zmprov.createdomains
done

########### Make Distribution list creation script
rm -f $migrationfolder/zmprov.createdistlists
for i in `cat $migrationfolder/distlist.txt`
do
echo zmprov cdl $i >> $migrationfolder/zmprov.createdistlists
members=""
for j in `cat $migrationfolder/distlist/$i.distlistdetails | grep zimbraMailForwardingAddress | cut -f2 -d " "`
do
members="$members$j "
done
displayName=`cat $migrationfolder/distlist/$i.distlistdetails | grep displayName | cut -f2 -d ":"|awk '{gsub(/^ +| +$/,"")}1'`
cn=`cat $migrationfolder/distlist/$i.distlistdetails | grep cn: | cut -f2 -d ":"|awk '{gsub(/^ +| +$/,"")}1'`

echo zmprov modifyDistributionList $i cn \"$cn\"  displayName \"$displayName\"  >> $migrationfolder/zmprov.createdistlists
echo zmprov adlm $i $members >> $migrationfolder/zmprov.createdistlists
for k in `cat $migrationfolder/distlist/$i.distlistdetails | grep zimbraMailAlias | cut -f2 -d " "`
do
echo zmprov adla $i $k  >> $migrationfolder/zmprov.createdistlists
done
done

########### Make Alias creation script

rm -f $migrationfolder/zmprov.createaliases
for i in `cat $migrationfolder/mailboxlist.txt`
do
for j in `cat $migrationfolder/accounts/$i.accountdetails | grep -i zimbraMailAlias | cut -f2 -d " "`
do
echo zmprov aaa $i $j >> $migrationfolder/zmprov.createaliases
done
done

########### Make Calendar Resource creation script  (Not 100% sure these imports are usable/correlatable to a calender after importing
##########                                           My migration had them in there, but they weren't using them - usage may vary)
rm -f $migrationfolder/zmprov.createcalres

for i in `cat $migrationfolder/resources.txt`
do
displayname=`cat resources/$i.resourcedetails| grep displayName| cut -f2 -d ":"|awk '{gsub(/^ +| +$/,"")}1'`
zimbraCalResType=`cat resources/$i.resourcedetails| grep zimbraCalResType| cut -f2 -d ":"|awk '{gsub(/^ +| +$/,"")}1'`
echo zmprov createCalendarResource $i $newTmpPass displayName \"$displayname\" zimbraCalResType \"$zimbraCalResType\">> $migrationfolder/zmprov.createcalres
done

########### Make zimbraMailSieveScript (webmail filters)
rm -f /tmp/tmpsieve

for i in `cat $migrationfolder/mailboxlist.txt`
do
echo -n "zmprov ma $i zimbraMailSieveScript '" >> /tmp/tmpsieve
cat accounts/$i.accountdetails | sed -n '/zimbraMailSieveScript: /,/zimbraMailSignatureMaxLength/p' | sed 's/zimbraMailSieveScript: //g'|grep -v zimbraMailSignatureMaxLength >> /tmp/tmpsieve
echo "'" >> /tmp/tmpsieve
done
cat /tmp/tmpsieve | grep -v "zimbraMailSieveScript '$" > zmprov.recreatesieve
## NOTE:  IF SOMEONE HAS A ' in their sieve script, this may not re-import right.. update the name of the filter manually in this script if that errors, and re-run

########### Force password Change script
for i in `cat $migrationfolder/mailboxlist.txt`
do
echo "zmprov ma $i zimbraPasswordMustChange TRUE " >> $migrationfolder/zmprov.zzz_forcepasswordchange
done

###########  Make user creation script

rm -f $migrationfolder/zmprov.createusers
rm -f $migrationfolder/zmprov.createsigs
echo "#Note, signature creation script may die if there is a ' in it." > $migrationfolder/zmprov.createsigs

for i in `cat $migrationfolder/mailboxlist.txt`
do
# For debugging
echo Working on $i

#### ************************* FOR DOING THIS YOU NEED TO READ HERE !!******************************
## FOR THIS MIGRATION, I WAS PROVIDED A TXT FILE userpass.txt THAT CONTAINED username|password
## IF YOU DO NOT HAVE SUCH A FILE, OR THE USER DOESN'T EXIST IN THAT FILE, IT WILL USE THE TMP PASS CONFIGURED ABOVE
## If you have access to the old LDAP, and are experienced with modifying LDAP, you may be able to migrate the passwords..
## 
touch $migrationfolder/userpass.txt
user=`echo $i|cut -f1 -d "@"`
passline=`cat $migrationfolder/userpass.txt | grep -i "$user|"`
password=`echo $passline|cut -f2 -d"|"`
if [ "x$password" == "x" ]; then
password=$newTmpPass
fi
displaynamea=`cat $migrationfolder/accounts/$i.accountdetails|grep displayName|cut -f2 -d ":"|awk '{gsub(/^ +| +$/,"")}1'`
givenname=`cat $migrationfolder/accounts/$i.accountdetails|grep givenName|cut -f2 -d ":"|awk '{gsub(/^ +| +$/,"")}1'`
lastname=`cat $migrationfolder/accounts/$i.accountdetails|grep sn:|cut -f2 -d ":"|awk '{gsub(/^ +| +$/,"")}1'`
zimbraMailQuota=`cat $migrationfolder/accounts/$i.accountdetails|grep zimbraMailQuota|cut -f2 -d ":"|awk '{gsub(/^ +| +$/,"")}1'`

signature=`cat $migrationfolder/accounts/$i.accountsignature | sed -n '/zimbraPrefMailSignature: /,/zimbraSignatureId/p' | sed 's/zimbraPrefMailSignature: //g'|grep -v zimbraSignatureId|while read line; do echo -n "$line\n"; done`

cos="zimbraCOSId $newcosID"

echo zmprov ca $i $password displayname \"$displaynamea\" givenName \"$givenname\" sn \"$lastname\" $cos zimbraMailQuota $zimbraMailQuota >> $migrationfolder/zmprov.createusers

if [ "x$signature" != "x" ]; then
signame=`cat $migrationfolder/accounts/$i.accountsignature|grep zimbraSignatureName|sed 's/zimbraSignatureName: //g'`
echo zmprov csig $i \"$signame\" zimbraPrefMailSignature \' >> $migrationfolder/zmprov.createsigs
cat $migrationfolder/accounts/$i.accountsignature | sed -n '/zimbraPrefMailSignature: /,/zimbraSignatureId/p' | sed 's/zimbraPrefMailSignature: //g'|grep -v zimbraSignatureId |while read line; do
echo "$line" >> $migrationfolder/zmprov.createsigs;
done
echo "'" >> $migrationfolder/zmprov.createsigs
echo zmprov ma $i zimbraSignatureName \"$signame\" >> $migrationfolder/zmprov.createsigs
fi

done




########### Make the Actual Migration Scripts to be used @ cutover time
##
mkdir $migrationfolder/migrationscripts
mkdir $migrationfolder/migrationmailboxes

migratescriptmaster="$migrationfolder/migratescriptmaster"
rm -f $migratescriptmaster
rm -f $migratescriptmaster.restore
touch $migratescriptmaster
touch $migratescriptmaster.restore
counter=1
for i in `cat $migrationfolder/mailboxlist.txt`
do
echo "echo $counter of 25" >> $migratescriptmaster
echo "echo working on Downloading $i" >> $migratescriptmaster
echo "zmmailbox -u https://$oldhost:7071 -a $oldadmin -p $oldadminpass -m $i getRestURL \"//?fmt=tgz\" > $migrationfolder/migrationmailboxes/$i.tgz" >> $migratescriptmaster
echo "echo $counter of 25" >> $migratescriptmaster.restore
echo "echo Working on Restoring $i" >> $migratescriptmaster.restore
echo "zmmailbox -z -m $i postRestURL \"//?fmt=tgz&resolve=replace\" $migrationfolder/migrationmailboxes/$i.tgz" >> $migratescriptmaster.restore
counter=`expr $counter + 1`
done
rm -f migrationscripts/*_*
# splitting by 75, as I wanted 25 users in a batch, and each one is 3 lines long in the script file
# Then I started a number of screen sessions, and started each one in a window, so I could run 15 or 20 scripts at a time
## See my Wiki page on using "screen"
split -l 75 $migratescriptmaster $migrationfolder/migrationscripts/getdata_
split -l 75 $migratescriptmaster.restore $migrationfolder/migrationscripts/restorelocally_
# Note:  This displays the numbering system slightly odd when running it, as it shows "304 of 25" but I wanted it that way, so I could also see the total process point

Process / What to do

Here is basically what I did:

Planning

Ensured I was able to get all the data I would need for the migration (Ran 01_getolddetails.sh)

Pre-Setup

  • Once the local Zimbra was installed, I manually created the LDAP, security, settings, and COS/domains that I wanted
  • Then I ran the 02_MakeMigrationScripts.sh and got my setup scripts read
  • Run the appropriate scripts (zmprov.*) scripts, I ran them in this order
    • zmprov.createusers
    • zmprov.createaliases
    • zmprov.createcalres
    • zmprov.createdistlists
    • zmprov.createdistlistaliasesonly
    • zmprov.createsigs
    • zmprov.recreatesievescript (I actually re-ran this at the end too, so it would hopefully link up to the right folders, after they exist- i.e. post import)
    • zmprov.zzz_forcepasswordchange is optional, if you were not provided passwords, however, it is highly suggested - it sets all accounts to be forced to change password on first login

Post-Setup testing

yes, you should test for mail flow, firewall access, etc, to be sure the server works, setup antispam, etc, etc.. once everything looks good, you should be ready.

Prep for Go-Live

Make sure you control DNS for this domain

  • Change TTL to say 5 minutes, to prepare for this migration
  • Be sure all users know when it is going to happen, and that right after it happens, they will NOT see any of their old data in their accounts immediately, but that it will slowly start re-appearing after a few hours. New email will not get lost.

Migration Day

  • At the point of "go-live", change the MX records, give it about 10 mins to ensure new mail is flowing to new server
  • Then Start running the migrationscripts/* scripts using screen (see link to screen page at end) (you can probably run a number of them at one time, depending on the size of the source mailboxes. Restoring will take 3x longer than downloading, so calculate entire time appropriately. (obviously, run the getdata_** scripts, before the restorelocally_** scripts.. I ran the getdata_** scripts in seperate screens, then as each one finished, I started the restorelocally_** script for that "batch" and recorded the times as I went, so I knew my entire process timeframe.
  • Watch the process.
  • watch the logs the entire time too for errors

Process

In a nutshell, you just change the MX records, after they have moved, start migrating old data, and start watching logs for anything odd.. after migration, take a long nap.

Don't Forget

More Info may be coming... The "$migrationfolder" obviously will get big, as it has a copy of everyone's old mailbox, so just keep it around as long as you need it. Or: delete the "migrationmailboxes" subfolder to save space (wait at least a day or two if possible), and keep the rest of the data for a few weeks (or longer), in case some detail was missed, and you need to see how it was on the old server in the config files that were saved.

Things that do not migrate (or do not migrate well or automatically anyway)

  • User Grants (folders that were shared with other users)
  • locked accounts (you have to run a script to unlock them remotely, then migrate, then re-local locally, if you need them - see script generated named zmprov.movelockedaccounts
  • Not 100% sure if calendar resources are going to properly be connected, I have not had a scenario where I could test this.. yet
  • ZCO users will NEED to uninstall and re-install, due to the internal configurations of the ZDB
  • Global Signatures
  • Custom (or additional) zimlets
  • Domain customizations (we downloaded all configs, so if we missed something we could easily find it in the configs)

Conclusion

Not much to say here.. Good luck!
I hope they help others
Created and run originally by user:Richardteachout

Notes

You will definitely need: Using_Screen_for_Session_Mgmt_(Never_have_your_scripts_die_because_you_lost_your_connection) to assist you, just in case.

Comments by other users

Section saved, so in this wiki others can add comments here of their results of using these script, and the scenarios that worked for them

Jump to: navigation, search