User Migration

Revision as of 21:37, 26 April 2009 by Gettyless (talk | contribs)


Migrating your users means you're going to have to deal with mail, calendar, contacts and passwords. This page used to be one monolith sized monstrosity -- but has now been broken down into sections:

Mail Migration

Mail Migration

Password Migration

Migrating Users Password from Postfix Admin

Save this script as something.php, make chmod +x something.php and run. Script will export all email accounts into designated file.

#!/usr/bin/php5

// Postfix.admin to Zimbra import
//
// Created by Jarosław Czarniak on 26-10-2008
// Trivial bug fixed by Luca G. on 18-01-2009

<?
/////////////////////////////////////////////////////////

$user="mysql_login";
$pass="mysql_pass";
$base="mysql_database";
$tabl="mailbox"; //table
$file="exported.sh";

/////////////////////////////////////////////////////////
echo "Usage: as \"zimbra\" user on destination server:\n";
echo "# sh ./exported.sh\n\n";
echo "";

$mydb = mysql_connect('localhost',$user, $pass) or die ('Błąd połączenia z serwerem');
mysql_select_db($base);
mysql_query("SET CHARACTER SET utf8");
mysql_query("SET NAMES utf8");

$query = "SELECT username,password,name,maildir,quota,domain FROM ".$tabl;
$dane = mysql_query($query) or die ('Błąd podczas zapytania do bazy1'.mysql_error());

$handle = fopen($file, "w");

while ($row = mysql_fetch_array($dane, MYSQL_NUM))
{
    $StringData = "zmprov ca ".$row[0]." dsfs123hsdyfgbsdgfbsd displayName '".$row[2]."'\n";
    fwrite($handle, $StringData);
    $StringData = "zmprov ma ".$row[0]." userPassword '{crypt}".$row[1]."'"."\n";
    fwrite($handle, $StringData);
}

?>

Migrating User Password from Shadow Password to Zimbra

Basically we will use the hash password from /etc/shadow. Get the hash values in between the first and second colon only. Eg:

user1:$1$MWAOIgPB$skcX.nKYV9JSctTR60uat/:13780:0:99999:7:::

Then run this as user zimbra.

zmprov ma user1@domain userPassword '{crypt}$1$MWAOIgPB$skcX.nKYV9JSctTR60uat/'

A simple batch script to migrate bulk shadow password into zimbra password as below:

#!/usr/bin/perl
# Usage: as root   # ./shadow2zm.pl /etc/shadow > shadow.zm
#        as zimbra # zmprov < shadow.zm

$domain="my.domain.com";

while(<>) {
    chomp;
    my ($uname,$pass) = split(/:/);

    print qq{zmprov ma $uname\@$domain userPassword '{crypt}$pass'\n};
    print qq{\n};
}

Credit to bewley from this forum post

./scalper

In some cases above perl script doesn't work very well. And if user doesn't already exist you will get en error. My script will import and create all users from shadow file (with exception of system accounts).

#!/bin/bash

# Shadow to Zimbra import
#
# Created by Jarosław Czarniak on 26-10-2008
#

clear
echo "Usage: as \"zimbra\" user on destination server"
echo "# zmprov < shadow.file"

domain="jarsat.pl"  # change to your domain!
file="shadow.file"
x=0
echo ''>$file

for linia in `cat /etc/shadow`
do
    user=`echo $linia|cut -f1 -d":"`
    pass=`echo $linia|cut -f2 -d":"`

    if [ "$pass" != "*" ]
    then
        if [ "$pass" != "!" ]
        then
            echo "zmprov ca $user@$domain temppasswordQAZXSW displayName $user">>$file
            echo "zmprov ma $user@$domain userPassword '{crypt}$pass'">>$file
            x=$[x+1]
        fi
    fi
done

echo
echo
echo "$x accounts exported to \"$PWD/$file\""

sleep 5

Calendar and Contacts Migration

If your mail migration strategy doesn't cover contacts and calendar, you can import via the REST (REpresentational State Transfer) [1] interface. (Currently, contacts and calendar are only imported with the Exchange migration tool and the PST Import Wizard, so this applies to most IMAP migrations).

Overview

The basic procedure is this:

  1. Export calendar or contact data from your existing server into a csv or ics file
  2. Migrate that data file to a host that can access the zimbra server
  3. Use the REST interface to insert the data into Zimbra

REST overview

Today, within the Zimbra Collaboration Suite we have a number of different server-side URLs that our client accesses to download an attachment, export contacts as CSV, export a calendar as an ICS, file etc. We are also adding sharing (what would collaboration be without sharing, after all) of calendars, contacts, etc. Not only within a particular Zimbra community, but between Zimbra communities and the public at large.

In order to facilitate this, we are coming up with a clean, consistent URL interface to all our resources. The best way to describe this is with some examples.

Lets say I want access to my calendar folder from within iCal. The URL would look like:

 http://server/zimbra/user/roland/calendar

The default format on calendar folders is ICS, so no need to specify the format.

Lets say I want to export my contacts folder so I can import them into another account:

 http://server/zimbra/user/roland/contacts

Contact folders have a default type of CSV, so like calendars, no need to specify the format.

How about an RSS feed of unread messages in my inbox:

 http://server/zimbra/user/roland/inbox.rss?query=is:unread

By specifying an extension of ".rss" on the inbox folder, the server will automatically generate an RSS feed on it. Adding the "query" parameter lets me further refine what gets returned. You can also specify "?fmt=rss" instead of using the ".rss" extension if you'd like. You can also use the encoding for double quotes, %22 (For example, inbox.rss?query=%22is:unread%22)

Lets do something a little more interesting. How about a zip file containing all messages in my talks/ajax folder:

 http://server/zimbra/user/roland/talks/ajax.zip

The server zips them all up and returns the zip file.

Another interesting example is say you have created a public calendar that you want to share with everyone. Once you have granted access to the calendar, it is up to the consumer to chose what format they want to view it in:

 http://server/zimbra/user/roland/calendar/talks.ics
 http://server/zimbra/user/roland/calendar/talks.html?view=month
 http://server/zimbra/user/roland/calendar/talks.atom
 http://server/zimbra/user/roland/calendar/talks.xml
 http://server/zimbra/user/roland/calendar/talks.txt

How about accessing another user's calendar/folder? Once they grant you access, you can use the same exact syntax:

 http://server/zimbra/user/janie/holidays.ics

One last interesting example to leave you with. Lets say you have a friend at widgets.com who you know is running Zimbra and who has shared their calendar with you, but you don't know the name of their public Zimbra server. As long as they publish some DNS SRV records for _zimbra._tcp.widgets.com, then you can access it directly from your Zimbra without needing to know his server's address:

 http://server/zimbra/user/friend@widgets.com/calendar

This last format (user@domain.com) is also necessary when accessing resources in a non-default domain on your local Zimbra server.

REST file formats

see this reference page: REST File Formats

Importing into Zimbra

We've added the ability to update/create content by POST'ing content to REST urls. GETs continue to remain read only, with no side-effects (as they should be).

For example, you can POST an RFC822 formatted message to your inbox REST url to append messages to the inbox folder. Using the popular curl program, this would look like the following:

 curl -u schemers:password --data-binary @/tmp/rfc822.txt https://server/service/home/schemers/inbox

Note, you currently have to use /service/home for POSTs instead of /zimbra/home, because /zimbra/home issues a redirect, which isn't allowed with POSTs. We'll be fixing that in an upcoming release.

Other items that can be updated this way are calendar appointments (ICS), and contacts (csv and the new vcf (vCard) format):

 curl -u username:password --data-binary @/tmp/new.csv http://server/service/home/username/contacts?fmt=csv
 curl -u username:password --data-binary @/tmp/new.ics http://server/service/home/username/calendar?fmt=ics
 curl -u username:password --data-binary @/tmp/new.vcf http://server/service/home/username/contacts?fmt=vcf

The full user name (user@domain.com) is required when importing to resources on a non-default domain on your local Zimbra server.

NOTE - this works for resource accounts as well.

The target "contacts" in the above command corresponds to the actual folder name on the Zimbra server, rather than a service. This means that you can import to user-defined folders by specifying the appropriate target name in the command. If you have created a folder named "SharedContacts", for example, the command to import into this folder would be

 curl -u username:password --data-binary @/tmp/new.csv http://server/service/home/username/sharedcontacts?fmt=csv


On later version's (such as 5.0.10+) instead of --data-binary, you should use --upload-file.

For example:

curl --insecure -u admin:password --upload-file /tmp/test.csv
https://server/service/home/test1@domain.com/contacts?fmt=csv

Alternatively, you can use zmmailbox pru:

zmmailbox -z -m test1@domain.com pru /Contacts /tmp/test.csv

Migrating Contacts

Eudora

  • In Eudora, select the Address Book that you wish to import to Zimbra. (Select Tools/Address Book) and then Click on the correct Book.
  • Select File/Save As, Save as type: CSV Files (*.csv) and type in an appropriate file name. Click Save
  • Open Notepad, Open the file.csv that you saved in step 2.
  • Insert the following line EXACTLY. (Must be the first line in the file.) nickname,email,fullName,firstName,lastName,homeStreet,homeCity,homeState,homeCountry,homePostalCode,homePhone,homeFax,homePhone2,homeURL,company,jobTitle,workStreet,workCity,workState,workCountry,workPostalCode,workPhone,workFax,workPhone2,workURL,email2,otherPhone,otherURL,notes
  • (Make sure to press enter after the above line and save and close notepad.)
  • Login to Zimbra, and go to Options/Address Book, Click Browse and select the file. Click import.
  • Select the address book to import into or select New to create new address book. Wait until box in bottom left hand corner says import complete.
  • Click Address Book and make sure the contacts imported.

Thunderbird

Zimbra To Zimbra Server Migration

as a whole method

Unofficial method (work with 5.0, may work with <5.0): http://www.zimbra.com/forums/migration/22697-zimbra-zimbra-migration-script.html

Official method (only work > 5.0.9) http://www.zimbrablog.com/blog/archives/2008/09/zcs-to-zcs-migrations.html

item to item method

Here is a script that is a modification of the script above for using curl to migrate users between zimbra boxes using imap protocol. It also transfers calendars and contacts from zimbra server to zimbra server. I just started working on it and here is what i have so far. It worked on two servers i just migrated last week, i hope it helps :)

NOTE: Modified perl below: was not using IO::Scalar or File::Path so no need to use; now supports user names with "@" via wget; support for command line arguments; support for SSL imap; support for various input file separators; other samll tweaks - scottp@dd.com.au

#!/usr/bin/perl
use strict;
use warnings;

#############################################################################
# Please make changes below to suit your system and requirements

# NOTE: hard code host1 & host2 - or pass in as first two parameters to script
my $host1=shift; #host1 is Source
my $host2=shift; #host2 is Dest

# NOTE: enable ssl for imapsync
my $imapsync = " -ssl1 -ssl2";

##############################
# NOTE: Do not set these here - they come from userlist.txt
my $user;
my $pass;

my $data_file="userlist.txt";
open(DAT, $data_file) || die("Could not open file!");
while (<DAT>) {
        chomp;
        # NOTE: userlist.txt can have username@domain password (or comma, tab or |)
        ($user,$pass)=split(/[\|\s\t,]+/,$_);

        ##############
        open (TESTOUT, "imapsync --buffersize 18192000 --nosyncacls --subscribe --syncinternaldates --noauthmd5 --host1 $host1 --user1 $user --password1 $pass --host2 $host2 --user2 $user --password2 $pass $imapsync|");
        while (<TESTOUT>){
                print $_;
        }
        close TESTOUT;

        ### get contacts
        open (TESTOUT, "wget --user $user --password $pass https://$host1/zimbra/user/$user/contacts.csv --no-check-certificate|");
        while (<TESTOUT>){
                print $_;
        }
        close TESTOUT;

        ### get calendar
        open (TESTOUT, "wget --user $user --password $pass https://$host1/zimbra/user/$user/calendar.ics --no-check-certificate |");
        while (<TESTOUT>){
                print $_;
        }
        close TESTOUT;

        ### import calendars
        open (TESTOUT, "curl -u $user:$pass --data-binary \@calendar.ics https://$host2/service/home/$user/calendar?fmt=ics --insecure|");
        while (<TESTOUT>){
                print $_;
        }
        close TESTOUT;

        ### import contacts
        open (TESTOUT, "curl -u $user:$pass --data-binary \@contacts.csv https://$host2/service/home/$user/contacts?fmt=csv --insecure|");
        while (<TESTOUT>){
                print $_;
        }
        close TESTOUT;

        ### remove files
        unlink 'calendar.ics';
        unlink 'contacts.csv';
        ###############
}

users and passwords go in a text file userlist.txt in the format: username|password.

With some versions of wget you may need to specify the username and password as command line arguements (ie --http-passwd=thepass --http-user=user@dom.com) instead of in the URL (ie https://usersname:pass@theurl). If you are getting the error Bad Port, then give this a try.

Copy Calendar From One Zimbra User to Another

In this example we will export a calendar called "LPO calendar" from one user to another user and name it "LPO" in the destination account.

  • Export the calendar "LPO calendar" from user SRC_USER:
[root@zimbra ~]# curl -u admin 'https://zimbra:7071/home/SRC_USER/LPO%20calendar?fmt=ics' > LPO.ics
  • Adjust all the meeting organizer addresses. If you don't do this, the destination user will see the appointments, but can not edit them.
[root@zimbra ~]# cat LPO.ics | sed 's/SRC_USER@/DST_USER@/g' > LPO-dest.ics
  • Create empty calendar in destination user account.
[root@zimbra ~]# su - zimbra -c 'zmmailbox -z -m DST_USER createFolder --view appointment /LPO'
  • Import modified ical data into new calendar.
[root@zebra ~]# curl -u "admin" --data-binary @LPO-dest.ics 'https://zimbra:7071/home/DST_USER/LPO?fmt=ics'


Copy All Messages, Contacts, Calendars, and Mail Filters

I have written a script, which can be run in batch, to copy user data between servers or between mailboxes. This will copy contacts and calendars from every folder no just /Contacts and /Calendar. I used this script two times to copy pretty much everything important when migrating from 4.5.8 to 5.0.2. It doesn't support shared folders, task lists, documents, briefcase, and some other things (I recreated shared folders manually with zmmailbox) but support could be added given enough motivation :)

ZCS Tools currently contains these Migration scripts written in Python/Bash. These scripts are packaged as zimbraMigration-0.x and require imapsync to be installed.

Import distribution list

I wrote a simply bash script to migrate distribution lists from one Zimbra server to another it creates a provisioning script for each list.

myPath=$(pwd)
/opt/zimbra/bin/zmprov gadl | while read listname;
do
   echo "/opt/zimbra/bin/zmprov cdl $listname" > $myPath/$listname
   /opt/zimbra/bin/zmprov gdl $listname | grep zimbraMailForwardingAddress >  $myPath/$listname.tmp
   cat $myPath/$listname.tmp | sed 's/zimbraMailForwardingAddress: //g' |
   while read member; do
     echo "/opt/zimbra/bin/zmprov adlm $listname $member" >> $myPath/$listname
   done
   /bin/rm $myPath/$listname.tmp
done

Migrating Sieve Filter Rules

You have a few options to migrate Sieve rules. Generally you can take existing rules and then just dump them into the account attribute zimbraMailSieveScript. The server will be able to process these rules. However, if you intend to enable the client to edit the rules you will need to make sure they conform to a ZCS-specific set of characteristics. There are two ways to do this.

Using zmmailbox addFilterRule

Within the zmmailbox command there is a "addFilterRule" command. This allows you to add a named filter to a particular user. Its usage defines the parameters that ZWC can understand for editing. Given that for any migration scenario you'll likely end up parsing the rules, this will let you reconstruct the rules in a syntax ZWC understands. You will need to find a "name" for each rule. It could be as simple as "rule1", "rule2", etc.

Here's the usage for addFilterRule [zimbra@qa03 ~]$ zmmailbox help filter

 addFilterRule(afrl)                    add filter rule
 deleteFilterRule(dfrl)                 add filter rule
 getFilterRules(gfrl)                   get filter rules
 modifyFilterRule(mfrl)                 add filter rule
 {conditions}:
   header "name" is|not_is|contains|not_contains|matches|not_matches "value"
   header "name" exists|not_exists
   date before|not_before|after|not_after "YYYYMMDD"
   size under|not_under|over|not_over "1|1K|1M"
   body contains|not_contains "text"
   addressbook in|not_in "header-name"
   attachment exists|not_exists
 {actions}:
   keep
   discard
   fileinto "/path"
   tag "/tag"
   mark read|flagged
   redirect "address"
   stop

Also, for add filter rule: mbox> afrl usage:

 addFilterRule(afrl)          [opts] {name}  [*active|inactive] [any|*all] {conditions}+ {actions}+
   -a/--after <arg>             add after filter-name
   -f/--first                   add as first filter rule
   -l/--last                    add as last filter rule
   -b/--before <arg>            add before filter-name


Each zmmailbox script invokes a Java VM (just like zmprov). So you'll want to create a script to be loaded into zmmailbox. That script would look something like
adminAuthenticate ...
selectMailbox user1 ...
addFilterRule rule1 header "from" contains "kevin" discard
addFilterRule rule2 ....
...
selectMailbox user2
addFilterRule ....
...

Then you can run it via zmmailbox < scriptname.

Processing a Sieve script and inserting it via zmprov

Alternately you can process the existing Sieve scripts to conform to ZCS expectations and then insert the whole script via zmprov ma account zimbraMailSieveScript ..... You'll need to restrict the entered rules to the Sieve commands as documented above (if ZWC or ZCO are expected to edit them). Each filter rule will need a comment above it; this comment is the filter's name as identified in addFilterRule. Creating one or two simple rules using the web client will give you a framework for the created rule.


Troubleshooting

Please see User Migration Troubleshooting if you are having problems migrating data to your Zimbra installation.

Jump to: navigation, search