Ajcody-Migration-Notes: Difference between revisions

mNo edit summary
Line 4: Line 4:


Please see [[Ajcody-Migration-Notes]]
Please see [[Ajcody-Migration-Notes]]
===RFE/Bug's For Migration Tools===
I'll try to keep track of items here when I find them.
* Migration Tool zimbra to zimbra
** http://bugzilla.zimbra.com/show_bug.cgi?id=19630
*** I've made comments to this one, please see [http://bugzilla.zimbra.com/show_bug.cgi?id=19630#c10 c#10]. Has some suggestions about expanding zmbackup and zmrestore for migrations.


===Server Migrations Terminology===
===Server Migrations Terminology===
Line 38: Line 46:
:* Can be setup fairly fast and easily.
:* Can be setup fairly fast and easily.
:* Good for testing since it requires very little modifications to the existing system.
:* Good for testing since it requires very little modifications to the existing system.
===RFE/Bug's For Migration Tools===
I'll try to keep track of items here when I find them.
* Migration Tool zimbra to zimbra
** http://bugzilla.zimbra.com/show_bug.cgi?id=19630
*** I've made comments to this one, please see [http://bugzilla.zimbra.com/show_bug.cgi?id=19630#c10 c#10]. Has some suggestions about expanding zmbackup and zmrestore for migrations.


===Staged or Split-Domain Migration===
===Staged or Split-Domain Migration===

Revision as of 21:17, 6 September 2008

Migration Issues

Actual Migration Notes Homepage

Please see Ajcody-Migration-Notes

RFE/Bug's For Migration Tools

I'll try to keep track of items here when I find them.

Server Migrations Terminology

Staged

Staged describes a migration will take place over time (days, weeks, months). Where users are moved to the "new" server over these time frames. This process might involve a Split-Domain or Sub-Domain configuration or the use of LDAP replica's that would integrate a new ZCS mailstore server.
Reasons For Staged Migrations

Normally, the split-domain option is used for one of these two reasons:

  • Very large users base and can't migrate everyone over is one downtime window. Must migrate over the course of multiple downtime windows spanning across weeks or months.
    • One of the mailserver might be over a remote network link that prevents migration data to be moved fast enough in one downtime window.
  • The "old" mail system must stay up because some users will not be migrated to Zimbra.

Split-Domain (Shared Mail Domain Between Multiple Email Server Types/Vendors

Split-Domain means that your going to have mail servers from different vendors (Zimbra, MS Exchange, Sendmail, Postfix, etc.). One of these servers+vendor-type will be authoritative (Primary) for the email domain. The other servers (Secondaries) will still be in the same email domain, rather than a sub-domain, and use the Primary server for authoritative type functions.
Reasons For Split-Domains
  • Allows the highest possible merger between the two systems to minimize the impact to the end users.
  • Doesn't require a final domain rename as a final step to the server migration.
    • This can cause some disruption to some user items like - calendars, appointments, client reconfiguration, etc.

Sub-Domain (New Sub-Domain For Mail To Integrate Into Existing Mail Root-Domain - Multiple Email Server Types/Vendors)

Sub-Domain means we'll add a new email sub-domain to the primary email domain to migrate users over to. It doesn't have to be an actual dns sub-domain, i.e. company.com > sub.company.com . This configuration will use forwarding & domain masquerading rules between the two email systems to allow users to move to the new email system over time. Once everyone is moved over, it can be reconfigured to use the primary domain only and drop the use of the sub-domain.
Reasons For Sub-Domains
  • Administrator wants to avoid the possibility of very complex configurations of the other options.
  • Can be setup fairly fast and easily.
  • Good for testing since it requires very little modifications to the existing system.

Staged or Split-Domain Migration

Required Reading Overview & Resources

Review the following articles:

Split-Domain Example 1 - Zimbra Primary & Exchange Secondary

My attempt to rewrite Split_Domain for customers that are having problems following it for whatever reasons.

Rough Notes - Working On It Now

Hostnames used for server configurations when referencing each other MUST be using internal ip addresses/hostnames for resolution. This will avoid unnecessary routing out of the local network and then back in through your firewall, thus getting a new IP address.


Details of Firewal and/or External Mail Filter :

  • Inbound email goes first to:
    • Firewall : homename =
      • external ip's =
      • internal ip's =
    • External Mail Filter Server : hostname = mailfilter.YYY.state.XX.us
      • external ip's =
      • internal ip's =
    • Will then forward email to primary.YYY.state.XX.us via ip address after implementation


Configuring The New Zimbra Server As The Primary :

  1. Primary Mail Server ("authoritative" system for domain YYY.state.XX.us)
    • Zimbra Server
      • Hostname is primary.YYY.state.XX.us
      • mail domain = YYY.state.XX.us
    • The primary MTA must
      1. Be aware of, and accept mail for, all accounts on the domain
        1. Create ALL accounts for Zimbra that are on existing Exchange box
          • $ zmprov ca secondary-user@YYY.state.XX.us <some_random_password>
          • Review wiki on Bulk_Create . Don't forget about system accounts that you have configured for email
      2. Forward mail to the secondary MTA for users hosted there.
        1. Setup Transport for users to point to the Exchange Server
          • $ zmprov ma secondary-user@YYY.state.XX.us zimbraMailTransport smtp:secondary.YYY.state.XX.us:25
          • Again, review wiki on Bulk_Create to this easier.
      3. Reject on RCPT TO or configure a catchall account for those accounts that do not exist in either system
      4. LAST STEP - Configure DNS, MX records, firewall to make Zimbra server authoritative for domain.
        1. You will bounce mail if you make this change before configuring the entire system and testing that mail flow is working as desired.)
        2. Change your MX record so mail from the internet flows into the Zimbra MTA first.


Re-configuring The Old Exchange Server As The Secondary :

  1. Seconardy Mail Server
    • Exchange Server
      • Hostname is secondary.YYY.state.XX.us
      • mail domain = YYY.state.XX.us
    • The secondary MTA must
      1. Accept mail for accounts that are hosted on the secondary
        1. If this was a Zimbra Server, you would use these commands (domain already exists in below example of course):
          • The first two commands (in combination) tell the Zimbra postfix to accept all addresses in the @example.com domain as valid addresses.
            • $ zmprov md YYY.state.XX.us zimbraMailCatchAllAddress @YYY.state.XX.us
            • $ zmprov md YYY.state.XX.us zimbraMailCatchAllForwardingAddress @YYY.state.XX.us
      2. But must forward all other mail for accounts on this domain to the primary system
        1. If this was a Zimbra Server, you would use this command (domain already exists in below example of course).
          • This third command establishes default mail routing for the domain. Any users that do not exist on the Zimbra system will have their mail routed according to this rule.
            • $ zmprov md YYY.state.XX.us zimbraMailTransport smtp:primary.YYY.state.XX.us
      3. We also highly recommend that you turn off DNS lookups and internet wide message routing from the secondary host and route all mail through the primary.
        1. on a secondary Zimbra system you would:
          • Relay mail to Primary with:
            • $ zmprov mcf zimbraMtaRelayHost primary.YYY.state.XX.us
          • Turn off DNS lookups with:
            • $ zmprov mcf zimbraMtaDnsLookupsEnabled FALSE
              • If you disable DNS Lookups (under the MTA tab of the admin console, or with zmprov), Zimbra will end up using (according to the postconf man page) the "gethostbyname() system library routine which normally also looks in /etc/hosts" (based on the entries on the "hosts" line in /etc/nsswitch.conf). If you do this but don't also specify an SMTP relay host (typically your ISP's SMTP server), which will take care of checking DNS, you will reverse your ability to send mail: suddenly you can send mail to other users on the Zimbra server, but you can't send to the internet (though you can still receive mail from the internet either way).
      4. accept mail relayed by primary.YYY.state.XX.us
      5. forward mail to primary.YYY.state.XX.us for accounts hosted on the primary server
      6. After configuration changes, restart services/server if needed.
        1. If this was a Zimbra Server, you would use this command (as zimbra):
          • $ postfix stop
          • $ postfix start


Migrating Users To Be On the Zimbra Server :

  • When you are ready to move a user from the old system to the new system just run this command
    1. (where zimbra.YYY.state.XX.us is the name of your Zimbra server):
    2. $ zmprov ma secondary-user@YYY.state.XX.us zimbraMailTransport lmtp:primary.YYY.state.XX.us:7025


If there is no account on either server, the mail will bounce back to Zimbra.YYY.state.XX.us to catchall@YYY.state.XX.us


Outbound email:

Either of the mail servers will be able to send email out independent of the other by allowing outbound smtp traffic from their respective ip addresses through the firewall via policy rule.

Split-Domain Example 2 - Exchange Primary & Zimbra Secondary

My attempt to rewrite Split_Domain for customers that are having problems following it for whatever reasons.

Unsetting Split-Domain Configurations (Migration is done and Split-Domain is no longer needed)

Simply null out the values/return them to their previous:

zmprov md domain.com zimbraMailCatchAllAddress 
zmprov md domain.com zimbraMailCatchAllForwardingAddress 
zmprov md domain.com zimbraMailTransport lmtp:primary.YYY.state.XX.us:7025

Accounts should already of been adjusted through the migration steps:

zmprov ma secondary-user@YYY.state.XX.us zimbraMailTransport lmtp:primary.YYY.state.XX.us:7025

Unset Mail Relay and DNS Lookups False:

zmprov mcf zimbraMtaRelayHost 
zmprov mcf zimbraMtaDnsLookupsEnabled TRUE

Staged Migration Of Complete Zimbra Server To New Zimbra Server

If your moving users to a new mailstore, please see King0770-Notes#Preferred_Method_Moving_Users_To_New_Machine for a possible solution. This process implements a new LDAP replica's with the new ZCS mailstore. Over time you migrate the account to new ZCS with zmmailboxmove. Once the accounts have all been moved, you prompt the LDAP Replica configuration on the new ZCS server to be the LDAP master. Adjust for the MTA configuration and then retire the old box.

The rsync method, described in Ajcody-Notes#Server_Move_To_Same_Platform_64bit_And_OS_Type_Version would require downtime.

Staged Migration of Zimbra Server Domains To Zimbra Server With Existing Domains

One Possibility Untested or QA'd - NOT FINISHED - DON'T USE

The process would involve migrating the LDAP data for the domain from the old zimbra server into the other existing Zimbra system (using slapcat or ldapsearch). You'll need to get the server entries, any customer COS entries, the domain entry, and all users under that domain. You would then load the information (ldif) into the existing Zimbra server with slapadd. You could then reconfigured the old server to become an ldap slave to the existing Zimbra box and to server simply as a mailstore for that domain data (users). One could also proceed with zmmailboxmove if you needed to migrate the user data onto the other box as well now.

Adjustments would also need to be done in regards to auth keys and server certs.

Second Possibility Untested or QA'd - NOT FINISHED - DON'T USE

Situation

Let's say you own a hosting company and a new customer says they are running Zimbra Network edition (same version as yours) and wants you to host them. They only have one domain and want to minimize downtime. Their server is in another country or state. They have 50 users.

Realistically, during the migration time frame, those accounts that will be using the temporary sub-domain until the switch is complete should only use the zimbra web client. Phones, thick clients, and so forth will introduce issues about synchronization and so forth that would be better to simply avoid.

Split-Domain Setup

Setup a split domain that you can use through the migration time frame.

They are company.com

You'll be migrate.company.com

See Ajcody-Notes#Staged_Migration_of_Non-Zimbra_or_ZCS_Server_to_New_ZCS_Server about Split-Domains and Domain Masquerading. Other adjustments might need to be applied.

The situation will be this:

  • All emails maintain their company.com domain and email comes and goes through customers ZCS server.
  • Customers ZCS will be using forward/alias for account to goto @migrate.company.com
    • User Forward/Alias
      • Need to figure out the right way to do this for this situation still.
    • Relaying/Domain Forwarding
      • zmprov
      • md example.com zimbraMailCatchAllAddress @migrate.company.com
      • md example.com zimbraMailCatchAllForwardingAddress @migrate.company.com
      • md example.com zimbraMailTransport smtp:mta.HOSTING-COMPANY.com
  • Accounts migrated to Hosting company server will have rewrite rules. So migrated accounts will not be exposed to "outside" world.
    • zmprov md migrate.company.com zimbraMailCatchAllAddress @migrate.company.com zimbraMailCatchAllCanonicalAddress @company.com
    • Need instructions to configure for this particular domain mta rules do basically do the below WITHOUT effecting the other domains with the hosting companies ZCS infrastructure.
      • $ zmprov mcf zimbraMtaRelayHost mta.COMPANY.com
      • $ zmprov mcf zimbraMtaDnsLookupsEnabled FALSE
Break Up User Migration

Determine group of users that make sense to migrate based upon their functions and mailbox sizes. Some items might not work 100% as expect while in transition - i.e. Calendars, Resources, Wiki, etc..

Set Accounts To Maintenance

Set the targeted accounts to Maintenance status. They should stay this way after the backup is completed and will stay like that until the migration to the new server is complete and the split-domain is setup.

Backup

On the customers ZCS server, do a backup of the selected users. Be careful of where you direct the backup to and that you'll have space.

mkdir /tmp/backup-accounts
zmbackup -f -a userA@company.com userB@company.com userC@company.com -t /tmp/backup-accounts
Problem With Backup

Ideally, we would then do something like this:

zmbackup -i -a userA@company.com userB@company.com userC@company.com -t /tmp/backup-accounts

But this will fail, with:

Error occurred: system failure: Customer backup target is not allowed for incremental backup

And if you don't use the customer target it seems like the incremental picks up all the accounts. The point of wanting this would be to send a full via mail if needed and to allow the incrementals to be small enough to transfer over the network between the customer site and the hosting company.

Set Company Domain Onto Hosting Server

You'll need to setup the companies domain on your server before the zmrestore will work. You might as well configure the COS to make what the customer is using.

If your using a non-default COS.

  • zmprov cc companyCOS
    • adjust in web admin console if needed
  • zmprov gc companyCOS | grep zimbraId
    • zimbraId: 420447db-ea43-4d92-b62d-0164767f516d
  • zmprov cd company.com
    • adjust in web admin console if needed
    • zmprov md company.com zimbraDomainDefaultCOSId 420447db-ea43-4d92-b62d-0164767f516d

Set this domain to Maintenance mode.

Restore Account Onto Hosting Server
zmrestore -a userA@company.com userB@company.com userC@company.com -t /tmp/backup-accounts
Move Restored Accounts Into Split-Domain

If split domain is already setup:

zmprov ra userA@company.com userA@migrate.company.com

If not, your could do a domainrename:

zmprov -l rd company.com migrate.company.com
Forward Accounts That Were Moved

On the customers server, you'll now setup those accounts to forward to the split-domain.

Email for userA@company.com forwards to userA@migrate.company.com

Here's another problem though. We need to find a way to switch to a forward/alias on the companies ZCS so that emails don't get bounced with "no such account". You can't setup an alias as long as the account exists on the box.

Third Possibility Untested or QA'd - NOT FINISHED - DON'T USE

Summary Of Idea
I'm sure there's a way you could have both servers hosting the company.com domain at the same time with separate users. Where you had all mail going to a shared non-zimbra MTA host that would direct delivery to either the Hosting or Client ZCS server depending on where the user was at. Of course, you would need to have DNS setup to handle this type of setup (both internal and external). Could your Postini, MailProtector, or other such device do this? Then you would need to figure out a way to handle internal domain delivery between the two systems. The defaults, would have lmtp rejecting the delivery thinking the account doesn't exists because it believes it's authoritative for the domain. Maybe the use of subdomains for aliases/forwarding work be the work around for this.
Diagram Of Flow
                            ALL Company.com
                           emails to and from

                               MTA HEAD
                     lookups username to determine
                          server to deliver to

                                either/or
 userA@company.com is                             userB@company.com is
 userA@hosting.company.com                       userB@client.company.com
                    (above is done on MTA device)

 hosting server has:                             client server has:

 hosting.company.com has a                        client.company.com has a
 domain alias pointing to company.com             domain alias pointing to company.com
  * see http://wiki.zimbra.com/index.php?title=ManagingDomains#Creating_a_Domain_Alias *

 hosting server knows (MX record)                client server knows (MX record)
 Eternity server for delivery of                 hosting server for delivery of  
 eternity.company.com accounts (alias)           volaria.company.com accounts (alias)
  * also see http://wiki.zimbra.com/index.php?title=ManagingDomains#Relaying.2FDomain_Forwarding *

 Summary of hosting.company.com ZCS server
 Has a domain called company.com
 Has a domain alias called hosting.company.com that points to company.com
 All users are setup within company.com
 Migrate user(s) via zmrestore command into the domain (company.com)
 Non-migrated users will have alias setup to forward to username@client.company.com
 Will have mail-relaying/forwarding setup for client.company.com

 Summary of client.company.com ZCS server
 Has a domain called company.com
 Has a domain alias called client.company.com that points to company.com
 All users all ready exist within company.com
 Non-migrated user(s) will remain untouched
 Migrate user(s) via zmbackup with individual or list of users as flag
 Migrated users will have alias setup to forward to username@hosting.company.com
 Will have mail-relaying/forwarding setup for client.company.com

I've only just started to think this one possible solution through, maybe you can pick up what I have and go with it. To see if it might or might not work. Of course there's going to be a performance overhead on this in regards to mail delivery - because of the MTA Head being needed for routing issues. And there's also any security (spam as well) issues to think through.

Some Zimbra Users From DomainA to DomainB On Same Zimbra Server

All Accounts From DomainA to DomainB

You would use [renameDomain]:

zmprov -l rd domainA.com domainB.com

Please review domainrename though, there's some outstanding issues with this depending on your ZCS version.

Account From DomainA to DomainB

You want to use [renameAccount]:

zmprov ra userA@domainA.com userA@domainB.com

This is same command to rename account within the same domain:

zmprov ra userA@domainA.com userB@domainA.com

Non-Zimbra IMAP Accounts To Zimbra

IMAPSYNC with admin login

Reference - http://wiki.zimbra.com/index.php?title=User_Migration

imapsync --buffersize 8192000 --nosyncacls --subscribe --syncinternaldates \
--host1 server1.test.org --user1 yourAccount --password1 yourPassword \
--host2 zimbra.test.org --user2 yourZimbraAccount --authuser2 admin \
--password2 adminZimbraPassword --authmech2 LOGIN

You can also try --authmech2 PLAIN

I found this description in one of the imapsync files:

"You may authenticate as one user (typically an admin user), but be authorized as someone else, which means you don't need to know every user's personal password. Specify --authuser1 "adminuser" to enable this on host1. In this case, --authmech1 PLAIN will be used, but otherwise, --authmech1 CRAM-MD5 is the default. Same behavior with the --authuser2 option."

Performance Issues With IMAPSYNC

Jump to: navigation, search