Ajcody-Notes-Server-Move

Attention.png - This article is NOT official Zimbra documentation. It is a user contribution and may include unsupported customizations, references, suggestions, or information.

Server Move Notes

Actual Server Move Home Page

Please see: Ajcody-Notes-Server-Move

Server Move To Same Platform (32/64bit) And OS Type & Version

Reference

Please review the main reference for this page. It will have additional information you should be aware of before following these steps.

To do a server move via zmrestoreoffline, see:

Related RFE's

The following, if and when done, should also address directly changes in the OS or architecture of the "new" server as compared to the old production server you were moving from.

Assumptions

  1. This article makes the assumption your moving to the same OS type & version. For example, your PROD machine is running RHEL4-64bit. So your new machine would be running the same and brought to the same patch level as well.
  2. New server is setup with the same HOSTNAME information as PROD but it will use a different IP until PROD can be shutoff/reconfigured (if needed)
  3. SAME AMOUNT OF MEMORY If your moving from 32bit to 32bit and the new system has more than 4GB of memory and the older one didn't you will most likely need to adjust mailboxd_java_heap_memory_percent . This problem will show up as the mailbox stop starting and /opt/zimbra/log/zmmailboxd.out logging errors about JVM memory heap. Try the following:
su - zimbra
zmlocalconfig -e mailboxd_java_heap_memory_percent=25
zmmailboxdctl restart
If Your Attempting A Switch From 32 to 64
Note - If you skipped this section for whatever reason and run into the problem later as
your attempting to start/install on the new box you can still fix it. Use rpm2cpio to extra the
needed data for the zimbramon/ directory. Move the existing one at /opt/zimbra/ to a tmp
location and then copy/move the one from your rpm2cpio extraction.
Get the path to the zimbra-core package within the extracted tarball you have for the zimbra installation :
ex: /root/zcs-NETWORK-6.0.6_GA_2324.RHEL5_64.20100406133038/packages
Change directory to a temporary location that you'll extract the data to :
ex: rpm2cpio /root/zcs-NETWORK-6.0.6_GA_2324.RHEL5_64.20100406133038/packages/zimbracore-6.0.6_GA_2324.RHEL5_64-20100406133038.x86_64.rpm | cpio -idmv
You'll be able to find the zimbramon directory under there now to copy over to the /opt/zimbra path.


The details of this document assume that both servers are of the same chip type - 32bit or 64bit. Below are notes to "tweak" this how-to to handle a switch as well from 32 to 64.

The two exceptions to this how-to's instructions would be handling the zimbramon direcotry and the inclusion of the 32>64 ldap export/import steps listed in Network_Edition:_Moving_from_32-bit_to_64-bit_Server :

  1. On the "new server" , install the 64-bit version of ZCS that is the same version that your old server is running. The only difference would be it's 64 rather than 32-bit tar ball of ZCS.
  2. Copy /opt/zimbra/zimbramon of the 64bit installed stuff to a tmp dir
    • [as root] cp -rp /opt/zimbra/zimbramon /opt/zimbramon-64bit
  3. On the "old server" do the zmldapcat steps listed in Network_Edition:_Moving_from_32-bit_to_64-bit_Server
  4. Proceed with the steps listed in the main body of this how-to
    1. When you start step 7 in Ajcody-Notes-Server-Move#The_Big_Day_-_PROD_Downtime_For_Switch section stop.
  5. Copy the zimbramon-tmp over the /opt/zimbra/zimbramon dir
    • [as root] cp -rp /opt/zimbramon-64bit/* /opt/zimbra/zimbramon 
      • This is to bypass an reported error, below, during the ./install.sh part . Reports so far have been against Ubuntu.
        • Can't load '/opt/zimbra/zimbramon/lib/i486-linux-gnu-thread-multi/auto/IO/IO.so' for module IO:
          /opt/zimbra/zimbramon/lib/i486-linux-gnu-thread-multi/auto/IO/IO.so: wrong ELF class:
          ELFCLASS32 at /usr/lib/perl/5.8/XSLoader.pm line 70.
          at /opt/zimbra/zimbramon/lib/i486-linux-gnu-thread-multi/IO.pm line 11
  6. Save a copy of your .saveconfig directory in case you need to fall back on it. It seems the ./install.sh -s removes the files in .saveconfig which means when you goto do the ./install.sh later it will not see your configuration variable for your server.
    • [as root] cp -rp /opt/zimbra/.saveconfig /opt/saveconfig-bak
  7. Do a ./install.sh -s from the zimbra installation extract directory as root and then copy back your files from the saveconfig-bak directory.
    • [as root] cp -rp  /opt/saveconfig-bak/* /opt/zimbra/.saveconfig/
  8. Then follow the steps listed in the 32>64 bit wiki page about restoring the ldap data
    • Either the 5.x or 6.x section entitled:
      • "5.0.x or previous LDAP setup: 1. Restore the LDAP data to the 64-bit server. As zimbra, type"
      • "6.0.x and later LDAP setup: 1. Restore the LDAP data to the 64-bit server. As zimbra, type"
    • Stop at the point you just finished doing the
      • "/opt/zimbra/openldap/sbin/slapadd -q -b "" -F /opt/zimbra/data/ldap/config -cv -l /backup/ldap.bak"
  9. Now proceed with the step 7 section of the server move wiki, which is the full install/upgrade -- ./install.sh .

Optional Use - For DR Prior To Upgrades

You could also use this method for a "copy" of your old version prior to doing an upgrade.

Other Sync Method Process - BackupPC

Some customers have been using BackupPC for server moves and roll backs from ZCS upgrades.

Please see:

Please Note

Attention.png - DO NOT USE CRON WHEN SYNCING YOUR SERVER! You should always run the rsync command manually. If you fail to do so, and then complete the migration without removing the cron job that did the sync's, and you leave the old server running, you will lose data. The sync will then kick off again when cron runs and overwrite your production data! This will very likely corrupt your installation and leave you with an unstable system!


Preparing NEWHOST Server

  1. Install Newer (supported) Operating System that matches PROD
  2. Set up newer ZCS Server’s Hostname as it was on the older server
  3. Configure BIND locally on NEWHOST to handle resolution issues (A, MX, etc.)
    1. On NEWHOST confirm /etc/hosts , /etc/resolv.conf
  4. Download the EXACT version that your PROD is using.
  5. ZIMBRA User And UID Match
    • On OLDHOST as ZIMBRA:
      zmlocalconfig zimbra_uid
    • On OLDHOST, note the zimbra entry in /etc/passwd and /etc/group.
    • On NEWHOST configure the /etc/passwd to be the same UID for zimbra as the OLDHOST had.
    • On NEWHOST configure the entry for zimbra in the OLDHOST's /etc/group to match for zimbra as well.
  6. On NEWHOST as ROOT: Run the installer with the -s option:
    ./install.sh -s
    • This tells the installer to only install the software, and not to configure the installation.
  7. On NEWHOST as Root: Remove the dummy install:
    rm -rf /opt/zimbra ; mkdir /opt/zimbra
  8. On NEWHOST, make any other mounts or directories you'll need as to match the PROD server.
    • Secondary mailstores, alternative backup directory paths, etc.

Please Note

Attention.png - DO NOT USE CRON WHEN SYNCING YOUR SERVER! You should always run the rsync command manually. If you fail to do so, and then complete the migration without removing the cron job that did the sync's, and you leave the old server running, you will lose data. The sync will then kick off again when cron runs and overwrite your production data! This will very likely corrupt your installation and leave you with an unstable system!

Sync PROD Data While PROD Is Still In Production Use

UPDATE : I've added the -H option to the rsync command to preserve hard links.

First, if your data is rather large, you might want to see the following first:

From RHEL5 nice man page:

Nicenesses range from -20 (most favorable scheduling) to 19 (least favorable).
-n, --adjustment=N add integer N to the niceness (default 10)


  1. First initial sync of PROD to NEWHOST
    • on PROD as Root
      nice -n +19 rsync -avzH -e ssh --progress /opt/zimbra/ root@NEWHOSTIP:/opt/zimbra
      • Do the same with other paths you might have - i.e. secondary volumes.
      • Some distro's, might want this format for nice :
        nice +19 rsync ....
  2. Sync daily until schedule downtime is available
    • On PROD as Root
      nice -n +19 rsync -avzH -e ssh --progress /opt/zimbra/ root@NEWHOSTIP:/opt/zimbra
      • Do the same with other paths you might have - i.e. secondary volumes.
      • Some distro's, might want this format for nice :
        nice +19 rsync ....

The Big Day - PROD Downtime For Switch

  1. Commercial Certificates
    • Are you prepared for the impact this server move has for your commercial certificate? This document doesn't address that.
  2. Block client access to the old server's IP address with firewall rules
    • If your remote, make sure to keep your access port open. We are just trying to prevent any changes to the machines while they are being reconfigured.
  3. Shut down Zimbra on PROD
    • On PROD as ZIMBRA
      su - zimbra
      zmcontrol stop

Please Note

Attention.png - YOU MUST PERFORM THIS STEP WITH ZIMBRA DOWN! Please ensure that you have the Zimbra server STOPPED before performing the final rsync. If you're using this method to test your server move, you must schedule downtime to stop the server before performing the final sync. This is required, and failure to do so will result in corruption of your mysql/ldap databases on the test server.
  1. Last rsync of PROD to NEWHOST
    • On Prod as ROOT
      nice -20 rsync -avzH -e ssh --delete --progress /opt/zimbra/ root@NEWHOSTIP:/opt/zimbra 
      UPDATE : I've added the -H option to the rsync command to preserve hard links.
      • Do the same with other paths you might have - i.e. secondary volumes.
      • Some distro's, SLES, might want this format for nice :
        nice –n -20 rsync ....
  2. Fix permissions on NEWHOST
    • On NEWHOST as ROOT run
      /opt/zimbra/libexec/zmfixperms --verbose --extended
  3. Turn off PROD and reconfigure NEWHOST
    • This is a good time to turn off PROD.
      • Reconfigure network interfaces so if someone turns on PROD later, it will not use the ip addresses that will now be used on NEWHOST.
      • Reconfigure any mounts (san, nfs, iscsi, etc.) so it will not mount anything that should only be mounted on our NEWHOST. Again, in case the machine is powered on accidentally later.
    • Reconfigure NEWHOST to take over ip addresses of PROD.
      • Make any firewall or other network changes that are necessary.
        • Remember about arp tables.
    • Reconfigure for any mounts that were on PROD that will be needed for NEWHOST.
  4. Install of Zimbra on NEWHOST
    • If you have a Split DNS install or use private LAN addresses on the server with a firewall front-ending the public addresses, you'll want to verify logical hostname resolution and hostname resolution
      • In some cases, you can move Zimbra to another server with a different hostname but keep the logical hostname the same. The logical hostname is what users know this server as, and it doesn't necessarily have to match the actual hostname.
        • For example, you might have "mail.mydomain.com" as the DNS name for the server, but the hostname is "web11233"
        • You need to have the server itself resolve the logical hostname, the old hostname and the new hostname as the internal private LAN address.
        • host `hostname`
        • host (logical hostname)
        • nslookup `hostname`
        • nslookup (logical hostname)
        • nslookup (old hostname)
        • nslookup (old hostname)
      • You want all these to look the same. You can follow instructions at Split_dns. Essentially what that does is have a local copy of bind (named) running that resolves just those names and forwards all other lookups to your normal DNS servers.
    • On NEWHOST as ROOT, rerun the installer without the -s option
      ./install.sh
      • It will detect ZCS already installed, and ask if you want to upgrade. Select Yes.
      • Within the installation script, you might want to choose the option that tells Zimbra to NOT automatically start upon completion of upgrade/install.
  5. Post-Install on NEWHOST
    • This document assumes you were going to get the same hostname and ip address once the final move was done. In case this isn't true, below are some follow up issues you might want to check. You might of done some of these already.
      • Do you need to make adjustments for commercial certificates?
      • Reconfigure any network interface/ip information that you need because of hardware move.
      • Make necessary adjustments you might need because of hostname changes. ( see ZmSetServerName )
      • Adjust any firewall settings
        • If ip address is going to be different, make sure you know the settings you'll need to adjust within Zimbra (if any).
        • If ip address is going to be the same, remember your network will take awhile to see change as the new MAC address gets updated to other devices arp table.
          • If you can, you can speed this along with changes on your switches.
  6. Start zimbra once you think everything is ready.
    • Do some client access tests within your LAN.
    • If testing looks good, the remove any firewall rules you might of done to block access from outside. Then confirm outside access and functionality.
      • Remember to check those mobile devices, certificates, and other access software/devices besides just the Zimbra webclient.

Zimbra Domains Into An Existing Zimbra Server

See Ajcody-Migration-Notes . There's a couple of possibilities being worked out there.

Jump to: navigation, search