Ajcody-Notes-Upgrade-Options

Revision as of 00:51, 26 August 2008 by Cfremon (talk | contribs)

How Do I Make Sure I Don't Lose Emails During Upgrade? If I Need To Fail Back To Old Version?

Actual Upgrade Options Notes Homepage

Please see: Ajcody-Notes-Upgrade-Options

Warning - Must Read!

These steps are the start of my notes/ideas. They have not been peer-reviewed or gone through any QA process.

Update I'll need to update some of the steps below in regards to mta errors until it can communicate with mailstores. Went through one dry run tonight (July 23, 08) with customer, I'll incorporate changes soon. See Ajcody-Notes-Of-Customer-Cluster-Upgrade and the follow up bug I did:

Bug filed for Documentation and QA issues:

Normal Order Of Upgrades

Update the servers in the following order:

All LDAP servers, MTA's, and dedicated Proxy servers must be updated in the same downtime window.

Update in this order.

  1. LDAP Master server
  2. LDAP replica servers. For upgrade details see LDAP_Replicas_4.5.x_to_5.0.x
  3. Zimbra MTA servers
  4. Dedicated Proxy servers
  5. Next upgrade the mailbox servers.
    • The first mailbox server to upgrade is the server where the Documents wiki account was created. If you do not upgrade and get this server working first, you will need to manually install the Documents templates on each of the other servers.

Starting Resources

Upgrade Steps for Multi-Servers (Non-Cluster)

  • Please review ALL documentation and make sure this process fits and makes sense for your environment. This is untested at this time.
  • Start or have a very recent full backup.
  • Incremental Backups
    • If you have a short time frame for your incremental backups and you've confirm the scheduled one ran the night before then:
      • Then you can proceed to Step 4
    • If you have a large time frame for your incremental backups and you've confirm the scheduled one ran the night before then:
      • Figure out a time frame to start another incremental mid-day that will complete shortly before your schedule downtime.
        • This is just to minimize the amount of time the mta's are offline.
      • Proceed to Step 4
  • Start of schedule downtime
  • Stop MTAs from delivering to the mailstore
    • There are a number of ways this can be done
      • Simply shutdown the zimbra mta server/services
        • We are relying on other MTAs to queue up your mail until your MTAs can be reached again. This variable is usually fairly long (hours/days) before MTAs will actually bounce back messages as undeliverable.
      • Use a non-zimbra device or server that can hold message queue, configure it to hold queue and not forward to zimbra MTAs
      • Other ideas?
        • Alter DNS/MX/Firewall to direct to a holding machine ?
    • Do a sanity check about disk space needed to queue up expected messages over time of upgrade.
      • Monitor space and be prepare to shut them back down again if disk issue becomes a problem
  • Do your last incremental backup of mailstores
  • Reference steps in multi-server upgrade doc's for technical details for below
    • This will recommend shutting down ALL Zimbra server
    • The release notes don't mention a particular order, but following from the "Rolling Upgrade" wiki this make sense
      • Upgrade LDAP Master server first
      • Upgrade LDAP replica servers second
      • Upgrade Zimbra MTA servers third
      • Upgrade dedicated Proxy servers fourth
        • In this case, the mailstore is assumed to be down already. The mailstores being the last to be upgraded.
      • Upgrade the Mailstore servers last
        • The first mailbox server to upgrade is the server where the Documents wiki account was created. If you do not upgrade and get this server working first, you will need to manually install the Documents templates on each of the other servers.
  • Follow up with Items listed in the official documentation about post upgrade steps.

Upgrade Steps for Multi-Servers with Clustered Mailstores

  • Please review ALL documentation and make sure this process fits and makes sense for your environment. This is untested at this time.
  • Start or have a very recent full backup.
  • Incremental Backups
    • If you have a short time frame for your incremental backups and you've confirm the scheduled one ran the night before then:
      • Then you can proceed to Step 4
    • If you have a large time frame for your incremental backups and you've confirm the scheduled one ran the night before then:
      • Figure out a time frame to start another incremental mid-day that will complete shortly before your schedule downtime.
        • This is just to minimize the amount of time the MTAs are offline.
      • Proceed to Step 4
  • Start of schedule downtime
  • Stop MTAs from delivering to the mailstore
    • There are a number of ways this can be done
      • Simply shutdown the Zimbra mta server/services
        • We are relying on other MTAs to queue up your mail until your MTAs can be reached again. This variable is usually fairly long (hours/days) before MTAs will actually bounce back messages as undeliverable.
      • Use a non-Zimbra device or server that can hold message queue, configure it to hold queue and not forward to Zimbra MTAs
      • Other ideas?
        • Alter DNS/MX/Firewall to direct to a holding machine?
    • Do a sanity check about disk space needed to queue up expected messages over time of upgrade.
      • Monitor space and be prepare to shut them back down again if disk issue becomes a problem
  • Do your last incremental backup of mailstores
  • Reference steps in multi-server & cluster upgrade doc's for technical details for below
  • Upgrade LDAP Master server first
  • Upgrade LDAP replica servers second
  • Upgrade Zimbra MTA servers third
  • Upgrade any dedicated Proxy servers fourth
    • Choose option in the installer config to not start after it's finish. We'll want them to stay down.
      • Confirm they are still down
  • Upgrade the "clustered" mailstore servers last
    • Follow steps as outline in cluster (single/multi-server) or multi-server upgrade doc's
      • This summary below is to brief to use, you MUST review the cluster upgrade documentation.
      • Please note "The first mailbox server to upgrade is the server where the Documents wiki account was created. If you do not upgrade and get this server working first, you will need to manually install the Documents templates on each of the other servers." Account for this in case there's multiple cluster mailstores.
        • Active Nodes upgraded first
          • The first one will effectively bring down the mailstore cluster - meaning it can't accept any mail deliveries
        • Standby Nodes upgrade second
        • When all are upgraded, you'll re-enabled the cluster
    • Once the upgrade has started (where the cluster is brought down with first upgrade), the "mailstore" should be offline and the mta should now be able to queue up your messages
      • Start up your mta's (or whatever method you used to delayed delivery to the Zimbra MTA's)
  • Follow up with items listed in the official documentation about post upgrade steps.

Notes I Took While Doing These Steps With A Customer

Please see Ajcody-Notes-Of-Customer-Cluster-Upgrade

Rolling Upgrade Outline

  • Please review ALL documentation and make sure this process fits and makes sense for your environment. This is untested at this time.
  • Start or have a very recent full backup.
  • Incremental Backups
    • If you have a short time frame for your incremental backups and you've confirm the scheduled one ran the night before then:
      • Then you can proceed to Step 4
    • If you have a large time frame for your incremental backups and you've confirm the scheduled one ran the night before then:
      • Figure out a time frame to start another incremental mid-day that will complete shortly before your schedule downtime.
        • This is just to minimize the amount of time the MTAs are offline.
      • Proceed to Step 4
  • Start of schedule downtime
  • Stop MTAs from delivering to the mailstore
    • There are a number of ways this can be done
      • Simply shutdown the Zimbra MTA server/services
        • We are relying on other MTAs to queue up your mail until your MTAs can be reached again. This variable is usually fairly long (hours/days) before mta's will actually bounce back messages as undeliverable.
      • Use a non-Zimbra device or server that can hold message queue, configure it to hold queue and not forward to Zimbra MTAs
      • Other ideas?
        • Alter DNS/MX/Firewall to direct to a holding machine?
      • Do a sanity check about disk space needed to queue up expected messages over time of upgrade.
        • Monitor space and be prepare to shut them back down again if disk issue becomes a problem
  • Do your last incremental backup of mailstores
  • Reference steps in the rolling upgrade wiki, multi-server, cluster upgrade doc's for technical details for below
  • Start of First Group Of Upgrades
    • From Rolling_Upgrades_for_ZCS , All LDAP servers and MTA servers must be updated in the same downtime window I'm including proxy servers below as well.
    • Update in this order
      • Upgrade LDAP Master server first
      • Upgrade LDAP replica servers second
      • Upgrade Zimbra MTA servers third
      • Upgrade all dedicated Proxy servers fourth
    • Allow messages to be delivered to mailstores (still running old version) once things look right.
  • Start of Second Group of Upgrades (Mailstores)
    • Repeat the steps about backups, stopping delivery of MTAs to mailstore, and then final incremental.
      • Do a sanity check about disk space needed to queue up expected messages over time of upgrade.
        • Monitor space and be prepare to shut them back down again if disk issue becomes a problem
    • Upgrade your mailstores based up your need/configuration
      • The first mailbox server to upgrade is the server where the Documents wiki account was created. If you do not upgrade and get this server working first, you will need to manually install the Documents templates on each of the other servers.
    • Once upgrade is done and things look right, allow delivery of mail from the MTAs to the mailstores.
  • Follow up with Items listed in the official documentation about post upgrade steps.
Jump to: navigation, search