Ajcody-Notes-Upgrade-Options: Difference between revisions

 
mNo edit summary
 
(29 intermediate revisions by 2 users not shown)
Line 1: Line 1:
==Actual Upgrade Options Notes Homepage==
{{BC|Zeta Alliance}}                        <!-- Note, this will also add [[Category: Zeta Alliance]] to bottom of wiki page. -->
__FORCETOC__                              <!-- Will force a TOC regards of size of article. __NOTOC__  if no TOC is wanted. -->
<div class="col-md-12 ibox-content">
==How Do I Make Sure I Don't Lose Emails During Upgrade? If I Need To Fail Back To Old Version?==             <!-- Normally will reflect page title. Is listed at very top of page. -->
{{KB|{{ZETA}}|{{ZCS 8.5}}|{{ZCS 8.0}}|{{ZCS 7.0}}|}}            <!-- Can only handle 3 ZCS versions. -->
{{WIP}}                                                <!-- For pages that are "work in progress". -->


Please see [[Ajcody-Notes-Upgrade-Options]]
===Actual Upgrade Options Notes Homepage===
 
Please see: [[Ajcody-Notes-Upgrade-Options]]


===Warning - Must Read!===
===Warning - Must Read!===


These steps are the start of my notes/ideas. They have not been peer-reviewed or gone through any QA process.
These steps are the start of my notes/ideas. They have not been peer-reviewed or gone through any QA process.
'''Update''' I'll need to update some of the steps below in regards to mta errors until it can communicate with mailstores. Went through one dry run tonight (July 23, 08) with customer, I'll incorporate changes soon. See [[Ajcody-Notes-Of-Customer-Cluster-Upgrade]] and the follow up bug I did:
Bug filed for Documentation and QA issues:
* "QA & Corrections For Cluster Upgrade Document"
** http://bugzilla.zimbra.com/show_bug.cgi?id=31026
====Normal Order Of Upgrades====
Update the servers in the following order:
:: '''All LDAP servers, MTA's, and dedicated Proxy servers must be updated in the same downtime window'''.
Update in this order.
# LDAP Master server
# LDAP replica servers. For upgrade details see [[LDAP_Replicas_4.5.x_to_5.0.x]]
# Zimbra MTA servers
# Dedicated Proxy servers
# Next upgrade the mailbox servers.
#* The first mailbox server to upgrade is the server where the Documents wiki account was created. If you do not upgrade and get this server working first, you will need to manually install the Documents templates on each of the other servers.


===Starting Resources===
===Starting Resources===
Line 16: Line 44:
*** Cluster Upgrade - Single and Multi Node
*** Cluster Upgrade - Single and Multi Node
** http://wiki.zimbra.com/index.php?title=Rolling_Upgrades_for_ZCS
** http://wiki.zimbra.com/index.php?title=Rolling_Upgrades_for_ZCS
* '''You should read this if your upgrading to ZCS 6 : "Optimizing 5.0 to 6.0 LDAP upgrade"'''
** http://wiki.zimbra.com/index.php?title=Optimizing_5.0_to_6.0_LDAP_upgrade


===Upgrade Steps for Multi-Servers (Non-Cluster)===
===Upgrade Steps for Multi-Servers (Non-Cluster)===
Line 28: Line 58:
**** This is just to minimize the amount of time the mta's are offline.
**** This is just to minimize the amount of time the mta's are offline.
*** Proceed to Step 4
*** Proceed to Step 4
* Start of schedule dowmtime
* Start of schedule downtime
* Stop mta's from delivering to the mailstore
* Stop MTAs from delivering to the mailstore
** There are a number of ways this can be done
** There are a number of ways this can be done
*** Simply shutdown the zimbra mta server/services
*** Simply shutdown the zimbra mta server/services
**** We are relying on other mta's to queue up your mail until your mta's can be reached again. This variable is usually fairly long (hours/days) before mta's will actually bounce back messages as undeliverable.
**** We are relying on other MTAs to queue up your mail until your MTAs can be reached again. This variable is usually fairly long (hours/days) before MTAs will actually bounce back messages as undeliverable.
*** Use a non-zimbra device or server that can hold message queue, configure it to hold queue and not forward to zimbra mta's
*** Use a non-zimbra device or server that can hold message queue, configure it to hold queue and not forward to zimbra MTAs
*** Other ideas?
*** Other ideas?
**** Alter DNS/MX/Firewall to direct to a holding machine ?
**** Alter DNS/MX/Firewall to direct to a holding machine ?
Line 40: Line 70:
* Do your last incremental backup of mailstores
* Do your last incremental backup of mailstores
* Reference steps in multi-server upgrade doc's for technical details for below
* Reference steps in multi-server upgrade doc's for technical details for below
** This will recommend shutting down ALL zimbra server
** This will recommend shutting down ALL Zimbra server
** The release notes don't mention a particular order, but following from the "Rolling Upgrade" wiki this make sense
** The release notes don't mention a particular order, but following from the "Rolling Upgrade" wiki this make sense
*** Upgrade LDAP Master server first
*** Upgrade LDAP Master server first
*** Upgrade LDAP replica servers second
*** Upgrade LDAP replica servers second
*** Upgrade Zimbra MTA servers third
*** Upgrade Zimbra MTA servers third
*** Upgrade dedicated Proxy servers fourth
**** In this case, the mailstore is assumed to be down already. The mailstores being the last to be upgraded.
**** In this case, the mailstore is assumed to be down already. The mailstores being the last to be upgraded.
*** Upgrade the Mailstore servers upgrades last  
*** Upgrade the Mailstore servers last
**** The first mailbox server to upgrade is the server where the Documents wiki account was created. If you do not upgrade and get this server working first, you will need to manually install the Documents templates on each of the other servers.
* Follow up with Items listed in the official documentation about post upgrade steps.
* Follow up with Items listed in the official documentation about post upgrade steps.


Line 58: Line 90:
** If you have a large time frame for your incremental backups and you've confirm the scheduled one ran the night before then:
** If you have a large time frame for your incremental backups and you've confirm the scheduled one ran the night before then:
*** Figure out a time frame to start another incremental mid-day that will complete shortly before your schedule downtime.
*** Figure out a time frame to start another incremental mid-day that will complete shortly before your schedule downtime.
**** This is just to minimize the amount of time the mta's are offline.
**** This is just to minimize the amount of time the MTAs are offline.
*** Proceed to Step 4
*** Proceed to Step 4
* Start of schedule dowmtime
* Start of schedule downtime
* Stop mta's from delivering to the mailstore
* Stop MTAs from delivering to the mailstore
** There are a number of ways this can be done
** There are a number of ways this can be done
*** Simply shutdown the zimbra mta server/services
*** Simply shutdown the Zimbra mta server/services
**** We are relying on other mta's to queue up your mail until your mta's can be reached again. This variable is usually fairly long (hours/days) before mta's will actually bounce back messages as undeliverable.
**** We are relying on other MTAs to queue up your mail until your MTAs can be reached again. This variable is usually fairly long (hours/days) before MTAs will actually bounce back messages as undeliverable.
*** Use a non-zimbra device or server that can hold message queue, configure it to hold queue and not forward to zimbra mta's
*** Use a non-Zimbra device or server that can hold message queue, configure it to hold queue and not forward to Zimbra MTAs
*** Other ideas?
*** Other ideas?
**** Alter DNS/MX/Firewall to direct to a holding machine ?
**** Alter DNS/MX/Firewall to direct to a holding machine?
** Do a sanity check about disk space needed to queue up expected messages over time of upgrade.
** Do a sanity check about disk space needed to queue up expected messages over time of upgrade.
*** Monitor space and be prepare to shut them back down again if disk issue becomes a problem
*** Monitor space and be prepare to shut them back down again if disk issue becomes a problem
Line 75: Line 107:
* Upgrade LDAP replica servers second
* Upgrade LDAP replica servers second
* Upgrade Zimbra MTA servers third
* Upgrade Zimbra MTA servers third
* Upgrade any dedicated Proxy servers fourth
** Choose option in the installer config to not start after it's finish. We'll want them to stay down.
** Choose option in the installer config to not start after it's finish. We'll want them to stay down.
*** Confirm they are still down
*** Confirm they are still down
* Upgrade the "clustered" Mailstore servers last
* Upgrade the "clustered" mailstore servers last
** Follow steps as outline in cluster (single/multi-server) or multi-server upgrade doc's
** Follow steps as outline in cluster (single/multi-server) or multi-server upgrade doc's
*** This summary below is to brief to use, you MUST review the cluster upgrade documentation.
*** This summary below is to brief to use, you MUST review the cluster upgrade documentation.
*** Please note "The first mailbox server to upgrade is the server where the Documents wiki account was created. If you do not upgrade and get this server working first, you will need to manually install the Documents templates on each of the other servers." Account for this in case there's multiple cluster mailstores.
**** Active Nodes upgraded first
**** Active Nodes upgraded first
***** The first one will effectively bring down the mailstore cluster - meaning it can't accept any mail deliveries
***** The first one will effectively bring down the mailstore cluster - meaning it can't accept any mail deliveries
Line 85: Line 119:
**** When all are upgraded, you'll re-enabled the cluster
**** When all are upgraded, you'll re-enabled the cluster
** Once the upgrade has started (where the cluster is brought down with first upgrade), the "mailstore" should be offline and the mta should now be able to queue up your messages
** Once the upgrade has started (where the cluster is brought down with first upgrade), the "mailstore" should be offline and the mta should now be able to queue up your messages
*** Start up your mta's (or whatever method you used to delayed delivery to the zimbra mta's)
*** Start up your mta's (or whatever method you used to delayed delivery to the Zimbra MTA's)
* Follow up with Items listed in the official documentation about post upgrade steps.
* Follow up with items listed in the official documentation about post upgrade steps.
 
====Notes I Took While Doing These Steps With A Customer====
 
Please see [[Ajcody-Notes-Of-Customer-Cluster-Upgrade]]


===Rolling Upgrade Outline===
===Rolling Upgrade Outline===


* Please review ALL documentation and make sure this process fits and makes sense for your environment. This is untested at this time.
* Please review ALL documentation and make sure this process fits and makes sense for your environment. This is untested at this time.
** Especially http://wiki.zimbra.com/index.php?title=Rolling_Upgrades_for_ZCS
* Start or have a very recent full backup.
* Start or have a very recent full backup.
* Incremental Backups
* Incremental Backups
Line 97: Line 136:
** If you have a large time frame for your incremental backups and you've confirm the scheduled one ran the night before then:
** If you have a large time frame for your incremental backups and you've confirm the scheduled one ran the night before then:
*** Figure out a time frame to start another incremental mid-day that will complete shortly before your schedule downtime.
*** Figure out a time frame to start another incremental mid-day that will complete shortly before your schedule downtime.
**** This is just to minimize the amount of time the mta's are offline.
**** This is just to minimize the amount of time the MTAs are offline.
*** Proceed to Step 4
*** Proceed to Step 4
* Start of schedule dowmtime
* Start of schedule downtime
* Stop mta's from delivering to the mailstore
* Stop MTAs from delivering to the mailstore
** There are a number of ways this can be done
** There are a number of ways this can be done
*** Simply shutdown the zimbra mta server/services
*** Simply shutdown the Zimbra MTA server/services
**** We are relying on other mta's to queue up your mail until your mta's can be reached again. This variable is usually fairly long (hours/days) before mta's will actually bounce back messages as undeliverable.
**** We are relying on other MTAs to queue up your mail until your MTAs can be reached again. This variable is usually fairly long (hours/days) before mta's will actually bounce back messages as undeliverable.
*** Use a non-zimbra device or server that can hold message queue, configure it to hold queue and not forward to zimbra mta's
*** Use a non-Zimbra device or server that can hold message queue, configure it to hold queue and not forward to Zimbra MTAs
*** Other ideas?
*** Other ideas?
**** Alter DNS/MX/Firewall to direct to a holding machine ?
**** Alter DNS/MX/Firewall to direct to a holding machine?
*** Do a sanity check about disk space needed to queue up expected messages over time of upgrade.
*** Do a sanity check about disk space needed to queue up expected messages over time of upgrade.
**** Monitor space and be prepare to shut them back down again if disk issue becomes a problem
**** Monitor space and be prepare to shut them back down again if disk issue becomes a problem
Line 112: Line 151:
* Reference steps in the rolling upgrade wiki, multi-server, cluster upgrade doc's for technical details for below
* Reference steps in the rolling upgrade wiki, multi-server, cluster upgrade doc's for technical details for below
* Start of First Group Of Upgrades
* Start of First Group Of Upgrades
** From rolling upgrade wiki, '''All LDAP servers and MTA servers must be updated in the same downtime window'''
** From [[Rolling_Upgrades_for_ZCS]] , '''All LDAP servers and MTA servers must be updated in the same downtime window''' I'm including proxy servers below as well.
** Update in this order
** Update in this order
*** Upgrade LDAP Master server first
*** Upgrade LDAP Master server first
*** Upgrade LDAP replica servers second
*** Upgrade LDAP replica servers second
*** Upgrade Zimbra MTA servers third
*** Upgrade Zimbra MTA servers third
*** Upgrade all dedicated Proxy servers fourth
** Allow messages to be delivered to mailstores (still running old version) once things look right.
** Allow messages to be delivered to mailstores (still running old version) once things look right.
* Start of Second Group of Upgrades (Mailstores)
* Start of Second Group of Upgrades (Mailstores)
** Repeat the steps about backups, stopping delivery of mta's to mailstore, and then final incremental.
** Repeat the steps about backups, stopping delivery of MTAs to mailstore, and then final incremental.
*** Do a sanity check about disk space needed to queue up expected messages over time of upgrade.
*** Do a sanity check about disk space needed to queue up expected messages over time of upgrade.
**** Monitor space and be prepare to shut them back down again if disk issue becomes a problem
**** Monitor space and be prepare to shut them back down again if disk issue becomes a problem
** Upgrade your Mailstores based up your need/configuration
** Upgrade your mailstores based up your need/configuration
** Once upgrade is done and things look right, allow delivery of mail from the mta's to the mailstores.
*** The first mailbox server to upgrade is the server where the Documents wiki account was created. If you do not upgrade and get this server working first, you will need to manually install the Documents templates on each of the other servers.
** Once upgrade is done and things look right, allow delivery of mail from the MTAs to the mailstores.
* Follow up with Items listed in the official documentation about post upgrade steps.
* Follow up with Items listed in the official documentation about post upgrade steps.
----
[[Category: Community Sandbox]]
[[Category:Installation]]
[[Category:Installation Guides]]
[[Category:Multi-Server]]
[[Category:Planning and Design]]
[[Category:Upgrade]]
[[Category: Author:Ajcody]]
[[Category: Zeta Alliance]]

Latest revision as of 03:44, 21 June 2016

How Do I Make Sure I Don't Lose Emails During Upgrade? If I Need To Fail Back To Old Version?

   KB 2502        Last updated on 2016-06-21  




0.00
(0 votes)
24px ‎  - This is Zeta Alliance Certified Documentation. The content has been tested by the Community.


Actual Upgrade Options Notes Homepage

Please see: Ajcody-Notes-Upgrade-Options

Warning - Must Read!

These steps are the start of my notes/ideas. They have not been peer-reviewed or gone through any QA process.

Update I'll need to update some of the steps below in regards to mta errors until it can communicate with mailstores. Went through one dry run tonight (July 23, 08) with customer, I'll incorporate changes soon. See Ajcody-Notes-Of-Customer-Cluster-Upgrade and the follow up bug I did:

Bug filed for Documentation and QA issues:

Normal Order Of Upgrades

Update the servers in the following order:

All LDAP servers, MTA's, and dedicated Proxy servers must be updated in the same downtime window.

Update in this order.

  1. LDAP Master server
  2. LDAP replica servers. For upgrade details see LDAP_Replicas_4.5.x_to_5.0.x
  3. Zimbra MTA servers
  4. Dedicated Proxy servers
  5. Next upgrade the mailbox servers.
    • The first mailbox server to upgrade is the server where the Documents wiki account was created. If you do not upgrade and get this server working first, you will need to manually install the Documents templates on each of the other servers.

Starting Resources

Upgrade Steps for Multi-Servers (Non-Cluster)

  • Please review ALL documentation and make sure this process fits and makes sense for your environment. This is untested at this time.
  • Start or have a very recent full backup.
  • Incremental Backups
    • If you have a short time frame for your incremental backups and you've confirm the scheduled one ran the night before then:
      • Then you can proceed to Step 4
    • If you have a large time frame for your incremental backups and you've confirm the scheduled one ran the night before then:
      • Figure out a time frame to start another incremental mid-day that will complete shortly before your schedule downtime.
        • This is just to minimize the amount of time the mta's are offline.
      • Proceed to Step 4
  • Start of schedule downtime
  • Stop MTAs from delivering to the mailstore
    • There are a number of ways this can be done
      • Simply shutdown the zimbra mta server/services
        • We are relying on other MTAs to queue up your mail until your MTAs can be reached again. This variable is usually fairly long (hours/days) before MTAs will actually bounce back messages as undeliverable.
      • Use a non-zimbra device or server that can hold message queue, configure it to hold queue and not forward to zimbra MTAs
      • Other ideas?
        • Alter DNS/MX/Firewall to direct to a holding machine ?
    • Do a sanity check about disk space needed to queue up expected messages over time of upgrade.
      • Monitor space and be prepare to shut them back down again if disk issue becomes a problem
  • Do your last incremental backup of mailstores
  • Reference steps in multi-server upgrade doc's for technical details for below
    • This will recommend shutting down ALL Zimbra server
    • The release notes don't mention a particular order, but following from the "Rolling Upgrade" wiki this make sense
      • Upgrade LDAP Master server first
      • Upgrade LDAP replica servers second
      • Upgrade Zimbra MTA servers third
      • Upgrade dedicated Proxy servers fourth
        • In this case, the mailstore is assumed to be down already. The mailstores being the last to be upgraded.
      • Upgrade the Mailstore servers last
        • The first mailbox server to upgrade is the server where the Documents wiki account was created. If you do not upgrade and get this server working first, you will need to manually install the Documents templates on each of the other servers.
  • Follow up with Items listed in the official documentation about post upgrade steps.

Upgrade Steps for Multi-Servers with Clustered Mailstores

  • Please review ALL documentation and make sure this process fits and makes sense for your environment. This is untested at this time.
  • Start or have a very recent full backup.
  • Incremental Backups
    • If you have a short time frame for your incremental backups and you've confirm the scheduled one ran the night before then:
      • Then you can proceed to Step 4
    • If you have a large time frame for your incremental backups and you've confirm the scheduled one ran the night before then:
      • Figure out a time frame to start another incremental mid-day that will complete shortly before your schedule downtime.
        • This is just to minimize the amount of time the MTAs are offline.
      • Proceed to Step 4
  • Start of schedule downtime
  • Stop MTAs from delivering to the mailstore
    • There are a number of ways this can be done
      • Simply shutdown the Zimbra mta server/services
        • We are relying on other MTAs to queue up your mail until your MTAs can be reached again. This variable is usually fairly long (hours/days) before MTAs will actually bounce back messages as undeliverable.
      • Use a non-Zimbra device or server that can hold message queue, configure it to hold queue and not forward to Zimbra MTAs
      • Other ideas?
        • Alter DNS/MX/Firewall to direct to a holding machine?
    • Do a sanity check about disk space needed to queue up expected messages over time of upgrade.
      • Monitor space and be prepare to shut them back down again if disk issue becomes a problem
  • Do your last incremental backup of mailstores
  • Reference steps in multi-server & cluster upgrade doc's for technical details for below
  • Upgrade LDAP Master server first
  • Upgrade LDAP replica servers second
  • Upgrade Zimbra MTA servers third
  • Upgrade any dedicated Proxy servers fourth
    • Choose option in the installer config to not start after it's finish. We'll want them to stay down.
      • Confirm they are still down
  • Upgrade the "clustered" mailstore servers last
    • Follow steps as outline in cluster (single/multi-server) or multi-server upgrade doc's
      • This summary below is to brief to use, you MUST review the cluster upgrade documentation.
      • Please note "The first mailbox server to upgrade is the server where the Documents wiki account was created. If you do not upgrade and get this server working first, you will need to manually install the Documents templates on each of the other servers." Account for this in case there's multiple cluster mailstores.
        • Active Nodes upgraded first
          • The first one will effectively bring down the mailstore cluster - meaning it can't accept any mail deliveries
        • Standby Nodes upgrade second
        • When all are upgraded, you'll re-enabled the cluster
    • Once the upgrade has started (where the cluster is brought down with first upgrade), the "mailstore" should be offline and the mta should now be able to queue up your messages
      • Start up your mta's (or whatever method you used to delayed delivery to the Zimbra MTA's)
  • Follow up with items listed in the official documentation about post upgrade steps.

Notes I Took While Doing These Steps With A Customer

Please see Ajcody-Notes-Of-Customer-Cluster-Upgrade

Rolling Upgrade Outline

  • Please review ALL documentation and make sure this process fits and makes sense for your environment. This is untested at this time.
  • Start or have a very recent full backup.
  • Incremental Backups
    • If you have a short time frame for your incremental backups and you've confirm the scheduled one ran the night before then:
      • Then you can proceed to Step 4
    • If you have a large time frame for your incremental backups and you've confirm the scheduled one ran the night before then:
      • Figure out a time frame to start another incremental mid-day that will complete shortly before your schedule downtime.
        • This is just to minimize the amount of time the MTAs are offline.
      • Proceed to Step 4
  • Start of schedule downtime
  • Stop MTAs from delivering to the mailstore
    • There are a number of ways this can be done
      • Simply shutdown the Zimbra MTA server/services
        • We are relying on other MTAs to queue up your mail until your MTAs can be reached again. This variable is usually fairly long (hours/days) before mta's will actually bounce back messages as undeliverable.
      • Use a non-Zimbra device or server that can hold message queue, configure it to hold queue and not forward to Zimbra MTAs
      • Other ideas?
        • Alter DNS/MX/Firewall to direct to a holding machine?
      • Do a sanity check about disk space needed to queue up expected messages over time of upgrade.
        • Monitor space and be prepare to shut them back down again if disk issue becomes a problem
  • Do your last incremental backup of mailstores
  • Reference steps in the rolling upgrade wiki, multi-server, cluster upgrade doc's for technical details for below
  • Start of First Group Of Upgrades
    • From Rolling_Upgrades_for_ZCS , All LDAP servers and MTA servers must be updated in the same downtime window I'm including proxy servers below as well.
    • Update in this order
      • Upgrade LDAP Master server first
      • Upgrade LDAP replica servers second
      • Upgrade Zimbra MTA servers third
      • Upgrade all dedicated Proxy servers fourth
    • Allow messages to be delivered to mailstores (still running old version) once things look right.
  • Start of Second Group of Upgrades (Mailstores)
    • Repeat the steps about backups, stopping delivery of MTAs to mailstore, and then final incremental.
      • Do a sanity check about disk space needed to queue up expected messages over time of upgrade.
        • Monitor space and be prepare to shut them back down again if disk issue becomes a problem
    • Upgrade your mailstores based up your need/configuration
      • The first mailbox server to upgrade is the server where the Documents wiki account was created. If you do not upgrade and get this server working first, you will need to manually install the Documents templates on each of the other servers.
    • Once upgrade is done and things look right, allow delivery of mail from the MTAs to the mailstores.
  • Follow up with Items listed in the official documentation about post upgrade steps.

Jump to: navigation, search