Steps To Rebuild ZCS Server: Difference between revisions
mNo edit summary |
No edit summary |
||
(12 intermediate revisions by 2 users not shown) | |||
Line 26: | Line 26: | ||
===(A) | ===(A) Rebuild New Server using Zextras Suite or NG Backup === | ||
Only for servers on where NG Backup/Zextras Backup is not already initialized and running. | Only for servers on where NG Backup/Zextras Backup is not already initialized and running. | ||
==== Steps needs to perform on Old Server | ==== Steps needs to perform on Old Server ==== | ||
'''1)''' Download and install latest Zextras Suite. | '''1)''' Download and install latest Zextras Suite. | ||
Line 42: | Line 42: | ||
'''2)''' Add a new network storage volume to take backup and mount it on folder "/NewBackup", make sure 'zimbra' user must have ownership on the backup path. <br/> | '''2)''' Add a new network storage volume to take backup and mount it on folder "/NewBackup", make sure 'zimbra' user must have ownership on the backup path. <br/> | ||
The size of backup volume must be equal to the size of store. | The size of backup volume must be equal to the size of store+hsm volume. | ||
[Run as ROOT] | [Run as ROOT] | ||
Line 70: | Line 70: | ||
==== Steps needs to perform on New Server | ==== Steps needs to perform on New Server ==== | ||
'''Note:''' Once the destination server is in production, existing connections will no longer be valid. Active users will need to login into the new system with the existing password. <br /> | |||
ZCO and Zimbra Desktop Client profiles will need to be recreated. IMAP, POP and active-sync accounts will reload existing data. | |||
Line 148: | Line 151: | ||
zmmailboxdctl restart | zmmailboxdctl restart | ||
===(B) Rebuild New Server using Legacy Backup with Disaster Recovery steps === | |||
==== Steps needs to perform on Old Server ==== | |||
==== Steps needs to perform on Old Server | |||
'''1)''' Add a new network storage volume to take backup and mount it on folder "/NewBackup", make sure 'zimbra' user must have ownership on the backup path. <br /> | '''1)''' Add a new network storage volume to take backup and mount it on folder "/NewBackup", make sure 'zimbra' user must have ownership on the backup path. <br /> | ||
Line 189: | Line 189: | ||
==== Steps needs to perform on New Server | ==== Steps needs to perform on New Server ==== | ||
'''1)''' Install same ZCS version as installed on old server. OS can be different but the ZCS version must be identical. | '''1)''' Install same ZCS version as installed on old server. OS can be different but the ZCS version must be identical. | ||
Line 213: | Line 213: | ||
su - zimbra | su - zimbra | ||
cd /opt/zimbra/data/ldap | cd /opt/zimbra/data/ldap | ||
mv config config.old1 | |||
mv mdb mdb.old1 | mv mdb mdb.old1 | ||
mv | mv accesslog accesslog.old1 | ||
mkdir config | |||
mkdir -p mdb/db | mkdir -p mdb/db | ||
mkdir | mkdir -p accesslog/db | ||
Line 246: | Line 248: | ||
[Run as ROOT] | [Run as ROOT] | ||
mkdir /opt/zimbra/hsm | mkdir /opt/zimbra/hsm | ||
mount /dev/<volume_name> /opt/zimbra/hsm | |||
chown zimbra:zimbra /opt/zimbra/hsm | |||
Line 255: | Line 255: | ||
su - zimbra | su - zimbra | ||
ldap start | |||
zmconvertctl start | |||
cd /tmp ; nohup zmrestoreoffline -sys -a all --ignoreRedoErrors -c -lb <full backup> -t /NewBackupData/ | cd /tmp ; nohup zmrestoreoffline -sys -a all --ignoreRedoErrors -c -lb <full backup> -t /NewBackupData/ | ||
Line 271: | Line 273: | ||
'''11)''' Once restore will be done, check and verify all services and then point mail flow to newly build server. | '''11)''' Once restore will be done, check and verify all services and then point mail flow to newly build server. | ||
<br/> | |||
'''12)''' Redeploy SSL certificate,<br/> | |||
If previously using a self-signed certificate then regenerate the new self-signed certificate. <br/> | |||
If previously using a commercial certificate then deploy same commercial certificate. | |||
[https://wiki.zimbra.com/wiki/Administration_Console_and_CLI_Certificate_Tools Refer Regenerate SSL certificate] | |||
===(C) | ===(C) Rebuild New Server using Rsync migration steps === | ||
'''1)''' On old server check LDAP | '''1)''' On old server check LDAP and MySQL passwords. Note down these details because we need to set same on new server. | ||
su - zimbra | su - zimbra | ||
zmlocalconfig -s | grep ldap | grep pass | zmlocalconfig -s | grep ldap | grep pass | ||
zmlocalconfig -s | grep mysql | grep pass | zmlocalconfig -s | grep mysql | grep pass | ||
Line 308: | Line 314: | ||
zmmypasswd --root <password> # set mysql_root_passwd | zmmypasswd --root <password> # set mysql_root_passwd | ||
zmmypasswd <password> # set zimbra_myql_password | zmmypasswd <password> # set zimbra_myql_password | ||
Line 325: | Line 328: | ||
'''6)''' Run following commands on old server to rsync MySQL, store and index data to new server. | '''6)''' Run following commands on old server to rsync MySQL, store and index data to new server. | ||
rsync - | rsync -avHpSKogP /opt/zimbra/db/data [NewServer's-IP-address]:/opt/data_from_old_server/ | ||
rsync - | rsync -avHpSKogP /opt/zimbra/store [NewServer's-IP-address]:/opt/data_from_old_server/ | ||
rsync - | rsync -avHpSKogP /opt/zimbra/index [NewServer's-IP-address]:/opt/data_from_old_server/ | ||
'''[Optional]''' If HSM volume configured on old server then rsync that volume data as well. <br /> | '''[Optional]''' If HSM volume configured on old server then rsync that volume data as well. <br /> | ||
[Create same mount point on new server for HSM and directly copy data into that volume] | [Create same mount point on new server for HSM and directly copy data into that volume] | ||
rsync - | rsync -avHpSKogP /<HSM_Volume_Path>/ [NewServer's-IP-address]:/opt/<HSM_Volume_Path>/ | ||
Line 342: | Line 345: | ||
/opt/zimbra/libexec/zmslapcat /tmp/ldap_backup/ | /opt/zimbra/libexec/zmslapcat /tmp/ldap_backup/ | ||
rsync - | rsync -avHpSKogP /tmp/ldap_backup [NewServer's-IP-address]:/opt/data_from_old_server/ | ||
Line 348: | Line 351: | ||
==== Steps needs to perform on New Server | ==== Steps needs to perform on New Server ==== | ||
'''1)''' Create a new directory and move out data of fresh installation. | '''1)''' Create a new directory and move out data of fresh installation. | ||
Line 374: | Line 377: | ||
mkdir config | mkdir config | ||
/opt/zimbra/libexec/zmslapadd -c /opt/data_from_old_server/ldap_backup/ldap-config.bak | /opt/zimbra/libexec/zmslapadd -c /opt/data_from_old_server/ldap_backup/ldap-config.bak | ||
/opt/zimbra/libexec/zmslapadd /opt/data_from_old_server/ldap_backup/ldap.bak | /opt/zimbra/libexec/zmslapadd /opt/data_from_old_server/ldap_backup/ldap.bak | ||
zmcontrol restart | |||
<br/> | |||
'''4)''' Redeploy SSL certificate,<br/> | |||
If previously using a self-signed certificate then regenerate the new self-signed certificate. <br/> | |||
If previously using a commercial certificate then deploy same commercial certificate. | |||
[https://wiki.zimbra.com/wiki/Administration_Console_and_CLI_Certificate_Tools Refer Regenerate SSL certificate] | |||
Revision as of 13:00, 30 December 2019
Steps To Rebuild ZCS Server
Problem
- If the ZCS server got compromised
- If the OS is not able to boot but we are able to save data of "/opt/zimbra" volume.
- If we wants to switch OS
Solution
There are 3 procedures to rebuild the ZCS server in above situations.
1) Using Zextras-Suite or NG Backup
2) Using legacy backup with Disaster Recovery steps
3) Using Rsync migration
(A) Rebuild New Server using Zextras Suite or NG Backup
Only for servers on where NG Backup/Zextras Backup is not already initialized and running.
Steps needs to perform on Old Server
1) Download and install latest Zextras Suite.
[Run as ROOT] wget http://download.zextras.com/zextras_suite-legacy.tgz tar zxvf zextras_suite-latest.tgz cd zextras_suite-[version] ./install.sh all
2) Add a new network storage volume to take backup and mount it on folder "/NewBackup", make sure 'zimbra' user must have ownership on the backup path.
The size of backup volume must be equal to the size of store+hsm volume.
[Run as ROOT] mkdir /NewBackup chown zimbra:zimbra /NewBackup/
3) Mount new network storage volume to "/NewBackup/"
4) Change backup path and disable RealTime Scanner and redolog generation then restart mailbox service to take effect.
su - zimbra zxsuite backup setProperty ZxBackup_RealTimeScanner false zxsuite backup setProperty ZxBackup_DestPath /NewBackup/ zmprov ms `zmhostname` zimbraRedoLogEnabled FALSE zmmailboxdctl restart
5) Run SmartScan and wait to finish both RealTime Scaning and SmartScan.
These processes will take time to finish, and the total time depend on the size of data in store.
zxsuite backup doSmartScan
6) Once backup is done, stop the services and mount the backup volume to newly installed server.
Steps needs to perform on New Server
Note: Once the destination server is in production, existing connections will no longer be valid. Active users will need to login into the new system with the existing password.
ZCO and Zimbra Desktop Client profiles will need to be recreated. IMAP, POP and active-sync accounts will reload existing data.
1) Install new server with ZCS v8.8.12. [Do not create any new account]
2) Start SmartScan [it will be done within a minute]
su - zimbra zxsuite backup doSmartScan
3) Now mount network storage volume from old server to path /NewBackupData/.
Ensure that the ‘zimbra’ user must have r/w permission on it.
4) Disable RealTime Scanner and redolog generation.
su - zimbra zxsuite backup setProperty ZxBackup_RealTimeScanner false zmprov ms `zmhostname` zimbraRedoLogEnabled FALSE zmmailboxdctl restart
5) If you are migrating a very large infrastructure where an export/import lasts for hours or even days, there is an alternative way to handle the migration from this point forward.
Instead of importing all of your data to the destination server, you can run a '"Provisioning Only"' import that will only create Domains, COSes and Accounts on the destination server, skipping all mailbox contents.
zxsuite backup doExternalRestore /NewBackupData/ provisioning_only TRUE
After doing this, switch the mailflow to the new server and, when the switch is completed, start the "real" data import.
zxsuite backup doExternalRestore /NewBackupData/
This way, your users will now connect to the new server where new emails will be delivered while old emails are being restored.
This approach has it's pros and cons, namely:
Pros:
- Since items are only imported once and never modified or deleted afterwards, using this method will result in less discrepancies than the "standard" incremental migration.
- This is the option that has less impact on the source server (e.g. good if you are in a hurry to decommission it).
Cons:
- Depending on the timing of the operation, this method has a higher impact on your users due to the fact that items are restored WHILE they work on their mailbox.
- Since the import is done on a running system, you might notice some slowdowns.
6) Post migration checks:
Run the following command to check and fix if there is any shares inconsistencies. Second command will parse the import mapfile used as the first argument and fix any broken share. Mapfiles can be found in the Backup Path of the destination server as "map_[source_serverID]".
zxsuite backup doCheckShares zxsuite backup doFixShares
7) After restoring data using the External Restore, there is a very high chance for the cache-based de-duplication not to catch many duplicates.
To fix message duplication we need to run following commands.
zxsuite hsm getAllVolumes zxsuite hsm doDeduplicate <volume name of the store>
8) Delete and re-create GAL Sync accounts.
/opt/zimbra/bin/zmgsautil deleteAccount -a galsync@DOMAIN.COM /opt/zimbra/bin/zmgsautil createAccount -a galsync@DOMAIN.COM -n InternalGAL --domain DOMAIN.COM -s <zmhostname of mailbox server> -t zimbra -f _InternalGAL /opt/zimbra/bin/zmgsautil forceSync -a galsync@DOMAIN.COM -n InternalGAL
9) Check and verify all services and then point mail flow to newly build server.
10) Once all data will be restored then enable RealTime Scanner and restart mailbox service.
su - zimbra zxsuite backup setProperty ZxBackup_RealTimeScanner true zmmailboxdctl restart
(B) Rebuild New Server using Legacy Backup with Disaster Recovery steps
Steps needs to perform on Old Server
1) Add a new network storage volume to take backup and mount it on folder "/NewBackup", make sure 'zimbra' user must have ownership on the backup path.
The size of backup volume must be equal to the size of store.
[Run as ROOT] mkdir /NewBackup chown zimbra:zimbra /NewBackup/
2) Mount new network storage volume to /NewBackup/
3) Change backup path in server level configuration:
su - zimbra zmprov -l ms `zmhostanme` zimbraBackupTarget /NewBackup zmmailboxdctl restart
4) Take a fresh full backup if there is no backup available.
/opt/zimbra/bin/zmbackup -f -a all
5) If previously full backup was taken on the server then take incremental backup.
/opt/zimbra/bin/zmbackup -i
Note: This step is optional, we can copy all available redologs /opt/zimbra/redolog/ to a safe location.
6) Once backup will be done. Stop all ZCS services, unmount backup volume and then shutdown the old server.
Steps needs to perform on New Server
1) Install same ZCS version as installed on old server. OS can be different but the ZCS version must be identical. Also, make sure that you configure the same domain, hostname, passwords as on the old server.
2) Once installation is done, stop all ZCS services.
zmcontrol stop
3) Take a backup of /opt/zimbra/conf/localconfig.xml, as you need to copy all the passwords from localconfig.xml (from the Full backup) and replace it in /opt/zimbra/conf/localconfig.xml file. [If same passwords were not set during installation]
4) Mount network backup volume on new server at folder /NewBackupData mount point. Make sure 'zimbra' user must have ownership on the backup path.
mkdir /NewBackupData chown zimbra:zimbra /NewBackupData/
5) Prepare directories to restore LDAP from full backup label.
su - zimbra cd /opt/zimbra/data/ldap mv config config.old1 mv mdb mdb.old1 mv accesslog accesslog.old1 mkdir config mkdir -p mdb/db mkdir -p accesslog/db
6) Get "ldap-config.bak" and "ldap.bak" from full backup label from the mounted backup directory. These files can be in .gz format, so extract these files in /tmp/ directory for import.
/NewBackupData/sessions/full-xxxxxxxx.xxxxxx.xxx/ldap/ OR /NewBackupData/sessions/incr-xxxxxxxx.xxxxxx.xxx/ldap/
7) Import LDAP data:
su - zimbra /opt/zimbra/libexec/zmslapadd -c /tmp/ldap-config.bak /opt/zimbra/libexec/zmslapadd /tmp/ldap.bak
8) Re-initialize MySQL database: (this step will clean old MySQL entries and initializes a fresh database)
su - zimbra mv /opt/zimbra/db/data /opt/zimbra/db/data_new source ~/bin/zmshutil ; zmsetvars /opt/zimbra/libexec/zmmyinit --sql_root_pw $mysql_root_password exit
9) If any HSM volume configured on old server then we need to configure HSM volume on new server at same path (mount point).
[Run as ROOT] mkdir /opt/zimbra/hsm mount /dev/<volume_name> /opt/zimbra/hsm chown zimbra:zimbra /opt/zimbra/hsm
10) Start offline restore using full backup.
su - zimbra ldap start zmconvertctl start cd /tmp ; nohup zmrestoreoffline -sys -a all --ignoreRedoErrors -c -lb <full backup> -t /NewBackupData/
Now play redo logs files which were copied from old server. We need to play redo logs in sequence like following example.
zmplayredo --logfiles /NewBackupData/redolog_old_server/archive/redo-20190522.193050.975-seq762.log zmplayredo --logfiles /NewBackupData/redolog_old_server/archive/redo-20190522.172507.329-seq761.log zmplayredo --logfiles /NewBackupData/redolog_old_server/archive/redo-20190523.102828.964-seq763.log zmplayredo --logfiles /NewBackupData/redolog_old_server/archive/redo-20190528.193039.638-seq764.log zmplayredo --logfiles /NewBackupData/redolog_old_server/archive/redo-20190529.193042.617-seq765.log zmplayredo --logfiles /NewBackupData/redolog_old_server/archive/redo-20190529.214208.456-seq766.log zmplayredo --logfiles /NewBackupData/redolog_old_server/redo.log
11) Once restore will be done, check and verify all services and then point mail flow to newly build server.
12) Redeploy SSL certificate,
If previously using a self-signed certificate then regenerate the new self-signed certificate.
If previously using a commercial certificate then deploy same commercial certificate.
Refer Regenerate SSL certificate
(C) Rebuild New Server using Rsync migration steps
1) On old server check LDAP and MySQL passwords. Note down these details because we need to set same on new server.
su - zimbra zmlocalconfig -s | grep ldap | grep pass zmlocalconfig -s | grep mysql | grep pass
2) Prepare a new server and install same ZCS version and same ZCS services as installed on old server.
Follow the directions in the ZCS single server installation guide to install ZCS. Make sure that you configure the same domain, hostname, passwords as on the old server. During ZCS install, the following settings must be changed to match the original server settings:
- Zimbra LDAP Server - For Domain to create - identify the same default domain as on the original server.
- Zimbra Mailbox Server - An administrator's account is automatically created.
- Make sure that the account name for Admin user to create is the same name as on the original server.
- Set the admin password.
- Change the Spam training user and the Non-spam (HAM) training user account names to be the same as the spam account names on the original server.
- Change any other settings on the new server to match the configuration on the original server.
3) After installation on new server set same LDAP and MySQL passwords as set on old server.
zmldappasswd <password> # set zimbra_ldap_password zmldappasswd -r <password> # set ldap_root_passwd zmldappasswd -a <password> # set ldap_amavis_password zmldappasswd -n <password> # set ldap_nginx_password zmldappasswd -p <password> # set ldap_postfix_password zmldappasswd -l <password> # set ldap_replication_password zmldappasswd -b <password> # set ldap_bes_searcher_password zmmypasswd --root <password> # set mysql_root_passwd zmmypasswd <password> # set zimbra_myql_password
4) Create a directory on new server to copy data from old server.
mkdir /opt/data_from_old_server/
5) Stop ZCS service on both servers.
zmcontrol stop
6) Run following commands on old server to rsync MySQL, store and index data to new server.
rsync -avHpSKogP /opt/zimbra/db/data [NewServer's-IP-address]:/opt/data_from_old_server/ rsync -avHpSKogP /opt/zimbra/store [NewServer's-IP-address]:/opt/data_from_old_server/ rsync -avHpSKogP /opt/zimbra/index [NewServer's-IP-address]:/opt/data_from_old_server/
[Optional] If HSM volume configured on old server then rsync that volume data as well.
[Create same mount point on new server for HSM and directly copy data into that volume]
rsync -avHpSKogP /<HSM_Volume_Path>/ [NewServer's-IP-address]:/opt/<HSM_Volume_Path>/
7) Take LDAP dump on old server and rsync it to new server:
su - zimbra mkdir /tmp/ldap_backup/ /opt/zimbra/libexec/zmslapcat -c /tmp/ldap_backup/ /opt/zimbra/libexec/zmslapcat /tmp/ldap_backup/
rsync -avHpSKogP /tmp/ldap_backup [NewServer's-IP-address]:/opt/data_from_old_server/
8) Shutdown old server.
Steps needs to perform on New Server
1) Create a new directory and move out data of fresh installation.
mkdir /opt/fresh_installation_data/ mv /opt/zimbra/store /opt/fresh_installation_data/ mv /opt/zimbra/index /opt/fresh_installation_data/ mv /opt/zimbra/db/data /opt/fresh_installation_data/
2) Now we will place old server's data on new server.
mv /opt/data_from_old_server/data /opt/zimbra/db/ mv /opt/data_from_old_server/store /opt/zimbra/store mv /opt/data_from_old_server/index /opt/zimbra/index
3) Prepare directories to restore LDAP data and import backup.
su - zimbra cd /opt/zimbra/data/ldap mv mdb mdb.old1 mv config config.old1 mkdir -p mdb/db mkdir config /opt/zimbra/libexec/zmslapadd -c /opt/data_from_old_server/ldap_backup/ldap-config.bak /opt/zimbra/libexec/zmslapadd /opt/data_from_old_server/ldap_backup/ldap.bak zmcontrol restart
4) Redeploy SSL certificate,
If previously using a self-signed certificate then regenerate the new self-signed certificate.
If previously using a commercial certificate then deploy same commercial certificate.
Refer Regenerate SSL certificate
5) Check and verify all services and then point mail flow to newly build server.
Submitted by: Heera Singh Koranga |