Steps To Rebuild ZCS Server: Difference between revisions
Line 187: | Line 187: | ||
'''6)''' Once backup will be done. Stop all ZCS services, unmount backup volume and then shutdown the old server. | '''6)''' Once backup will be done. Stop all ZCS services, unmount backup volume and then shutdown the old server. | ||
==== Steps needs to perform on New Server: ==== | ==== Steps needs to perform on New Server: ==== |
Revision as of 23:30, 29 May 2019
Steps To Rebuild ZCS Server
Problem
- If the ZCS server got compromised
- If the OS is not able to boot but we are able to save data of "/opt/zimbra" volume.
- If we wants to switch OS
Solution
There are 3 procedures to rebuild the ZCS server in above situations.
1) Using Zextras-Suite or NG Backup
2) Using legacy backup with Disaster Recovery steps
3) Using Rsync migration
(A) Rebuid New Server using Zextras Suite or NG Backup :
Only for servers on where NG Backup/Zextras Backup is not already initialized and running.
Steps needs to perform on Old Server:
1) Download and install latest Zextras Suite.
[Run as ROOT] wget http://download.zextras.com/zextras_suite-legacy.tgz tar zxvf zextras_suite-latest.tgz cd zextras_suite-[version] ./install.sh all
2) Add a new network storage volume to take backup and mount it on folder "/NewBackup", make sure 'zimbra' user must have ownership on the backup path.
The size of backup volume must be equal to the size of store.
[Run as ROOT] mkdir /NewBackup chown zimbra:zimbra /NewBackup/
3) Mount new network storage volume to "/NewBackup/"
4) Change backup path and disable RealTime Scanner and redolog generation then restart mailbox service to take effect.
su - zimbra zxsuite backup setProperty ZxBackup_RealTimeScanner false zxsuite backup setProperty ZxBackup_DestPath /NewBackup/ zmprov ms `zmhostname` zimbraRedoLogEnabled FALSE zmmailboxdctl restart
5) Run SmartScan and wait to finish both RealTime Scaning and SmartScan.
These processes will take time to finish, and the total time depend on the size of data in store.
zxsuite backup doSmartScan
6) Once backup is done, stop the services and mount the backup volume to newly installed server.
Steps needs to perform on New Server:
1) Install new server with ZCS v8.8.12. [Do not create any new account]
2) Start SmartScan [it will be done within a minute]
su - zimbra zxsuite backup doSmartScan
3) Now mount network storage volume from old server to path /NewBackupData/.
Ensure that the ‘zimbra’ user must have r/w permission on it.
4) Disable RealTime Scanner and redolog generation.
su - zimbra zxsuite backup setProperty ZxBackup_RealTimeScanner false zmprov ms `zmhostname` zimbraRedoLogEnabled FALSE zmmailboxdctl restart
5) If you are migrating a very large infrastructure where an export/import lasts for hours or even days, there is an alternative way to handle the migration from this point forward.
Instead of importing all of your data to the destination server, you can run a '"Provisioning Only"' import that will only create Domains, COSes and Accounts on the destination server, skipping all mailbox contents.
zxsuite backup doExternalRestore /NewBackupData/ provisioning_only TRUE
After doing this, switch the mailflow to the new server and, when the switch is completed, start the "real" data import.
zxsuite backup doExternalRestore /NewBackupData/
This way, your users will now connect to the new server where new emails will be delivered while old emails are being restored.
This approach has it's pros and cons, namely:
Pros:
- Since items are only imported once and never modified or deleted afterwards, using this method will result in less discrepancies than the "standard" incremental migration.
- This is the option that has less impact on the source server (e.g. good if you are in a hurry to decommission it).
Cons:
- Depending on the timing of the operation, this method has a higher impact on your users due to the fact that items are restored WHILE they work on their mailbox.
- Since the import is done on a running system, you might notice some slowdowns.
6) Post migration checks:
Run the following command to check and fix if there is any shares inconsistencies. Second command will parse the import mapfile used as the first argument and fix any broken share. Mapfiles can be found in the Backup Path of the destination server as "map_[source_serverID]".
zxsuite backup doCheckShares zxsuite backup doFixShares
7) After restoring data using the External Restore, there is a very high chance for the cache-based de-duplication not to catch many duplicates.
To fix message duplication we need to run following commands.
zxsuite hsm getAllVolumes zxsuite hsm doDeduplicate <volume name of the store>
8) Delete and re-create GAL Sync accounts.
/opt/zimbra/bin/zmgsautil deleteAccount -a galsync@DOMAIN.COM /opt/zimbra/bin/zmgsautil createAccount -a galsync@DOMAIN.COM -n InternalGAL --domain DOMAIN.COM -s <zmhostname of mailbox server> -t zimbra -f _InternalGAL /opt/zimbra/bin/zmgsautil forceSync -a galsync@DOMAIN.COM -n InternalGAL
9) Check and verify all services and then point mail flow to newly build server.
10) Once all data will be restored then enable RealTime Scanner and restart mailbox service.
su - zimbra zxsuite backup setProperty ZxBackup_RealTimeScanner true zmmailboxdctl restart
(B) Rebuid New Server using Legacy Backup with Disaster Recovery steps :
Steps needs to perform on Old Server:
1) Add a new network storage volume to take backup and mount it on folder "/NewBackup", make sure 'zimbra' user must have ownership on the backup path.
The size of backup volume must be equal to the size of store.
[Run as ROOT] mkdir /NewBackup chown zimbra:zimbra /NewBackup/
2) Mount new network storage volume to /NewBackup/
3) Change backup path in server level configuration:
su - zimbra zmprov -l ms `zmhostanme` zimbraBackupTarget /NewBackup zmmailboxdctl restart
4) Take a fresh full backup if there is no backup available.
/opt/zimbra/bin/zmbackup -f -a all
5) If previously full backup was taken on the server then take incremental backup.
/opt/zimbra/bin/zmbackup -i
Note: This step is optional, we can copy all available redologs /opt/zimbra/redolog/ to a safe location.
6) Once backup will be done. Stop all ZCS services, unmount backup volume and then shutdown the old server.
Steps needs to perform on New Server:
1) Install same ZCS version as installed on old server. OS can be different but the ZCS version must be identical. Also, make sure that you configure the same domain, hostname, passwords as on the old server.
2) Once installation is done, stop all ZCS services.
zmcontrol stop
3) Take a backup of /opt/zimbra/conf/localconfig.xml, as you need to copy all the passwords from localconfig.xml (from the Full backup) and replace it in /opt/zimbra/conf/localconfig.xml file. [If same passwords were not set during installation]
4) Mount network backup volume on new server at folder /NewBackupData mount point. Make sure 'zimbra' user must have ownership on the backup path.
mkdir /NewBackupData chown zimbra:zimbra /NewBackupData/
5) Prepare directories to restore LDAP from full backup label.
su - zimbra cd /opt/zimbra/data/ldap mv mdb mdb.old1 mv config config.old1 mkdir -p mdb/db mkdir config
6) Get "ldap-config.bak" and "ldap.bak" from full backup label from the mounted backup directory. These files can be in .gz format, so extract these files in /tmp/ directory for import.
/NewBackupData/sessions/full-xxxxxxxx.xxxxxx.xxx/ldap/ OR /NewBackupData/sessions/incr-xxxxxxxx.xxxxxx.xxx/ldap/
7) Import LDAP data:
su - zimbra /opt/zimbra/openldap/sbin/slapadd -q -n 0 -F /opt/zimbra/data/ldap/config -cv -l /tmp/ldap-config.bak /opt/zimbra/openldap/sbin/slapadd -q -b "" -F /opt/zimbra/data/ldap/config -cv -l /tmp/ldap.bak
8) Re-initialize MySQL database: (this step will clean old MySQL entries and initializes a fresh database)
su - zimbra mv /opt/zimbra/db/data /opt/zimbra/db/data_new source ~/bin/zmshutil ; zmsetvars /opt/zimbra/libexec/zmmyinit --sql_root_pw $mysql_root_password exit
9) Start offline restore using full backup.
su - zimbra cd /tmp ; nohup zmrestoreoffline -sys -a all --ignoreRedoErrors -c -lb <full backup> -t /NewBackupData/
Now play redo logs files which were copied from old server. We need to play redo logs in sequence like following example.
zmplayredo --logfiles /NewBackupData/redolog_old_server/archive/redo-20190522.193050.975-seq762.log zmplayredo --logfiles /NewBackupData/redolog_old_server/archive/redo-20190522.172507.329-seq761.log zmplayredo --logfiles /NewBackupData/redolog_old_server/archive/redo-20190523.102828.964-seq763.log zmplayredo --logfiles /NewBackupData/redolog_old_server/archive/redo-20190528.193039.638-seq764.log zmplayredo --logfiles /NewBackupData/redolog_old_server/archive/redo-20190529.193042.617-seq765.log zmplayredo --logfiles /NewBackupData/redolog_old_server/archive/redo-20190529.214208.456-seq766.log zmplayredo --logfiles /NewBackupData/redolog_old_server/redo.log
10) Once restore will be done, check and verify all services and then point mail flow to newly build server.
(C) Rebuid New Server using Rsync migration steps :
1) On old server check LDAP, MySQL passwords and account id of 'zimbra' user on OS level. Note down these details because we need to set same on new server.
su - zimbra zmlocalconfig -s | grep ldap | grep pass zmlocalconfig -s | grep mysql | grep pass id zimbra
2) Prepare a new server and install same ZCS version and same ZCS services as installed on old server.
Follow the directions in the ZCS single server installation guide to install ZCS. Make sure that you configure the same domain, hostname, passwords as on the old server. During ZCS install, the following settings must be changed to match the original server settings:
- Zimbra LDAP Server - For Domain to create - identify the same default domain as on the original server.
- Zimbra Mailbox Server - An administrator's account is automatically created.
- Make sure that the account name for Admin user to create is the same name as on the original server.
- Set the admin password.
- Change the Spam training user and the Non-spam (HAM) training user account names to be the same as the spam account names on the original server.
- Change any other settings on the new server to match the configuration on the original server.
3) After installation on new server set same LDAP and MySQL passwords as set on old server.
zmldappasswd <password> # set zimbra_ldap_password zmldappasswd -r <password> # set ldap_root_passwd zmldappasswd -a <password> # set ldap_amavis_password zmldappasswd -n <password> # set ldap_nginx_password zmldappasswd -p <password> # set ldap_postfix_password zmldappasswd -l <password> # set ldap_replication_password zmldappasswd -b <password> # set ldap_bes_searcher_password zmmypasswd --root <password> # set mysql_root_passwd zmmypasswd <password> # set zimbra_myql_password zmlocalconfig -e zimbra_uid="XXXX" #set same uid of zimbra user from old server zmlocalconfig -e zimbra_gid="XXXX" #set same gid of zimbra user from old server
4) Create a directory on new server to copy data from old server.
mkdir /opt/data_from_old_server/
5) Stop ZCS service on both servers.
zmcontrol stop
6) Run following commands on old server to rsync MySQL, store and index data to new server.
rsync -avzHSKP /opt/zimbra/db/data [NewServer's-IP-address]:/opt/data_from_old_server/ rsync -avzHSKP /opt/zimbra/store [NewServer's-IP-address]:/opt/data_from_old_server/ rsync -avzHSKP /opt/zimbra/index [NewServer's-IP-address]:/opt/data_from_old_server/
If HSM volume configured on old server then rsync that volume data as well. [Create same mount point on new server for HSM and directly copy data into that volume]
rsync -avzHSKP /opt/zimbra/hsm [NewServer's-IP-address]:/opt/<HSM_Volume_Path>/
7) Take LDAP dump on old server and rsync it to new server:
su - zimbra mkdir /tmp/ldap_backup/ /opt/zimbra/libexec/zmslapcat -c /tmp/ldap_backup/ /opt/zimbra/libexec/zmslapcat /tmp/ldap_backup/
rsync -avzHSKP /tmp/ldap_backup [NewServer's-IP-address]:/opt/data_from_old_server/
8) Shutdown old server.
Steps needs to perform on New Server:
1) Create a new directory and move out data of fresh installation.
mkdir /opt/fresh_installation_data/ mv /opt/zimbra/store /opt/fresh_installation_data/ mv /opt/zimbra/index /opt/fresh_installation_data/ mv /opt/zimbra/db/data /opt/fresh_installation_data/
2) Now we will place old server's data on new server.
mv /opt/data_from_old_server/data /opt/zimbra/db/ mv /opt/data_from_old_server/store /opt/zimbra/store mv /opt/data_from_old_server/index /opt/zimbra/index
3) Prepare directories to restore LDAP data and import backup.
su - zimbra cd /opt/zimbra/data/ldap mv mdb mdb.old1 mv config config.old1 mkdir -p mdb/db mkdir config /opt/zimbra/libexec/zmslapcat -c /opt/data_from_old_server/ldap_backup/ldap-config.bak /opt/zimbra/libexec/zmslapcat /opt/data_from_old_server/ldap_backup/ldap.bak
4) Start ZCS service on new server.
zmcontrol start
5) Check and verify all services and then point mail flow to newly build server.
Submitted by: Heera Singh Koranga |