Steps To Rebuild ZCS Server

Steps To Rebuild ZCS Server


   KB 23853        Last updated on 2023-09-18  




0.00
(0 votes)


Problem

  • If the ZCS server got compromised
  • If the OS is not able to boot but we are able to save data of "/opt/zimbra" volume.
  • If we wants to switch OS


Solution

There are 3 procedures to rebuild the ZCS server in above situations.

1) Using Zextras-Suite or NG Backup

2) Using legacy backup with Disaster Recovery steps

3) Using Rsync migration


(A) Rebuild New Server using Zextras Suite or NG Backup

Only for servers on where NG Backup/Zextras Backup is not already initialized and running.

Steps needs to perform on Old Server

1) Download and install latest Zextras Suite.

[Run as ROOT] 
wget http://download.zextras.com/zextras_suite-legacy.tgz 
tar zxvf zextras_suite-latest.tgz 
cd zextras_suite-[version] 
./install.sh all   


2) Add a new network storage volume to take backup and mount it on folder "/NewBackup", make sure 'zimbra' user must have ownership on the backup path.
The size of backup volume must be equal to the size of store+hsm volume.

[Run as ROOT]
mkdir /NewBackup 
chown zimbra:zimbra /NewBackup/  


3) Mount new network storage volume to "/NewBackup/"


4) Change backup path and disable RealTime Scanner and redolog generation then restart mailbox service to take effect.

su - zimbra 
zxsuite backup setProperty ZxBackup_RealTimeScanner false  
zxsuite backup setProperty ZxBackup_DestPath /NewBackup/  
zmprov ms `zmhostname` zimbraRedoLogEnabled FALSE 
zmmailboxdctl restart 


5) Run SmartScan and wait to finish both RealTime Scaning and SmartScan.
These processes will take time to finish, and the total time depend on the size of data in store.

zxsuite backup doSmartScan  


6) Once backup is done, stop the services and mount the backup volume to newly installed server.


Steps needs to perform on New Server

Note: Once the destination server is in production, existing connections will no longer be valid. Active users will need to login into the new system with the existing password.
ZCO and Zimbra Desktop Client profiles will need to be recreated. IMAP, POP and active-sync accounts will reload existing data.


1) Install new server with ZCS v8.8.12. [Do not create any new account]


2) Start SmartScan [it will be done within a minute]

su - zimbra 
zxsuite backup doSmartScan


3) Now mount network storage volume from old server to path /NewBackupData/. Ensure that the ‘zimbra’ user must have r/w permission on it.


4) Disable RealTime Scanner and redolog generation.

su - zimbra 
zxsuite backup setProperty ZxBackup_RealTimeScanner false  
zmprov ms `zmhostname` zimbraRedoLogEnabled FALSE 
zmmailboxdctl restart 


5) If you are migrating a very large infrastructure where an export/import lasts for hours or even days, there is an alternative way to handle the migration from this point forward.
Instead of importing all of your data to the destination server, you can run a '"Provisioning Only"' import that will only create Domains, COSes and Accounts on the destination server, skipping all mailbox contents.

zxsuite backup doExternalRestore /NewBackupData/ provisioning_only TRUE  

After doing this, switch the mailflow to the new server and, when the switch is completed, start the "real" data import.

zxsuite backup doExternalRestore /NewBackupData/   

This way, your users will now connect to the new server where new emails will be delivered while old emails are being restored.

This approach has it's pros and cons, namely:

Pros:

  • Since items are only imported once and never modified or deleted afterwards, using this method will result in less discrepancies than the "standard" incremental migration.
  • This is the option that has less impact on the source server (e.g. good if you are in a hurry to decommission it).

Cons:

  • Depending on the timing of the operation, this method has a higher impact on your users due to the fact that items are restored WHILE they work on their mailbox.
  • Since the import is done on a running system, you might notice some slowdowns.


6) Post migration checks:

Run the following command to check and fix if there is any shares inconsistencies. Second command will parse the import mapfile used as the first argument and fix any broken share. Mapfiles can be found in the Backup Path of the destination server as "map_[source_serverID]".

zxsuite backup doCheckShares  
zxsuite backup doFixShares


7) After restoring data using the External Restore, there is a very high chance for the cache-based de-duplication not to catch many duplicates. To fix message duplication we need to run following commands.

zxsuite hsm getAllVolumes 
zxsuite hsm doDeduplicate <volume name of the store>  


8) Delete and re-create GAL Sync accounts.

/opt/zimbra/bin/zmgsautil deleteAccount -a galsync@DOMAIN.COM 
/opt/zimbra/bin/zmgsautil createAccount -a galsync@DOMAIN.COM -n InternalGAL --domain DOMAIN.COM -s <zmhostname of mailbox server> -t zimbra -f _InternalGAL
/opt/zimbra/bin/zmgsautil forceSync -a galsync@DOMAIN.COM -n InternalGAL 


9) Check and verify all services and then point mail flow to newly build server.


10) Once all data will be restored then enable RealTime Scanner and restart mailbox service.

su - zimbra 
zxsuite backup setProperty ZxBackup_RealTimeScanner true    
zmmailboxdctl restart

(B) Rebuild New Server using Legacy Backup with Disaster Recovery steps

Steps needs to perform on Old Server

1) Add a new network storage volume to take backup and mount it on folder "/NewBackup", make sure 'zimbra' user must have ownership on the backup path.
The size of backup volume must be equal to the size of store.

[Run as ROOT]
mkdir /NewBackup 
chown zimbra:zimbra /NewBackup/  


2) Mount new network storage volume to /NewBackup/


3) Change backup path in server level configuration:

su - zimbra 
zmprov -l ms `zmhostanme` zimbraBackupTarget /NewBackup 
zmmailboxdctl restart 


4) Take a fresh full backup if there is no backup available.

/opt/zimbra/bin/zmbackup -f -a all  


5) If previously full backup was taken on the server then take incremental backup.

/opt/zimbra/bin/zmbackup -i  

Note: This step is optional, we can copy all available redologs /opt/zimbra/redolog/ to a safe location.


6) Once backup will be done. Stop all ZCS services, unmount backup volume and then shutdown the old server.


Steps needs to perform on New Server

1) Install same ZCS version as installed on old server. OS can be different but the ZCS version must be identical. Also, make sure that you configure the same domain, hostname, passwords as on the old server.


2) Once installation is done, stop all ZCS services.

zmcontrol stop 


3) Take a backup of /opt/zimbra/conf/localconfig.xml, as you need to copy all the passwords from localconfig.xml (from the Full backup) and replace it in /opt/zimbra/conf/localconfig.xml file. [If same passwords were not set during installation]


4) Mount network backup volume on new server at folder /NewBackupData mount point. Make sure 'zimbra' user must have ownership on the backup path.

mkdir /NewBackupData 
chown zimbra:zimbra /NewBackupData/  


5) Prepare directories to restore LDAP from full backup label.

su - zimbra 
cd /opt/zimbra/data/ldap  
mv config config.old1 
mv mdb mdb.old1 
mv accesslog accesslog.old1
mkdir config 
mkdir -p mdb/db 
mkdir -p accesslog/db 


6) Get "ldap-config.bak" and "ldap.bak" from full backup label from the mounted backup directory. These files can be in .gz format, so extract these files in /tmp/ directory for import.

/NewBackupData/sessions/full-xxxxxxxx.xxxxxx.xxx/ldap/
OR 
/NewBackupData/sessions/incr-xxxxxxxx.xxxxxx.xxx/ldap/


7) Import LDAP data:

su - zimbra  
/opt/zimbra/libexec/zmslapadd -c /tmp/ldap-config.bak
/opt/zimbra/libexec/zmslapadd /tmp/ldap.bak  


8) Re-initialize MySQL database: (this step will clean old MySQL entries and initializes a fresh database)

su - zimbra 
mv /opt/zimbra/db/data /opt/zimbra/db/data_new 
source ~/bin/zmshutil ; zmsetvars
/opt/zimbra/libexec/zmmyinit --sql_root_pw $mysql_root_password   
exit 


9) If any HSM volume configured on old server then we need to configure HSM volume on new server at same path (mount point).

[Run as ROOT]
mkdir /opt/zimbra/hsm
mount /dev/<volume_name> /opt/zimbra/hsm 
chown zimbra:zimbra /opt/zimbra/hsm 


10) Start offline restore using full backup.

su - zimbra  
ldap start
zmconvertctl start 
cd /tmp ; nohup zmrestoreoffline -sys -a all --ignoreRedoErrors -c -lb <full backup> -t /NewBackupData/    

Now play redo logs files which were copied from old server. We need to play redo logs in sequence like following example.

zmplayredo --logfiles /NewBackupData/redolog_old_server/archive/redo-20190522.193050.975-seq762.log
zmplayredo --logfiles /NewBackupData/redolog_old_server/archive/redo-20190522.172507.329-seq761.log
zmplayredo --logfiles /NewBackupData/redolog_old_server/archive/redo-20190523.102828.964-seq763.log
zmplayredo --logfiles /NewBackupData/redolog_old_server/archive/redo-20190528.193039.638-seq764.log
zmplayredo --logfiles /NewBackupData/redolog_old_server/archive/redo-20190529.193042.617-seq765.log
zmplayredo --logfiles /NewBackupData/redolog_old_server/archive/redo-20190529.214208.456-seq766.log
zmplayredo --logfiles /NewBackupData/redolog_old_server/redo.log 


11) Once restore will be done, check and verify all services and then point mail flow to newly build server.


12) Redeploy SSL certificate,
If previously using a self-signed certificate then regenerate the new self-signed certificate.
If previously using a commercial certificate then deploy same commercial certificate. Refer Regenerate SSL certificate


(C) Rebuild New Server using Rsync migration steps

1) On old server check LDAP and MySQL passwords. Note down these details because we need to set same on new server.

su - zimbra 
zmlocalconfig -s | grep ldap | grep pass 
zmlocalconfig -s | grep mysql | grep pass  


2) Prepare a new server and install same ZCS version and same ZCS services as installed on old server.

Follow the directions in the ZCS single server installation guide to install ZCS. Make sure that you configure the same domain, hostname, passwords as on the old server. During ZCS install, the following settings must be changed to match the original server settings:

  • Zimbra LDAP Server - For Domain to create - identify the same default domain as on the original server.
  • Zimbra Mailbox Server - An administrator's account is automatically created.
  • Make sure that the account name for Admin user to create is the same name as on the original server.
  • Set the admin password.
  • Change the Spam training user and the Non-spam (HAM) training user account names to be the same as the spam account names on the original server.
  • Change any other settings on the new server to match the configuration on the original server.


3) After installation on new server set same LDAP and MySQL passwords as set on old server.

zmldappasswd <password>                   # set zimbra_ldap_password 
zmldappasswd -r <password>                # set ldap_root_passwd 
zmldappasswd -a <password>                # set ldap_amavis_password 
zmldappasswd -n <password>                # set ldap_nginx_password
zmldappasswd -p <password>                # set ldap_postfix_password 
zmldappasswd -l <password>                # set ldap_replication_password 
zmldappasswd -b <password>                # set ldap_bes_searcher_password  

zmmypasswd --root <password>              # set mysql_root_passwd   
zmmypasswd <password>                     # set zimbra_myql_password 


4) Create a directory on new server to copy data from old server.

mkdir /opt/data_from_old_server/ 


5) Stop ZCS service on both servers.

zmcontrol stop 


6) Run following commands on old server to rsync MySQL, store and index data to new server.

rsync -avHpSKogP  /opt/zimbra/db/data [NewServer's-IP-address]:/opt/data_from_old_server/ 
rsync -avHpSKogP  /opt/zimbra/store [NewServer's-IP-address]:/opt/data_from_old_server/ 
rsync -avHpSKogP  /opt/zimbra/index [NewServer's-IP-address]:/opt/data_from_old_server/  

[Optional] If HSM volume configured on old server then rsync that volume data as well.
[Create same mount point on new server for HSM and directly copy data into that volume]

rsync -avHpSKogP  /<HSM_Volume_Path>/  [NewServer's-IP-address]:/opt/<HSM_Volume_Path>/   


7) Take LDAP dump on old server and rsync it to new server:

su - zimbra  
mkdir /tmp/ldap_backup/ 
/opt/zimbra/libexec/zmslapcat -c /tmp/ldap_backup/ 
/opt/zimbra/libexec/zmslapcat /tmp/ldap_backup/  
rsync -avHpSKogP  /tmp/ldap_backup [NewServer's-IP-address]:/opt/data_from_old_server/  


8) Shutdown old server.


Steps needs to perform on New Server

1) Create a new directory and move out data of fresh installation.

mkdir /opt/fresh_installation_data/ 
mv /opt/zimbra/store /opt/fresh_installation_data/  
mv /opt/zimbra/index /opt/fresh_installation_data/ 
mv /opt/zimbra/db/data /opt/fresh_installation_data/ 


2) Now we will place old server's data on new server.

mv /opt/data_from_old_server/data  /opt/zimbra/db/ 
mv /opt/data_from_old_server/store /opt/zimbra/store 
mv /opt/data_from_old_server/index /opt/zimbra/index 


3) Prepare directories to restore LDAP data and import backup.

su - zimbra 
cd /opt/zimbra/data/ldap 
mv mdb mdb.old1 
mv config config.old1 
mkdir -p mdb/db 
mkdir config  
/opt/zimbra/libexec/zmslapadd -c /opt/data_from_old_server/ldap_backup/ldap-config.bak
/opt/zimbra/libexec/zmslapadd /opt/data_from_old_server/ldap_backup/ldap.bak
zmcontrol restart 


4) Redeploy SSL certificate,
If previously using a self-signed certificate then regenerate the new self-signed certificate.
If previously using a commercial certificate then deploy same commercial certificate. Refer Regenerate SSL certificate


5) Check and verify all services and then point mail flow to newly build server.

Note: Check and verify If the source server has a NG module, then proceed with the following steps to obtain the NG module configuration on new server. Copy NG-drive DB data and zextras folder from old server to the new server.

  • Execute the provided command to stop the services and move the existing folder to .old on the new server.
su - zimbra
zmcontrol stop
mv /opt/zimbra/conf/zextras /opt/zimbra/conf/zextras.old
mv /opt/zimbra/data/drive  /opt/zimbra/data/drive.old
  • Execute the provided command on the old server to transfer the zextras and drive folders to the new server. Please ensure that services are stopped on the old server
rsync -avzHSKP /opt/zimbra/conf/zextras [NewServer's-IP-address]:/opt/zimbra/conf
rsync -avzHSKP /opt/zimbra/data/drive [NewServer's-IP-address]:/opt/zimbra/data
  • Restart the services on the new server.
zmcontrol restart
Submitted by: Heera Singh Koranga
Verified Against: ZCS 8.8.x, 8.7.x, 8.6, 8.5.x Date Created: 2019-05-30
Article ID: https://wiki.zimbra.com/index.php?title=Steps_To_Rebuild_ZCS_Server Date Modified: 2023-09-18



Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »


Jump to: navigation, search