ZCS88-zxbackup-Disaster-Recovery: Difference between revisions
| (36 intermediate revisions by the same user not shown) | |||
| Line 119: | Line 119: | ||
At this point, you should be up and running; however, as an option, you may want to check the user BLOBs, you can refer to this [https://wiki.zimbra.com/wiki/ZCS88-zxbackup-Missing-BLOB guide].<br> | At this point, you should be up and running; however, as an option, you may want to check the user BLOBs, you can refer to this [https://wiki.zimbra.com/wiki/ZCS88-zxbackup-Missing-BLOB guide].<br> | ||
==<font size="6">Scenario II - | ==<font size="6">Scenario II - Mysql Server with corrupted mysql data that is beyond repair</font>== | ||
You have been running ZCS-88 using the Next Generation Suite, and you have just discovered a lot of errors in the /opt/zimbra/mysql_error.log file<br> | You have been running ZCS-88 using the Next Generation Suite, and you have just discovered a lot of errors in the /opt/zimbra/mysql_error.log file, and the mysql service is not running<br> | ||
You even tried to [https://wiki.zimbra.com/wiki/Mysql_Crash_Recovery export and reimport] the mysql data to no avail.<br> | You even tried to [https://wiki.zimbra.com/wiki/Mysql_Crash_Recovery export and reimport] the mysql data to no avail.<br> | ||
===Stop Zimbra Services=== | |||
=== | |||
<code><pre> | <code><pre> | ||
zmcontrol stop | zmcontrol stop | ||
</pre></code> | </pre></code> | ||
===Move aside | ===Move aside your old mysql directory=== | ||
<code><pre> | <code><pre> | ||
mv /opt/zimbra/ | mv /opt/zimbra/db/data/ /opt/zimbra/db/OLD_data/ | ||
mkdir /opt/zimbra/db/data/ | |||
</pre></code> | </pre></code> | ||
=== | ===Start up mysql in a clean state=== | ||
<code><pre> | <code><pre> | ||
/opt/zimbra/ | /opt/zimbra/zmmyinit | ||
</pre></code> | </pre></code> | ||
=== | ===Start up the rest of the ZCS services=== | ||
<code><pre> | <code><pre> | ||
zmcontrol start | |||
zmcontrol status | |||
</pre></code> | </pre></code> | ||
<strong><span style="color:red">REMEMBER!</span></strong><br> | |||
The mailboxd service must be running in order to use the <strong>zxsuite</strong> command.<br> | |||
=== | ===Run the external restore command to rebuild the mysql data=== | ||
<code><pre> | <code><pre> | ||
zxsuite backup doExternalRestore /opt/zimbra/backup/ | |||
</pre></code> | </pre></code> | ||
=== | ===Scenario II: Summary=== | ||
If mysql service is not running because of corruption, simply move aside the data directory, and run the zmmyinit script to bring up mysql service into a new clean state. Followed up by running the doExternalRestore option. | |||
==<font size="6">Scenario III - Multi Server: Restore LDAP Service</font>== | ==<font size="6">Scenario III - Multi Server: Restore LDAP Service</font>== | ||
<font size="5">Two Node Setup</font><br> | <font size="5">Two Node Setup</font><br> | ||
Server 1 | <strong>Server 1</strong> - LDAP + Proxy + MTA<br> | ||
Server 2 | <strong>Server 2</strong> - Mailboxd (User mail store)<br> | ||
If LDAP services are needed to be restored, you can do the following.<br> | If LDAP services are needed to be restored, you can do the following.<br> | ||
===Stop ldap and move the old database aside=== | ===Method A=== | ||
====Stop ldap and move the old database aside==== | |||
<code><pre> | <code><pre> | ||
ldap stop | ldap stop | ||
| Line 275: | Line 174: | ||
mkdir -p /opt/zimbra/data/ldap/mdb/db | mkdir -p /opt/zimbra/data/ldap/mdb/db | ||
</ | </pre></code> | ||
===Copy the backed up ldap data from Server 2=== | ====Copy the backed up ldap data from Server 2==== | ||
<code><pre> | <code><pre> | ||
mkdir /tmp/ldap/ | mkdir /tmp/ldap/ | ||
| Line 286: | Line 185: | ||
</pre></code> | </pre></code> | ||
===Extract the ldap.ldif file=== | ====Extract the ldap.ldif file AND ldap-config file==== | ||
<code><pre> | <code><pre> | ||
tar -zxvf ldap_23_08_17#04_01_00.tar.gz ldap.ldif | tar -zxvf ldap_23_08_17#04_01_00.tar.gz ldap.ldif | ||
tar -zxvf ldap_23_08_17#04_01_00.tar.gz ldap-config.ldif | |||
</pre></code> | </pre></code> | ||
===Restore the ldap-config=== | ====Restore the ldap-config==== | ||
<code><pre> | <code><pre> | ||
cd /tmp/ldap/ | cd /tmp/ldap/ | ||
mv /opt/zimbra/data/ldap/config/ /opt/zimbra/data/ldap/OLDconfig/ | mv /opt/zimbra/data/ldap/config/ /opt/zimbra/data/ldap/OLDconfig/ | ||
| Line 304: | Line 203: | ||
</pre></code> | </pre></code> | ||
===Use zmslapadd to add the ldap data=== | ====Use zmslapadd to add the ldap data==== | ||
<code><pre> | <code><pre> | ||
/opt/zimbra/libexec/zmslapadd /tmp/ldap/ldap.ldif | /opt/zimbra/libexec/zmslapadd /tmp/ldap/ldap.ldif | ||
| Line 311: | Line 210: | ||
</pre></code> | </pre></code> | ||
====If ldap is not starting, use the long command to start ldap | ===Method B=== | ||
====Use zmrestoreldap==== | |||
List all labels<br> | |||
<code><pre> | |||
zmrestoreldap -lbs | |||
**Should see something like...** | |||
full-20170824.050005.077 | |||
full-20170823.050004.754 | |||
full-20170822.050004.779 | |||
full-20170821.050004.736 | |||
full-20170820.050005.331 | |||
</pre></code> | |||
Then use zmrestoreldap with the label name to restore the ldap data.<br> | |||
<code><pre> | |||
zmrestoreldap -lb full-20170824.050005.077 | |||
</pre></code> | |||
===If ldap is not starting, use the long command to start ldap=== | |||
<code><pre> | <code><pre> | ||
sudo /opt/zimbra/libexec/zmslapd -l LOCAL0 -u zimbra -h 'ldap://ldap.example.com:389 ldapi:///' -F /opt/zimbra/data/ldap/config -d -4 | sudo /opt/zimbra/libexec/zmslapd -l LOCAL0 -u zimbra -h 'ldap://ldap.example.com:389 ldapi:///' -F /opt/zimbra/data/ldap/config -d -4 | ||
</pre></code> | </pre></code> | ||
The bottom portion of the output will usually provide a clue on why ldap is not starting.<br> | The bottom portion of the output will usually provide a clue on why ldap is not starting.<br> | ||
===Section III: Summary=== | |||
<strong>Method A</strong> is to illustrate you can copy the backed up ldap from another node with Next Generation Module installed, extract the ldap data, and manually add ldap back into the ldap service.<br> | |||
<strong>Method B</strong> is to illustrate the old standard way of restoring ldap service using the zmrestoreldap command.<br> | |||
<strong><span style="color:red">REMEMBER!</span></strong><br> | |||
Next Generation modules are only installed on Zimbra nodes running mailboxd service; and by convention, nodes running mailboxd will need to remove zmbackup from cron.<br> | |||
However, in a multi node setup, Next Generation modules would not be installed on a node running ldap service. Therefore, the zmbackup command can still be in place in cron to save the ldap config.<br> | |||
Latest revision as of 16:14, 5 September 2017
- This article is a Work in Progress, and may be unfinished or missing sections.
Scenario I - Single Server
You have been running ZCS-88 using the Next Generation Suite, and you have a dedicated disk for /opt/zimbra/, something like...
/dev/mapper/vg_zcs-lv_root
xxG xxxG xxG xx% /
tmpfs /dev/shm
/dev/sda1 /boot
/dev/sda2 /opt/zimbra/
But something has happened to the server node, effecting the operating system, which will require a reinstall of the operating system, and ZCS-88.
Reinstall the Operating System
When you reinstall the operating system, make sure the disk containing /opt/zimbra/ is not mounted yet.
Reinstall ZCS-88 with Next Generation Modules
When you are ready to reinstall ZCS-88, make sure the /opt/zimbra/ is now mounted.
If it's possible, rename your production /opt/zimbra/ directory.
mv /opt/zimbra/ mv /opt/zimbraSAVE/
Now, run the installer as the root user using the software option.
./install.sh -s
*OR*
./install.sh --software
During the ZCS-88 Software Reinstall
Observe at the end of the software install, you will see the following
Running Post Installation Configuration:
Software Installation complete!
Operations logged to /tmp/install.log.SviTipi8
Now observe, you have two zimbra directories
drwxr-xr-x 30 root root 4096 Aug 24 14:29 zimbra
drwxr-xr-x 30 root root 4096 Aug 17 13:20 zimbraSAVE
Since the software install had created the /opt/zimbra/ directory, lets move it out of the way
mv /opt/zimbra/ /opt/zimbraSOFTWARE_INSTALL/
Now, lets put our production directory back into place
mv /opt/zimbraSAVE/ /opt/zimbra/
After putting your production /opt/zimbra/ directory back into place, switch to the zimbra account, and start Zimbra
su - zimbra
zmcontrol start
Host test3.zextras.local
Starting ldap...Done.
Starting zmconfigd...Done.
Starting logger...Done.
Starting convertd...Done.
Starting mailbox...Done.
Starting memcached...Done.
Starting proxy...Done.
Starting amavis...Done.
Starting antispam...Done.
Starting antivirus...Done.
Starting opendkim...Done.
Starting snmp...Done.
Starting spell...Done.
Starting mta...Done.
Starting stats...Done.
Starting service webapp...Done.
Starting zimbra webapp...Done.
Starting zimbraAdmin webapp...Done.
Starting zimlet webapp...Done.
Starting imapd...Done.
Check the status
zmcontrol status
Host test3.zextras.local
amavis Running
antispam Running
antivirus Running
convertd Running
imapd Running
ldap Running
logger Running
mailbox Running
memcached Running
mta Running
opendkim Running
proxy Running
service webapp Running
snmp Running
spell Running
stats Running
zimbra webapp Running
zimbraAdmin webapp Running
zimlet webapp Running
zmconfigd Running
Scenario I - Summary
Operating system needed to be installed, moving aside production Zimbra, and doing a software install, and placing production Zimbra directory back into place.
At this point, you should be up and running; however, as an option, you may want to check the user BLOBs, you can refer to this guide.
Scenario II - Mysql Server with corrupted mysql data that is beyond repair
You have been running ZCS-88 using the Next Generation Suite, and you have just discovered a lot of errors in the /opt/zimbra/mysql_error.log file, and the mysql service is not running
You even tried to export and reimport the mysql data to no avail.
Stop Zimbra Services
zmcontrol stop
Move aside your old mysql directory
mv /opt/zimbra/db/data/ /opt/zimbra/db/OLD_data/
mkdir /opt/zimbra/db/data/
Start up mysql in a clean state
/opt/zimbra/zmmyinit
Start up the rest of the ZCS services
zmcontrol start
zmcontrol status
REMEMBER!
The mailboxd service must be running in order to use the zxsuite command.
Run the external restore command to rebuild the mysql data
zxsuite backup doExternalRestore /opt/zimbra/backup/
Scenario II: Summary
If mysql service is not running because of corruption, simply move aside the data directory, and run the zmmyinit script to bring up mysql service into a new clean state. Followed up by running the doExternalRestore option.
Scenario III - Multi Server: Restore LDAP Service
Two Node Setup
Server 1 - LDAP + Proxy + MTA
Server 2 - Mailboxd (User mail store)
If LDAP services are needed to be restored, you can do the following.
Method A
Stop ldap and move the old database aside
ldap stop
mv /opt/zimbra/data/ldap/mdb/db /opt/zimbra/data/ldap/mdb/OLD_db/
mkdir -p /opt/zimbra/data/ldap/mdb/db
Copy the backed up ldap data from Server 2
mkdir /tmp/ldap/
scp root@server2.example.com:/opt/zimbra/backup/ldap/ldap_23_08_17#04_01_00.tar.gz /tmp/ldap/
chown -R zimbra:zimbra /tmp/ldap/
Extract the ldap.ldif file AND ldap-config file
tar -zxvf ldap_23_08_17#04_01_00.tar.gz ldap.ldif
tar -zxvf ldap_23_08_17#04_01_00.tar.gz ldap-config.ldif
Restore the ldap-config
cd /tmp/ldap/
mv /opt/zimbra/data/ldap/config/ /opt/zimbra/data/ldap/OLDconfig/
mkdir /opt/zimbra/data/ldap/config/
/opt/zimbra/libexec/zmslapadd -c /tmp/ldap/ldap-config.ldif
Use zmslapadd to add the ldap data
/opt/zimbra/libexec/zmslapadd /tmp/ldap/ldap.ldif
ldap start
Method B
Use zmrestoreldap
List all labels
zmrestoreldap -lbs
**Should see something like...**
full-20170824.050005.077
full-20170823.050004.754
full-20170822.050004.779
full-20170821.050004.736
full-20170820.050005.331
Then use zmrestoreldap with the label name to restore the ldap data.
zmrestoreldap -lb full-20170824.050005.077
If ldap is not starting, use the long command to start ldap
sudo /opt/zimbra/libexec/zmslapd -l LOCAL0 -u zimbra -h 'ldap://ldap.example.com:389 ldapi:///' -F /opt/zimbra/data/ldap/config -d -4
The bottom portion of the output will usually provide a clue on why ldap is not starting.
Section III: Summary
Method A is to illustrate you can copy the backed up ldap from another node with Next Generation Module installed, extract the ldap data, and manually add ldap back into the ldap service.
Method B is to illustrate the old standard way of restoring ldap service using the zmrestoreldap command.
REMEMBER!
Next Generation modules are only installed on Zimbra nodes running mailboxd service; and by convention, nodes running mailboxd will need to remove zmbackup from cron.
However, in a multi node setup, Next Generation modules would not be installed on a node running ldap service. Therefore, the zmbackup command can still be in place in cron to save the ldap config.
