View source for ZCS88-zxbackup-Disaster-Recovery

{{ZCS 8.8}}
{{WIP}}

==<font size="6">Scenario I - Single Server</font>==

You have been running ZCS-88 using the Next Generation Suite, and you have a dedicated disk for /opt/zimbra/, something like...<br>
<code><pre>
/dev/mapper/vg_zcs-lv_root
xxG  xxxG   xxG  xx% /
tmpfs                /dev/shm
/dev/sda1            /boot
/dev/sda2            /opt/zimbra/
</pre></code>

But something has happened to the server node, effecting the operating system, which will require a reinstall of the operating system, and ZCS-88. 

===<font size="5">Reinstall the Operating System</font>===
When you reinstall the operating system, make sure the disk containing /opt/zimbra/ is not mounted yet.<br>

===<font size="5">Reinstall ZCS-88 with Next Generation Modules</font>===
When you are ready to reinstall ZCS-88, make sure the /opt/zimbra/ is now mounted.<br>

If it's possible, rename your production /opt/zimbra/ directory.<br>
<code><pre>
mv /opt/zimbra/    mv /opt/zimbraSAVE/
</pre></code>


Now, run the installer as the root user using the software option.<br>
<code><pre>
./install.sh -s

*OR*

./install.sh --software
</pre></code>

====<font size="5">During the ZCS-88 Software Reinstall</font>====
Observe at the end of the software install, you will see the following<br>
<code><pre>
Running Post Installation Configuration:

Software Installation complete!

Operations logged to /tmp/install.log.SviTipi8
</pre></code>

Now observe, you have two zimbra directories<br>
<code><pre>
drwxr-xr-x  30 root root 4096 Aug 24 14:29 zimbra
drwxr-xr-x  30 root root 4096 Aug 17 13:20 zimbraSAVE
</pre></code>

Since the software install had created the /opt/zimbra/ directory, lets move it out of the way<br>
<code><pre>
mv /opt/zimbra/   /opt/zimbraSOFTWARE_INSTALL/
</pre></code>

Now, lets put our production directory back into place<br>
<code><pre>
mv /opt/zimbraSAVE/   /opt/zimbra/
</pre></code>

After putting your production /opt/zimbra/ directory back into place, switch to the zimbra account, and start Zimbra<br>
<code><pre>
su - zimbra
zmcontrol start
Host test3.zextras.local
	Starting ldap...Done.
	Starting zmconfigd...Done.
	Starting logger...Done.
	Starting convertd...Done.
	Starting mailbox...Done.
	Starting memcached...Done.
	Starting proxy...Done.
	Starting amavis...Done.
	Starting antispam...Done.
	Starting antivirus...Done.
	Starting opendkim...Done.
	Starting snmp...Done.
	Starting spell...Done.
	Starting mta...Done.
	Starting stats...Done.
	Starting service webapp...Done.
	Starting zimbra webapp...Done.
	Starting zimbraAdmin webapp...Done.
	Starting zimlet webapp...Done.
	Starting imapd...Done.
</pre></code>

Check the status<br>
<code><pre>
zmcontrol status
Host test3.zextras.local
	amavis                  Running
	antispam                Running
	antivirus               Running
	convertd                Running
	imapd                   Running
	ldap                    Running
	logger                  Running
	mailbox                 Running
	memcached               Running
	mta                     Running
	opendkim                Running
	proxy                   Running
	service webapp          Running
	snmp                    Running
	spell                   Running
	stats                   Running
	zimbra webapp           Running
	zimbraAdmin webapp      Running
	zimlet webapp           Running
	zmconfigd               Running
</pre></code>
===Scenario I - Summary===
Operating system needed to be installed, moving aside production Zimbra, and doing a software install, and placing production Zimbra directory back into place.<br> 

At this point, you should be up and running; however, as an option, you may want to check the user BLOBs, you can refer to this [https://wiki.zimbra.com/wiki/ZCS88-zxbackup-Missing-BLOB guide].<br>

==<font size="6">Scenario II - Mysql Server with corrupted mysql data that is beyond repair</font>==
You have been running ZCS-88 using the Next Generation Suite, and you have just discovered a lot of errors in the /opt/zimbra/mysql_error.log file, and the mysql service is not running<br>

You even tried to [https://wiki.zimbra.com/wiki/Mysql_Crash_Recovery export and reimport] the mysql data to no avail.<br>

===Stop Zimbra Services===
<code><pre>
zmcontrol stop
</pre></code>

===Move aside your old mysql directory===
<code><pre>
mv /opt/zimbra/db/data/   /opt/zimbra/db/OLD_data/

mkdir /opt/zimbra/db/data/
</pre></code>

===Start up mysql in a clean state===
<code><pre>
/opt/zimbra/zmmyinit
</pre></code>

===Start up the rest of the ZCS services===
<code><pre>
zmcontrol start

zmcontrol status    
</pre></code>
<strong><span style="color:red">REMEMBER!</span></strong><br>
The mailboxd service must be running in order to use the <strong>zxsuite</strong> command.<br>

===Run the external restore command to rebuild the mysql data===
<code><pre>
zxsuite backup doExternalRestore /opt/zimbra/backup/
</pre></code>

===Scenario II: Summary===
If mysql service is not running because of corruption, simply move aside the data directory, and run the zmmyinit script to bring up mysql service into a new clean state. Followed up by running the doExternalRestore option.

==<font size="6">Scenario III - Multi Server: Restore LDAP Service</font>==
<font size="5">Two Node Setup</font><br>
<strong>Server 1</strong> - LDAP + Proxy + MTA<br>
<strong>Server 2</strong> - Mailboxd  (User mail store)<br>

If LDAP services are needed to be restored, you can do the following.<br>

===Method A===

====Stop ldap and move the old database aside====
<code><pre>
ldap stop

mv /opt/zimbra/data/ldap/mdb/db  /opt/zimbra/data/ldap/mdb/OLD_db/

mkdir -p  /opt/zimbra/data/ldap/mdb/db
</pre></code>

====Copy the backed up ldap data from Server 2====
<code><pre>
mkdir /tmp/ldap/

scp root@server2.example.com:/opt/zimbra/backup/ldap/ldap_23_08_17#04_01_00.tar.gz  /tmp/ldap/

chown -R zimbra:zimbra /tmp/ldap/
</pre></code>

====Extract the ldap.ldif file AND ldap-config file====
<code><pre>
tar -zxvf ldap_23_08_17#04_01_00.tar.gz ldap.ldif

tar -zxvf ldap_23_08_17#04_01_00.tar.gz ldap-config.ldif
</pre></code>

====Restore the ldap-config====
<code><pre>
cd /tmp/ldap/

mv /opt/zimbra/data/ldap/config/  /opt/zimbra/data/ldap/OLDconfig/

mkdir /opt/zimbra/data/ldap/config/

/opt/zimbra/libexec/zmslapadd -c /tmp/ldap/ldap-config.ldif
</pre></code>

====Use zmslapadd to add the ldap data====
<code><pre>
/opt/zimbra/libexec/zmslapadd /tmp/ldap/ldap.ldif

ldap start
</pre></code>

===Method B===

====Use zmrestoreldap====
List all labels<br>
<code><pre>
zmrestoreldap -lbs

**Should see something like...**
full-20170824.050005.077
full-20170823.050004.754
full-20170822.050004.779
full-20170821.050004.736
full-20170820.050005.331
</pre></code>
Then use zmrestoreldap with the label name to restore the ldap data.<br>
<code><pre>
zmrestoreldap -lb full-20170824.050005.077
</pre></code>

===If ldap is not starting, use the long command to start ldap===
<code><pre>
sudo /opt/zimbra/libexec/zmslapd -l LOCAL0 -u zimbra -h 'ldap://ldap.example.com:389 ldapi:///' -F /opt/zimbra/data/ldap/config -d -4
</pre></code>
The bottom portion of the output will usually provide a clue on why ldap is not starting.<br>

===Section III: Summary===
<strong>Method A</strong> is to illustrate you can copy the backed up ldap from another node with Next Generation Module installed, extract the ldap data, and manually add ldap back into the ldap service.<br>

<strong>Method B</strong> is to illustrate the old standard way of restoring ldap service using the zmrestoreldap command.<br>

<strong><span style="color:red">REMEMBER!</span></strong><br>
Next Generation modules are only installed on Zimbra nodes running mailboxd service; and by convention, nodes running mailboxd will need to remove zmbackup from cron.<br>
However, in a multi node setup, Next Generation modules would not be installed on a node running ldap service. Therefore, the zmbackup command can still be in place in cron to save the ldap config.<br>