My Wonderful adventure in backing stuff Up
This is a work in progress, it may and probably does contain errors, misspellings, bad code, and bad grammar. Be warned.
Ok, so, I originally had a "Cold Backup Server" that I setup using some info gleaned from the forums. The solution that was provided didn't quite work for me. For one, it didn't provide a fast enough fail over AND due to misspellings and other steps left out of the script, it didn't work either.
I decided to try something that was probably not the ideal way, but, it was effective. I used a Dell workstation with a big HDD and a decent amount of ram, installed ubuntu 6.02lts (same as our live server) and installed Zimbra. The following is from my own documentation of this process.
Install Ubuntu 6.0 LTS Server
I'm not going into detail on how to install ubuntu, but, rest assured it is a required step. One thing to note however, the hostname should be identical to your LIVE server.
After you have ubuntu installed, you may proceed to the next step.
Add the “universe” depository to the apt-get updater
- Open /etc/apt/sources.list – “pico /etc/apt/sources.list”
- Remove the # in front of :
deb http://us.archive.ubuntu.com/ubuntu/ dapper universe deb-src http://us.archive.ubuntu.com/ubuntu/ dapper universe
- Ctrl-X and press Y to save.
Run the following commands
apt-get install dnsmasq pico /etc/dnsmasq.conf
- Add the following lines:
address=/your.maildomain.com/X.X.X.X mx-host=maildomain.com,your.maildomain.com,10 mx-host=your.maildomain.com,your.maildomain.com,15 mx-target=your.maildomain.com
- Press Ctrl+X and Y to save
Install Zimbra Prerequisites
apt-get install libidn11 fetchmail libxml2 libstdc++5 openssl libltdl3 perl
- Set the DNS Resolver to point to itself.
mv /etc/resolv.conf resolv.conf.BU pico /etc/resolv.conf
- Add the following:
Nameservers X.X.X.X #where x is your local ip
- Press “ctrl+X” then Y to save.
Install the zimbra version that matches what is currently running on the LIVE server.
- Copy the .tgz file downloaded from Zimbra.com .
- Copy the file to a directory on the server. I’ve been using /ZIMBRAINSTALL/
- Decompress the file: “gunzip FILENAME”
- Extract the resulting tar file: “tar –xvf FILENAME”
- Go into the directory that was created. “cd FILENAME”
Ok, I stop at this point because this is where my original and new methodologies begin to take different roads.
To make a long story short, I was rsyncing the /opt/zimbra directory from my live server to the local system. It did this once every hour. I didn't shutdown anything, I just had it syncing. The standby server was not active (meaning, zimbra was not running). However, thanks to a modified version of the script provided by fisch09 in the forums, I could quickly switch it from cold standby to Hot mode with a single command.
This past Christmas eve, our mail server crashed. I was 1500 miles away at the time and was able to remote into the backup machine, run the script, reboot and was back in action. So, it DOES work, but I've never been comfortable it. This fear was realized when we had a LDAP corruption and that corruption was synced with the backup server. This put an end to THAT particular setup.
This method also meant that after my Zimbra install, I would essentially move it aside and sync over the existing install. Since that is no longer the case, the installing of zimbra now takes on a few more steps. For this, the Disaster recovery Wiki entry was very helpful. I will be detailing that below and adding in a few things that were not quite covered. So, I revisited fisch09's method. Basically, taking backups from the live server, syncing them over and restoring them on the backup system. So, now to continue our setup.
After more review, research, scripting, testing, more testing, and then more testing after that; I've come up with a Hybrid approach that takes the best of both worlds.
Basically, we ARE going to sync the /opt/zimbra directory (and any directories that you are using that exist outside that directory) to a cold standby server. This initial sync will give you a mirror of the live server as it sits, it however still has some side effects that we will address. This also keeps your incremental/full backups synced as well. So, in the event that the sync is BAD, we can still fall back on the zimbra made backups. I've divided this into two methods.
Installing Zimbra for Cold-Standby Operation (Method 1)
Use this method if you are NOT going to rsync your existing install to the cold standby server.
The process is pretty much identical to the method described for Disaster Recovery of the Network Edition. The following is essentially a cut and paste of the internal documentation I made for the company I work for.
- Go to the extracted Zimbra install Directory and Run the installer
- Pay attention to any errors.
- It will Confirm what packages you are installing.
- Use whatever packages you use on the Live Server
- After this, it will test to make sure the DNS is working properly
If it isn’t you need to trouble shoot the dnsmasq settings and resolv.conf
- Assuming all goes ok, it will stop at the config screen where you need to set Additional parameters. See next step.
IMPORTANT – At this step, you need to set the system to match the EXISTING server your backing up. You need to set:
- LDAP Passwords: There are 5 different ones. All of them are identical (typically).To find the existing passwords:
- Log into the existing email server.
su zimbra zmlocalconfig -s | grep -i pass
- Write down the password used for the LDAP_ entries.
- Admin Password
- SPAM and HAM Email Addresses
- Admin email address
- wiki document email Address.
11) Enable default backup schedule: no r) Start servers after configuration no
- Apply the config and continue install.
After it’s complete, we need to sync the backup directory and zimlets.
Sync the Backup Directory
- Establish a directory to place the backup files. I’m using an identical structure as the source server.
- Type: mkdir /DRIVE2/backup
- Type: chmod 777 /DRIVE2/backup
- To sync the files, use the following command.
rsync -e 'ssh -ax -c blowfish' -av 220.127.116.11:/DRIVE2/backup/ /DRIVE2/backup/
- Enter the root password for the source server when prompted. This will take awhile if it’s the first time.
Sync the Zimlets
- We need to sync the zimlets on the source server with the new target server. Use the following command:
rsync -e 'ssh -ax -c blowfish' -av 18.104.22.168:/opt/zimbra/mailboxd/webapps/service/zimlet /opt/zimbra/mailboxd/webapps/service/
Since we haven’t yet, Lets reboot. It may not be absolutely needed, but It seems to solve issues I sometimes get after a clean install..
Now we are going to restore our LDAP to the latest incremental backup. Type:
- in the list, will be a bunch of directories with a INCR prefix. Look for the newest one. Copy the whole directory name, this is our incremental backup label.
For this example, we will use: incr-20090203.200008.925
zmrestoreldap -lb incr-20090203.200008.925
- This should show an output of the accounts being restored. Look for any errors. It will be done quickly.
- LDAP Server should be started at this point.
mysql.server start zmmailboxdctl start
- Now to restore the main message store.
zmrestore -restoreToIncrLabel incr-20090204.170011.969 -a "all" -t /Path/To/Backup
- It won’t give you any feedback, so you need to open a second ssh session:
- Login and Type:
tail –f /opt/zimbra/log/mailbox.log
- You can monitor its progress here. This will take a while. About 1 – 2 hours.
Once the restore is complete:
zmcontrol stop zmcontrol start
- Use the IP address of the server to access it and login. Verify the data.
Hybrid Backup - Cold Standby Server (Method 2)
This method involves:
1. Installing a dummy copy of Zimbra (still must be the identical version your using on your live server).
2. Syncing your existing live Zimbra installation to the Cold Backup server. This includes your backups.
3. Running the mysql database recovery script/procedure to keep the database from getting out of sequence.
Using my 3rBackup and a new DumpandRestore script, This whole thing can be automated when put in a cron job. Currently, I have about 60 users with ~38GB of data in total. When ran hourly, the whole process from Sync to Database Repair takes about 11 mins. This doesn't stress the live server at all so I have it running on an hourly basis. I also do hourly incremental backups so I have something to fall back on in case the synced copy is screwed.
In addition, I use the change.zimbra.status.sh script to switch the system over to live mode rapidly in the event of a mail server crash. Below are the scripts used and how.
The files involved:
3rbackup.sh -- The syncing script. This file is used by pairing it with a config file that defines the connection to the live server.
example: ./3rbackup.sh ../MYSERVER/MYSERVERVAR.TXT
MYSERVERVAR.TXT -- This is the variable file. It can be named whatever you want.
dumpandrestore.sh -- This script dumps out the entire database to files, drops them, deletes the ibdata and ib_log files, then reimports the data. This keeps you from getting the out of sequence errors in sql after doing a live sync. Is it smart? Well, it works for me.
mirror.sh -- Wraps all the scripts up into a single command that can be called in a cronjob.
sshsetup.sh -- This is not used during the backup, but can be used to create a ssh key to be used by rsync. Without setting up a key-pair, you CANNOT automate this in cron.