Scripts to sync to a remote Zimbra backup machine

From Zimbra :: Wiki

Jump to: navigation, search
   Article-alert.png  - This article is a community contribution and may include unsupported customizations for network edition customers.



Contents

Synchronising to a Remote Backup Machine

Scenario

A single remote machine is running the Zimbra Open Source Edition (OSE), providing services for a whole load of users (user1, user2, userx) in various virtual domains (domain1, domain2 and domainy).

The zimbra server will have a public IP address and is in operation 24 hours-a-day 7 days-a-week.

The source machine or zimbra server, in this fictitious example, will be known as 'zimbra.server.com' and the backup machine as 'backup.server.com'.

Aims

  • To provide a cost-effective backup solution, ensuring the main server (zimbra.server.com) is on-line at all times.
  • The backup solution must stay up-to-date, so in case of an emergency, users do not lose large chunks of their emails.
  • The solution must be customisable, automated and require a minimal amount of time for administrators to implement.

System Requirements

  • Imapsync
  • Zimbra OSE
  • Reasonable amount of disk space to ensure room for emails and their attachments
  • Familiarity with crontab

Time To Implement

After Zimbra and imapsync installations - just the time needed to input your user names and passwords and setup a cron job.

So, depending on your experience of Linux and Zimbra installations and the size of your user base, anything between say 20 minutes to an hour for the configuration and then probably over night depending on how many users and how many emails you are backing up.

Advice

Read ahead and ensure you follow any instructions as stated, complex machines require delicate coding and the authors/devlopers try their best to publish instructions to the letter.

If you do find a genuine error please post a message highlighting the problematic area in the Zimbra Forums.

Don't forget to thank the guys for their hard work too - they sure do appreciate a pat on the back once in a while.

Top

Laying the foundations

Plan of action

  • Install Zimbra OSE on a local machine, in this case it will be known as 'backup.server.com' which does not have to be on the same Linux platform or even the same Zimbra version ... but of course, for the sake of completion it may as well be a mirrored setup. The 'backup.server.com' can be implemented anywhere .. a local LAN, a second remote site or right next to our source server 'zimbra.server.com'. In this scenario the zimbra.server.com is located in the docklands, London and our backup.server.com is located out in the country at an office.

Step 1

  • Create a directory named 'Zimbra_Scripts' and change into that directory.
mkdir Zimbra_Scripts; cd Zimbra_Scripts

Top

Step 2

  • Create and edit a file called 'imap_users' and enter the domain names, the user names and passwords of all your domains and users on zimbra.server.com, using the syntax as follows (obviously substituting domains, users and passwords appropriately):
nano imap_users
domains=(
	domain1 \
	domain2 \
	domainy)

users=(
	user1@domain1 \
	user2@domain1 \
	userx@domain1 \
	user1@domain2 \
	user2@domain2 \
	userx@domain2 \
	user1@domainy \
	user2@domainy \
	userx@domainy)

passwords=(
	u1d1passwd \
	u2d1passwd \
	uxd1passwd \
	u1d2passwd \
	u2d2passwd \
	uxd2passwd \
	u1dypasswd \
	u2dypasswd \
	uxdypasswd)

N.B. - if you don't know all the user passwords see the extra section below about setting the imapsync authuser !!

Top

Step 3

  • Create another file in the 'Zimbra_Scripts' folder - called 'zimbra_migration' and enter the main body of our code here.

This code recreates the domains currently residing on zimbra.server.com onto the backup.server.com, ready to receive the emails in our next step.

For admins with a lot of domains, recreating them one-by-one can be a time consuming process.

N.B. - The first line does start with a period . and ensures the file 'imap_users' is included into 'zimbra_migration' script.

This was done in regards to object oriented programming, for portability and security.

nano zimbra_migration
. imap_users

for ((i = 0 ; i < ${#domains[@]} ; i++ ))
do
echo $i: /opt/zimbra/bin/zmprov cd ${domains[$i]}
done

for ((i = 0 ; i < ${#users[@]} ; i++ ))
do
echo $i: /opt/zimbra/bin/zmprov ca ${users[$i]}
/opt/zimbra/bin/zmprov ca ${users[$i]} ${passwords[$i]}
done
Breakdown

Uses two for loops, iterating through the array of domains firstly.

A little debugging code, helps us by showing the id number of the domain in the loop and then creates the domain by calling
zmprov cd

The domain value is taken from the 'domains' array located in the 'imap_users' file.

N.B. - You may re-run this script at any time, if the domain already exists - it will NOT be overwritten but a message informing you that the domain aready exists will appear here.

In the second loop, again a little debug code lets us know where we are in the list of users and which user is being created.

zmprov ca
creates the user, taking its arguments from the 'users' array in the 'imap_users' file, and the password from the same array index position of the 'passwords' array, again located in the 'imap_users' file.

N.B. - You may re-run this script at any time, if the user already exists - it will NOT be overwritten but a message informing you that the user aready exists will appear here

Top

Step 4

There are two options here I suggest the first as a semi-automated way to setup a new server with accounts and the second when either migrating/backing-up between existing zimbra install on diverse platforms.

Using the script

Set the permissions on all of the files so that we can execute the files correctly:

chmod +x *

So you are now 'good-to-go' - you can invoke this script by running, from within the 'Zimbra_Scripts' directory, the command:

./zimbra_migration

Top


Using LDAP

Syncing between platforms favours exporting your ldap from the main server, achieved via:

(this example works from x86_64 => x86 - taken from here)

'The following commands should execute on the main/source server'

Logging in as zimbra user

su - zimbra

Export LDAP data to an ldif file. For 5.0.x and previous:

./openldap/sbin/slapcat -f /opt/zimbra/conf/slapd.conf -b "" -l /tmp/zimbra-ldap.ldif

For 6.0.x and later:

./openldap/sbin/slapcat -F /opt/zimbra/data/ldap/config -b "" -l /tmp/zimbra-ldap.ldif

'The following command should execute on the destination/backup server'

Logging in as zimbra user

su - zimbra

Shutting zimbra server down

zmcontrol stop

Manually check for any imbra services - (kill if neccessary)

ps auxx | grep slapd

The following removes any current data - you could just move it until back in the 'safe zone'. Do not delete recursively here.

For 5.0.x and previous:

rm -f openldap-data/*

Import user accounts and passwords from the LDAP file zimbra-ldap.ldif

./openldap/sbin/slapadd -f /opt/zimbra/conf/slapd.conf -b "" -q -l /tmp/zimbra-ldap.ldif

For 6.0.x and later:

rm -f data/ldap/hdb/db/__db.* data/ldap/hdb/db/*.bdb data/ldap/hdb/db/alock data/ldap/hdb/logs/*

Import user accounts and passwords from the LDAP file zimbra-ldap.ldif

./openldap/sbin/slapadd -F /opt/zimbra/data/ldap/config -b "" -q -l /tmp/zimbra-ldap.ldif

That's that you may now proceed to run your imapsync_crontab file

(It is suggested and personally recommended to do things this way - however - the choice is your own and the instructions on the rest of this page also successfully work)

Top

IMAPSYNC & Cronjob

IMAPSYNC

So now to automate the sync on a regular basis, between source and destination servers. The nwer version of imapsync removed various options as stated in the original Wiki page.

Option A -- For imapsync versions around 1.172

Option B -- For imapsync version above and beyond no doubt.

For both options:

Create a file for our cron job called 'imapsync_crontab' - again within the 'Zimbra_Scripts' directory.

nano imapsync_crontab

Option A

Edit it to include the following:

. imap_users #if in the same folder as imap_users else full-path to imap_users

src_srv = zimbra.server.com
dest_srv = backup.server.com

for ((i = 0 ; i < ${#users[@]} ; i++ ))
do
/usr/bin/imapsync --noauthmd5 --syncinternaldates --buffersize 65535000 --subscribe \
--host1 $src_srv --ssl1 --user1 ${users[$i]} --password1 ${passwords[$i]} --nofastio1 --authmech1 LOGIN \
--host2 $dest_srv --ssl2 --user2 ${users[$i]} --password2 ${passwords[$i]} --nofastio2 --authmech2 LOGIN
done
Breakdown

We set the source server address as 'src_srv' and the destination server address as 'dest_srv' - for use in the call to imapsync.

A single for loop iterates through the array containing our list of 'users' and their passwords.

The following imapsync options are used:

  • noauthmd5 - no use of md5
  • syncinternaldates - to keep the correct dates and times emails were sent/received
  • buffersize 65535000 - ensures if a large attachment is encountered there is room in the buffer
  • subscribe - automatically creates any additional user folders on the destination machine (keeping the account directory structure)
  • host1 - this is our source server 'zimbra.server.com'
  • host2 - this is our destination server 'backup.server.com'
  • ssl1/2 - uses a secure connection on both machines
  • nofastio1/2 - helps to reduce errors when syncing
  • authmech1/2 - the method used to authenticate at both ends (LOGIN is commonly successful)
  • password1/2 - should probably be the same on both machines

Top

Option B

Edit it to include the following:

. imap_users #if in the same folder as imap_users else full-path to imap_users (e.g = " . /path/to/imap_users")

src_srv = zimbra.server.com
dest_srv = backup.server.com

for ((i = 0 ; i < ${#users[@]} ; i++ ))
do
/usr/bin/imapsync --noauthmd5 --syncinternaldates --subscribe \
--host1 $src_srv --ssl1 --user1 ${users[$i]} --password1 ${passwords[$i]}  \
--host2 $dest_srv --ssl2 --user2 ${users[$i]} --password2 ${passwords[$i]} 
done
Breakdown

We set the source server address as 'src_srv' and the destination server address as 'dest_srv' - for use in the call to imapsync.

A single for loop iterates through the array containing our list of 'users' and their passwords.

The following imapsync options are used:

  • noauthmd5 - no use of md5
  • syncinternaldates - to keep the correct dates and times emails were sent/received
  • subscribe - automatically creates any additional user folders on the destination machine (keeping the account directory structure)
  • host1 - this is our source server 'zimbra.server.com'
  • host2 - this is our destination server 'backup.server.com'
  • ssl1/2 - uses a secure connection on both machines
  • password1/2 - should probably be the same on both machines

Option C

Edit to include the following if each user password is not known **this is the recommended option** (much easier for us admins:):

. imap_users #if in the same folder as imap_users else full-path to imap_users (e.g = " . /path/to/imap_users")

src_srv = zimbra.server.com
dest_srv = backup.server.com

for ((i = 0 ; i < ${#users[@]} ; i++ ))
do
/usr/bin/imapsync --noauthmd5 --syncinternaldates --subscribe \
--host1 $src_srv --ssl1 --user1 ${users[$i]} --authuser1 adminusername --password1 adminpassword  \
--host2 $dest_srv --ssl2 --user2 ${users[$i]} --authuser2 adminusername --password2 adminpassword 
done
Breakdown

The same as option B but eliminates the need to know each users password - this is achieved by passing the authuser credentials.

Top

Crontab

Create a cron job by using the following command (firstly setting the editor to use nano):

export EDITOR=$(type -P nano);crontab -e

Now, set an hourly synchronisation to the destination server. This assumes the 'Zimbra_Scripts' directory is located in '/opt/Zimbra_Scripts'.

The output is emailed to admin@zimbra.server.com.

MAILTO=admin@zimbra.server.com
@hourly /opt/Zimbra_Scripts/imapsync_crontab

This ensures that users emails are backed up on an hourly basis.

N.B. - if you have a large number of users, it would be advisable to run the job at greater intervals. This will alleviate system resources and ensure no overlapping.

Top

Created by Costa-101

External Links

Verified Against: unknown Date Created: 7/16/2007
Article ID: http://wiki.zimbra.com/index.php?title=Scripts_to_sync_to_a_remote_Zimbra_backup_machine Date Modified: 07/3/2012
Personal tools