Scripts to sync to a remote Zimbra backup machine: Difference between revisions
(7 intermediate revisions by 3 users not shown) | |||
Line 165: | Line 165: | ||
<code><pre>su - zimbra</pre></code> | <code><pre>su - zimbra</pre></code> | ||
Export LDAP data to an ldif file | Export LDAP data to an ldif file. | ||
<code><pre>openldap/sbin/slapcat -f /opt/zimbra/conf/slapd.conf -l /tmp/zimbra-ldap.ldif</pre></code> | For 5.0.x and previous: | ||
<code><pre>./openldap/sbin/slapcat -f /opt/zimbra/conf/slapd.conf -b "" -l /tmp/zimbra-ldap.ldif</pre></code> | |||
For 6.0.x and later: | |||
<code><pre>./openldap/sbin/slapcat -F /opt/zimbra/data/ldap/config -b "" -l /tmp/zimbra-ldap.ldif</pre></code> | |||
''''The following command should execute on the destination/backup server'''' | ''''The following command should execute on the destination/backup server'''' | ||
Line 179: | Line 183: | ||
<code><pre>ps auxx | grep slapd</pre></code> | <code><pre>ps auxx | grep slapd</pre></code> | ||
The following removes any current data - you could just move it until back in the 'safe zone'. | The following removes any current data - you could just move it until back in the 'safe zone'. Do not delete recursively here. | ||
For 5.0.x and previous: | |||
<code><pre>rm -f openldap-data/*</pre></code> | <code><pre>rm -f openldap-data/*</pre></code> | ||
Import user accounts and passwords from the LDAP file zimbra-ldap.ldif | Import user accounts and passwords from the LDAP file zimbra-ldap.ldif | ||
<code><pre>openldap/sbin/slapadd -f /opt/zimbra/conf/slapd.conf -l /tmp/zimbra-ldap.ldif</pre></code> | <code><pre>./openldap/sbin/slapadd -f /opt/zimbra/conf/slapd.conf -b "" -q -l /tmp/zimbra-ldap.ldif</pre></code> | ||
For 6.0.x and later: | |||
<code><pre>rm -f data/ldap/hdb/db/__db.* data/ldap/hdb/db/*.bdb data/ldap/hdb/db/alock data/ldap/hdb/logs/*</pre></code> | |||
Import user accounts and passwords from the LDAP file zimbra-ldap.ldif | |||
<code><pre>./openldap/sbin/slapadd -F /opt/zimbra/data/ldap/config -b "" -q -l /tmp/zimbra-ldap.ldif</pre></code> | |||
That's that you may now proceed to run your imapsync_crontab file | That's that you may now proceed to run your imapsync_crontab file | ||
Line 316: | Line 328: | ||
[[#Synchronising to a Remote Backup Machine|Top]] | [[#Synchronising to a Remote Backup Machine|Top]] | ||
Created by [ | Created by [http://wiki.zimbra.com/wiki/User:Costa-101 Costa-101] | ||
=External Links= | =External Links= |
Latest revision as of 14:24, 3 July 2012
Synchronising to a Remote Backup Machine
Scenario
A single remote machine is running the Zimbra Open Source Edition (OSE), providing services for a whole load of users (user1, user2, userx) in various virtual domains (domain1, domain2 and domainy).
The zimbra server will have a public IP address and is in operation 24 hours-a-day 7 days-a-week.
The source machine or zimbra server, in this fictitious example, will be known as 'zimbra.server.com' and the backup machine as 'backup.server.com'.
Aims
- To provide a cost-effective backup solution, ensuring the main server (zimbra.server.com) is on-line at all times.
- The backup solution must stay up-to-date, so in case of an emergency, users do not lose large chunks of their emails.
- The solution must be customisable, automated and require a minimal amount of time for administrators to implement.
System Requirements
- Imapsync
- Zimbra OSE
- Reasonable amount of disk space to ensure room for emails and their attachments
- Familiarity with crontab
Time To Implement
After Zimbra and imapsync installations - just the time needed to input your user names and passwords and setup a cron job.
So, depending on your experience of Linux and Zimbra installations and the size of your user base, anything between say 20 minutes to an hour for the configuration and then probably over night depending on how many users and how many emails you are backing up.
Advice
Read ahead and ensure you follow any instructions as stated, complex machines require delicate coding and the authors/devlopers try their best to publish instructions to the letter.
If you do find a genuine error please post a message highlighting the problematic area in the Zimbra Forums.
Don't forget to thank the guys for their hard work too - they sure do appreciate a pat on the back once in a while.
Laying the foundations
Plan of action
- Install Zimbra OSE on a local machine, in this case it will be known as 'backup.server.com' which does not have to be on the same Linux platform or even the same Zimbra version ... but of course, for the sake of completion it may as well be a mirrored setup. The 'backup.server.com' can be implemented anywhere .. a local LAN, a second remote site or right next to our source server 'zimbra.server.com'. In this scenario the zimbra.server.com is located in the docklands, London and our backup.server.com is located out in the country at an office.
- Install imapsync.
Step 1
- Create a directory named 'Zimbra_Scripts' and change into that directory.
mkdir Zimbra_Scripts; cd Zimbra_Scripts
Step 2
- Create and edit a file called 'imap_users' and enter the domain names, the user names and passwords of all your domains and users on zimbra.server.com, using the syntax as follows (obviously substituting domains, users and passwords appropriately):
nano imap_users
domains=(
domain1 \
domain2 \
domainy)
users=(
user1@domain1 \
user2@domain1 \
userx@domain1 \
user1@domain2 \
user2@domain2 \
userx@domain2 \
user1@domainy \
user2@domainy \
userx@domainy)
passwords=(
u1d1passwd \
u2d1passwd \
uxd1passwd \
u1d2passwd \
u2d2passwd \
uxd2passwd \
u1dypasswd \
u2dypasswd \
uxdypasswd)
N.B. - if you don't know all the user passwords see the extra section below about setting the imapsync authuser !!
Step 3
- Create another file in the 'Zimbra_Scripts' folder - called 'zimbra_migration' and enter the main body of our code here.
This code recreates the domains currently residing on zimbra.server.com onto the backup.server.com, ready to receive the emails in our next step.
For admins with a lot of domains, recreating them one-by-one can be a time consuming process.
N.B. - The first line does start with a period . and ensures the file 'imap_users' is included into 'zimbra_migration' script.
This was done in regards to object oriented programming, for portability and security.
nano zimbra_migration
. imap_users
for ((i = 0 ; i < ${#domains[@]} ; i++ ))
do
echo $i: /opt/zimbra/bin/zmprov cd ${domains[$i]}
done
for ((i = 0 ; i < ${#users[@]} ; i++ ))
do
echo $i: /opt/zimbra/bin/zmprov ca ${users[$i]}
/opt/zimbra/bin/zmprov ca ${users[$i]} ${passwords[$i]}
done
Breakdown
Uses two for loops, iterating through the array of domains firstly.
A little debugging code, helps us by showing the id number of the domain in the loop and then creates the domain by calling
zmprov cd
The domain value is taken from the 'domains' array located in the 'imap_users' file.
N.B. - You may re-run this script at any time, if the domain already exists - it will NOT be overwritten but a message informing you that the domain aready exists will appear here.
In the second loop, again a little debug code lets us know where we are in the list of users and which user is being created.
zmprov ca
creates the user, taking its arguments from the 'users' array in the 'imap_users' file, and the password from the same array index position of the 'passwords' array, again located in the 'imap_users' file.
N.B. - You may re-run this script at any time, if the user already exists - it will NOT be overwritten but a message informing you that the user aready exists will appear here
Step 4
There are two options here I suggest the first as a semi-automated way to setup a new server with accounts and the second when either migrating/backing-up between existing zimbra install on diverse platforms.
Using the script
Set the permissions on all of the files so that we can execute the files correctly:
chmod +x *
So you are now 'good-to-go' - you can invoke this script by running, from within the 'Zimbra_Scripts' directory, the command:
./zimbra_migration
Using LDAP
Syncing between platforms favours exporting your ldap from the main server, achieved via:
(this example works from x86_64 => x86 - taken from here)
'The following commands should execute on the main/source server'
Logging in as zimbra user
su - zimbra
Export LDAP data to an ldif file. For 5.0.x and previous:
./openldap/sbin/slapcat -f /opt/zimbra/conf/slapd.conf -b "" -l /tmp/zimbra-ldap.ldif
For 6.0.x and later:
./openldap/sbin/slapcat -F /opt/zimbra/data/ldap/config -b "" -l /tmp/zimbra-ldap.ldif
'The following command should execute on the destination/backup server'
Logging in as zimbra user
su - zimbra
Shutting zimbra server down
zmcontrol stop
Manually check for any imbra services - (kill if neccessary)
ps auxx | grep slapd
The following removes any current data - you could just move it until back in the 'safe zone'. Do not delete recursively here.
For 5.0.x and previous:
rm -f openldap-data/*
Import user accounts and passwords from the LDAP file zimbra-ldap.ldif
./openldap/sbin/slapadd -f /opt/zimbra/conf/slapd.conf -b "" -q -l /tmp/zimbra-ldap.ldif
For 6.0.x and later:
rm -f data/ldap/hdb/db/__db.* data/ldap/hdb/db/*.bdb data/ldap/hdb/db/alock data/ldap/hdb/logs/*
Import user accounts and passwords from the LDAP file zimbra-ldap.ldif
./openldap/sbin/slapadd -F /opt/zimbra/data/ldap/config -b "" -q -l /tmp/zimbra-ldap.ldif
That's that you may now proceed to run your imapsync_crontab file
(It is suggested and personally recommended to do things this way - however - the choice is your own and the instructions on the rest of this page also successfully work)
IMAPSYNC & Cronjob
IMAPSYNC
So now to automate the sync on a regular basis, between source and destination servers. The nwer version of imapsync removed various options as stated in the original Wiki page.
Option A -- For imapsync versions around 1.172
Option B -- For imapsync version above and beyond no doubt.
For both options:
Create a file for our cron job called 'imapsync_crontab' - again within the 'Zimbra_Scripts' directory.
nano imapsync_crontab
Option A
Edit it to include the following:
. imap_users #if in the same folder as imap_users else full-path to imap_users
src_srv = zimbra.server.com
dest_srv = backup.server.com
for ((i = 0 ; i < ${#users[@]} ; i++ ))
do
/usr/bin/imapsync --noauthmd5 --syncinternaldates --buffersize 65535000 --subscribe \
--host1 $src_srv --ssl1 --user1 ${users[$i]} --password1 ${passwords[$i]} --nofastio1 --authmech1 LOGIN \
--host2 $dest_srv --ssl2 --user2 ${users[$i]} --password2 ${passwords[$i]} --nofastio2 --authmech2 LOGIN
done
Breakdown
We set the source server address as 'src_srv' and the destination server address as 'dest_srv' - for use in the call to imapsync.
A single for loop iterates through the array containing our list of 'users' and their passwords.
The following imapsync options are used:
- noauthmd5 - no use of md5
- syncinternaldates - to keep the correct dates and times emails were sent/received
- buffersize 65535000 - ensures if a large attachment is encountered there is room in the buffer
- subscribe - automatically creates any additional user folders on the destination machine (keeping the account directory structure)
- host1 - this is our source server 'zimbra.server.com'
- host2 - this is our destination server 'backup.server.com'
- ssl1/2 - uses a secure connection on both machines
- nofastio1/2 - helps to reduce errors when syncing
- authmech1/2 - the method used to authenticate at both ends (LOGIN is commonly successful)
- password1/2 - should probably be the same on both machines
Option B
Edit it to include the following:
. imap_users #if in the same folder as imap_users else full-path to imap_users (e.g = " . /path/to/imap_users")
src_srv = zimbra.server.com
dest_srv = backup.server.com
for ((i = 0 ; i < ${#users[@]} ; i++ ))
do
/usr/bin/imapsync --noauthmd5 --syncinternaldates --subscribe \
--host1 $src_srv --ssl1 --user1 ${users[$i]} --password1 ${passwords[$i]} \
--host2 $dest_srv --ssl2 --user2 ${users[$i]} --password2 ${passwords[$i]}
done
Breakdown
We set the source server address as 'src_srv' and the destination server address as 'dest_srv' - for use in the call to imapsync.
A single for loop iterates through the array containing our list of 'users' and their passwords.
The following imapsync options are used:
- noauthmd5 - no use of md5
- syncinternaldates - to keep the correct dates and times emails were sent/received
- subscribe - automatically creates any additional user folders on the destination machine (keeping the account directory structure)
- host1 - this is our source server 'zimbra.server.com'
- host2 - this is our destination server 'backup.server.com'
- ssl1/2 - uses a secure connection on both machines
- password1/2 - should probably be the same on both machines
Option C
Edit to include the following if each user password is not known **this is the recommended option** (much easier for us admins:):
. imap_users #if in the same folder as imap_users else full-path to imap_users (e.g = " . /path/to/imap_users")
src_srv = zimbra.server.com
dest_srv = backup.server.com
for ((i = 0 ; i < ${#users[@]} ; i++ ))
do
/usr/bin/imapsync --noauthmd5 --syncinternaldates --subscribe \
--host1 $src_srv --ssl1 --user1 ${users[$i]} --authuser1 adminusername --password1 adminpassword \
--host2 $dest_srv --ssl2 --user2 ${users[$i]} --authuser2 adminusername --password2 adminpassword
done
Breakdown
The same as option B but eliminates the need to know each users password - this is achieved by passing the authuser credentials.
Crontab
Create a cron job by using the following command (firstly setting the editor to use nano):
export EDITOR=$(type -P nano);crontab -e
Now, set an hourly synchronisation to the destination server. This assumes the 'Zimbra_Scripts' directory is located in '/opt/Zimbra_Scripts'.
The output is emailed to admin@zimbra.server.com.
MAILTO=admin@zimbra.server.com
@hourly /opt/Zimbra_Scripts/imapsync_crontab
This ensures that users emails are backed up on an hourly basis.
N.B. - if you have a large number of users, it would be advisable to run the job at greater intervals. This will alleviate system resources and ensure no overlapping.
Created by Costa-101
External Links
- Installing imapsync
- Crontab manual
- Zimbra Migration Forum
- Zimbra Installation Forum
- LCRM Zimbra Support
- Zimbra Admin Powertips