HOT Backup and HOT Restore

Revision as of 12:21, 27 October 2010 by Bggo (talk | contribs)

Admin Article

Article Information

This article applies to the following ZCS versions.

ZCS 5.0 Article ZCS 5.0 ZCS 5.0 Article ZCS 5.0


NOTE

The zmbkpose works just fine for ZCS 6.0, Needs report in another versions, please contact bruno@4linux.com.br, rubens@4linux.com.br, alan.milani@4linux.com.br

Releases

The zmbpkose has been upgrated and has a newer version:

Editing ...

zmbkpose

Original Version: ___

by Rubens Alonso Filho <rubens@4linux.com.br>

The zmbkpose tool is a shell script that does hot backup and hot restore of ZCS Opensource accounts

This tool DOES NOT DO disaster recovery.

It can run from any host in the net, which means that it can be set on a backup server already existent.

It was concepted after Zimbra released ZCS 5.0.12, where a new import/export feature was exposed to user can be able to do his archiving.

From that point it was clear for me that I just need to write a code to automatically get all user data from server, remotely via HTTP SOAP. Plus LDAP data.

So, it is possible to run full backups and restore even deleted accounts.

In the version 1.0.0 of zmbkpose, the tool provides:

  • Full backup of all accounts.
  • Full backup of any account.
  • Incremental backup of all accounts.
  • Incremental backup of any account.
  • Lists all backup sessions done.
  • Restore all contents (mail, contacts, appointments...) to any account.
  • Restore deleted accounts back to directory and all of its contents.
  • Restore only account attributes, like password, class of service, etc

To run zmbkpose it is necessary

  1. Install ldap-utils e curl utils
  2. Create /etc/zmbkpose
  3. Config /etc/zmbkpose/zmbkpose.conf file (bellow)
  4. Create the script from code bellow, giving execute permissions
  5. To see zmbkpose syntax, type just zmbkpose

You can find more informations and instructions (in portuguese) at http://www.kyapanel.com/wiki/doku.php?id=zimbra:zmbkpose

Config and Scripts files from zmbkpose

/etc/zmbkpose/zmbkpose.conf

# This file is part of zmbkpose.

#    zmbkopse is free software: you can redistribute it and/or modify
#    it under the terms of the GNU General Public License as published by
#    the Free Software Foundation, either version 3 of the License, or
#    (at your option) any later version.

#    zmbkopse is distributed in the hope that it will be useful,
#    but WITHOUT ANY WARRANTY; without even the implied warranty of
#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#    GNU General Public License for more details.

#    You should have received a copy of the GNU General Public License
#    along with zmbkpose.  If not, see <http://www.gnu.org/licenses/>

# Configuration file for ZCS Opensource Edition Backup Tool
# Lines begining with "#" will not be used
# Values must have to be filled without spaces, quotes, or any kind of special characters

# Privileged user that will run zmbkpose and write in the work directory
BACKUPUSER=

# Work directory where will be the backup sessions (all user data)
# 	# Keep the directory access strict to the backup user as long as it contains all user data.
WORKDIR=

# Must have be filled with an administrator account
# Hint: If you have deleted admin, or missed its password, you can create a new admin directly from command line:
#	# zmprov ca newadmin@exemplo.com password
#	# zmprov ma newadmin@exemplo.com zimbraIsAdminAccount TRUE
ADMINUSER=

# Must be filled with ADMINUSER password
ADMINPASS=

# Must be filled with ldap url from Zimbra-LDAP master host
# Example: ldap://hostname:389
LDAPMASTERSERVER=

# Must be filled with zimbra_ldap_userdn key's value 
# Hint: To get this value, at Zimbra's server, run:
#	# zmlocalconfig zimbra_ldap_userdn
LDAPZIMBRADN=

# Must be filled with zimbra_ldap_password key's value 
# Hint: To get this value, at Zimbra's server, run:
#       # zmlocalconfig -s zimbra_ldap_password
LDAPZIMBRAPASS=

# Log file location. It must to have write permission to BACKUPUSER
LOGFILE=


/usr/local/zmbkpose

#!/bin/bash
#
# zmbkpose
#
# Bash script to hot backup and hot restore Zimbra Collaboration Suite Opensource
#
# Copyright (C) 2007 Rubens Alonso Filho <rubens@harv.com.br>
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of version 2 of the GNU General Public
# License as published by the Free Software Foundation.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307
# USA
#
# Version: 1.0.0


show_help ()
{
echo "Uses: 
     zmbkpose -f 	
			Starts a full backup of all accounts.
     zmbkpose -f mail1,mail2,...,mailn
			Starts a full backup of any account specified at command line.
     zmbkpose -i 	
			Starts an incremental backup of all accounts.
			It needs a previous full backup.
     zmbkpose -i mail1,mail2,...,mailn	
			Starts an incremental backup of any account specified at command line.
			A full account backup will run if it doesnt have any previous full backup.
     zmbkpose -l
			Lists all backup sessions done.
     zmbkpose -r mail1,mail2,...,mailn session_name
			Restore all contents (mail, contacts, appointments...) to any account from session specifieds at command line.
			If session is not specified, all sessions will be restored from oldest to newest session. It may take longer.
     zmbkpose -restoreAccount mail1,mail2,...,mailn
			Restore deleted accounts back to directory and all of its contents from oldest to newest session. It may take longer.
     zmbkpose -restoreAccount mail1,mail2,...,mailn --LDAPOnly session_name
			Restore only account attributes, like password, class of service, etc; from specified session.
			HINT: It may be necessary to flush server's cache to apply imediatelly recovered attributes. So at Zimbra server, run:
			      zmprov fc account account_name
"

exit 0
}

all_accounts_backup ()
{
ACCOUNTSLIST=$(mktemp)
$(which ldapsearch) -x -H $LDAPMASTERSERVER -D $LDAPZIMBRADN -w $LDAPZIMBRAPASS -b  -LLL "(objectclass=zimbraAccount)" zimbraMailDeliveryAddress zimbraMailHost | grep ^zimbraMail | awk '{print $2}' > "$ACCOUNTSLIST"
SESSION="full-"$(date  +%Y%m%d%H%M%S)
echo "SESSION: $SESSION started at $(date)" >> $WORKDIR/sessions.txt
TEMPDIR=$(mktemp -d $WORKDIR/XXXX)
for MAIL in $(grep @ $ACCOUNTSLIST); do 
	MAILHOST=$(grep -A1 $MAIL $ACCOUNTSLIST| grep -v @)
	$(which ldapsearch) -x -H $LDAPMASTERSERVER -D $LDAPZIMBRADN -w $LDAPZIMBRAPASS -b  -LLL "(zimbraMailDeliveryAddress=$MAIL)" > $TEMPDIR/$MAIL.ldiff
	$(which curl) -k -u $ADMINUSER:$ADMINPASS https://$MAILHOST:7071/home/$MAIL/?fmt=tgz > $TEMPDIR/$MAIL.tgz
	echo $SESSION:$MAIL:$(date +%m/%d/%y) >> $WORKDIR/sessions.txt
done
mv "$TEMPDIR" "$WORKDIR/$SESSION" && rm -rf "$TEMPDIR"
echo "SESSION: $SESSION ended at $(date)" >> $WORKDIR/sessions.txt
exit 0
}

all_accounts_incremental ()
{
FULLSESSIONLABEL=$(grep "SESSION: full-" $WORKDIR/sessions.txt | tail -1 | awk '{print $2}')
if ! [ -z "$FULLSESSIONLABEL" ]; then
        if ! [ -d "$WORKDIR/$FULLSESSIONLABEL" ]; then
                echo "$WORKDIR/$FULLSESSIONLABEL directory doesnt exist. Impossible to proceed."
                exit 0
        fi
else
	echo "No full backups found. Impossible to proceed."
	exit 0
fi
INCFROM=$(grep INCFROM: $WORKDIR/sessions.txt | tail -1 | awk '{print $2}')
ACCOUNTSLIST=$(mktemp)
$(which ldapsearch) -x -H $LDAPMASTERSERVER -D $LDAPZIMBRADN -w $LDAPZIMBRAPASS -b  -LLL "(objectclass=zimbraAccount)" zimbraMailDeliveryAddress zimbraMailHost | grep ^zimbraMail | awk '{print $2}' > "$ACCOUNTSLIST"
SESSION="inc"-$(date  +%Y%m%d%H%M%S)
echo "SESSION: $SESSION started at $(date)" >> $WORKDIR/sessions.txt
TEMPDIR=$(mktemp -d $WORKDIR/XXXX)
for MAIL in $(grep @ $ACCOUNTSLIST); do 
	EXISTFULL=$(grep $MAIL $WORKDIR/sessions.txt | grep ^full)
        if [ -z $EXISTFULL ]; then
        	echo "$MAIL doesnt have any previous full backup. Running now..."
                account_backup $MAIL
        else
		INCFROM=$(grep $MAIL $WORKDIR/sessions.txt | grep -v ^WARN | tail -1 | awk -F: '{print $3}')
		if [ "$INCFROM" = "$(date +%m/%d/%y)" ]; then
			echo "WARN: $MAIL was already backed up today. Nothing to do." | tee -a $WORKDIR/sessions.txt
		else
			MAILHOST=$(grep -A1 $MAIL $ACCOUNTSLIST| grep -v @)
			$(which ldapsearch) -x -H $LDAPMASTERSERVER -D $LDAPZIMBRADN -w $LDAPZIMBRAPASS -b  -LLL "(zimbraMailDeliveryAddress=$MAIL)" > $TEMPDIR/$MAIL.ldiff
			$(which curl) -k -u $ADMINUSER:$ADMINPASS https://$MAILHOST:7071/home/$MAIL/?fmt=tgz\&query=after:\"$INCFROM\" > $TEMPDIR/$MAIL.tgz
			echo $SESSION:$MAIL:$(date +%m/%d/%y) >> $WORKDIR/sessions.txt
		fi
	fi
done
mv "$TEMPDIR" "$WORKDIR/$SESSION" && rm -rf "$TEMPDIR"
echo "SESSION: $SESSION ended at $(date)" >> $WORKDIR/sessions.txt
exit 0
}

account_backup ()
{
if [ -z $SESSION ]; then
	SESSION="full-"$(date  +%Y%m%d%H%M%S)
	echo "SESSION: $SESSION started at $(date)" >> $WORKDIR/sessions.txt
	TEMPDIR=$(mktemp -d $WORKDIR/XXXX)
else
	SUBSESSION="$SESSION"
	SESSION="full-"$(date  +%Y%m%d%H%M%S)
fi
K=1
while true; do
	MAIL=$(echo $1, | cut -d, -f$K)
	if [ -z $MAIL ]; then
		break
	fi
	$(which ldapsearch) -x -H $LDAPMASTERSERVER -D $LDAPZIMBRADN -w $LDAPZIMBRAPASS -b  -LLL "(zimbraMailDeliveryAddress=$MAIL)" > $TEMPDIR/$MAIL.ldiff
	MAILHOST=$(grep ^zimbraMailHost $TEMPDIR/$MAIL.ldiff | awk '{print $2}')
	$(which curl) -k -u $ADMINUSER:$ADMINPASS https://$MAILHOST:7071/home/$MAIL/?fmt=tgz > $TEMPDIR/$MAIL.tgz
	echo $SESSION:$MAIL:$(date +%m/%d/%y) >> $WORKDIR/sessions.txt
	((K = K+1))
	unset MAIL
	sleep 1
done
if [ -z $SUBSESSION ]; then
	mv "$TEMPDIR" "$WORKDIR/$SESSION" && rm -rf "$TEMPDIR"
	echo "SESSION: $SESSION ended at $(date)" >> $WORKDIR/sessions.txt
	exit 0
else
	SESSION="$SUBSESSION"
fi
}

account_incremental ()
{
SESSION="inc-"$(date  +%Y%m%d%H%M%S)
echo "SESSION: $SESSION started at $(date)" >> $WORKDIR/sessions.txt
TEMPDIR=$(mktemp -d $WORKDIR/XXXX)
K=1
while true; do
	MAIL=$(echo $1, | cut -d, -f$K)
	if [ -z $MAIL ]; then
		break
	else
		EXISTFULL=$(grep $MAIL $WORKDIR/sessions.txt | grep ^full)
		if [ -z $EXISTFULL ]; then
			echo " $MAIL doesnt have any previous full backup. Running now..."
			account_backup $MAIL
			((K = K+1))
		else
			$(which ldapsearch) -x -H $LDAPMASTERSERVER -D $LDAPZIMBRADN -w $LDAPZIMBRAPASS -b  -LLL "(zimbraMailDeliveryAddress=$MAIL)" > $TEMPDIR/$MAIL.ldiff
			INCFROM=$(grep $MAIL $WORKDIR/sessions.txt | grep -v ^WARN | tail -1 | awk -F: '{print $3}')
			if [ "$INCFROM" = "$(date +%m/%d/%y)" ]; then
				echo "WARN:  $MAIL was already backed up today. Nothing to do." | tee -a $WORKDIR/sessions.txt
				((K = K+1))
			else
				MAILHOST=$(grep ^zimbraMailHost $TEMPDIR/$MAIL.ldiff | awk '{print $2}')
				$(which curl) -k -u $ADMINUSER:$ADMINPASS https://$MAILHOST:7071/home/$MAIL/?fmt=tgz\&query=after:\"$INCFROM\" > $TEMPDIR/$MAIL.tgz
				echo $SESSION:$MAIL:$(date +%m/%d/%y) >> $WORKDIR/sessions.txt
				((K = K+1))
			fi
		fi
	fi
	unset MAIL
done
mv "$TEMPDIR" "$WORKDIR/$SESSION" && rm -rf "$TEMPDIR"
echo "SESSION: $SESSION ended at $(date)" >> $WORKDIR/sessions.txt
exit 0
}

list_sessions ()
{
grep SESSION: $WORKDIR/sessions.txt| grep started | awk '{print $2}'
exit 0
}

account_restore ()
{
ACCOUNTBKPS=$(mktemp)
K=1
while true; do
        MAIL=$(echo $1, | cut -d, -f$K)
        if [ -z $MAIL ]; then
                break
        fi
	grep $MAIL $WORKDIR/sessions.txt | grep -v ^WARN: > $ACCOUNTBKPS
	if ! [ -s $ACCOUNTBKPS ]; then
		echo "$MAIL: No backups found. Impossible to restore"
		((K = K+1))
	else
		if [ -z $2 ]; then
		echo "Not Implemented."
		# Complete restore from oldest to newest
		((K = K+1))
		else
			ACCOUNTSESSION=$(grep $2 $WORKDIR/sessions.txt | tail -1 | awk '{print $2}')
			if [ -z $ACCOUNTSESSION ]; then
				echo "$MAIL: $2 session doesnt exists. Impossible to proceed..."
				break
			else
				ACCOUNTINSESSIO=$(grep $MAIL $ACCOUNTBKPS | grep $ACCOUNTSESSION)
				if [ -z $ACCOUNTINSESSIO ]; then
					echo "$MAIL not found in session $ACCOUNTSESSION. Impossible to restore."
        				((K = K+1))
				else
					MAILHOST=$(grep ^zimbraMailHost $WORKDIR/$ACCOUNTSESSION/$MAIL.ldiff | awk '{print $2}')
        				$(which curl) -k --data-binary @$WORKDIR/$ACCOUNTSESSION/$MAIL.tgz -u $ADMINUSER:$ADMINPASS https://$MAILHOST:7071/home/$MAIL/?fmt=tgz
        				((K = K+1))
				        unset MAIL
				fi
			fi
		fi
	fi
done
exit 0
}
LDAP_content_restore ()
{
ACCOUNTBKPS=$(mktemp)
K=1
while true; do
        MAIL=$(echo $1, | cut -d, -f$K)
        if [ -z $MAIL ]; then
                break
	fi
	if [ -z $2 ]; then
		EXIST=$($(which ldapsearch) -x -H $LDAPMASTERSERVER -D $LDAPZIMBRADN -w $LDAPZIMBRAPASS -b  -LLL "(&(objectclass=zimbraAccount)(zimbraMailDeliveryAddress=$MAIL))" uid)
		if ! [ -z "$EXIST" ]; then
			echo "$MAIL account exists. Run zmbkpose -r $MAIL session_name."
			((K = K+1))
		else
		       	grep $MAIL $WORKDIR/sessions.txt | grep -e ^inc- -e ^full- > $ACCOUNTBKPS
		        if ! [ -s $ACCOUNTBKPS ]; then
        		        echo "$MAIL: No backups found. Impossible to restore"
                		((K = K+1))
		        else
				echo "Sessions found
$(cat $ACCOUNTBKPS | awk -F: '{print $1}')"
				for ACCOUNTSESSION in $(cat $ACCOUNTBKPS | awk -F: '{print $1}'); do
					echo "Restoring from $ACCOUNTSESSION"
		                	MAILHOST=$(grep ^zimbraMailHost $WORKDIR/$ACCOUNTSESSION/$MAIL.ldiff | awk '{print $2}')
					$(which ldapdelete) -r -x -H $LDAPMASTERSERVER -D $LDAPZIMBRADN -c -w $LDAPZIMBRAPASS $(grep ^dn: $WORKDIR/$ACCOUNTSESSION/$MAIL.ldiff | awk '{print $2}') 2>/dev/null
					$(which ldapadd) -x -H $LDAPMASTERSERVER -D $LDAPZIMBRADN -c -w $LDAPZIMBRAPASS -f $WORKDIR/$ACCOUNTSESSION/$MAIL.ldiff
        	        	        $(which curl) -k --data-binary @$WORKDIR/$ACCOUNTSESSION/$MAIL.tgz -u $ADMINUSER:$ADMINPASS https://$MAILHOST:7071/home/$MAIL/?fmt=tgz
					echo "$MAIL restored from $ACCOUNTSESSION"
				done
	        	        ((K = K+1))
        	        	unset MAIL
			fi
		fi
	else
		ACCOUNTSESSION=$(grep $2 $WORKDIR/sessions.txt | grep $MAIL | tail -1 | awk -F: '{print $1}')
	        if [ -z $ACCOUNTSESSION ]; then
                	echo "$MAIL: Session $2 doesnt exist or not found. Impossible to restore..."
        	        ((K = K+1))
                else
			USERDN=$(grep ^dn: $WORKDIR/$ACCOUNTSESSION/$MAIL.ldiff | awk '{print $2}')
			$(which ldapdelete) -r -x -H $LDAPMASTERSERVER -D $LDAPZIMBRADN -c -w $LDAPZIMBRAPASS $USERDN
			$(which ldapadd) -x -H $LDAPMASTERSERVER -D $LDAPZIMBRADN -c -w $LDAPZIMBRAPASS -f $WORKDIR/$ACCOUNTSESSION/$MAIL.ldiff
			echo "User profile and settings restored from $ACCOUNTSESSION"
        	        ((K = K+1))
       	        	unset MAIL
		fi
        fi
done
exit 0
}

# This option is not yet exposed at script's help cause it need more consistences so
# DO NOT USE IT AT A PRODUCTION ENVIRONMENT
# DeusMeAjude ()
# {
# for BKP in $(grep -e ^full- -e ^inc- $WORKDIR/sessions.txt); do
#	ACCOUNTSESSION=$(echo $BKP | awk -F: '{print $1}')
#	MAIL=$(echo $BKP | awk -F: '{print $2}')
#	USERDN=$(grep ^dn: $WORKDIR/$ACCOUNTSESSION/$MAIL.ldiff | awk '{print $2}')
#	MAILHOST=$(grep ^zimbraMailHost $WORKDIR/$ACCOUNTSESSION/$MAIL.ldiff | awk '{print $2}')
#	echo "Restaurando $MAIL"
#	$(which ldapdelete) -r -x -H $LDAPMASTERSERVER -D $LDAPZIMBRADN -c -w $LDAPZIMBRAPASS $USERDN 2>/dev/null
#	$(which ldapadd) -x -H $LDAPMASTERSERVER -D $LDAPZIMBRADN -c -w $LDAPZIMBRAPASS -f $WORKDIR/$ACCOUNTSESSION/$MAIL.ldiff
#	$(which curl) -k --data-binary @$WORKDIR/$ACCOUNTSESSION/$MAIL.tgz -u $ADMINUSER:$ADMINPASS https://$MAILHOST:7071/home/$MAIL/?fmt=tgz
#	echo "$MAIL restaurado"
#done
#exit 0
#}

# Loading config file
source /etc/zmbkpose/zmbkpose.conf

if ! [ -z "$BACKUPUSER" ]; then
	if [ "$(id -u)" != "$(id -u $BACKUPUSER)" ]; then
		echo "You must be $BACKUPUSER to run this script"
		exit 0
	fi
else
	echo "You must set BACKUPUSER"
	exit 0
fi

if ! [ -z "$WORKDIR" ]; then
        if ! [ -d "$WORKDIR" ]; then
                echo "$WORKDIR doesnt exist"
                exit 0
        fi
else
        echo "You must set WORKDIR"
        exit 0
fi

if [ -z "$ADMINUSER" ]; then
        echo "You must set ADMINUSER"
        exit 0
fi

if [ -z "$ADMINPASS" ]; then
        echo "You must set ADMINPASS"
        exit 0
fi

if [ -z "$LDAPMASTERSERVER" ]; then
        echo "You must set LDAPMASTERSERVER"
        exit 0
fi

if [ -z "$LDAPZIMBRADN" ]; then
        echo "You must set LDAPZIMBRADN"
        exit 0
fi

if [ -z "$LDAPZIMBRAPASS" ]; then
        echo "You must set LDAPZIMBRAPASS"
        exit 0
fi

if [ -z "$LOGFILE" ]; then
        echo "You must set LOGFILE"
        exit 0
fi

# Criticar os parametros passados na linha de comando

case "$1" in
"-f" )
	if [ -z "$2" ]; then
		all_accounts_backup
	else
		if [ -z "$3" ]; then
			account_backup $2
		fi
		echo "Incorrect parameters $@. See help."
        	show_help
	fi
;;
"-i" )
	if [ -z "$2" ]; then
		all_accounts_incremental
	else
		if [ -z "$3" ]; then
			account_incremental $2
		fi
		echo "Incorrect parameters $@. See help."
        	show_help
	fi
;;
"-l" )
        if [ -z "$2" ]; then
                list_sessions
	else
                echo "Incorrect parameters $@. See help."
                show_help
	fi
;;
"-r" )
        if [ -z "$2" ]; then
                echo "Incorrect parameters $@. See help."
                show_help
	else
		if [ -z "$4" ]; then
			account_restore $2 $3
		else
                	echo "Incorrect parameters $@. See help."
	                show_help
		fi
	fi
;;
"-restoreAccount" )
        if [ -z "$2" ]; then
                echo "Incorrect parameters $@. See help."
                show_help
	else
		if [ -z "$3" ]; then
			LDAP_content_restore $2
		else
			if [ "$3" = "--LDAPOnly" ]; then
				LDAP_content_restore $2 $4
			else
	                	echo "Incorrect parameters $@. See help."
		                show_help
			fi
		fi
	fi
	
;;
# "--DeusMeAjude" )
#	if ! [ -z "$2" ]; then
#	        echo "Incorrect parameters $@. See help."
#        	show_help
#	else
#		DeusMeAjude
#	fi
# ;;
* )
	echo "Incorrect parameters $@. See help."
	show_help
;;
esac
exit 0

You can find more informations and instructions (in portuguese) at http://www.kyapanel.com/wiki/doku.php?id=zimbra:zmbkpose


Verified Against: unknown Date Created: 12/15/2009
Article ID: https://wiki.zimbra.com/index.php?title=HOT_Backup_and_HOT_Restore Date Modified: 2010-10-27



Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »

Jump to: navigation, search