HOT Backup and HOT Restore: Difference between revisions
Line 12: | Line 12: | ||
= Releases = | = Releases = | ||
1.0.0 - [ http://wiki.zimbra.com/images/2/2c/Zmbkpose-1.0.0.tar.gz ] | 1.0.0 - [ http://wiki.zimbra.com/images/2/2c/Zmbkpose-1.0.0.tar.gz | a] | ||
= zmbkpose = | = zmbkpose = |
Revision as of 12:28, 27 October 2010
Article Information |
---|
This article applies to the following ZCS versions. |
NOTE
ZMBKPOSE is a bash script to hot backup and hot restore Zimbra Collaboration Suite Opensource. ZMBKPOSE was first developed by Rubens Alonso Filho and improved by many other people.
Would be nice if we get some reports to be always improveing the tool,please contact bruno@4linux.com.br, rubens@4linux.com.br, alan.milani@4linux.com.br for bugs, suggestions and others.
Releases
1.0.0 - [ http://wiki.zimbra.com/images/2/2c/Zmbkpose-1.0.0.tar.gz | a]
zmbkpose
Original Version: ___
by Rubens Alonso Filho <rubens@4linux.com.br>
The zmbkpose tool is a shell script that does hot backup and hot restore of ZCS Opensource accounts
This tool DOES NOT DO disaster recovery.
It can run from any host in the net, which means that it can be set on a backup server already existent.
It was concepted after Zimbra released ZCS 5.0.12, where a new import/export feature was exposed to user can be able to do his archiving.
From that point it was clear for me that I just need to write a code to automatically get all user data from server, remotely via HTTP SOAP. Plus LDAP data.
So, it is possible to run full backups and restore even deleted accounts.
In the version 1.0.0 of zmbkpose, the tool provides:
- Full backup of all accounts.
- Full backup of any account.
- Incremental backup of all accounts.
- Incremental backup of any account.
- Lists all backup sessions done.
- Restore all contents (mail, contacts, appointments...) to any account.
- Restore deleted accounts back to directory and all of its contents.
- Restore only account attributes, like password, class of service, etc
To run zmbkpose it is necessary
- Install ldap-utils e curl utils
- Create /etc/zmbkpose
- Config /etc/zmbkpose/zmbkpose.conf file (bellow)
- Create the script from code bellow, giving execute permissions
- To see zmbkpose syntax, type just zmbkpose
You can find more informations and instructions (in portuguese) at http://www.kyapanel.com/wiki/doku.php?id=zimbra:zmbkpose
Config and Scripts files from zmbkpose
/etc/zmbkpose/zmbkpose.conf
# This file is part of zmbkpose. # zmbkopse is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # zmbkopse is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # You should have received a copy of the GNU General Public License # along with zmbkpose. If not, see <http://www.gnu.org/licenses/> # Configuration file for ZCS Opensource Edition Backup Tool # Lines begining with "#" will not be used # Values must have to be filled without spaces, quotes, or any kind of special characters # Privileged user that will run zmbkpose and write in the work directory BACKUPUSER= # Work directory where will be the backup sessions (all user data) # # Keep the directory access strict to the backup user as long as it contains all user data. WORKDIR= # Must have be filled with an administrator account # Hint: If you have deleted admin, or missed its password, you can create a new admin directly from command line: # # zmprov ca newadmin@exemplo.com password # # zmprov ma newadmin@exemplo.com zimbraIsAdminAccount TRUE ADMINUSER= # Must be filled with ADMINUSER password ADMINPASS= # Must be filled with ldap url from Zimbra-LDAP master host # Example: ldap://hostname:389 LDAPMASTERSERVER= # Must be filled with zimbra_ldap_userdn key's value # Hint: To get this value, at Zimbra's server, run: # # zmlocalconfig zimbra_ldap_userdn LDAPZIMBRADN= # Must be filled with zimbra_ldap_password key's value # Hint: To get this value, at Zimbra's server, run: # # zmlocalconfig -s zimbra_ldap_password LDAPZIMBRAPASS= # Log file location. It must to have write permission to BACKUPUSER LOGFILE=
/usr/local/zmbkpose
#!/bin/bash # # zmbkpose # # Bash script to hot backup and hot restore Zimbra Collaboration Suite Opensource # # Copyright (C) 2007 Rubens Alonso Filho <rubens@harv.com.br> # # This program is free software; you can redistribute it and/or # modify it under the terms of version 2 of the GNU General Public # License as published by the Free Software Foundation. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 # USA # # Version: 1.0.0 show_help () { echo "Uses: zmbkpose -f Starts a full backup of all accounts. zmbkpose -f mail1,mail2,...,mailn Starts a full backup of any account specified at command line. zmbkpose -i Starts an incremental backup of all accounts. It needs a previous full backup. zmbkpose -i mail1,mail2,...,mailn Starts an incremental backup of any account specified at command line. A full account backup will run if it doesnt have any previous full backup. zmbkpose -l Lists all backup sessions done. zmbkpose -r mail1,mail2,...,mailn session_name Restore all contents (mail, contacts, appointments...) to any account from session specifieds at command line. If session is not specified, all sessions will be restored from oldest to newest session. It may take longer. zmbkpose -restoreAccount mail1,mail2,...,mailn Restore deleted accounts back to directory and all of its contents from oldest to newest session. It may take longer. zmbkpose -restoreAccount mail1,mail2,...,mailn --LDAPOnly session_name Restore only account attributes, like password, class of service, etc; from specified session. HINT: It may be necessary to flush server's cache to apply imediatelly recovered attributes. So at Zimbra server, run: zmprov fc account account_name " exit 0 } all_accounts_backup () { ACCOUNTSLIST=$(mktemp) $(which ldapsearch) -x -H $LDAPMASTERSERVER -D $LDAPZIMBRADN -w $LDAPZIMBRAPASS -b -LLL "(objectclass=zimbraAccount)" zimbraMailDeliveryAddress zimbraMailHost | grep ^zimbraMail | awk '{print $2}' > "$ACCOUNTSLIST" SESSION="full-"$(date +%Y%m%d%H%M%S) echo "SESSION: $SESSION started at $(date)" >> $WORKDIR/sessions.txt TEMPDIR=$(mktemp -d $WORKDIR/XXXX) for MAIL in $(grep @ $ACCOUNTSLIST); do MAILHOST=$(grep -A1 $MAIL $ACCOUNTSLIST| grep -v @) $(which ldapsearch) -x -H $LDAPMASTERSERVER -D $LDAPZIMBRADN -w $LDAPZIMBRAPASS -b -LLL "(zimbraMailDeliveryAddress=$MAIL)" > $TEMPDIR/$MAIL.ldiff $(which curl) -k -u $ADMINUSER:$ADMINPASS https://$MAILHOST:7071/home/$MAIL/?fmt=tgz > $TEMPDIR/$MAIL.tgz echo $SESSION:$MAIL:$(date +%m/%d/%y) >> $WORKDIR/sessions.txt done mv "$TEMPDIR" "$WORKDIR/$SESSION" && rm -rf "$TEMPDIR" echo "SESSION: $SESSION ended at $(date)" >> $WORKDIR/sessions.txt exit 0 } all_accounts_incremental () { FULLSESSIONLABEL=$(grep "SESSION: full-" $WORKDIR/sessions.txt | tail -1 | awk '{print $2}') if ! [ -z "$FULLSESSIONLABEL" ]; then if ! [ -d "$WORKDIR/$FULLSESSIONLABEL" ]; then echo "$WORKDIR/$FULLSESSIONLABEL directory doesnt exist. Impossible to proceed." exit 0 fi else echo "No full backups found. Impossible to proceed." exit 0 fi INCFROM=$(grep INCFROM: $WORKDIR/sessions.txt | tail -1 | awk '{print $2}') ACCOUNTSLIST=$(mktemp) $(which ldapsearch) -x -H $LDAPMASTERSERVER -D $LDAPZIMBRADN -w $LDAPZIMBRAPASS -b -LLL "(objectclass=zimbraAccount)" zimbraMailDeliveryAddress zimbraMailHost | grep ^zimbraMail | awk '{print $2}' > "$ACCOUNTSLIST" SESSION="inc"-$(date +%Y%m%d%H%M%S) echo "SESSION: $SESSION started at $(date)" >> $WORKDIR/sessions.txt TEMPDIR=$(mktemp -d $WORKDIR/XXXX) for MAIL in $(grep @ $ACCOUNTSLIST); do EXISTFULL=$(grep $MAIL $WORKDIR/sessions.txt | grep ^full) if [ -z $EXISTFULL ]; then echo "$MAIL doesnt have any previous full backup. Running now..." account_backup $MAIL else INCFROM=$(grep $MAIL $WORKDIR/sessions.txt | grep -v ^WARN | tail -1 | awk -F: '{print $3}') if [ "$INCFROM" = "$(date +%m/%d/%y)" ]; then echo "WARN: $MAIL was already backed up today. Nothing to do." | tee -a $WORKDIR/sessions.txt else MAILHOST=$(grep -A1 $MAIL $ACCOUNTSLIST| grep -v @) $(which ldapsearch) -x -H $LDAPMASTERSERVER -D $LDAPZIMBRADN -w $LDAPZIMBRAPASS -b -LLL "(zimbraMailDeliveryAddress=$MAIL)" > $TEMPDIR/$MAIL.ldiff $(which curl) -k -u $ADMINUSER:$ADMINPASS https://$MAILHOST:7071/home/$MAIL/?fmt=tgz\&query=after:\"$INCFROM\" > $TEMPDIR/$MAIL.tgz echo $SESSION:$MAIL:$(date +%m/%d/%y) >> $WORKDIR/sessions.txt fi fi done mv "$TEMPDIR" "$WORKDIR/$SESSION" && rm -rf "$TEMPDIR" echo "SESSION: $SESSION ended at $(date)" >> $WORKDIR/sessions.txt exit 0 } account_backup () { if [ -z $SESSION ]; then SESSION="full-"$(date +%Y%m%d%H%M%S) echo "SESSION: $SESSION started at $(date)" >> $WORKDIR/sessions.txt TEMPDIR=$(mktemp -d $WORKDIR/XXXX) else SUBSESSION="$SESSION" SESSION="full-"$(date +%Y%m%d%H%M%S) fi K=1 while true; do MAIL=$(echo $1, | cut -d, -f$K) if [ -z $MAIL ]; then break fi $(which ldapsearch) -x -H $LDAPMASTERSERVER -D $LDAPZIMBRADN -w $LDAPZIMBRAPASS -b -LLL "(zimbraMailDeliveryAddress=$MAIL)" > $TEMPDIR/$MAIL.ldiff MAILHOST=$(grep ^zimbraMailHost $TEMPDIR/$MAIL.ldiff | awk '{print $2}') $(which curl) -k -u $ADMINUSER:$ADMINPASS https://$MAILHOST:7071/home/$MAIL/?fmt=tgz > $TEMPDIR/$MAIL.tgz echo $SESSION:$MAIL:$(date +%m/%d/%y) >> $WORKDIR/sessions.txt ((K = K+1)) unset MAIL sleep 1 done if [ -z $SUBSESSION ]; then mv "$TEMPDIR" "$WORKDIR/$SESSION" && rm -rf "$TEMPDIR" echo "SESSION: $SESSION ended at $(date)" >> $WORKDIR/sessions.txt exit 0 else SESSION="$SUBSESSION" fi } account_incremental () { SESSION="inc-"$(date +%Y%m%d%H%M%S) echo "SESSION: $SESSION started at $(date)" >> $WORKDIR/sessions.txt TEMPDIR=$(mktemp -d $WORKDIR/XXXX) K=1 while true; do MAIL=$(echo $1, | cut -d, -f$K) if [ -z $MAIL ]; then break else EXISTFULL=$(grep $MAIL $WORKDIR/sessions.txt | grep ^full) if [ -z $EXISTFULL ]; then echo " $MAIL doesnt have any previous full backup. Running now..." account_backup $MAIL ((K = K+1)) else $(which ldapsearch) -x -H $LDAPMASTERSERVER -D $LDAPZIMBRADN -w $LDAPZIMBRAPASS -b -LLL "(zimbraMailDeliveryAddress=$MAIL)" > $TEMPDIR/$MAIL.ldiff INCFROM=$(grep $MAIL $WORKDIR/sessions.txt | grep -v ^WARN | tail -1 | awk -F: '{print $3}') if [ "$INCFROM" = "$(date +%m/%d/%y)" ]; then echo "WARN: $MAIL was already backed up today. Nothing to do." | tee -a $WORKDIR/sessions.txt ((K = K+1)) else MAILHOST=$(grep ^zimbraMailHost $TEMPDIR/$MAIL.ldiff | awk '{print $2}') $(which curl) -k -u $ADMINUSER:$ADMINPASS https://$MAILHOST:7071/home/$MAIL/?fmt=tgz\&query=after:\"$INCFROM\" > $TEMPDIR/$MAIL.tgz echo $SESSION:$MAIL:$(date +%m/%d/%y) >> $WORKDIR/sessions.txt ((K = K+1)) fi fi fi unset MAIL done mv "$TEMPDIR" "$WORKDIR/$SESSION" && rm -rf "$TEMPDIR" echo "SESSION: $SESSION ended at $(date)" >> $WORKDIR/sessions.txt exit 0 } list_sessions () { grep SESSION: $WORKDIR/sessions.txt| grep started | awk '{print $2}' exit 0 } account_restore () { ACCOUNTBKPS=$(mktemp) K=1 while true; do MAIL=$(echo $1, | cut -d, -f$K) if [ -z $MAIL ]; then break fi grep $MAIL $WORKDIR/sessions.txt | grep -v ^WARN: > $ACCOUNTBKPS if ! [ -s $ACCOUNTBKPS ]; then echo "$MAIL: No backups found. Impossible to restore" ((K = K+1)) else if [ -z $2 ]; then echo "Not Implemented." # Complete restore from oldest to newest ((K = K+1)) else ACCOUNTSESSION=$(grep $2 $WORKDIR/sessions.txt | tail -1 | awk '{print $2}') if [ -z $ACCOUNTSESSION ]; then echo "$MAIL: $2 session doesnt exists. Impossible to proceed..." break else ACCOUNTINSESSIO=$(grep $MAIL $ACCOUNTBKPS | grep $ACCOUNTSESSION) if [ -z $ACCOUNTINSESSIO ]; then echo "$MAIL not found in session $ACCOUNTSESSION. Impossible to restore." ((K = K+1)) else MAILHOST=$(grep ^zimbraMailHost $WORKDIR/$ACCOUNTSESSION/$MAIL.ldiff | awk '{print $2}') $(which curl) -k --data-binary @$WORKDIR/$ACCOUNTSESSION/$MAIL.tgz -u $ADMINUSER:$ADMINPASS https://$MAILHOST:7071/home/$MAIL/?fmt=tgz ((K = K+1)) unset MAIL fi fi fi fi done exit 0 } LDAP_content_restore () { ACCOUNTBKPS=$(mktemp) K=1 while true; do MAIL=$(echo $1, | cut -d, -f$K) if [ -z $MAIL ]; then break fi if [ -z $2 ]; then EXIST=$($(which ldapsearch) -x -H $LDAPMASTERSERVER -D $LDAPZIMBRADN -w $LDAPZIMBRAPASS -b -LLL "(&(objectclass=zimbraAccount)(zimbraMailDeliveryAddress=$MAIL))" uid) if ! [ -z "$EXIST" ]; then echo "$MAIL account exists. Run zmbkpose -r $MAIL session_name." ((K = K+1)) else grep $MAIL $WORKDIR/sessions.txt | grep -e ^inc- -e ^full- > $ACCOUNTBKPS if ! [ -s $ACCOUNTBKPS ]; then echo "$MAIL: No backups found. Impossible to restore" ((K = K+1)) else echo "Sessions found $(cat $ACCOUNTBKPS | awk -F: '{print $1}')" for ACCOUNTSESSION in $(cat $ACCOUNTBKPS | awk -F: '{print $1}'); do echo "Restoring from $ACCOUNTSESSION" MAILHOST=$(grep ^zimbraMailHost $WORKDIR/$ACCOUNTSESSION/$MAIL.ldiff | awk '{print $2}') $(which ldapdelete) -r -x -H $LDAPMASTERSERVER -D $LDAPZIMBRADN -c -w $LDAPZIMBRAPASS $(grep ^dn: $WORKDIR/$ACCOUNTSESSION/$MAIL.ldiff | awk '{print $2}') 2>/dev/null $(which ldapadd) -x -H $LDAPMASTERSERVER -D $LDAPZIMBRADN -c -w $LDAPZIMBRAPASS -f $WORKDIR/$ACCOUNTSESSION/$MAIL.ldiff $(which curl) -k --data-binary @$WORKDIR/$ACCOUNTSESSION/$MAIL.tgz -u $ADMINUSER:$ADMINPASS https://$MAILHOST:7071/home/$MAIL/?fmt=tgz echo "$MAIL restored from $ACCOUNTSESSION" done ((K = K+1)) unset MAIL fi fi else ACCOUNTSESSION=$(grep $2 $WORKDIR/sessions.txt | grep $MAIL | tail -1 | awk -F: '{print $1}') if [ -z $ACCOUNTSESSION ]; then echo "$MAIL: Session $2 doesnt exist or not found. Impossible to restore..." ((K = K+1)) else USERDN=$(grep ^dn: $WORKDIR/$ACCOUNTSESSION/$MAIL.ldiff | awk '{print $2}') $(which ldapdelete) -r -x -H $LDAPMASTERSERVER -D $LDAPZIMBRADN -c -w $LDAPZIMBRAPASS $USERDN $(which ldapadd) -x -H $LDAPMASTERSERVER -D $LDAPZIMBRADN -c -w $LDAPZIMBRAPASS -f $WORKDIR/$ACCOUNTSESSION/$MAIL.ldiff echo "User profile and settings restored from $ACCOUNTSESSION" ((K = K+1)) unset MAIL fi fi done exit 0 } # This option is not yet exposed at script's help cause it need more consistences so # DO NOT USE IT AT A PRODUCTION ENVIRONMENT # DeusMeAjude () # { # for BKP in $(grep -e ^full- -e ^inc- $WORKDIR/sessions.txt); do # ACCOUNTSESSION=$(echo $BKP | awk -F: '{print $1}') # MAIL=$(echo $BKP | awk -F: '{print $2}') # USERDN=$(grep ^dn: $WORKDIR/$ACCOUNTSESSION/$MAIL.ldiff | awk '{print $2}') # MAILHOST=$(grep ^zimbraMailHost $WORKDIR/$ACCOUNTSESSION/$MAIL.ldiff | awk '{print $2}') # echo "Restaurando $MAIL" # $(which ldapdelete) -r -x -H $LDAPMASTERSERVER -D $LDAPZIMBRADN -c -w $LDAPZIMBRAPASS $USERDN 2>/dev/null # $(which ldapadd) -x -H $LDAPMASTERSERVER -D $LDAPZIMBRADN -c -w $LDAPZIMBRAPASS -f $WORKDIR/$ACCOUNTSESSION/$MAIL.ldiff # $(which curl) -k --data-binary @$WORKDIR/$ACCOUNTSESSION/$MAIL.tgz -u $ADMINUSER:$ADMINPASS https://$MAILHOST:7071/home/$MAIL/?fmt=tgz # echo "$MAIL restaurado" #done #exit 0 #} # Loading config file source /etc/zmbkpose/zmbkpose.conf if ! [ -z "$BACKUPUSER" ]; then if [ "$(id -u)" != "$(id -u $BACKUPUSER)" ]; then echo "You must be $BACKUPUSER to run this script" exit 0 fi else echo "You must set BACKUPUSER" exit 0 fi if ! [ -z "$WORKDIR" ]; then if ! [ -d "$WORKDIR" ]; then echo "$WORKDIR doesnt exist" exit 0 fi else echo "You must set WORKDIR" exit 0 fi if [ -z "$ADMINUSER" ]; then echo "You must set ADMINUSER" exit 0 fi if [ -z "$ADMINPASS" ]; then echo "You must set ADMINPASS" exit 0 fi if [ -z "$LDAPMASTERSERVER" ]; then echo "You must set LDAPMASTERSERVER" exit 0 fi if [ -z "$LDAPZIMBRADN" ]; then echo "You must set LDAPZIMBRADN" exit 0 fi if [ -z "$LDAPZIMBRAPASS" ]; then echo "You must set LDAPZIMBRAPASS" exit 0 fi if [ -z "$LOGFILE" ]; then echo "You must set LOGFILE" exit 0 fi # Criticar os parametros passados na linha de comando case "$1" in "-f" ) if [ -z "$2" ]; then all_accounts_backup else if [ -z "$3" ]; then account_backup $2 fi echo "Incorrect parameters $@. See help." show_help fi ;; "-i" ) if [ -z "$2" ]; then all_accounts_incremental else if [ -z "$3" ]; then account_incremental $2 fi echo "Incorrect parameters $@. See help." show_help fi ;; "-l" ) if [ -z "$2" ]; then list_sessions else echo "Incorrect parameters $@. See help." show_help fi ;; "-r" ) if [ -z "$2" ]; then echo "Incorrect parameters $@. See help." show_help else if [ -z "$4" ]; then account_restore $2 $3 else echo "Incorrect parameters $@. See help." show_help fi fi ;; "-restoreAccount" ) if [ -z "$2" ]; then echo "Incorrect parameters $@. See help." show_help else if [ -z "$3" ]; then LDAP_content_restore $2 else if [ "$3" = "--LDAPOnly" ]; then LDAP_content_restore $2 $4 else echo "Incorrect parameters $@. See help." show_help fi fi fi ;; # "--DeusMeAjude" ) # if ! [ -z "$2" ]; then # echo "Incorrect parameters $@. See help." # show_help # else # DeusMeAjude # fi # ;; * ) echo "Incorrect parameters $@. See help." show_help ;; esac exit 0
You can find more informations and instructions (in portuguese) at http://www.kyapanel.com/wiki/doku.php?id=zimbra:zmbkpose