Harley77-Scripts
Preface I have no formal training in bash scripting. I learn as I go, so you can no doubt find all sorts of wonderful things wrong with my scripts. By all means, make comments and suggestions, just don't get all insulting because I will just ignore it.
Scripts I use for Backing up Zimbra
The following is cut and pasted from bash script files I created to assist in setting up automated backups. They are overkill for what I use them for in Zimbra, but, it is still very effective and maby someone else out there can use them as well.
I also wanted to separate this from my notes page so I could just reference it instead of having all this code in that page.
3rbackup
Remote Rsync and Rotation backup. I spent so much time making this I needed to title it.
The system is broken up into multiple files.
The Actual Scripts
- 3rbackup.sh (The Main File)
- setup3r.sh (Quick and Easy setup file)
- masterbackup.run (use this to wrap multiple backups into a single script)
Variable Files BACKUPVAR.TXT (The name will vary depending on what you name your backup) BACKUPEXCLUDES.TXT (See above) override.txt (allows you to place this file into the BACKUP config directory and override settings without modifying the actual VAR file.)
The first step is using setup3r.sh to create a backup scheme. It will walk you through setting up source and target directories, the login, passwordless keyfile, etc. Once it's done and if all went well, you would have a fully functional script you could run to sync a remote directory to a local directory and have it rotate if you wish. For zimbra, we won't be using the rotation, so during setup, you would choose 0 when it asks how many days do you want to keep in rotation.
The setup3r.sh will give you the option in the beginning to do SSH, SAMBA, or NOVELL backups. They all work to a degree, but SSH is by far the simplest, there are quirks to the other two that I'm not going into right now.
3rbackup.sh The Code
#!/bin/bash # Remote Rsync and Rotation # Brian Davidson bsdavidson@gmail.com # v2.5.9 01/28/2008 # # For SSH use, this script works best when a public key is shared between the two computers # that are involved. See my sshkeygencopy.sh script to assist in this or use 3rsetup.sh # that will walk you through the complete process of setting up a new machine to be backed up. # # Credit to www.mikerubel.org for inspiration and technique. # # Variables that this script uses are pulled from a separate file. Type the path and file name # as an argument when running this script. ex. ./3rbackup.sh /backup/mailserver/var.txt # For logging to a file, add a 1 on the end of the command string. ex. ./3rbackup.sh /backup/mailserver/var.txt 1 # # Run setup3r.sh to create a variable file, otherwise, # A Fill-in-the-blank copy of the Variable file can be found under the file: blankvar.txt # Enable the next line for debugging #set -x # ****************************** Program and Path Setup ********************************** # The defaults should work for most cases, but you never know. They're here for your convenience. # Path to Rsync (VERY IMPORTANT!!!) RSYNC="/usr/bin/rsync" # Path to Mail program MAIL="/usr/bin/mail" # Path to samba mount MOUNT="/bin/mount" # Path to Novell Mount and ipx_configure NCPMOUNT="/usr/bin/ncpmount" IPXCONFIGURE="/usr/bin/ipx_configure" # Path to umount command UMOUNT="/bin/umount" # SSH Program path SSH="/usr/bin/ssh" # The following sets the results log directory. It is shared among all backups and is used to store # the overall pass/fail log of a days backups. It's usefull when running a series of diffrent backups in a given day/night # since you can see in one log whether there were problems with any of them. # ex. RESULTSLOGDIR="/BACKUP/resultslog" RESULTSLOGDIR="/BACKUP/resultslog" # This is the email address that will appear in the FROM part of any emails sent to the backup administrator. SERVERMAIL="email@youremail.com" # ****************************** END Program and Path Setup ********************************** # _____________________________________________________________________________ # The following should not need to be altered if [ -f /root/coldstandby ]; then echo "server is in cold standby mode, proceeding with backup" else echo "The server is in Live operation, backup aborted." exit 127 fi RUNTIMECHECK=1 if [ "$2" = "" ]; then clear fi # Checking out the argument to make sure its all good. if ! [ "$1" ]; then echo "Please specify a config file to load!" exit 127 fi if [ -f $1 ]; then echo "Config File OK" else echo "Invalid config file specified!" exit 127 fi # Loading the variables.. source $1 # Loading any override variables if [ -f "override.txt" ]; then echo "Loading any override vars" source override.txt fi #This is used for defining log files. logdate=`date +'%m%d%Y'`; logtime=`date +'%R'`; # Results Log - Records the overall summary at the end of the backup. # Basically a single line that represents a Pass or Fail. # Its filename is based on the date so any backups that are run on the same date will append to this file. RESULTSLOG="$RESULTSLOGDIR/r$logdate.log" if [ -d $RESULTSLOGDIR ]; then echo "Skipping Resultlog directory creation (already present)" else mkdir $RESULTSLOGDIR fi echo "Writing Backup Result log to $RESULTSLOG" touch $RESULTSLOG # This determines if the detailed log output goes to screen or a file. If "1" is used as the second variable when running # the backup command, then it is all logged to a file, else, it goes to the screen with no logfile kept. if [ "$2" = "1" ]; then LOGFILE="$LOGDIR/$BUNAME$logdate.log" LOGERR="$LOGDIR/error$BUNAME$logdate.log" echo "Logging to file $LOGDIR/$BUNAME$logdate.log" # IO redirection for logging. touch $LOGFILE exec 6>&1 # Saves stdout. exec > $LOGFILE touch $LOGERR exec 7>&2 # Saves stderr. exec 2> $LOGERR else echo "Logging output to screen" fi echo "-------- Starting Backup for $BUNAME ------------" if [ "$SNAPSHOTNUM" = "0" ]; then RTARGET=""$BUDIR"/" else RTARGET=""$BUDIR"/snapshot.0/" # The following should auto create the snapshot.0 directory if it doesn't exist. # Its not foolproof so I have an additional check later to make sure # everything is all kosher. if [ -d $BUDIR ]; then if [ -d $RTARGET ]; then echo "Snapshot creation not needed" else mkdir $RTARGET fi fi fi if [ $BUTYPE = "SAMBA" ]; then echo "Mounting SAMBA Source" $UMOUNT "$SMBMOUNT" 2> /dev/null $MOUNT -t smbfs -o username=$USERNAME,password=$PASSWORD $SSOURCE $SMBMOUNT if [ "$?" = "0" ]; then echo "Samba Mounted OK" echo " -------------- $SMBMOUNT directory listing -----------------------" ls $SMBMOUNT echo "------------------------------------------------------------------ Testing individual sources to be backed up... " # Loop though the sources and test each one NUMCOUNT=1 eval SRCLIST=\$SOURCE$NUMCOUNT cd $SMBMOUNT until [ "$SRCLIST" = "" ] do if [ -d "$SRCLIST" ]; then echo "$SRCLIST OK" else echo "$SRCLIST FAIL" SMBERRORDETECT=1 fi NUMCOUNT=`expr $NUMCOUNT + 1` eval SRCLIST=\$SOURCE$NUMCOUNT done else SMBERRORDETECT=1 fi fi if [ $BUTYPE = "NOVELL" ]; then echo "Mounting NOVELL Source" $UMOUNT "$SMBMOUNT" 2> /dev/null # Pre-connection stuff to help ensure the novell mount works properly. $IPXCONFIGURE --auto-primary=on $IPXCONFIGURE --auto-interface=on echo "Waiting 5 Seconds for IPX" sleep 5 $NCPMOUNT -S "$SSOURCE" -U "$USERNAME" -P "$PASSWORD" "$SMBMOUNT" if [ "$?" = "0" ]; then echo "Novell Mounted OK" echo " -------------- $SMBMOUNT directory listing -----------------------" ls $SMBMOUNT echo "------------------------------------------------------------------ Testing individual sources to be backed up... " # Loop though the sources and test each one NUMCOUNT=1 eval SRCLIST=\$SOURCE$NUMCOUNT SRCLIST2=$SMBMOUNT$SRCLIST cd $SMBMOUNT until [ "$SRCLIST" = "" ] do if [ -d "$SRCLIST2" ]; then echo "$SRCLIST OK" else echo "$SRCLIST FAIL" NOVERRORDETECT=1 fi NUMCOUNT=`expr $NUMCOUNT + 1` eval SRCLIST=\$SOURCE$NUMCOUNT SRCLIST2=$SMBMOUNT$SRCLIST done else NOVERRORDETECT=1 fi fi echo " " # Checking to make sure some of the paths and variables are valid echo "******************* SELF TESTING ************************" if [ -d $BUDIR ] ; then echo "Backup Directory: OK" else echo "Backup Directory: FAIL" ERRORDETECT=1 fi if [ -d $RTARGET ] ; then echo "Target Directory: OK" else echo "Target Directory: FAIL" ERRORDETECT=1 fi if [ -d $LOGDIR ] ; then echo "Log Directory: OK" else echo "Log Directory: FAIL" ERRORDETECT=1 fi if ! [ "$EXCLUDES" ]; then echo "Excludes File: N/A" EXCLUDESCMD="" else if [ -f $EXCLUDES ] ; then echo "Excludes File: OK" EXCLUDESCMD="--exclude-from="$EXCLUDES"" else echo "Excludes File: FAIL" ERRORDETECT=1 fi fi if ! [ "$INCLUDES" ]; then echo "Includes File: N/A" INCLUDESCMD="" else if [ -f $INCLUDES ] ; then echo "Includes File: OK" INCLUDESCMD="--include-from="$INCLUDES"" else echo "Includes File: FAIL" ERRORDETECT=1 fi fi if [ $BUTYPE = "SSH" ] ; then if [ ! -f $RKEY ] ; then echo "SSH KEY: FAIL" ERRORDETECT=1 else if ! $SSH -2 -T -i $RKEY "$USERNAME"@$RHOST "test -x ~/"; then echo "SSH LOGIN: FAIL" ERRORDETECT=1 else echo "SSH LOGIN: OK" fi fi fi if [ "$BUTYPE" = "SAMBA" ] ; then if [ "$SMBERRORDETECT" = "1" ]; then echo "Samba Connection: FAIL" ERRORDETECT=1 else echo "Samba Connection: OK" fi fi if [ "$BUTYPE" = "NOVELL" ] ; then if [ "$NOVERRORDETECT" = "1" ]; then echo "Novell Connection: FAIL" ERRORDETECT=1 else echo "Novell Connection: OK" fi fi echo "******************* END TESTING ************************ " # If any errors are detected, it will add it to the Overall backup log and also create or append to an existing # Error Log file. At the end of the backup, it will send two emails to the admin, one that indicates a backup error # and another that gives the overall listing of Pass/Fail. if [ "$ERRORDETECT" = "1" ] ; then echo "Errors were detected, please check the error log above and verify" echo "that your settings are correct" if [ "$2" = "1" ]; then echo "$logtime - Backup $BUNAME Error - FAILED TO START " >> $RESULTSLOG echo "$logtime - Backup $BUNAME Error - FAILED TO START " >> $LOGERR fi exit 1 else echo "ALL PASS" fi # END CHECKS #****************************************************************** #----------------------Backup Rotation----------------------------- #****************************************************************** #Only run if snapshot count is 1 or higher if [ "$SNAPSHOTNUM" = "0" ]; then echo "Backup set for NO rotation, skipping..." else DAYS=$SNAPSHOTNUM echo "Backup is configured for a $DAYS day rotation" echo "Begin Rotation Procedure ..... " if [ -d $BUDIR/snapshot.$DAYS ] ; then echo "Deleting oldest snapshot" rm -Rf $BUDIR/snapshot.$DAYS DAYS=`expr $DAYS - 1` else DAYS=`expr $DAYS - 1` fi ; echo "Verifying/Creating Snapshot directories" until [ "$DAYS" = "0" ] do if [ -d $BUDIR/snapshot.$DAYS ] ; then echo "snapshot.$DAYS exists" else echo "snapshot.$DAYS Doesnt exist, Creating.. " mkdir $BUDIR/snapshot.$DAYS ; fi ; DAYS=`expr $DAYS - 1` done #Reset DAYS Variable DAYS=$SNAPSHOTNUM DAYS=`expr $DAYS - 1` echo "Shifting snapshots to their older slots" until [ "$DAYS" = "0" ] do echo "snapshot.$DAYS ==> snapshot.`expr $DAYS + 1`" mv $BUDIR/snapshot.$DAYS $BUDIR/snapshot.`expr $DAYS + 1` ; DAYS=`expr $DAYS - 1` done # I'm using a hard link as opposed to the --link-dest method since the results have proven (at least in my tests) # to be more reliable and consistant with its behavior. echo "Making a hard link from snapshot.0 to snapshot.1" if [ -d $BUDIR/snapshot.0 ] ; then cp -al $BUDIR/snapshot.0 $BUDIR/snapshot.1 ; fi ; echo "Rotation Complete" fi # --------------------- Rotation Complete ------------------------ # Time to compile the sources into a single variable that's rsync compatible. # reseting the count NUMCOUNT=1 # Establish our first entry and # Set the initial Source Directory in the Var eval SRCLIST=\$SOURCE$NUMCOUNT # Replace the space with a ? for compatibility with rsync (applies to Novell and Samba only). if [ $BUTYPE == "SAMBA" -o $BUTYPE == "NOVELL" ]; then SRCLIST=`echo "$SRCLIST" | sed s'/ /?/'` fi # Adding so if the source has a space, rsync won't have a fit (SSH ONLY). if [ $BUTYPE == "SSH" ]; then SRCLIST="'$SRCLIST'" fi # This is the first SOURCE Directory. SOURCEVAR="$SMBMOUNT/$SRCLIST" # The Idea here is to take each source (SOURCE1, SOURCE2, SOURCE3, ect) and append it to a single SOURCEVAR. until [ "$SRCLIST" = "" ] do # Since we have already established SOURCEVAR1 earlier, I'm adding a 1 to make SOURCEVAR2. NUMCOUNT=`expr $NUMCOUNT + 1` # I'm now refrencing the Next SOURCE in the list. eval SRCLIST=\$SOURCE$NUMCOUNT # echo "$SRCLIST" # If the SOURCEVAR does NOT exist, I know that we have reached the end of the list. Otherwise, I will have to modify # the path a little so it will work properly when RSYNC gets ahold of it. if [ "$SRCLIST" != "" ]; then if [ $BUTYPE == "SAMBA" -o $BUTYPE == "NOVELL" ]; then # Replace the space with a ? for compatibility with rsync (applies to Novell and Samba only). SRCLIST=`echo "$SRCLIST" | sed s'/ /?/'` fi if [ $BUTYPE == "SSH" ]; then # Adding so if the source has a space, rsync won't have a fit SRCLIST="'$SRCLIST'" fi # I now take the existing SOURCEVAR and append the next SOURCE SOURCEVAR="$SOURCEVAR $SRCLIST" fi done # Added to show the sources as rsync will see it. Good for debugging and show you that it did things right. echo "Compiled Source(s):" echo "$SOURCEVAR " # ------------------------- Compile complete -------------------------- # ------------------------- Backup Actually Begins -------------------------- echo "Performing Sync. This may take a few minutes to several hours depending on data size." echo "Rsync Started `date`" if [ "$BUTYPE" = "LOCAL" ] ; then $RSYNC $EXCLUDESCMD $RVAR $SOURCEVAR $RTARGET fi if [ "$BUTYPE" = "SAMBA" ] ; then cd $SMBMOUNT $RSYNC $EXCLUDESCMD $RVAR $SOURCEVAR $RTARGET # This backs us out of the directory so it can be dismounted. cd / $UMOUNT "$SMBMOUNT" fi if [ "$BUTYPE" = "NOVELL" ] ; then cd $SMBMOUNT $RSYNC $EXCLUDESCMD $RVAR $SOURCEVAR $RTARGET # This backs us out of the directory so it can be dismounted. cd / $UMOUNT "$SMBMOUNT" fi if [ "$BUTYPE" = "SSH" ] ; then # A fine looking command if I say so myself. $RSYNC $EXCLUDESCMD $RVAR -e "$SSH -T -i $RKEY" "$USERNAME"@$RHOST:"$SOURCEVAR" "$RTARGET" fi # ------------------------- Backup Actually Ends -------------------------- # Setting the permissions to make it web-readable. I use this in combination with a PHP Filemanager so I can login from any workstation # to download a needed file. It's not required for anything other than that, so if you don't need it you can set # the option CHMODDIR in the var file to "1". #if [ "$CHMODDIR" = "" ]; then # chmod -R 755 $RTARGET #fi # Make an empty file with the name of the date/time to identify the date the snapshot was made. # This is just to have a hard indicator of the actual date/time if the file/dir timestamp gets messed up. # It can be excluded in the VAR file by setting NOTIMEFILE="1" if [ "$NOTIMEFILE" = "" ]; then echo "Creating timestamp file" rm -Rf $RTARGET/backupdate* touch $RTARGET/"backupdate$logdate-$logtime" echo "Rsync Ended `date`" fi # This is the end-of-backup log handler. if [ "$2" = "1" ]; then if [ -s "$LOGERR" ]; then echo "Error Reported!" echo "$logdate $logtime - BACKUP $BUNAME ERROR. LOGGED ERRORS DURING BACKUP" >> $RESULTSLOG echo "`cat $LOGERR`" | mail -s "$BUNAME ERROR REPORTED" -r "$SERVERMAIL" $NOTIFY else echo "No Errors" echo "$logdate $logtime - BACKUP $BUNAME SUCCESSFUL" >> $RESULTSLOG rm -Rf $LOGERR fi fi # The END. I hope you had a great time, I know I did. exit 0