Home » Blogs » Kiran's blog

Automatically backup your website to Dropbox

25 Nov 2012
Posted by Kiran

Until recently I have been periodically taking manual backups of the content on my website. However given that life has been extremely busy in recent times, I have often missed my backup schedule. That I had to login onto my CPanel to take the backups, then manually download them - all time consuming activities - did not help. I kept deferring my backups reasoning to myself that I hadn't made any recent updates in any case.

One day however, I found that some comments that had been left on my blogs were missing. I tried to investigate but couldn't figure out what had happened. I then encountered some missing files - again without explanation. Then my website just rolled over and died! Every time I got unsatisfactory explanations from my webhost: a planned server migraton had been cancelled; they had performed some security updates at the server level; etc. All the time they assured me that my content should be intact - well, I felt different.

That was when I decided that an Automated Backup solution was a necessity. I needed something that had minimal necessity of manual intervention.

I started with a search on Google and found various scriptlets that would help. The most promising was one I found at LifeHacker

I used the script they had provided on that article as a base and decided to build on it to make it more configurable.

The script I ended up building was:

Script: databackup.sh

  1. #!/bin/sh
  2. #
  3. # Script: databackup.sh
  4. # Version: 1.2
  5. #
  6. # Purpose:
  7. # Perform an Automatic Backup based on inputs provided in a Config file.
  8. #
  9. # License: LGPLv2
  10. # Author: Kiran J. Holla
  11. #         http://www.kiranjholla.com/
  12. #
  13. # Description:
  14. #    This script provides a utility that can be used to automatically backup various
  15. #    elements of a website. Individual TAR files are created as per configuration and
  16. #    then optionally uploaded to Dropbox.
  17. #
  18. #    The upload to Dropbox functionality makes use of the script provided by Andrea Fabrizi
  19. #    at http://www.andreafabrizi.it/?dropbox_uploader
  20. #
  21. #
  22. # Modification Log:
  23. #    v1.0      Nov 25, 2012    Initial version
  24. #
  25. #    v1.1      Jan 14, 2013    Fixed some bugs where the usage was getting printed multiple
  26. #                              times and the number of days for the directory backup was
  27. #                              yielding negative numbers when the year changes.
  28. #
  29. #    v1.2      Mar 16, 2013    Included the --create-options option in the mysqldump command
  30. #                              to ensure that the auto increment parameter is not skipped in
  31. #                              dump file.
  32. #
  33.  
  34. print_usage()
  35. {
  36.    # Function to print the Usage instructions for this script.
  37.    #
  38.    echo "Usage:"
  39.    echo "   databackup.sh <Backup Name> <Param 1> [<Param 2> [<Param 3> [. . .]]]"
  40.    echo " "
  41.    echo "   Backup Name: The name by which the consolidated backup should be named. A file by name"
  42.    echo "                BACKUP_<Backup Name>_<Date & Time>.tar is created."
  43.    echo " "
  44.    echo "    Parameters: Each parameter marks a set of Backup instructions that need to be documented"
  45.    echo "                in the .databackup_config file that should exist in the same directory as this"
  46.    echo "                script."
  47.    echo " "
  48.    echo "Sample Configuration File Contents:"
  49.    echo "   PARAM1:BKPTYPE:DIR"
  50.    echo "   PARAM1:BKPFORI:FULL"
  51.    echo "   PARAM1:BKPATTR1:WebFiles"
  52.    echo "   PARAM1:BKPATTR2:/home/user/httpd/www"
  53.    echo "   PARAM1:BKPATTR3:/home/user/backups"
  54.    echo " "
  55.    echo "   PARAM2:BKPTYPE:DB"
  56.    echo "   PARAM2:BKPFORI:FULL"
  57.    echo "   PARAM2:BKPATTR1:WebDatabase"
  58.    echo "   PARAM2:BKPATTR2:my_database"
  59.    echo "   PARAM2:BKPATTR3:/home/user/backups"
  60.    echo " "
  61.    echo "Then, run the script as below:"
  62.    echo "   databackup.sh WebBackup PARAM1 PARAM2"
  63.    echo " "
  64.  
  65. }
  66.  
  67. backup_db()
  68. {
  69.    # Function to dump a MYSQL database and then package the resulting
  70.    # SQL file in a tar.
  71.    #
  72.    # This function assumes that the configuration parameter ${DB_BACKUP_USER}
  73.    # is set to any user name that possesses sufficient privileges on the DB
  74.    # being backed up using mysqldump.
  75.    #
  76.    # This function further assumes that a .my.cnf file has been created and
  77.    # placed in the home directory of the user running this script and that
  78.    # file contains the correct password for the user name being used.
  79.    #
  80.  
  81.  
  82.    DB_BACKUP_USER=`grep 'DB_BACKUP_USER' ${CNFGFILE} | cut -d":" -f2`
  83.    DB_HOST=`grep 'DB_HOST' ${CNFGFILE} | cut -d":" -f2`
  84.  
  85.    echo "Dumping Database " >> ${MAILFILE}
  86.    echo " " >> ${MAILFILE}
  87.  
  88.    mysqldump -u ${DB_BACKUP_USER} -h ${DB_HOST} --skip-opt --add-drop-table --create-options --complete-insert --extended-insert --single-transaction --result-file="${BKPATTR3}/BKPFULL_${BKPTYPE}_${BKPATTR1}_${TMTODAY}.sql" ${BKPATTR2}
  89.  
  90.    cd ${BKPATTR3}
  91.    tar --create --verbose --file "BKPFULL_${BKPTYPE}_${BKPATTR1}_${TMTODAY}.tar" "BKPFULL_${BKPTYPE}_${BKPATTR1}_${TMTODAY}.sql"
  92.  
  93.    echo " " >> ${MAILFILE}
  94.  
  95.    if [ -f "${BKPATTR3}/BKPFULL_${BKPTYPE}_${BKPATTR1}_${TMTODAY}.tar" ]
  96.    then
  97.       echo "Database backed up to ${BKPATTR3}/BKP${BKPTYPE}_${BKPATTR1}_${TMTODAY}.tar" >> ${MAILFILE}
  98.  
  99.       rm "${BKPATTR3}/BKPFULL_${BKPTYPE}_${BKPATTR1}_${TMTODAY}.sql"
  100.       return 0
  101.    else
  102.       echo "Error in Database backup." >> ${MAILFILE}
  103.       return 1
  104.    fi
  105.  
  106. }
  107.  
  108. backup_dir()
  109. {
  110.    # Function to backup a physical directory. There are two options;
  111.    #
  112.    #    1: Full Backup - this option creates a full backup for the
  113.    #       entire directory including any subdirectories within it.
  114.    #
  115.    #    2: Incremental Backup - this option takes a backup of only
  116.    #       those files that were modified after the LASTRUN date
  117.    #
  118.  
  119.    echo "Backing up directory ${BKPATTR2}" >> ${MAILFILE}
  120.    BKPOK=1
  121.  
  122.    LASTRUN=`grep ${BKPMODE} ${CNFGFILE} | grep 'LASTRUN' | cut -d":" -f3`           #Last Run Date in YYYYMMDD
  123.  
  124.    if [ "${BKPFORI}" = "FULL" ] || [ -z ${LASTRUN} ]
  125.    then
  126.       # Perform Full Backup
  127.  
  128.       tar --create --verbose --recursion --file ${BKPATTR3}/BKP${BKPFORI}_${BKPTYPE}_${BKPATTR1}_${TMTODAY}.tar ${BKPATTR2}
  129.  
  130.       BKPOK=$?
  131.       echo "Tar Return Status ${BKPOK}" >> ${MAILFILE}
  132.  
  133.    else
  134.       # Perform Incremental Backup
  135.  
  136.       echo "Backup was last run on ${LASTRUN}" >> ${MAILFILE}
  137.       DAYSLAST=$((($(date -u -d "${DTTODAY}" +%s) - $(date -u -d "${LASTRUN}" +%s)) / 86400))
  138.  
  139.       echo "Days since last run ${DAYSLAST}" >> ${MAILFILE}
  140.  
  141.  
  142.       #Create an empty TAR file, which can then be used to add the modified files
  143.       tar --create --verbose --file ${BKPATTR3}/BKP${BKPFORI}_${BKPTYPE}_${BKPATTR1}_${TMTODAY}.tar
  144.       find ${BKPATTR2} -type f -mtime -${DAYSLAST} -exec tar --append --verbose --dereference --file ${BKPATTR3}/BKP${BKPFORI}_${BKPTYPE}_${BKPATTR1}_${TMTODAY}.tar "{}" ;
  145.  
  146.       BKPOK=$?
  147.       echo "Tar Return Status ${BKPOK}" >> ${MAILFILE}
  148.  
  149.    fi
  150.  
  151.    if [ ${BKPOK} -eq 0 ]
  152.    then
  153.  
  154.       echo " " >> ${MAILFILE}
  155.       echo "TAR created." >> ${MAILFILE}
  156.  
  157.       #Backup has been successful
  158.       #Replace the LASTRUN date in Config file with today's date
  159.       cat ${CNFGFILE} | grep -v "${BKPMODE}:LASTRUN" > ${CNFGTEMP}
  160.       echo "${BKPMODE}:LASTRUN:${DTTODAY}" >> ${CNFGTEMP}
  161.       cat ${CNFGTEMP} > ${CNFGFILE}
  162.  
  163.       rm -f ${CNFGTEMP}
  164.  
  165.       echo "Backed up the directory." >> ${MAILFILE}
  166.       return 0
  167.    else
  168.       echo "Error while backing up directory!" >> ${MAILFILE}
  169.       return 1
  170.    fi
  171. }
  172.  
  173. export CNFGFILE
  174. export CNFGTEMP
  175. export MAILFILE
  176. export TMTODAY
  177. export DTTODAY
  178. export BKPFORI
  179. export BKPATTR1
  180. export BKPATTR2
  181. export BKPATTR3
  182. export BKPMODE
  183.  
  184. SCRPTDIR=`dirname $0`
  185. CNFGFILE=${SCRPTDIR}/.databackup_config
  186. CNFGTEMP=${SCRPTDIR}/.databackup_conftemp
  187. MAILFILE=${SCRPTDIR}/mailfile_temp.txt
  188. MAILADDR=`grep 'BKP_MAIL_RECIPIENT' ${CNFGFILE} | cut -d":" -f2`
  189.  
  190.  
  191. trap 'cat ${MAILFILE} | mail -s "Backup Error!" ${MAILADDR}; rm -f ${MAILFILE}; rm -f ${CNFGTEMP}; exit' 1 2 3 15
  192.  
  193. TMTODAY=`date +%Y%m%d%H%M%S`
  194. DTTODAY=`date +%Y%m%d`
  195.  
  196.  
  197.  
  198. # The name that is to be used to create the consolidated backup file
  199. if [ "$1" ]
  200. then
  201.    BKPNAME=${1}
  202.    shift
  203. else
  204.    print_usage >> ${MAILFILE}
  205.    exit 1
  206. fi
  207.  
  208. if [ "$1" ]
  209. then
  210.    echo "Running Backup at ${TMTODAY} for ${BKPNAME}" > ${MAILFILE}
  211.    echo " " >> ${MAILFILE}
  212.  
  213.    while [ "$1" ]
  214.    do
  215.  
  216.       echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~" >> ${MAILFILE}
  217.       echo "Running for $1 . . ." >> ${MAILFILE}
  218.  
  219.       BKPMODE=${1}
  220.       BKPTYPE=`grep ${BKPMODE} ${CNFGFILE} | grep 'BKPTYPE' | cut -d":" -f3`           #DB or DIR
  221.       BKPFORI=`grep ${BKPMODE} ${CNFGFILE} | grep 'BKPFORI' | cut -d":" -f3`           #Full or Incremental
  222.       BKPATTR1=`grep ${BKPMODE} ${CNFGFILE} | grep 'BKPATTR1' | cut -d":" -f3`         #Site Name
  223.       BKPATTR2=`grep ${BKPMODE} ${CNFGFILE} | grep 'BKPATTR2' | cut -d":" -f3`         #DB or DIR
  224.       BKPATTR3=`grep ${BKPMODE} ${CNFGFILE} | grep 'BKPATTR3' | cut -d":" -f3`         #Backup Directory
  225.  
  226.       echo "BKPTYPE = ${BKPTYPE}" >> ${MAILFILE}
  227.       echo "BKPFORI = ${BKPFORI}" >> ${MAILFILE}
  228.       echo "BKPATTR1 = ${BKPATTR1}" >> ${MAILFILE}
  229.       echo "BKPATTR2 = ${BKPATTR2}" >> ${MAILFILE}
  230.       echo "BKPATTR3 = ${BKPATTR3}" >> ${MAILFILE}
  231.  
  232.       echo " " >> ${MAILFILE}
  233.  
  234.       if [ "$BKPTYPE" ]
  235.       then
  236.          case $BKPTYPE
  237.          in
  238.             "DB") backup_db ;;
  239.             "DIR") backup_dir ;;
  240.          esac
  241.       fi
  242.  
  243.       shift
  244.    done
  245.  
  246.    echo "Creating Consolidated Backup TAR . . ." >> ${MAILFILE}
  247.  
  248.    cd ${BKPATTR3}
  249.    tar --create --verbose --file BACKUP_${BKPNAME}_${TMTODAY}.tar BKP*${TMTODAY}.tar
  250.  
  251.    if [ -f ${BKPATTR3}/BACKUP_${BKPNAME}_${TMTODAY}.tar ]
  252.    then
  253.       echo "Consolidated Backup TAR created successfully!" >> ${MAILFILE}
  254.       echo " " >> ${MAILFILE}
  255.       echo "Removing intermediate backups . . ." >> ${MAILFILE}
  256.  
  257.       rm ${BKPATTR3}/BKP*${TMTODAY}.tar >> ${MAILFILE}
  258.  
  259.       cd ${SCRPTDIR}
  260.  
  261.       if [ -x ./dropbox_uploader.sh ]
  262.       then
  263.          echo "Backing up ${BKPATTR3}/BACKUP_${BKPNAME}_${TMTODAY}.tar to Dropbox. . ." >> ${MAILFILE}
  264.          ./dropbox_uploader.sh upload ${BKPATTR3}/BACKUP_${BKPNAME}_${TMTODAY}.tar BACKUP_${BKPNAME}_${TMTODAY}.tar >> ${MAILFILE}
  265.          if [ $(grep -c 'DONE' ${MAILFILE}) -ne 0 ]
  266.          then
  267.             rm ${BKPATTR3}/BACKUP_${BKPNAME}_${TMTODAY}.tar
  268.             echo "Removed Consolidated TAR after upload to Dropbox" >> ${MAILFILE}
  269.          fi
  270.       else
  271.          echo "Dropbox uploader not found. Skipping Dropbox backup!" >> ${MAILFILE}
  272.       fi
  273.    fi
  274. else
  275.    print_usage >> ${MAILFILE}
  276.    exit 1
  277. fi
  278.  
  279. cat ${MAILFILE} | mail -s "Backup Completed!" ${MAILADDR}
  280.  
  281. rm -f ${MAILFILE}
  282. rm -f ${CNFGTEMP}

This script is capable of performing the following types of backups of a Linux-MySQL based website:

  1. Full Directory Backup
  2. Incremental Directory Backup
  3. Full Database Backup

You can use the script to perform multiple such backups and package all of them together into a consolidated TAR file. This consolidated backup can be uploaded to Dropbox using the dropbox_uploader.sh provided by Andrea Fabrizi.

To setup the automated backup using this script, simply follow the steps below:

1. Download the script and sample configuration file

This script and the associated configuration file can be downloaded from here. Once you have the script and associated configuration file, upload them on to a directory on your server. Don't forget to make the file databackup.sh executable.

chmod +x databackup.sh

2. Setup the databackup configuration file

You can setup the script to run multiple types of backups. Each backup needs to be identified by a Backup Parameter. This Backup Parameter can be any random string that you choose. It is only used to identify the group of configuration parameters that are associated with the particular backup you want the script to execute. The Backup Parameters need to be sent to the script as arguments on the command line.

For each Backup Parameter the script looks for the following Configuration Parameters:

  1. BKPTYPE - The backup type; it could be a directory or a database (DIR or DB)
  2. BKPFORI - Full backup or Incremental (FULL or INCR). This configuration parameters is irrelevant and is ignored for database backups; it is considered only when the BKPTYPE is DIR
  3. BKPATTR1 - the individual backup name; this is included in the individual TAR files to help you distinguish what files are in the TAR
  4. BKPATTR2 - The backup source. In case of a DIR backup, this would be the directory that needs to be backed up. In case of a DB backup, this would be the database that needs to be backed up.
  5. BKPATTR3 - The backup destination. The directory where the consolidated backup TAR files should be stored.

In addition, you will need to also setup a couple of generic parameters within the configuration file:

  1. DB_BACKUP_USER - A MySQL database user having the privileges to run the mysqldump command.
  2. DB_HOST - The hostname of the database server. In most cases, this would be localhost
  3. BKP_MAIL_RECIPIENT - An email id where the resulting log file should be sent to after the backup script completes.

The various parameters are delimited by the character ":".

A sample configuration file could be:

TEST_DIR_FULL:BKPTYPE:DIR
TEST_DIR_FULL:BKPFORI:FULL
TEST_DIR_FULL:BKPATTR1:Test
TEST_DIR_FULL:BKPATTR2:/user/test/httpd/web
TEST_DIR_FULL:BKPATTR3:/user/backup

TEST_DB_FULL:BKPTYPE:DB
TEST_DB_FULL:BKPFORI:FULL
TEST_DB_FULL:BKPATTR1:Test
TEST_DB_FULL:BKPATTR2:the_db_name
TEST_DB_FULL:BKPATTR3:/user/backup

DB_BACKUP_USER:backup_db_username
DB_HOST:servername
BKP_MAIL_RECIPIENT:emailid@example.com

Once the configuration file has been setup, running the script would follow the format as below:

databackup.sh BackupName BackupParam1 BackupParam2 BackupParam3

Here:

  • BackupName is a generic name that is given to the entire consolidated backup that is created.
  • BackupParam1, BackupParam2, are all individual Backup Parameters. They all need to have the 5 configuration parameters defined within the .databackup_config configuration file.

3. Setup access parameters for the database access

One of my key considerations while writing this script was that I did not want to have the database password included in the call to the mysqldump command. This was because there are chances that the command line parameters that are used while calling mysqldump could appear in server log files.

To better control this, I decided to use the approach suggested at Techie Corner to setup a mysqldump command without passing in the password using the -p option.

For setting up the access parameters for the database, please create a .my.cnf file within your home directory as follows:

[mysqldump]
user = backup_db_username
password = some_complicated_password

Please take care to ensure that the user mentioned in this file is the same as the one specified against the parameter DB_BACKUP_USER in step 2 above.

4. Setup the Dropbox Uploader script and configuration

If you want the functionality to upload the backup to Drobox, you will also need to download the scripts from Andrea's website. Andrea provides a script named dropbox_uploader.sh that is capable of uploading files from Linux to Dropbox. He also provides detailed setup instructions.

Using shell, manually run the dropbox_uploader.sh script once. This will prompt you through a series of configuration steps that help you setup the script as a Dropbox application on your Dropbox account; you will then authorize it to access your Dropbox account.

If you don't have shell access this can be more tricky. To get around this limitation, I edited the dropbox_uploader.sh script and piped commands at appropriate stages within the script to the mail program, thus mailing the output to myself. I then ran the script via cron and waited for the mail to arrive and then took the recommended steps. This approach however is not for everyone as it needs a fair degree of expertise in shell-scripting and technical know-how to understand what's happening.

5. Setting up a cron to run this script

The script by itself solves only half the problem. To be a truly automatic solution, it needs to be run without our intervention. To do this, we simply setup the script to run this as part of the crontab.

0 1 * * * /home/path/to/script/databackup.sh MyWebBackup BKPPARM1 BKPPARAM2 BKPPARAM3

That's about it folks. If everything has been setup correctly, the directories and databases configured in Step 2 should get backed up into a TAR file such as BAKUP_MyWebBackup_20121125170101.tar and uploaded to your Dropbox account.

Feel free to use the scripts provided. If you have any questions, do leave a comment and I will do my best to answer them. If you have any comments about these scripts, I would love to hear them — do leave a comment.

Disclaimer:
At this stage, I would like to remind you that I am providing these scripts with no warranties. Please use your own judgement and discretion before downloading and making use of these scripts. I will not accept liabilities should anything unexpected happen.

Tags: