Category Archives: amazon

Configure Amazon S3 backup with Backupninja and duplicity

I’m responsible for launching a new web page for Pedalibre, one of the oldest and biggest Madrid cyclist associations. It uses Drupal as CMS running on Apache httpd and storing everything in an MySQL database. We launched a proof-of-concept version on Amazon EC2 in the cheapest instance (micro, US-east zone). Works smoothly. Now a content work group must migrate the articles from the old site to the new one. Before that, I need a backup system.

My backup plan was the following:

  • create an image (EC AMI). In case of instance crash, I can restore the instance from the AMI and apply backups
  • backup Apache and MySQL configuration (in principle the entire /etc)
  • backup Drupal installation directory (/var/www)
  • backup Drupal database

I checked Google, but I didn’t find any article or a blog post describing how to setup a complete backup using Amazon ecosystem and Amazon Simple Storage Service (S3) in particular. I had to mix various articles and finally reached a solution.

What I used

  • Backupninja – for MySQL database backup and backup orchestration
  • duplicity – filesystem backup (Amazon S3 capable)
  • dt-s3-backup (check also author’s blog post) – shell script performing a backup to an S3 bucket using duplicity
  • s3cmd – command line tool for managing data in Amazon S3; required by dt-s3-backup

Step 1. Install the tools

apt-get install backupninja s3cmd duplicity
download zip from http://github.com/thornomad/dt-s3-backup/zipball/master
extract dt-s3-backup.sh script

Remeber to run everything as super user.

Step 2. Configure Backupninja schedule

Check Backupninja configuration (/etc/backupninja.conf) if it suits your needs. The most important parameter is when – it controls the backup schedule (check the documentation for possible values). I left it with the default value (everyday at 01:00). If you want to get backup report, fill at least reportemail parameter value.

Step 3. Backup MySQL database

Create a Backupninja backup action for MySQL database dump. The easiest way is to start ninjahelper tool and go through create a new backup action/mysql database backup wizard (no tough questions asked :)). Ninjahelper creates the action configuration in the /etc/backup.d/20.mysql file.

Perform a test:

backupninja --now --debug

Check if there has been no error messages and if the dump has been created in /var/backups/mysql/sqldump.

Step 4. Configure s3cmd

I assume that you are registered in Amazon Web Services platform and you have the Access Key ID and the Secret Access Key.

Type:

s3cmd --configure

When asked, provice Access Key and Secret Key. At the end of the configuration process, s3cmd will do the configuration test.

Step 5. Create an S3 bucket

In order to store your backup archives on Amazon S3, you have to create a bucket (storage space). You can use AWS Management Console or s3cmd:

s3cmd mb s3://my-backup-bucket

Step 6. Generate a GPG key (if you don’t have one)

Duplicity encrypts your backups with GPG. If you don’t have the GPG key or you want to create a new one for the backup purposes:

gpg --gen-key

and go though the wizard. Check if it has been created and write down the GPG key ID:

gpg --list-keys

pub   2048R/<key ID> 2011-04-03

Step 7. Backup file system and transfer the archive to S3 (MySQL backup included)

Now, edit Damon’s S3 backup script (dt-s3-backup) and fill configuration parameters. The inline documentation is self-explanatory; here I listed only values I had to adapt:

export AWS_ACCESS_KEY_ID="<your AWS Access Key ID>"
export AWS_SECRET_ACCESS_KEY="<your AWS Secret Access Key>"
export PASSPHRASE="<GPG key passphrase>"
GPG_KEY="<GPG key ID>"
ROOT="/"
DEST="s3+http://my-backup-bucket/"
INCLIST=( "/etc" "/var/www" "/var/backups/mysql" )
EXCLIST=()
STATIC_OPTIONS="--full-if-older-than 7D"
CLEAN_UP_TYPE="remove-older-than"
CLEAN_UP_VARIABLE="14D"
LOGDIR="/var/log/duplicity/"
LOG_FILE="duplicity-`date +%Y-%m-%d-%M`.txt"
LOG_FILE_OWNER="root:root"
VERBOSITY="-v4"

Test the backup script:

./dt-s3-backup.sh --backup
s3cmd ls s3://my-backup-bucket

You should get the a list similar to this one:

2011-04-04 15:21   1219566   s3://pedalibre-backup/duplicity-full-signatures.20110404T152112Z.sigtar.gpg
2011-04-04 15:21       801   s3://pedalibre-backup/duplicity-full.20110404T152112Z.manifest.gpg
2011-04-04 15:21  17990849   s3://pedalibre-backup/duplicity-full.20110404T152112Z.vol1.difftar.gpg

Check if you can restore from the backup (quite important issue :)):

mkdir /tmp/restore
./dt-s3-backup.sh --restore /tmp/restore

If everything works fine, orchestrate duplicity S3 backup from Backupninja. In /etc/backup.d, create a new action, e.g. 30.filesystem_to_s3.sh with this content:

<path to dt-s3-backup.sh>/dt-s3-backup.sh --backup

Don’t forget to set the owner and group to root:root and permissions to 700. The numerical prefix must be higher than of the MySQL backup action. In this configuration, first a MySQL dump is created and then, together with other directories (/etc and /var/www), is pushed to the S3 bucket.

Step 8. Test everything together

backupninja --now --debug

and check if there has been no error messages, the backup is in the S3 bucket and it can be restored.

Es todo amigos!

Follow

Get every new post delivered to your Inbox.

Join 383 other followers