Configure Amazon S3 backup with Backupninja and duplicity

I’m responsible for launching a new web page for Pedalibre, one of the oldest and biggest Madrid cyclist associations. It uses Drupal as CMS running on Apache httpd and storing everything in an MySQL database. We launched a proof-of-concept version on Amazon EC2 in the cheapest instance (micro, US-east zone). Works smoothly. Now a content work group must migrate the articles from the old site to the new one. Before that, I need a backup system.

My backup plan was the following:

  • create an image (EC AMI). In case of instance crash, I can restore the instance from the AMI and apply backups
  • backup Apache and MySQL configuration (in principle the entire /etc)
  • backup Drupal installation directory (/var/www)
  • backup Drupal database

I checked Google, but I didn’t find any article or a blog post describing how to setup a complete backup using Amazon ecosystem and Amazon Simple Storage Service (S3) in particular. I had to mix various articles and finally reached a solution.

What I used

  • Backupninja – for MySQL database backup and backup orchestration
  • duplicity – filesystem backup (Amazon S3 capable)
  • dt-s3-backup (check also author’s blog post) – shell script performing a backup to an S3 bucket using duplicity
  • s3cmd – command line tool for managing data in Amazon S3; required by dt-s3-backup

Step 1. Install the tools

apt-get install backupninja s3cmd duplicity
download zip from http://github.com/thornomad/dt-s3-backup/zipball/master
extract dt-s3-backup.sh script

Remeber to run everything as super user.

Step 2. Configure Backupninja schedule

Check Backupninja configuration (/etc/backupninja.conf) if it suits your needs. The most important parameter is when – it controls the backup schedule (check the documentation for possible values). I left it with the default value (everyday at 01:00). If you want to get backup report, fill at least reportemail parameter value.

Step 3. Backup MySQL database

Create a Backupninja backup action for MySQL database dump. The easiest way is to start ninjahelper tool and go through create a new backup action/mysql database backup wizard (no tough questions asked :)). Ninjahelper creates the action configuration in the /etc/backup.d/20.mysql file.

Perform a test:

backupninja --now --debug

Check if there has been no error messages and if the dump has been created in /var/backups/mysql/sqldump.

Step 4. Configure s3cmd

I assume that you are registered in Amazon Web Services platform and you have the Access Key ID and the Secret Access Key.

Type:

s3cmd --configure

When asked, provice Access Key and Secret Key. At the end of the configuration process, s3cmd will do the configuration test.

Step 5. Create an S3 bucket

In order to store your backup archives on Amazon S3, you have to create a bucket (storage space). You can use AWS Management Console or s3cmd:

s3cmd mb s3://my-backup-bucket

Step 6. Generate a GPG key (if you don’t have one)

Duplicity encrypts your backups with GPG. If you don’t have the GPG key or you want to create a new one for the backup purposes:

gpg --gen-key

and go though the wizard. Check if it has been created and write down the GPG key ID:

gpg --list-keys

pub   2048R/<key ID> 2011-04-03

Step 7. Backup file system and transfer the archive to S3 (MySQL backup included)

Now, edit Damon’s S3 backup script (dt-s3-backup) and fill configuration parameters. The inline documentation is self-explanatory; here I listed only values I had to adapt:

export AWS_ACCESS_KEY_ID="<your AWS Access Key ID>"
export AWS_SECRET_ACCESS_KEY="<your AWS Secret Access Key>"
export PASSPHRASE="<GPG key passphrase>"
GPG_KEY="<GPG key ID>"
ROOT="/"
DEST="s3+http://my-backup-bucket/"
INCLIST=( "/etc" "/var/www" "/var/backups/mysql" )
EXCLIST=()
STATIC_OPTIONS="--full-if-older-than 7D"
CLEAN_UP_TYPE="remove-older-than"
CLEAN_UP_VARIABLE="14D"
LOGDIR="/var/log/duplicity/"
LOG_FILE="duplicity-`date +%Y-%m-%d-%M`.txt"
LOG_FILE_OWNER="root:root"
VERBOSITY="-v4"

Test the backup script:

./dt-s3-backup.sh --backup
s3cmd ls s3://my-backup-bucket

You should get the a list similar to this one:

2011-04-04 15:21   1219566   s3://pedalibre-backup/duplicity-full-signatures.20110404T152112Z.sigtar.gpg
2011-04-04 15:21       801   s3://pedalibre-backup/duplicity-full.20110404T152112Z.manifest.gpg
2011-04-04 15:21  17990849   s3://pedalibre-backup/duplicity-full.20110404T152112Z.vol1.difftar.gpg

Check if you can restore from the backup (quite important issue :)):

mkdir /tmp/restore
./dt-s3-backup.sh --restore /tmp/restore

If everything works fine, orchestrate duplicity S3 backup from Backupninja. In /etc/backup.d, create a new action, e.g. 30.filesystem_to_s3.sh with this content:

<path to dt-s3-backup.sh>/dt-s3-backup.sh --backup

Don’t forget to set the owner and group to root:root and permissions to 700. The numerical prefix must be higher than of the MySQL backup action. In this configuration, first a MySQL dump is created and then, together with other directories (/etc and /var/www), is pushed to the S3 bucket.

Step 8. Test everything together

backupninja --now --debug

and check if there has been no error messages, the backup is in the S3 bucket and it can be restored.

Es todo amigos!

About these ads

9 thoughts on “Configure Amazon S3 backup with Backupninja and duplicity

  1. Benjamin C. says:

    Thanks a lot! I didn’t expected that it would be so easy!

  2. Michael says:

    Hola!
    Do you use use one of the duplicity options “remove-older-than” or “remove-all-but-n-full”??
    They do not seem to work for me. Backup on S3 backend grows forever.

    Saludos a Madrid desde Berlin!

  3. gamerxgirl says:

    Thanks for posting this! However I can’t seem to run the .sh script:

    ./dt-s3-backup.sh –backup
    I get this error message:

    ./dt-s3-backup.sh: Permission denied

    Any suggestions? Thanks!

  4. This also needs the package: python-boto

  5. [...] quite cost-effective way to back up things and can easily be automated. I used the good tutorial at Marcin Gryszko’s blog for setup. It’s quite good, so I only will add here some tips to get around problems and to [...]

  6. Grimneko says:

    Thanks, was easier then expected. Most other guides where quite confusing/complicated, but this one was nice and clear. Added a IAM policy to improve security a bit, and its perfect.

  7. Derek says:

    Thanks for putting together this valuable guide. Much appreciated.

    By the way, I am using S3 as automatic backup for my wordpress blog using “Automatic Backup” plugin. I have uploaded all images on S3, and so my image URL looks like –
    http://bucketname.s3.amazonaws.com/wp-content/uploads/2010/12/Picture.png

    How would i go about changing the URL of the image OR swapping the image if I have to? What changes do i make to 1) change the image and 2) change the URL of the image back to http://www.domain.com/wp-content/uploads/2010/12/Picture.png?

    Thanks in advance.

  8. ritmon says:

    It seems that the duplicity-full.*.vol*.difftar.gpg files are not compressed?

Comments are closed.

Follow

Get every new post delivered to your Inbox.

Join 389 other followers

%d bloggers like this: