I’m responsible for launching a new web page for Pedalibre, one of the oldest and biggest Madrid cyclist associations. It uses Drupal as CMS running on Apache httpd and storing everything in an MySQL database. We launched a proof-of-concept version on Amazon EC2 in the cheapest instance (micro, US-east zone). Works smoothly. Now a content work group must migrate the articles from the old site to the new one. Before that, I need a backup system.
My backup plan was the following:
- create an image (EC AMI). In case of instance crash, I can restore the instance from the AMI and apply backups
- backup Apache and MySQL configuration (in principle the entire
- backup Drupal installation directory (
- backup Drupal database
I checked Google, but I didn’t find any article or a blog post describing how to setup a complete backup using Amazon ecosystem and Amazon Simple Storage Service (S3) in particular. I had to mix various articles and finally reached a solution.
What I used
- Backupninja – for MySQL database backup and backup orchestration
- duplicity – filesystem backup (Amazon S3 capable)
- dt-s3-backup (check also author’s blog post) – shell script performing a backup to an S3 bucket using duplicity
- s3cmd – command line tool for managing data in Amazon S3; required by dt-s3-backup
Step 1. Install the tools
apt-get install backupninja s3cmd duplicity download zip from http://github.com/thornomad/dt-s3-backup/zipball/master extract dt-s3-backup.sh script
Remeber to run everything as super user.
Step 2. Configure Backupninja schedule
Check Backupninja configuration (
/etc/backupninja.conf) if it suits your needs. The most important parameter is
when – it controls the backup schedule (check the documentation for possible values). I left it with the default value (
everyday at 01:00). If you want to get backup report, fill at least
reportemail parameter value.
Step 3. Backup MySQL database
Create a Backupninja backup action for MySQL database dump. The easiest way is to start
ninjahelper tool and go through create a new backup action/mysql database backup wizard (no tough questions asked :)). Ninjahelper creates the action configuration in the
Perform a test:
backupninja --now --debug
Check if there has been no error messages and if the dump has been created in
Step 4. Configure s3cmd
I assume that you are registered in Amazon Web Services platform and you have the Access Key ID and the Secret Access Key.
When asked, provice Access Key and Secret Key. At the end of the configuration process, s3cmd will do the configuration test.
Step 5. Create an S3 bucket
In order to store your backup archives on Amazon S3, you have to create a bucket (storage space). You can use AWS Management Console or s3cmd:
s3cmd mb s3://my-backup-bucket
Step 6. Generate a GPG key (if you don’t have one)
Duplicity encrypts your backups with GPG. If you don’t have the GPG key or you want to create a new one for the backup purposes:
and go though the wizard. Check if it has been created and write down the GPG key ID:
gpg --list-keys pub 2048R/<key ID> 2011-04-03
Step 7. Backup file system and transfer the archive to S3 (MySQL backup included)
Now, edit Damon’s S3 backup script (
dt-s3-backup) and fill configuration parameters. The inline documentation is self-explanatory; here I listed only values I had to adapt:
export AWS_ACCESS_KEY_ID="<your AWS Access Key ID>" export AWS_SECRET_ACCESS_KEY="<your AWS Secret Access Key>" export PASSPHRASE="<GPG key passphrase>" GPG_KEY="<GPG key ID>" ROOT="/" DEST="s3+http://my-backup-bucket/" INCLIST=( "/etc" "/var/www" "/var/backups/mysql" ) EXCLIST=() STATIC_OPTIONS="--full-if-older-than 7D" CLEAN_UP_TYPE="remove-older-than" CLEAN_UP_VARIABLE="14D" LOGDIR="/var/log/duplicity/" LOG_FILE="duplicity-`date +%Y-%m-%d-%M`.txt" LOG_FILE_OWNER="root:root" VERBOSITY="-v4"
Test the backup script:
./dt-s3-backup.sh --backup s3cmd ls s3://my-backup-bucket
You should get the a list similar to this one:
2011-04-04 15:21 1219566 s3://pedalibre-backup/duplicity-full-signatures.20110404T152112Z.sigtar.gpg 2011-04-04 15:21 801 s3://pedalibre-backup/duplicity-full.20110404T152112Z.manifest.gpg 2011-04-04 15:21 17990849 s3://pedalibre-backup/duplicity-full.20110404T152112Z.vol1.difftar.gpg
Check if you can restore from the backup (quite important issue :)):
mkdir /tmp/restore ./dt-s3-backup.sh --restore /tmp/restore
If everything works fine, orchestrate duplicity S3 backup from Backupninja. In
/etc/backup.d, create a new action, e.g.
30.filesystem_to_s3.sh with this content:
<path to dt-s3-backup.sh>/dt-s3-backup.sh --backup
Don’t forget to set the owner and group to root:root and permissions to 700. The numerical prefix must be higher than of the MySQL backup action. In this configuration, first a MySQL dump is created and then, together with other directories (
/var/www), is pushed to the S3 bucket.
Step 8. Test everything together
backupninja --now --debug
and check if there has been no error messages, the backup is in the S3 bucket and it can be restored.
Es todo amigos!