Disaster Recovery for Developers using Automated Snapshotting

Devlopment in the Cloud

Many employers are now using cloud instances to perform development tasks. Like anything there are pros and cons for this. The upsides include portability and uniformity. A developer can go to any customer building have full access to their exact workstation and setup. The common counter argument to this is that some type of disaster scenario will take down these cloud instances and developers will waste time trying to recover not only their work but any notes or files they had on their machines. There are steps to remediate loss such as being sure push commits to remote repositories or even periodically upload some files to S3. In this post I'll show you an automated back up strategy that is robust for these types of disaster scenarios.

Use EBS to Backup

When you launch a new Ec2 instance a root volume is automatically attached to it. To keep you work backed up you can attach a secondary volume that will persist even after termination of the instance. You can put a file system on that volume and mount it. This will keep your work safe. Then, using Cloudwatch you can created automated snapshots of that volume. You can optimize this process in a few different ways but part of what I'm sharing here is the volume snapshot automation.

Launch your Dev Instance

When you launch your dev instance be sure to add another storage volume to it like this.

Notice the second volume does not delete on termination by default. Once you are on that instance you can install a file system and create a mount point. You can assess your storage on the instance using the lsblk command and you'll see your secondary storage is not mounted.


Install a file system and mount that volume using these commands

sudo file -s /dev/xvdb  
sudo mkfs -t ext4 /dev/xvdb  
sudo mkdir -p /data/1  
sudo mount /dev/xvdb /data/1/  

Running lsblk again you can see the volume is mounted and ready to be written to.

I can do all my work from /data/1 now and persist it via automated snapshot. Let's write one file here and see how it is preserved through snapshot.

Here you can see I have created a file on /data/1 and put some text in it. Let's see how we can manage backups of this information.

Automated Snapshot with CloudWatch

Using the volume that we mounted previously we can use CloudWatch events to create daily snapshots of it.
Here is the setup.

Notice that I have it scheduled to happen daily on the left side and that I'm using the volume id on the right side of the setup. At this point if our dev machine were to fail, we could take the latest snapshot, generate a volume from it and attach it to a new instance. This will preserve all uncommitted work or any other working files that may be of unique interest to a developer.

Recovering your Work

At this point we could terminate the original instance and spin up a whole new instance to be our new development workstation. From here all we have to do is create a volume for our snapshot and attach it to our new instance. It's critically important that the new volume is in the same availability zone as your new instance as this could prevent attachment. Here is how the attachment looks. Notice that the availability zones of the instance and volume match.

Once we have the new volume attached we need to shell into that instance and mount it. We won't need to put a file system on it however, because it will be preserved from snapshotting along with a file we created.

In the following screen shot you'll notice

1. Originally xvdf has no mount point

2. We are on a new instance and the IP is different than the previous screen shots

3. The contents of our original file have been fully recovered after successful mounting.  

Have a Strategy

Most people gloss over disaster recovery strategies because they think the have a great plan already. It's one thing to have a strategy it's another to make sure it's up to date and actually works.