Backup & Restore of AWS Parameter Store aka DR Solution

It all started when I was creating a new AWS region deployment for a project, it took me a week to understand the process and implement it but that’s not important for this article, let’s move to an interesting topic. My Project was storing all its credentials to AWS Parameter Store only because it works with most of the tools and ease of implementation for storing sensitive data.

www.davehall.com.au

As per my memory, I never thought about DR of AWS Parameter store as was a very subtle topic and never came into my mind at least.

AWS Parameter store does have versioning but that’s not DR, it only provides an option to recover if someone overwrites by mistake so you can recover it at any point in time.

To Implement DR, we do need to save all of the stored key-pair to offsite location so in case of AWS region failure, we don’t lose our data. To keep the same thing in mind, I thought of creating a solution that will take offsite backup and restore it.

Honestly, this was one of the simplest solutions, I have implemented in python. let’s talk about a solution now :)

Requirements:

1. Take Backup of Given /SSM path and its sub-paths.
2. Must store the backup on reliable storage like AWS S3.

Implementation:

Source Code: 116davinder/aws-parameter-store-dr-solution
Packages: Python3 + boto3

Backup
It was super easy to implement because boto3 have one function called get_parameters_by_path which support recursion so it f**king simple, I just had to handle input, a little bit of filtering on data + output to S3 so I decided to use python temp-file for storing values till it’s collecting data and later upload to AWS S3. Everything was stored in JSON format as it can be loaded as a dictionary in python.

Example Backup Run

$ python3 backup.py '/Test' 'us-east-1' 'test-davinder-s3' 'SSM/'
{"@timestamp": "2021-03-23 11:27:49,565","level": "INFO","thread": "MainThread","name": "root","message": "SSM Parameter Path: /Test and its sub paths"}
{"@timestamp": "2021-03-23 11:27:49,574","level": "INFO","thread": "MainThread","name": "botocore.credentials","message": "Found credentials in environment variables."}
{"@timestamp": "2021-03-23 11:27:51,525","level": "INFO","thread": "MainThread","name": "root","message": "taking backup of key-pair at: /Test/xxxxxxx/API_KEY"}
{"@timestamp": "2021-03-23 11:27:51,525","level": "INFO","thread": "MainThread","name": "root","message": "taking backup of key-pair at: /Test/xxxxxxx/PASSWORD"}
{"@timestamp": "2021-03-23 11:27:51,525","level": "INFO","thread": "MainThread","name": "root","message": "taking backup of key-pair at: /Test/xxxxxxx/USERNAME"}
{"@timestamp": "2021-03-23 11:27:54,716","level": "INFO","thread": "MainThread","name": "root","message": "backup upload successful at s3://test-davinder-s3/SSM/21-03-23.json"}
{"@timestamp": "2021-03-23 11:27:54,716","level": "INFO","thread": "MainThread","name": "root","message": "cleaned temp files."}

Restore
It was a little tricky to implement because S3 doesn’t have an inbuilt sorting method so I have to download all the meta-data and do sorting on a local machine, Once I have sorted backup files, Currently, I just blindly select the last created backup file. Download the backup file as python temp-file for storing values and start parsing and writing to AWS Parameter store with a bit of customization like you can add a prefix to SSM restore path + can restore into different AWS region.

Example Restore Run

$ python3 restore.py '/DAV' 'us-east-1' 'test-davinder-s3' 'SSM/'
{"@timestamp": "2021-03-23 11:22:15,773","level": "INFO","thread": "MainThread","name": "botocore.credentials","message": "Found credentials in environment variables."}
{"@timestamp": "2021-03-23 11:22:17,599","level": "INFO","thread": "MainThread","name": "root","message": "Selected Backup File: SSM/21-03-23.json"}
{"@timestamp": "2021-03-23 11:22:18,426","level": "INFO","thread": "MainThread","name": "root","message": "restoring key-pair at: /DAV/xxxxxxxxx/API_KEY"}
{"@timestamp": "2021-03-23 11:22:21,180","level": "INFO","thread": "MainThread","name": "root","message": "restoring key-pair at: /DAV/xxxxxxxxx/PASSWORD"}
{"@timestamp": "2021-03-23 11:22:21,589","level": "INFO","thread": "MainThread","name": "root","message": "restoring key-pair at: /DAV/xxxxxxxxx/USERNAME"}
{"@timestamp": "2021-03-23 11:22:22,001","level": "INFO","thread": "MainThread","name": "root","message": "cleaned up temp files"}

My implementation does have minor problems which can be solved by other means but I do want to mention them here so anyone going to use this solution will also know about them.

1. Application won’t handle credentials of AWS. You must check the boto3 guide for it.
2. Application won’t create an s3 bucket, it assumes you have it will right bucket policies.
3. Application won’t encrypt any data. It assumes you have enabled AWS S3 Server Side Encryption.
4. S3 Bucket Prefix should be unique because the application assumes backup files that end with jsononly.
5. PEP8 Rules are not followed by me.
6. Application won’t take backup of SSM Parameter description and tags currently.

I hope, it would ease your pain a bit for AWS Cloud. Thank You!