Last updated on July 17, 2024
The Automated Daily RDS Export ensures that daily snapshots of Amazon RDS instances are created and made shareable. While RDS takes daily snapshots automatically, these are not shareable by default. To address this, we copy the snapshots and share the copies with a designated backup account. This process uses AWS Lambda Functions and Amazon EventBridge to automate the task. EventBridge triggers the Lambda function daily, which handles copying and sharing the snapshots. This automation ensures that shareable snapshots are created every day without manual intervention, enhancing the reliability and accessibility of database backups.
The Automated Daily RDS Export has several important benefits for managing database backups. By automating the creation of shareable snapshots, it reduces the risk of mistakes and ensures backups are made consistently. This process saves time and allows teams to focus on other important work. Shareable snapshots make it easier to recover data quickly in case of a disaster. Storing backups in a separate account also improves security, reducing the risk of data loss or corruption. Overall, this automated process provides a reliable and efficient way to handle database backups.
Implementation Steps
Backup Account
Step 1: On the backup account, create a Lambda Function that will copy the newly shared RDS Snapshot to a different region.
Function name: td-shared-rds-snapshot (Replace with desired name)
Runtime: Python 3.9
Ensure to replace the following:
- actual source region: us-east-2
- AWS account ID: YOUR AWS ACCOUNT ID
- desired target region: us-east-1
import boto3 import time def lambda_handler(event, context): try: # Wait for 10 seconds - This will wait for the RDS Snapshot to be shared. time.sleep(10) # Initialize the RDS client for the source region (where the snapshot is shared with you) rds_source_region = boto3.client('rds', region_name='us-east-2') # Replace with the actual source region # Specify the source region (us-east-2) and the source AWS account ID source_region = 'us-east-2' # Replace with the actual source region source_account_id = 'YOUR_SOURCE_ACCOUNT_ID' # Replace with your AWS account ID # Initialize the RDS client for the target region (where you want to copy the snapshot) rds_target_region = boto3.client('rds', region_name='us-east-1') # Replace with the desired target region # Specify the target region (us-east-1) target_region = 'us-east-1' # Replace with the desired target region # List shared snapshots for the specified source region and account response = rds_source_region.describe_db_snapshots( SnapshotType='shared', MaxRecords=500, # Increase the max records to ensure coverage IncludeShared=True, IncludePublic=False ) # Check if there are any shared snapshots found snapshots = response['DBSnapshots'] if not snapshots: return { 'statusCode': 404, 'body': 'No shared snapshots found for the specified source region and account.' } # Sort the snapshots by creation time (most recent first) snapshots.sort(key=lambda x: x['SnapshotCreateTime'], reverse=True) # Get the most recently created shared snapshot identifier source_snapshot_identifier = snapshots[0]['DBSnapshotIdentifier'] # Specify the target snapshot identifier based on the source identifier target_snapshot_identifier = 'copy-' + source_snapshot_identifier.replace(':', '-') # Copy the shared RDS snapshot from the source region (us-east-1) to the target region (us-east-2) copy_response = rds_target_region.copy_db_snapshot( SourceDBSnapshotIdentifier=source_snapshot_identifier, TargetDBSnapshotIdentifier=target_snapshot_identifier, SourceRegion=source_region ) # Get the identifier of the newly created snapshot from the copy response new_snapshot_identifier = copy_response['DBSnapshot']['DBSnapshotIdentifier'] return { 'statusCode': 200, 'body': f'Shared snapshot "{source_snapshot_identifier}" is now copying to {target_region} region as "{new_snapshot_identifier}".' } except Exception as e: return { 'statusCode': 500, 'body': f'Error: {str(e)}' }
Step 2: Add this role to the Lambda Function.
Name it as td-shared-rds-snapshot (Replace it with your desired function name)
{ "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "rds:DescribeDBSnapshots", "rds:CopyDBSnapshot", "rds:ModifyDBSnapshotAttribute" ], "Resource": "*" } ] }Step 3: Create an Event Bus
Note: As per the AWS documentation, RDS Shared Snapshot event is currently unavailable. Therefore, we will opt for cross-account event buses using EventBridge. To implement this, we will create an EventBridge rule that will transfer snapshot copy events to the backup account’s EventBridge bus. Then, we can attach a Lambda function to the Event bus in the backup account.
- Go to Amazon EventBridge → Event buses → Create event bus
- Name: td-shared-rds-snapshot-bus (Replace with your desired name)
- Resource-based policy
PROD Account: “”
Backup Account: “”{ "Version": "2012-10-17", "Statement": [{ "Sid": "sid1", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::54645683453:root" }, "Action": "events:PutEvents", "Resource": "arn:aws:events:us-east-1:54645683453:event-bus/td-shared-rds-snapshot-bus" (Replace with your desired name) }] }Step 4: Under the Created Event bus (td-shared-rds-snapshot-bus), create an Amazon Eventbridge (CloudWatch Events) rule to trigger the Lambda function.
![]()
Name: td-shared-rds-snapshot (Replace with your desired name)
Description: Copy the shared RDS Snapshot to a different region.
{ "source": [ "aws.rds" ], "detail-type": [ "RDS DB Snapshot Event" ], "detail": { "SourceType": [ "SNAPSHOT" ], "EventID": [ "RDS-EVENT-0197" ] } }Step 5: Copy the ARN of the created Event Bus. We will be using it later on the source account.
Backup Account:
Deletion of RDS Snapshots (Backup Account):
-
In terms of cost management, it is recommended to keep only one snapshot at the end of each month.
-
Based on the information provided, it is expected that there will be 12 snapshots per year.
-
Move the 12 RDS snapshots to the Amazon S3 Glacier.
Step 1: Create a Lambda Function to keep only one RDS snapshot at the end of each month.
Function name: td-keep-one-rds-snapshot-each-month
Runtime: Python 3.9
import boto3 from datetime import datetime, timedelta def lambda_handler(event, context): # Initialize the RDS client rds = boto3.client('rds') # Specify the maximum number of snapshots to keep each month max_snapshots_to_keep = 1 # Number of snapshots to keep each month # Target region target_region = 'us-east-2' # Initialize the list of snapshots to delete snapshots_to_delete = [] # Calculate the current date current_date = datetime.now() # Calculate the last day of the previous month last_day_of_previous_month = current_date.replace(day=1) - timedelta(days=1) # Calculate the date format for snapshot identifier (e.g., 2023-08-31) snapshot_identifier_date = last_day_of_previous_month.strftime('%Y-%m-%d') # List all manual snapshots from the AWS account response = rds.describe_db_snapshots( SnapshotType='manual' ) # Sort the snapshots by creation time (most recent first) snapshots = response['DBSnapshots'] snapshots.sort(key=lambda x: x['SnapshotCreateTime'], reverse=True) # Identify the snapshots from the last day of the month snapshots_to_keep = [snapshot for snapshot in snapshots if snapshot['SnapshotCreateTime'].date() == last_day_of_previous_month.date()] # Delete extra snapshots if there are more than max_snapshots_to_keep if len(snapshots_to_keep) > max_snapshots_to_keep: snapshots_to_delete = snapshots_to_keep[max_snapshots_to_keep:] for snapshot in snapshots_to_delete: rds.delete_db_snapshot(DBSnapshotIdentifier=snapshot['DBSnapshotIdentifier'], RegionName=target_region) return { 'statusCode': 200, 'body': f'Kept {len(snapshots_to_keep)} snapshots from {snapshot_identifier_date} and deleted {len(snapshots_to_delete)} snapshots in region {target_region}.' }Step 2: Add this policy to the Lambda Function.
Name it as td-move-rds-snapshots-to-s3-glacier (Replace with your desired name)
{ "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": "rds:DescribeDBSnapshots", "Resource": "*" } ] }Step 3: Create a CloudWatch Event to run this function every month.
- Go to Lambda Configuration → Triggers
- name: keep-one-rds-snapshot-each-month (Replace with your desired name)
- Schedule expression: cron(0 15 L * ? *)
Step 4: Create AWS KMS – We will be needing this in Step 7.
-
Exporting RDS Snapshots to Amazon S3 requires KMS even if the RDS Snapshots are not encrypted.
-
Ensure that the KMS matches the region where the RDS Snapshots are stored.
-
Refer to this KMS: https://us-east-1.console.aws.amazon.com/kms/home?region=us-east-1#/kms/keys
Step 5: Create an Amazon S3 Bucket.
Note: Ensure that the region in Amazon S3 matches the region where the RDS Snapshots are stored.
- Name: td-rds-snapshots (Replace with your desired name)
- Region: us-east-1
Step 6: Once created, go to Management → Create Lifecycle rule
Follow these configurations:
Once done, click the Create rule button.
Step 7: Create a Lambda Function that will move the 12 RDS snapshots to the Amazon S3 Glacier every year.
Function name: td-move-rds-snapshots-to-s3-glacier-yearly (Replace with your desired name)
Runtime: Python 3.9
Note: Make sure to replace the following:
- source region: us-east-2
- S3 bucket name: td-rds-snapshots
- IAM role ARN
- kms_key_id
import boto3 def lambda_handler(event, context): # Initialize AWS clients rds = boto3.client('rds', region_name='us-east-2') # Replace with your s3_bucket = 'td-rds-snapshots' # Replace with your S3 bucket name try: # List all manual snapshots response = rds.describe_db_snapshots(SnapshotType='manual') # Export snapshots to Amazon S3 for snapshot in response['DBSnapshots']: snapshot_arn = snapshot['DBSnapshotArn'] export_task_id = snapshot['DBSnapshotIdentifier'] iam_role_arn = '' # Replace with your IAM role ARN kms_key_id = '' s3_prefix = f'rds-snapshots/{export_task_id}' # Replace with your desired S3 prefix rds.start_export_task( ExportTaskIdentifier=export_task_id, SourceArn=snapshot_arn, S3BucketName=s3_bucket, IamRoleArn=iam_role_arn, KmsKeyId=kms_key_id, S3Prefix=s3_prefix ) return { 'statusCode': 200, 'body': f'Started export tasks for {len(response["DBSnapshots"])} snapshots.' } except Exception as e: return { 'statusCode': 500, 'body': f'Error: {str(e)}' }
Step 8: Add these policies to the Lambda Function
- Name it as td-move-rds-snapshots-to-s3-glacier (Replace with your desired name)
{ "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "s3:ListAccessPointsForObjectLambda", "s3:PutObject", "s3:GetObject", "rds:DescribeDBSnapshots", "rds:DescribeExportTasks", "rds:StartExportTask", "s3:ListBucket", "s3:DeleteObject", "s3:GetBucketLocation" ], "Resource": "*" } ] }Create another one, name it as td-pass-role (Replace with your desired name)
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "iam:PassRole", "Resource": "*" } ] }Step 9: Create a CloudWatch Event to run this function every year.
- Go to Lambda Configuration → Triggers
- name: td-move-rds-snapshots-to-s3-glacier-yearly (Replace with your desired name)
- Schedule expression: cron(0 15 1 1 ? *)
Copy the Automated RDS Snapshot to the same region
Step 1: Create a Lambda Function that copies the system snapshots for the purpose of making them shareable.
-
Function name: td-copy-system-rds-snapshot (Replace with your desired name)
-
Runtime: Python 3.9
Ensure to replace the following:
- RDS instance ID: prod-tutorialsdojo-portal-db-v2
- desired region: us-east-2
import boto3 def lambda_handler(event, context): # Initialize the RDS client rds = boto3.client('rds') # Specify the RDS instance identifier rds_instance_identifier = 'prod-tutorialsdojo-portal-db-v2' # Replace with your RDS instance ID try: # List automated snapshots for the specified RDS instance response = rds.describe_db_snapshots( DBInstanceIdentifier=rds_instance_identifier, SnapshotType='automated', MaxRecords=500, # Increase the max records to ensure coverage IncludeShared=False, IncludePublic=False ) # Check if there are any snapshots found snapshots = response['DBSnapshots'] if not snapshots: return { 'statusCode': 404, 'body': 'No completed automated snapshots found for the specified RDS instance.' } # Sort the snapshots by creation time (most recent first) snapshots.sort(key=lambda x: x['SnapshotCreateTime'], reverse=True) # Get the most recently created snapshot identifier source_snapshot_identifier = snapshots[0]['DBSnapshotIdentifier'] # Initialize the RDS client for the same region rds_same_region = boto3.client('rds', region_name='us-east-2') # Replace with your desired region # Specify the target region (same region) target_region = 'us-east-2' # Replace with your desired region # Modify the identifier to meet naming conventions target_snapshot_identifier = 'copy-' + source_snapshot_identifier.replace(':', '-') # Copy the most recently created automated RDS snapshot to the same region copy_response = rds_same_region.copy_db_snapshot( SourceDBSnapshotIdentifier=source_snapshot_identifier, TargetDBSnapshotIdentifier=target_snapshot_identifier, SourceRegion=target_region ) # Get the identifier of the newly created snapshot from the copy response new_snapshot_identifier = copy_response['DBSnapshot']['DBSnapshotIdentifier'] return { 'statusCode': 200, 'body': f'Newly created snapshot "{new_snapshot_identifier}" is now copying.' } except Exception as e: return { 'statusCode': 500, 'body': f'Error: {str(e)}' }Step 2: Add these policies to the Lambda Function
- Name it as td-copy-system-rds-snapshot (Replace with your desired name)
{ "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "rds:DescribeDBSnapshots", "rds:CopyDBSnapshot" ], "Resource": "*" } ] }Step 3: Utilize Amazon Eventbridge (CloudWatch Events) to trigger this function.
-
Name: td-copy-system-rds-snapshot (Replace with your desired name)
-
Description: Create a copy of the automatically generated RDS snapshot.
{ "source": [ "aws.rds" ], "detail-type": [ "RDS DB Snapshot Event" ], "detail": { "Message": [ "Automated snapshot created" ] } }Share the RDS Snapshot to the Backup Account
Step 4: Create a Lambda Function that will send the new RDS snapshot to the backup account.
- Function name: share-rds-snapshot (Replace with your desired name)
- Runtime: Python 3.9
import boto3 def lambda_handler(event, context): # Initialize the RDS client rds = boto3.client('rds') # Specify the AWS account ID source_aws_account_id = 'YOUR_SOURCE_ACCOUNT_ID' # Replace with PROD AWS account ID target_aws_account_id = 'YOUR_TARGET_ACCOUNT_ID' # Replace with the Backup Account AWS account ID # Specify the RDS instance identifier rds_instance_identifier = 'YOUR_RDS_INSTANCE_IDENTIFIER' # Replace with your RDS instance ID try: # List manual snapshots for the specified RDS instance response = rds.describe_db_snapshots( DBInstanceIdentifier=rds_instance_identifier, SnapshotType='manual', # Specify 'manual' for manual snapshots MaxRecords=500, # Increase the max records to ensure coverage IncludeShared=False, IncludePublic=False ) # Check if there are any snapshots found snapshots = response['DBSnapshots'] if not snapshots: return { 'statusCode': 404, 'body': 'No completed manual snapshots found for the specified RDS instance.' } # Sort the snapshots by creation time (most recent first) snapshots.sort(key=lambda x: x['SnapshotCreateTime'], reverse=True) # Get the most recently created snapshot identifier source_snapshot_identifier = snapshots[0]['DBSnapshotIdentifier'] # Share the snapshot with the target AWS account rds.modify_db_snapshot_attribute( DBSnapshotIdentifier=source_snapshot_identifier, AttributeName='restore', ValuesToAdd=[target_aws_account_id] ) return { 'statusCode': 200, 'body': f'Manual snapshot "{source_snapshot_identifier}" is now shared with Backup Account.' } except Exception as e: return { 'statusCode': 500, 'body': f'Error: {str(e)}' }Step 5: Add these policies to the Lambda Function
- Name it as td-share-rds-snapshot (Replace with your desired name)
{ "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "rds:DescribeDBSnapshots", "rds:CopyDBSnapshot", "rds:ModifyDBSnapshotAttribute" ], "Resource": "*" } ] }Step 6: Utilize Amazon Eventbridge (CloudWatch Events) to trigger this function.
-
Name: td-share-rds-snapshot (Replace with your desired name)
-
Description: Share the RDS Snapshot with the Backup Account
{ "source": [ "aws.rds" ], "detail-type": [ "RDS DB Snapshot Event" ], "detail": { "SourceType": [ "SNAPSHOT" ], "EventID": [ "RDS-EVENT-0197" ] } }Step 7: Add the created custom event bus from the Backup Account:
Conclusion
The Automated Daily RDS Export is a vital solution for managing database backups efficiently. It ensures that daily snapshots of Amazon RDS instances are created and made shareable. While RDS takes daily snapshots automatically, these aren’t shareable by default. To fix this, we copy the snapshots and share the copies with a backup account. This process uses AWS Lambda Functions and Amazon EventBridge to automate the task. EventBridge triggers the Lambda function daily, which handles copying and sharing the snapshots. Automating this process reduces mistakes and ensures backups are made consistently. It saves time and allows teams to focus on other important tasks. Shareable snapshots make it easier to recover data quickly in case of a disaster, and storing backups in a separate account improves security. Overall, this solution provides a reliable and efficient way to manage database backups, enhancing both security and data recovery.
AWS, Azure, and GCP Certifications are consistently among the top-paying IT certifications in the world, considering that most companies have now shifted to the cloud. Earn over $150,000 per year with an AWS, Azure, or GCP certification!
Follow us on LinkedIn, YouTube, Facebook, or join our Slack study group. More importantly, answer as many practice exams as you can to help increase your chances of passing your certification exams on your first try!
View Our AWS, Azure, and GCP Exam Reviewers Check out our FREE courses