Automatic SSH Key Pair Rotation via AWS Systems Manager Fleet Manager

AWS Systems Manager has an extensive collection of features and capabilities to manage your AWS and hybrid systems. This service allows you to centralize operational data from various AWS services and automate manual tasks on both your on-premises and AWS resources. One of its capabilities is the AWS Systems Manager Fleet Manager, which provides a unified user interface experience to help organizations remotely manage their servers (nodes) that are running on AWS or on-premises. The AWS Systems Manager Fleet Manager lets you view the performance status and overall health of your entire server fleet from a centralized web console for easier management.

As more organizations leverage the benefits of moving to the cloud, more opportunities for cloud professionals can also grow, resulting in high demand. However, it also means many employees will leave their current responsibilities for greener pastures. This can significantly impact companies in terms of security as it can introduce a vulnerability, especially if the resigning resource keeps company assets.

Suppose a DevOps Engineer at an enterprise frequently accesses their fleet of Linux-based EC2 instances on a day-to-day basis. To securely SSH, the server will require a PEM key from the employee to connect successfully. This PEM key is being kept by the DevOps Engineer and can be easily distributed via email or any other means of file sharing. One day, the resource abruptly left the enterprise. This scenario can open a security vulnerability for the company as the DevOps Engineer might still possess the PEM key. This means the employee can still SSH to the fleet of instances even after leaving the enterprise.

To address the issue, we can create a solution to rotate the authorized_keys file of the fleet of EC2 servers to revoke the SSH connection from the old PEM key using the AWS Systems Manager Fleet Manager.

Let’s get started.

1.  Create the new PEM key
In your local machine, generate the new PEM key using the command:
ssh-keygen -f <replace with name>.pem -t rsa

For this example, let’s name it new.pem for simplicity

This will create the new private and public keys:

We have to get the value of the public key by running the command:
cat <replace with name>.pem.pub

For this instance, new.pem.pub

Copy and save the value for future use.

2. Execute the command to the fleet of instances to rotate the PEM key
In your AWS Console, go to AWS Systems Manager Fleet Manager and choose the instance you want to rotate. For this example, let’s start with Ubuntu Instance.

Click Node actions > Execute run command

This will open the AWS Systems Manager – Run Command on a new tab.

Filter AWS-RunShellScript then select.

Scroll down until you find Command parameters

Paste the following command and replace the public key value with the generated from step 1 and the instance username.

(new.pem.pub):

sudo su
echo "<replace with public key value>" > /home/<replace with instance username>/.ssh/authorized_keys

For this example, the username is ubuntu
sudo su
echo "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDduEnWhAuZFZ3jzPvmINzwvWwwmM3rX0dblLh1PnUQHBJwPZ/q09iiOrhussQyVnJp

+Gi6lnE4JGARhauZ13cxYZ4y2dp9tlGvdG+lAgAJJx/j6nXGbkYahUWsP5OURU1NZRUfefajE+PBihIHtb/eYaAGvPkLPgwI

YFONtoVRkuNYH79JTsWKWKrQCnvOuE9ITYcg4EddR6VRhNhEGtR6j5M1lA8t/VzulzA3wT01ecOOAgqjZEWsZequo73gao2g

4J/Ep1Aa7QWaY/DXuUH6JpZeBHyvASKtpuSs+N99Xo9UZcQ5YqkWjJgOQMQlFKl2hzG8/WzSINULxnYZYwN/" >

/home/ubuntu/.ssh/authorized_keys

Scroll down and under Target selection, make sure the correct instance to be rotated is selected.

Since this is a simple command, uncheck Enable an S3 bucket

Scroll down and click Run

This will run the rotate command to the Ubuntu instance. Click the refresh button until it’s finished.

3. Test if the PEM key rotation is successful
Let’s test first if the old PEM key will still connect to the Ubuntu instance.

As expected, the old PEM key has been revoked from accessing the Ubuntu instance.

Now let’s test SSH using the new PEM key.

Congratulations! We have successfully accessed the Ubuntu instance using the new PEM key.

With a few simple steps, we have completed the whole PEM key rotation process. By leveraging AWS Systems Manager, we can easily rotate the PEM key of the other instances in batches using the same process.

Note: the selected instances share the same public key (new.pem.pub) and user name (ec2-user), therefore, we can use the same command.

Assuming all the PEM key of the fleet of instances has been rotated, the resigned DevOps Engineer will now be prevented from remotely accessing the company servers, eliminating the security vulnerability. This type of scenario is some of the examples that you can find from the AWS Certified Solutions Architect – Professional (SAP-C02) certification exam.

Aside from automatically rotating the PEM keys of your server fleet, the AWS Systems Manager Fleet Manager can also be used to access the Red Hat Knowledgebase Portal, manage node status, view instance information, manage accounts and registry as well as view the edge device information.