In AWS, it’s of utmost importance to consider the cost in every aspect. In fact, cost optimization is one of the 6 pillars of a well-architected framework, therefore, it must be included in every step in building solutions in AWS. There are infinite ways to reduce bills but one of the most popular ones for EC2 instances is to start or stop them, especially if they don’t need to run 24/7.

But constantly doing this manually can be quite tedious and can significantly increase the operational load. It involves human intervention and hence, piling up its risks for human errors. To resolve this, the conventional way is to automate the start and stop of EC2 instances using Lambda functions together with CloudWatch Events and IAM roles.

The solution is simple enough to implement in a standalone EC2 instance, but what if it’s included in an auto-scaling group? Things could get really complicated. The default behavior of the Auto-Scaling group will terminate the instance as stopping it without the proper steps will mark it as unhealthy and for replacement. For some non-distributed and traditional applications, replacing an instance has undesirable behavior and should only be done if needed, i.e. for disaster recovery.

For this article, we are going to create an automated solution that will start and stop an instance under auto-scaling group without terminating it, and will dynamically support a scenario where the instance was terminated and replaced.

SUMMARY:

1. Time-based CloudWatch Event rules will trigger the Lambda Functions based on the specified time e.g. every 8:00 AM for start-instance and every 8:00 PM stop-instance.
2. The Lambda functions will retrieve the current instance ID that is stored in SSM Parameter Store and will execute the start/stop process of instances under the Auto-Scaling group i.e. setting lifecycle to InService/StandBy etc.
3. Dynamically update the SSM Parameter Store value and send an SNS notification in case the instance was terminated and replaced.

STEPS:

IAM Policy for Lambda Schedulers

      1. Create an IAM policy for stop and start Lambda execution role. Go to IAM Console > Policies > Create Policy.

      2. Attach the IAM permissions in the JSON editor and replace “test-asg” with your auto-scaling group name > Hit Next:Tags > Next:Review to proceed.

      3. Enter name for the policy role ex. asg-stop-start-policy > Create policy

IAM Role for Lambda Schedulers

      4. Create the execution role by going to the IAM Console > Roles > Create role

      5. Choose AWS service > Lambda > Click Next

      6. Find the recently created IAM policy (asg-stop-start-policy) > Tick the checkbox > Click Next

      7. Enter Role name ex. asg-stop-start-role > Scroll down to the bottom > Click Create role

SSM Parameter Store

      8. Take note of the application instance id from the auto-scaling group for future use.

      9. Go to Systems Manager > Parameter Store > Create parameter

      10. Enter a Name for the parameter /asg/app/instance-id > Paste the instance ID in the Value > Click Create parameter

AWS Lambda Schedulers

      11. First we’ll create the stop scheduler function. Go to Lambda > Functions > Create function

      12. Author from scratch > Enter Function name > Choose Python 3.9 > Change default execution role > Use an existing role > Choose the recently created IAM Role (asg-stop-start-role) > Create function

      13. Replace the code in lambda_function.py with the following code > Click Deploy

      14. Add the following Environment variables. 

              a. ASG_NAME is the name of the auto-scaling group the instance belongs to. 

              b. REGION is where the auto-scaling group is located (ex. Singapore – ap-southeast-1)

      15. Go to Configuration > General configuration > Adjust Timeout based on how long the instance stops (max of 15 mins)

      16. Repeat the process to create the start scheduler function and use the code below:

CloudWatch Events Rules (time-based)

      17. We will be creating the stop rule. Go to CloudWatch > Rules > Create rule

      18. Choose Schedule > Enter the cron expression: 0 12 ? * * * (every 8 pm in PH time) > Add target > Lambda function > Choose the asg-stop-instance function > Configure details

      19. Enter Name of the rule > make sure State is Enabled > Create rule

      20. Create the start rule using the same process > use the cron expression 0 0 ? * * * (every 8 am in PH time) > Choose the asg-start-instance

IAM Role for Lambda SSM Parameter Store Updater

      21. Create the updater role by going to the IAM Console >  Roles > Create role

      22. Choose AWS service > Lambda > Click Next

      23. Skip attaching policy for now by clicking Next

      24. Enter role Name (asg-ssm-parameter-store-updater-role) then scroll down > Create role

      25. Open the created IAM role > Add permissions > Create inline policy

      26. Choose JSON > enter the policy below > Review policy

      27. Enter Name > Create policy

AWS Lambda SSM Parameter Store Updater function

      28. Go to Lambda > Create function

      29. Author from scratch > enter function name (ex. asg-ssm-parameter-store-updater) > Choose Python 3.9 > Use an existing role > Choose the previously created updater IAM role (ex. asg-ssm-parameter-store-updater-role) > Create function

      30. Replace the code in lambda_function.py with the following code > Click Deploy

      31. Add the following Environment variables. 

                 a. ASG_NAME is the name of the auto-scaling group the instance belongs to. 

                 b. REGION is where the auto-scaling group is located (ex. Singapore – ap-southeast-1)

                 c. SSM_PARAMETER_NAME is the name of the SSM parameter store that contains the current instance id (ex. /asg/app/instance-id)

      32. Go to Configuration > General configuration > Adjust Timeout to 10 seconds

SNS to send email notifications when the instance is terminated

      33. Go to SNS > Topics > Create topic

      34. Choose Standard > Enter Name > Create topic

      35. Create subscription > Choose  Email in Protocol > Provide the email address in Endpoint that will receive the notification when the instance is terminated > Create subscription (repeat this process to add more recipients) > Recipients will receive an initial email and must Confirm subscription

CloudWatch Events Rules (event-based)

      36. Go to CloudWatch > Rules > Create rule

      37. Choose Auto Scaling > Instance Launch and Terminate > Specific instance event(s) > EC2 Instance Launch Successful > Name of the auto-scaling group (test-asg) > Add target > Choose Lambda function > Choose the lambda function ssm parameter store value updater (ex. asg-ssm-parameter-store-updater) > Configure details

      38. Enter Name for the rule > State enabled > Create rule

      39. Create another rule > Choose Auto Scaling > Instance Launch and Terminate > Specific instance event(s) > EC2 Instance Terminate Successful > Name of the auto-scaling group (test-asg) > Add target > Choose SNS topic > Choose the recently created topic (ex. InstanceTerminationNotification) > Configure details

      40. Enter Name for the rule > State enabled > Create rule

This completes the automated solution for stopping/starting an instance under auto-scaling group. The solution will set the auto-scaling instance lifecycle to Standby first so that it can safely stop the instance without the auto-scaling recognizing it as unhealthy and for replacement. The automated solution will also make sure that the instance is started properly before bringing it back to InService. In case of instance termination, the solution will send an email notification for early remediation. It will also support the new instance as the instance ID is dynamically stored and retrieved in SSM Parameter Store by the Lambda functions.