Logging Using awslogs Log Driver in Amazon ECS

Home » AWS Cheat Sheets » AWS Compute Services » Compute Related Notes » Logging Using awslogs Log Driver in Amazon ECS

Logging Using awslogs Log Driver in Amazon ECS

Last updated on May 2, 2023

Monitoring and logging can be as important as developing and deploying applications. Imagine managing an enterprise-level distributed application that runs on hundreds of containers. You must place your containers cleverly on instances to achieve a certain level of resilience and performance. On top of that, you would want to have visibility into CPU and memory usage so you’ll have an idea which task/tasks are resource-hungry at a given time. You may also want to check the disk space of your instance and see if you still have enough storage for your application.

Fortunately, monitoring distributed applications hosted in AWS has never been easier. Cloudwatch natively integrates with ECS so you can send and view ECS metrics on your Cloudwatch dashboard.

So when the time comes where your CPU usage overloads and your applications start to crawl and slow down, CloudWatch will alarm and notify you what is happening. But what if you suddenly get a spike in request rates and you need to know which customer is causing it, or there have been multiple error rates and you need to identify which API calls are responsible for it? Sure, you’ll be notified about it, but now your boss wants you to troubleshoot and find the cause of it. This is where a logging driver comes into play. 

Why Do I Need A Log Driver?:

Running your application on a Docker container in Amazon ECS (with EC2 instance) does not produce a log file in the conventional way. You have to SSH on the EC2 instance running the container and run the “docker log <container name>” command to view the logs generated by your container. This is troublesome and not efficient if you need to view Docker logs on several ECS instances. 

It’s even harder on ECS Fargate tasks because you don’t have the ability to SSH on the containers. The logs generated by your Fargate tasks are not automatically shown on the AWS Console. In order to view logs of ECS containers and Fargate tasks, you will need to configure a log driver.

A log driver will:

  • Aggregate log data across all running tasks  and send them to a central service like CloudWatch Logs where you can view, search and filter them.
  • Saves space by preventing container logs from taking up disk space on your instances when using EC2 launch type.
Tutorials dojo strip

Amazon ECS supports multiple log drivers and awslogs is one of them. Awslogs sends logs from a container to Cloudwatch logs enabling you to conveniently view different logs from your containers on a central location.

Here’s how you can enable awslogs log driver on your ECS Task definitions.

Enabling awslogs:

  • If you are configuring a task definition on a JSON format, you must include the logConfiguration section wherein you will define your parameters such as logDriver, CloudWatch Log Group, AWS region, and the Log group stream prefix. .
    Note: The awslogs log driver can send log streams to an existing log group in CloudWatch Logs or it can create a new log group on your behalf as long as the instance IAM permission has enough CloudWatch permission to create a log group. 

LoggingUsing awslogsLog Driver inAmazo ECS1

  • You must include the logs:CreateLogStream & logs:PutLogEvents permissions on the IAM role associated with your container instances for them to be able to send logs to CloudWatch.
  • If you are using an EC2 launch type, your Amazon ECS container instances should be at least version 1.9.0 of the container agent.
  • The awslogs can either deliver logs to an existing Cloudwatch log group or it can create a log group on your behalf using the auto-configure option.

Enabling Auto-Configure

You can use auto-configure on the Amazon ECS Console. Note that auto-configure with awslogs is enabled by default when creating a task definition using a Fargate Launch type

Step 1: Head over to the ECS Console and click “Create new Task Definition”

LoggingUsing awslogsLog Driver inAmazo ECS2

Step 2:  Choose a launch type. We will use Fargate but you are free to use EC2. Click “Next step”.

LoggingUsing awslogsLog Driver inAmazo ECS3

Step 3: Scroll down and find Container Definitions. Click “Add container”.

LoggingUsing awslogsLog Driver inAmazo ECS4

Step 4: Scroll down until you find Storage and Logging. You will find the auto-configure for Cloudwatch Logs beside the Log configuration option. This is enabled by default on a Fargate launch type.

LoggingUsing awslogsLog Driver inAmazo ECS5

There are three required parameters when configuring awslogs log driver using Fargate:

  1. awslogs-group – the name of the log group to which awslogs driver sends its log streams
  2. awslogs-region – the region to which your awslogs driver sends its logs
  3. awslogs-stream-prefix – allows you to associate a log stream with the specified prefix, the container name, and the ID of the ECS task.

Note: awslogs-stream-prefix is optional when using the EC2 launch type. However, it is required in the Fargate launch type. 

Tip: If you don’t define a value for awslogs-stream-prefix, the log stream will take its name from the container ID which is a long random string. Tracing logs from the container that sent them using only a container ID is difficult, so whenever possible, use awslogs-stream-prefix. It will give you the ability to trace logs by the server name using the prefix name, the container name, and the ID of the ECS task it was sent from.

Viewing Log Group on CloudWatch:

Go to the CloudWatch Console. Click “Log groups”. On the main window, you’ll find the log group that was created for us by Amazon ECS.

AWS Exam Readiness Courses

LoggingUsing awslogsLog Driver inAmazo ECS6

You should now see the container logs on your AWS CloudWatch log group. You can filter the logs based on what you need to investigate, you can configure an expiry for your old container logs, or you can create a CloudWatch Alarm that will detect application log error and send you a notification. 

References:

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_awslogs.html
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-update.html

Tutorials Dojo portal

Be Inspired and Mentored with Cloud Career Journeys!

Tutorials Dojo portal

Enroll Now – Our Azure Certification Exam Reviewers

azure reviewers tutorials dojo

Enroll Now – Our Google Cloud Certification Exam Reviewers

Tutorials Dojo Exam Study Guide eBooks

tutorials dojo study guide eBook

FREE AWS Exam Readiness Digital Courses

Subscribe to our YouTube Channel

Tutorials Dojo YouTube Channel

FREE Intro to Cloud Computing for Beginners

FREE AWS, Azure, GCP Practice Test Samplers

Recent Posts

Written by: Carlo Acebedo

Carlo is a cloud engineer and a content creator at Tutorials Dojo. He's also a member of the AWS Community builder and holds 5 AWS Certifications. Carlo specializes in building and automating solutions in the Amazon Web Services Cloud.

AWS, Azure, and GCP Certifications are consistently among the top-paying IT certifications in the world, considering that most companies have now shifted to the cloud. Earn over $150,000 per year with an AWS, Azure, or GCP certification!

Follow us on LinkedIn, YouTube, Facebook, or join our Slack study group. More importantly, answer as many practice exams as you can to help increase your chances of passing your certification exams on your first try!

View Our AWS, Azure, and GCP Exam Reviewers Check out our FREE courses

Our Community

~98%
passing rate
Around 95-98% of our students pass the AWS Certification exams after training with our courses.
200k+
students
Over 200k enrollees choose Tutorials Dojo in preparing for their AWS Certification exams.
~4.8
ratings
Our courses are highly rated by our enrollees from all over the world.

What our students say about us?