Ends in
00
days
00
hrs
00
mins
00
secs
LEARN MORE

SALE! AWS Specialty Practice Exams at $15.99 USD each ONLY instead of $17.99

Logging Using awslogs Log Driver in Amazon ECS

Monitoring and logging can be as important as developing and deploying applications. Imagine managing an enterprise-level distributed application that runs on hundreds of containers. You must place your containers cleverly on instances to achieve a certain level of resilience and performance. On top of that, you would want to have visibility into CPU and memory usage so you’ll have an idea which task/tasks are resource-hungry at a given time. You may also want to check the disk space of your instance and see if you still have enough storage for your application.

Fortunately, monitoring distributed applications hosted in AWS has never been easier. Cloudwatch natively integrates with ECS so you can send and view ECS metrics on your Cloudwatch dashboard.

So when the time comes where your CPU usage overloads and your applications start to crawl and slow down, CloudWatch will alarm and notify you what is happening. But what if you suddenly get a spike in request rates and you need to know which customer is causing it, or there have been multiple error rates and you need to identify which API calls are responsible for it? Sure, you’ll be notified about it, but now your boss wants you to troubleshoot and find the cause of it. This is where a logging driver comes into play. 

Why Do I Need A Log Driver?:

Running your application on a Docker container in Amazon ECS (with EC2 instance) does not produce a log file in the conventional way. You have to SSH on the EC2 instance running the container and run the “docker log <container name>” command to view the logs generated by your container. This is troublesome and not efficient if you need to view Docker logs on several ECS instances. 

It’s even harder on ECS Fargate tasks because you don’t have the ability to SSH on the containers. The logs generated by your Fargate tasks are not automatically shown on the AWS Console. In order to view logs of ECS containers and Fargate tasks, you will need to configure a log driver.

A log driver will:

  • Aggregate log data across all running tasks  and send them to a central service like CloudWatch Logs where you can view, search and filter them.
  • Saves space by preventing container logs from taking up disk space on your instances when using EC2 launch type.

Amazon ECS supports multiple log drivers and awslogs is one of them. Awslogs sends logs from a container to Cloudwatch logs enabling you to conveniently view different logs from your containers on a central location.

IT Certification Category (English)728x90

Here’s how you can enable awslogs log driver on your ECS Task definitions.

Enabling awslogs:

  • If you are configuring a task definition on a JSON format, you must include the logConfiguration section wherein you will define your parameters such as logDriver, CloudWatch Log Group, AWS region, and the Log group stream prefix. .
    Note: The awslogs log driver can send log streams to an existing log group in CloudWatch Logs or it can create a new log group on your behalf as long as the instance IAM permission has enough CloudWatch permission to create a log group.

LoggingUsing awslogsLog Driver inAmazo ECS1

  • You must include the logs:CreateLogStream & logs:PutLogEvents permissions on the IAM role associated with your container instances for them to be able to send logs to CloudWatch.
  • If you are using an EC2 launch type, your Amazon ECS container instances should be at least version 1.9.0 of the container agent.
  • The awslogs can either deliver logs to an existing Cloudwatch log group or it can create a log group on your behalf using the auto-configure option.

Enabling Auto-Configure

You can use auto-configure on the Amazon ECS Console. Note that auto-configure with awslogs is enabled by default when creating a task definition using a Fargate Launch type

Step 1: Head over to the ECS Console and click “Create new Task Definition”

LoggingUsing awslogsLog Driver inAmazo ECS2

Step 2:  Choose a launch type. We will use Fargate but you are free to use EC2. Click “Next step”.

LoggingUsing awslogsLog Driver inAmazo ECS3

Step 3: Scroll down and find Container Definitions. Click “Add container”.

LoggingUsing awslogsLog Driver inAmazo ECS4

Step 4: Scroll down until you find Storage and Logging. You will find the auto-configure for Cloudwatch Logs beside the Log configuration option. This is enabled by default on a Fargate launch type.

LoggingUsing awslogsLog Driver inAmazo ECS5

There are three required parameters when configuring awslogs log driver using Fargate:

  1. awslogs-group – the name of the log group to which awslogs driver sends its log streams
  2. awslogs-region – the region to which your awslogs driver sends its logs
  3. awslogs-stream-prefix – allows you to associate a log stream with the specified prefix, the container name, and the ID of the ECS task.

Note: awslogs-stream-prefix is optional when using the EC2 launch type. However, it is required in the Fargate launch type. 

Tip: If you don’t define a value for awslogs-stream-prefix, the log stream will take its name from the container ID which is a long random string. Tracing logs from the container that sent them using only a container ID is difficult, so whenever possible, use awslogs-stream-prefix. It will give you the ability to trace logs by the server name using the prefix name, the container name, and the ID of the ECS task it was sent from.

Viewing Log Group on CloudWatch:

Go to the CloudWatch Console. Click “Log groups”. On the main window, you’ll find the log group that was created for us by Amazon ECS.

LoggingUsing awslogsLog Driver inAmazo ECS6

You should now see the container logs on your AWS CloudWatch log group. You can filter the logs based on what you need to investigate, you can configure an expiry for your old container logs, or you can create a CloudWatch Alarm that will detect application log error and send you a notification. 

Sources:
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_awslogs.html
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-update.html

AWS Specialty Practice Exams SALE!

Tutorials Dojo Study Guide and Cheatsheet

NEW! AWS Certified Developer Associate Video Course (Early Access Release)

NEW! AWS Certified Solutions Architect Associate Video Course [Early Access Release]

Pass your AWS, Azure, and Google Cloud Certifications with the Tutorials Dojo Portal

Tutorials Dojo portal

Our Bestselling AWS Certified Solutions Architect Associate Practice Exams

AWS Certified Solutions Architect Associate Practice Exams

Enroll Now – Our AWS Practice Exams with 95% Passing Rate

AWS Practice Exams Tutorials Dojo

FREE AWS Cloud Practitioner Essentials Course!

Enroll Now – Our Azure Certification Exam Reviewers

azure reviewers tutorials dojo

Enroll Now – Our Google Cloud Certification Exam Reviewers

Tutorials Dojo Exam Study Guide eBooks

tutorials dojo study guide eBook

Subscribe to our YouTube Channel

Tutorials Dojo YouTube Channel

FREE Intro to Cloud Computing for Beginners

FREE AWS, Azure, GCP Practice Test Samplers

Browse Other Courses

Generic Category (English)300x250

Recent Posts

AWS, Azure, and GCP Certifications are consistently among the top-paying IT certifications in the world, considering that most companies have now shifted to the cloud. Earn over $150,000 per year with an AWS, Azure, or GCP certification!

Follow us on LinkedIn, YouTube, Facebook, or join our Slack study group. More importantly, answer as many practice exams as you can to help increase your chances of passing your certification exams on your first try!

View Our AWS, Azure, and GCP Exam Reviewers

Our Community

~98%
passing rate
Around 95-98% of our students pass the AWS Certification exams after training with our courses.
200k+
students
Over 200k enrollees choose Tutorials Dojo in preparing for their AWS Certification exams.
~4.8
ratings
Our courses are highly rated by our enrollees from all over the world.

What our students say about us?

error: Content is protected !!