Monitoring and logging can be as important as developing and deploying applications. Imagine managing an enterprise-level distributed application that runs on hundreds of containers. You must place your containers cleverly on instances to achieve a certain level of resilience and performance. On top of that, you would want to have visibility into CPU and memory usage so you’ll have an idea which task/tasks are resource-hungry at a given time. You may also want to check the disk space of your instance and see if you still have enough storage for your application.

Fortunately, monitoring distributed applications hosted in AWS has never been easier. Cloudwatch natively integrates with ECS so you can send and view ECS metrics on your Cloudwatch dashboard.

So when the time comes where your CPU usage overloads and your applications start to crawl and slow down, CloudWatch will alarm and notify you what is happening. But what if you suddenly get a spike in request rates and you need to know which customer is causing it, or there have been multiple error rates and you need to identify which API calls are responsible for it? Sure, you’ll be notified about it, but now your boss wants you to troubleshoot and find the cause of it. This is where a logging driver comes into play. 

Why Do I Need A Log Driver?:

Running your application on a Docker container in Amazon ECS (with EC2 instance) does not produce a log file in the conventional way. You have to SSH on the EC2 instance running the container and run the “docker log <container name>” command to view the logs generated by your container. This is troublesome and not efficient if you need to view Docker logs on several ECS instances. 

It’s even harder on ECS Fargate tasks because you don’t have the ability to SSH on the containers. The logs generated by your Fargate tasks are not automatically shown on the AWS Console. In order to view logs of ECS containers and Fargate tasks, you will need to configure a log driver.

A log driver will:

• Aggregate log data across all running tasks  and send them to a central service like CloudWatch Logs where you can view, search and filter them.
• Saves space by preventing container logs from taking up disk space on your instances when using EC2 launch type.

Amazon ECS supports multiple log drivers and awslogs is one of them. Awslogs sends logs from a container to Cloudwatch logs enabling you to conveniently view different logs from your containers on a central location.

Here’s how you can enable awslogs log driver on your ECS Task definitions.

Enabling awslogs:

• If you are configuring a task definition on a JSON format, you must include the logConfiguration section wherein you will define your parameters such as logDriver, CloudWatch Log Group, AWS region, and the Log group stream prefix. .
  Note: The awslogs log driver can send log streams to an existing log group in CloudWatch Logs or it can create a new log group on your behalf as long as the instance IAM permission has enough CloudWatch permission to create a log group. 

• You must include the logs:CreateLogStream & logs:PutLogEvents permissions on the IAM role associated with your container instances for them to be able to send logs to CloudWatch.
• If you are using an EC2 launch type, your Amazon ECS container instances should be at least version 1.9.0 of the container agent.
• The awslogs can either deliver logs to an existing Cloudwatch log group or it can create a log group on your behalf using the auto-configure option.

Enabling Auto-Configure

You can use auto-configure on the Amazon ECS Console. Note that auto-configure with awslogs is enabled by default when creating a task definition using a Fargate Launch type. 

Step 1: Head over to the ECS Console and click “Create new Task Definition”

Step 2:  Choose a launch type. We will use Fargate but you are free to use EC2. Click “Next step”.

Step 3: Scroll down and find Container Definitions. Click “Add container”.

Step 4: Scroll down until you find Storage and Logging. You will find the auto-configure for Cloudwatch Logs beside the Log configuration option. This is enabled by default on a Fargate launch type.

There are three required parameters when configuring awslogs log driver using Fargate:

1. awslogs-group – the name of the log group to which awslogs driver sends its log streams
2. awslogs-region – the region to which your awslogs driver sends its logs
3. awslogs-stream-prefix – allows you to associate a log stream with the specified prefix, the container name, and the ID of the ECS task.

Note: awslogs-stream-prefix is optional when using the EC2 launch type. However, it is required in the Fargate launch type. 

Tip: If you don’t define a value for awslogs-stream-prefix, the log stream will take its name from the container ID which is a long random string. Tracing logs from the container that sent them using only a container ID is difficult, so whenever possible, use awslogs-stream-prefix. It will give you the ability to trace logs by the server name using the prefix name, the container name, and the ID of the ECS task it was sent from.

Viewing Log Group on CloudWatch:

Go to the CloudWatch Console. Click “Log groups”. On the main window, you’ll find the log group that was created for us by Amazon ECS.

You should now see the container logs on your AWS CloudWatch log group. You can filter the logs based on what you need to investigate, you can configure an expiry for your old container logs, or you can create a CloudWatch Alarm that will detect application log error and send you a notification. 

References:

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_awslogs.html
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-update.html