Generally, an Amazon ECS task definition allows us to specify a single log configuration object for a given container. However, to send Container Logs to Multiple Destinations in ECS, we can use FireLens.

Here, at Bobcares, we assist our customers with several AWS queries as part of our AWS Support Services.

Today. let us see methods to send Container Logs to Multiple Destinations.

Send Container Logs to Multiple Destinations in ECS

FireLens creates a configuration file on your behalf, but you can also specify a custom configuration file.

We can host this configuration file in either Amazon Simple Storage Service (Amazon S3), or create a custom Fluent Bit Docker image with the custom output configuration file added to it.

Moving ahead, let us see how our Support Techs perform this query for our customers.

Create AWS Identity and Access Management (IAM) permissions

To allow the task role to route the logs to different destinations, we need to create IAM permissions.

For example, suppose the destination is Kinesis Data Firehose. Then we give the task permission to call the firehose:PutRecordBatch API.

Create a Fluent Bit Docker image with a custom output configuration file

1. We need to create a custom Fluent Bit configuration file, logDestinations.conf with our choice of [OUTPUT] definitions defined in it.

For example, here have configurations defined for CloudWatch, Kinesis Data Firehose, and Splunk.

[OUTPUT]
Name firehose
Match YourContainerName*
region us-west-2
delivery_stream nginx-stream
[OUTPUT]
Name cloudwatch
Match YourContainerName*
region us-east-1
log_group_name firelens-nginx-container
log_stream_prefix from-fluent-bit
auto_create_group true
[OUTPUT]
Name splunk
Match *
Host 127.0.0.1
Splunk_Token xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx
Splunk_Send_Raw On

2. Then we create a Docker image with a custom Fluent Bit output configuration file like the following:

FROM amazon/aws-for-fluent-bit:latest
ADD logDestinations.conf /logDestinations.conf

3. With this Dockerfile we then create the custom fluent-bit Docker image. To do so, we run:

docker build -t custom-fluent-bit:latest .

We ensure to run the docker build command in the same location as the Dockerfile.

4. To confirm that the Docker image is available to Amazon ECS, we push the Docker image to Amazon ECR or our own Docker registry.

For example, to push a local Docker image to Amazon ECR, we run:

docker push aws_account_id.dkr.ecr.region.amazonaws.com/custom-fluent-bit:latest

5. After that, in the task definition (TaskDefinition), we update the options for the FireLens configuration. For example:

{
"containerDefinitions":[
{
"essential":true,
"image":"aws_account_id.dkr.ecr.region.amazonaws.com/custom-fluent-bit:latest",
"name":"log_router",
"firelensConfiguration":{
"type":"fluentbit",
"options":{
"config-file-type":"file",
"config-file-value":"/logDestinations.conf"
}
}
}
]
}

While on it, we need to consider the following:

We need to specify a custom configuration file, include the config-file-type and config-file-value options in the FireLens configuration file.

Then we must modify the image property in the containerDefinition section of the configuration to reflect a valid Amazon ECR image location.

To specify images in Amazon ECR repositories we can use the full registry/repository:tag naming convention.

For example,

aws_account_id.dkr.ecr.region.amazonaws.com/custom-fluent-bit:latest

[Stuck in between? We’d be happy to assist]

Conclusion

In short, we saw how our Support Techs Send Container Logs to Multiple Destinations in ECS.