Splunk log driver with ECS task on Fargate helps us to send the container logs to the Enterprise Splunk log collector or Splunk Cloud.

Here, at Bobcares, we assist our customers with several AWS queries as part of our AWS Support Services.

Today, let us see how to use Splunk Log Driver.

Use Splunk Log Driver with ECS task on Fargate

In this article, our Support Techs recommends steps to set up and configure the same.

• Set up the Splunk Cloud environment to receive Fargate logs

1. In order to create a Splunk Cloud account we use the Free Splunk option.

2. We sign in to the account.

3. Then from the navigation menu, we select Settings.

4. And in the DATA section, we select Data inputs.

5. We then go to the Local inputs section, in the Actions column, we select, Add new.

6. Here, we can enter a name for the token, and to go ahead, we select, Next.

7. We go through the rest of setup wizard, and then select Submit to create the HEC token.

8. After that, we test the new token using either curl or Docker.

Via curl:

curl -k https://prd-p-9l79p.splunkcloud.com:8088/services/collector -H "Authorization: Splunk splunk-token" -d '{"event": "Put this message in Splunk"}'

Via Docker:

docker run --publish 80:80 --log-driver=splunk --log-opt splunk-token=splunk token --log-opt splunk-url=https://prd-p-u7z1u.splunkcloud.com:8088 --log-opt splunk-insecureskipverify=true --log-opt splunk-sourcetype=docker-test --log-opt splunk-index=main nginx

• Configure AWS Secrets Manager to use the HEC token

We store the HEC token in AWS Secrets Manager. By doing so Fargate can use it.

To use it securely, we run the following command:

aws secretsmanager create-secret --name splunk-token --secret-string token-value

• Configure a Task Execution role with permission to access secrets

1. Initially, we open the AWS IAM console.

2. Then we select, Roles. From the list, we search and select, ecsTaskExecutionRole.

4. Eventually, we select Add inline policy.

5. We select the JSON tab of the editor and enter the following policy:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"secretsmanager:GetSecretValue",
"kms:Decrypt"
],
"Resource": [
"arn:aws:secretsmanager:<region>:<aws_account_id>:secret:<secret_name>",
"arn:aws:kms:<region>:<aws_account_id>:key/<key_id>"
]
}
]
}

In case of a custom AWS KMS key, we must provide the kms:Decrypt permission and ARN for the KMS key.

6. Then we select Review policy.

7. For Name, we enter a name for the policy.

8. Finally, we go ahead and click, Create policy.

• Configure the task definition for the Amazon ECS task on Fargate

1. Firstly, we open the Amazon ECS console.

2. Then we select, Amazon ECS > Task Definitions > Create new Task Definition.

4. Select Fargate > Next step.

5. In the Configure task and container definitions section, for Task definition name, we provide a name.

6. For Task Role, we select the role that we configured earlier.

7. In the Task size section, for Task memory (GB) and Task CPU (vCPU), we enter the appropriate values.

8. Then in the Container Definitions section, we select Add container.

9. Here, we can select appropriate options for the container needs.

10. In the STORAGE AND LOGGING section, we clear the Log configuration check box.

11. For Log driver, we select splunk.

12. After that, for the key and value options in Log options, we enter:

Key: splunk-url
Value: Your splunk endpoint (For example: https://prd-p-9l79p.splunkcloud.com:8088/)

Key: splunk-insecureskipverify
Value: True

Key: splunk-token
Value: The Secret ARN that you created earlier

13. Eventually, we select, Add.

14. Then to create the task definition, we click on Create.

15. Now, we can create an Amazon ECS service or run a task that uses the task definition.

• Check the logs in Splunk Cloud

In order to check if the logs in Splunk Cloud, we need to make sure the task runs.

1. Initially, we connect to the Splunk Cloud account.

2. Then from the navigation menu, we select, Search.

3. Eventually, we enter index=”main” in the search box.

Now, we can see the logs in the Splunk log collector.

[Stuck with the procedures? We are here for you]

Conclusion

In short, we saw how our Support Techs configure Splunk log driver.