Wondering how to configure AWS MSK authentication? We can help you.

Here, at Bobcares, we assist our customers with several AWS queries as part of our AWS Support Services.

Today, let us see how our Support techs assist with this query.

How to configure AWS MSK authentication?

AWS Lambda is introducing mutual TLS (mTLS) authentication for Amazon MSK and self-managed Kafka as an event source.

By default, the TLS protocol only requires a server to authenticate itself to the client.

The authentication of the client to the server is manage by the application layer.

The TLS protocol also offers the ability for the server to request that the client send an X.509 certificate to prove its identity.

This is called mutual TLS as both parties are authenticated via certificates with TLS.

Today, let us see the steps followed by our Support Techs for using mutual TLS authentication for Amazon MSK as event source for Lambda are:

1. Firstly, create a private certificate authority (CA) using AWS Certificate Manager (ACM) Private Certificate Authority (PCA).
2. Next, create a client certificate and private key. Store them as secret in AWS Secrets Manager.
3. Then, create an Amazon MSK cluster and a consuming Lambda function using the AWS Serverless Application Model (AWS SAM).
4. Finally, cttach the event source mapping.

1. Creating a private CA.

To use mutual TLS client authentication with Amazon MSK, create a root CA using AWS ACM Private Certificate Authority (PCA).

1. Firstly, from the AWS Certificate Manager console, choose Create a Private CA.
2. Then, in the Select CA type panel, select Root CA and choose Next.
3. Next, in the Configure CA subject name panel, provide your certificate details, and choose Next.
4. Then, from the Configure CA key algorithm panel, choose the key algorithm for your CA and choose Next.
5. From the Configure revocation panel, choose any optional certificate revocation options you require and choose Next.
6. Continue through the screens to add any tags required, allow ACM to renew certificates, review your options, and confirm pricing. Choose Confirm and create.
7. Once the CA is create, choose Install CA certificate to activate your CA. Configure the validity of the certificate and the signature algorithm and choose Next.
8. Finally, review the certificate details and choose Confirm and install.

2. Creating a client certificate.

You generate a client certificate using the root certificate you previously create, which is used to authenticate the client with the Amazon MSK cluster using mutual TLS.

1. On your local machine, run the following command to create a private key and certificate signing request using OpenSSL. Enter your certificate details. This creates a private key file and a certificate signing request file in the current directory.
2. Use the AWS CLI to sign your certificate request with the private CA previously created. Replace Private-CA-ARN with the ARN of your private CA. The certificate validity value is set to 300, change this if necessary. Save the certificate ARN provided in the response.
3. Then, retrieve the certificate that ACM signed for you. Replace the Private-CA-ARN and Certificate-ARN with the ARN you obtained from the previous commands. This creates a signed certificate file called client_cert.pem.
4. Create a new file called secret.json with the following structure
5. Copy the contents of the client_cert.pem in certificate and the content of key.pem in privatekey. Ensure that there are no extra spaces added.The file structure looks like this:
6. Finally, create the secret and save the ARN for the next section.

aws secretsmanager create-secret --name msk/mtls/lambda/clientcert --secret-string file://secret.json

3. Setting up an Amazon MSK cluster with AWS Lambda as a consumer.

Amazon MSK is a highly available service, so it must be configured to run in a minimum of two Availability Zones in your preferred Region.

To comply with security best practice, the brokers are usually configured in private subnets in each Region.

You can use AWS CLI, AWS Management Console, AWS SDK and AWS CloudFormation to create the cluster and the Lambda functions.

This blog uses AWS SAM to create the infrastructure and the associated code is available in the GitHub repository.

The AWS SAM template creates the following resources:

1. Firstly, amazon Virtual Private Cloud (VPC).
2. Secondly, amazon MSK cluster with mutual TLS authentication.
3. Then, lambda function for consuming the records from the Amazon MSK cluster.
4. IAM roles.
5. ambda function for testing the Amazon MSK integration by publishing messages to the topic.

The VPC has public and private subnets in two Availability Zones with the private subnets configured to use a NAT Gateway.

You can also set up VPC endpoints with PrivateLink to allow the Amazon MSK cluster to communicate with Lambda.

The Lambda function requires permission to describe VPCs and security groups, and manage elastic network interfaces to access the Amazon MSK data stream.

The Lambda function also needs two Kafka permissions: kafka:DescribeCluster and kafka:GetBootstrapBrokers.

The policy template AWSLambdaMSKExecutionRole includes these permissions.

The Lambda function also requires permission to get the secret value from AWS Secrets Manager for the secret you configure in the event source mapping.

ConsumerLambdaFunctionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service: lambda.amazonaws.com
Action: sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaMSKExecutionRole
Policies:
- PolicyName: SecretAccess
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action: "SecretsManager:GetSecretValue"
Resource: "*"

YAML

This release adds two new SourceAccessConfiguration types to the Lambda event source mapping:

1. CLIENT_CERTIFICATE_TLS_AUTH – (Amazon MSK, Self-managed Apache Kafka)

2. SERVER_ROOT_CA_CERTIFICATE – This is only for self-managed Apache Kafka.

[Need help with the process? We’d be happy to assist]

Conclusion

In short, we saw steps followed by our Support Techs to configure AWS MSK authentication