Export Kubernetes Pod Logs to Upstash Kafka Using Fluent Bit

Learn how to collect and stream Fluent Bit Kubernetes pod logs to an Upstash Kafka topic to build an efficient log pipeline for cloud monitoring and management. Optimize your Kubernetes logging workflow with our Cloud Management services.

Log pipelines play a critical part in Cloud Management because they deliver the visibility needed to detect issues, automate responses, and maintain platform health. Fluent Bit, Kafka, and Upstash together offer a simple way to create a cloud-native logging flow that pushes Kubernetes logs into a scalable event stream without managing heavy infrastructure.

Today, we will walk through setting up Fluent Bit inside a cluster, forwarding selected pod logs to an Upstash Kafka topic, and consuming these records through a lightweight Go server.

Why Fluent Bit and Upstash Fit Well in Cloud Management

Fluent Bit is a compact log processor designed for cloud environments. It supports a large collection of input and output plugins, which allows teams to collect logs, shape them as needed, and forward them to various storage and analytics systems.

Kafka output support is already included in Fluent Bit. This means that any log source collected inside the cluster can be shipped directly into a Kafka topic. Upstash provides a fully managed serverless Kafka service with a pay-per-message model, which helps teams avoid the operational load of maintaining brokers. The cost scales down to zero when there is no traffic, and the first ten thousand messages each day are free.

Together, this combination gives Cloud Management teams an efficient way to create a scalable log pipeline without provisioning or tuning cluster resources.

Prerequisites:

To follow the walkthrough, prepare the following:

• An Upstash account
• Any Kubernetes setup, such as minikube or Docker Desktop
• Helm installed locally

Step 1. Create the Kafka Topic in Upstash

Open the Upstash console, go to the Kafka section, and create a new cluster. Add a topic named logs, which will serve as the destination for the pod log stream.

Step 2. Deploy Fluent Bit to Kubernetes

Use the official Helm charts:

helm repo add fluent https://fluent.github.io/helm-charts

Before installation, prepare custom configuration files. Fluent Bit needs three sets of definitions:

Input Configuration

[INPUT]

Name tail

Path /var/log/containers/*_upstashed_*.log

multiline.parser docker, cri

Tag kube.*

Mem_Buf_Limit 5MB

Skip_Long_Lines On

This setup watches only pod logs inside the upstashed namespace. The tail plugin tracks file updates and ships each new log line into the pipeline.

Filter Configuration

[FILTER]

Name kubernetes

Match kube.*

Merge_Log On

Keep_Log Off

K8S-Logging.Parser On

K8S-Logging.Exclude On

[FILTER]

Name nest

Match *

Operation lift

Nested_under kubernetes

Add_prefix k8s_

The first filter enriches each record with Kubernetes metadata. The second filter lifts these fields to the top level with a prefix. This adjustment allows the Kafka output plugin to reference fields such as `k8s_pod_name` directly as message keys.

Output Configuration

[OUTPUT]

Name kafka

Match kube.*

Brokers <broker-url>

Topics logs

Message_Key_Field k8s_pod_name

rdkafka.security.protocol sasl_ssl

rdkafka.sasl.mechanism SCRAM-SHA-256

rdkafka.sasl.username <username>

rdkafka.sasl.password <password>

Use the topic created earlier and set the pod name as the message key to help downstream consumers group log entries.

Step 3. Install Fluent Bit

Deploy Fluent Bit as a DaemonSet:

helm install fluent-bit -n fluent-bit --create-namespace fluent/fluent-bit \

--set-file config.inputs=input.conf \

--set-file config.filters=filter.conf \

--set-file config.outputs=output.conf

Verify that the DaemonSet is running:

kubectl get ds -n fluent-bit

Check the logs to ensure a successful Kafka connection:

kubectl logs -f ds/fluent-bit -n fluent-bit

Step 4. Generate Logs in a Test Namespace

Create a namespace and deploy an application that prints random logs:

kubectl create ns upstashed

kubectl create deployment random-logger -n upstashed \

--image=chentex/random-logger:v1.0.1 -- /entrypoint.sh 7500 7500 1000

This container emits log lines at intervals, which you can watch through:

kubectl logs -f deployment/random-logger -n upstashed

Once active, these records start flowing into the Kafka topic in your Upstash console. When managing Kubernetes environments, balancing performance and costs is essential. Read our article on Kubernetes cost optimization for cloud savings to find techniques that align scaling with efficient resource use.

Conclusion

This setup creates a log pipeline that forwards Kubernetes pod logs into a managed Kafka service without the overhead of broker maintenance. It helps Cloud Management teams enhance observability, scale easily through serverless Kafka, work with metadata-rich records suitable for alerting and automation, and stay flexible since Fluent Bit can deliver logs to many different backends.