Deploy and Run Filebeat on Kubernetes for Complete Log Monitoring

Run Filebeat on Kubernetes to collect container logs and monitor your cluster easily. Our Kubernetes Support team is ready to assist you. 

Deploy and Run Filebeat on Kubernetes for Complete Log Monitoring

Monitoring logs from containers in a Kubernetes cluster can be challenging, especially when dealing with multiple nodes and dynamic workloads. Filebeat simplifies this process by collecting logs from all containers, adding Kubernetes metadata, and sending them to Elasticsearch for easy analysis. This guide explains how to deploy and configure Filebeat on Kubernetes and Red Hat OpenShift, load dashboards in Kibana, and ensure logs flow smoothly from every node in your cluster.

Deploy Filebeat on Kubernetes Using Docker

Filebeat collects logs from all containers in a Kubernetes cluster and sends them to Elasticsearch. Running Filebeat as a DaemonSet ensures every node in the cluster has an active instance capturing logs. Our expert take is that following Kubernetes security best practices is essential when configuring log collection to avoid exposing sensitive data.

1. Configure Filebeat

Create a configuration file that tells Filebeat which logs to collect, how to add Kubernetes metadata, and where to send the logs.

2. Store Configuration in Kubernetes

Use a ConfigMap to make the Filebeat configuration accessible to all pods. This allows the DaemonSet to use the configuration on every node.

3. Deploy as a DaemonSet

Set up Filebeat as a DaemonSet so it runs on every node. Make sure the pods have access to log directories on the host and the configuration from the ConfigMap. This process becomes particularly straightforward if you are performing a Kubernetes cluster deployment on Proxmox 8, as Proxmox provides flexible virtualization for the nodes

4. Set Permissions

Create a ServiceAccount with proper permissions and attach it to Filebeat. This lets it read Kubernetes resources such as pods, nodes, and namespaces for metadata enrichment.

5. Apply the Deployment

Deploy Filebeat in the cluster by applying the manifests. Once running, Filebeat will collect logs from all containers, enrich them with metadata, and forward them to Elasticsearch or the configured output.

Configuring Filebeat on Red Hat OpenShift

Setting up Filebeat on OpenShift requires a few key adjustments to collect logs from all nodes effectively.

• Grant Elevated Permissions

Filebeat must run with root-level access to read log directories on the host. This allows it to access system files and collect container logs across the cluster.

• Provide Service Account Access

The Filebeat service account needs permission to run privileged containers. Assigning these rights ensures Filebeat can operate under OpenShift security policies.

• Allow Scheduling on All Nodes

Some nodes, including master nodes, are often excluded from scheduling. Adjust the namespace settings to remove restrictions so Filebeat can run on every node.

• Set Paths for Container Logs

Specify where container logs are stored, especially for CRI-O or containerd environments. Logs are typically located in a container logs directory, often linked to pod log folders.

• Configure Autodiscovery

If Filebeat uses autodiscovery, make sure it follows the same log paths. This allows automatic detection and collection of logs from all containers without manual configuration.

Loading Filebeat Dashboards for Kubernetes Logs in Kibana

Filebeat offers ready-made dashboards that make it easy to visualize Kubernetes logs. These dashboards help you monitor container activity, pod performance, and cluster health.

1. Install Filebeat

Set up Filebeat on a system that can connect to Elasticsearch and Kibana. Make sure the system has proper network access so data flows without interruption.

2. Confirm Kibana is Running

Check that your Kibana instance is operational and accessible. If it requires authentication, ensure Filebeat has the correct credentials to connect.

3. Load Dashboards

Use Filebeat’s setup feature to import dashboards, visualizations, and index patterns into Kibana and Elasticsearch. This step also adds the recommended index template to organize incoming data effectively.

4. Special Case for Logstash

If Filebeat sends logs to Logstash, dashboards and pipelines do not load automatically. You must import index templates, dashboards, and ingest pipelines separately after configuring Filebeat for Logstash output.

5. Verify in Kibana

Open Kibana and navigate to the Dashboards section. You should see dashboards for various Filebeat modules. For Kubernetes logs, ensure the relevant module is active so you can view container and cluster metrics immediately.

Deploying Filebeat on Kubernetes

Deploy Filebeat to collect logs from all nodes in your Kubernetes cluster. This ensures every container’s activity is tracked and sent to Elasticsearch with detailed metadata.

1. Deploy Filebeat

Apply the Filebeat manifest to your cluster to create a DaemonSet. This runs Filebeat on every node, collecting logs automatically from all containers.

2. Verify Deployment

Check that all Filebeat pods are running and ready across the cluster. This confirms that log collection is active and functioning properly.

3. Log Flow to Elasticsearch

Once Filebeat is active, logs start flowing into Elasticsearch. Each event carries Kubernetes metadata, providing context such as pod names, namespaces, and container details. This makes it easier to monitor cluster activity and troubleshoot issues quickly.

[Need assistance with a different issue? Our team is available 24/7.]

Conclusion 

Running Filebeat on Kubernetes collects logs from all nodes and containers, adds metadata, and provides clear insights through Kibana dashboards. This setup ensures efficient monitoring and easy troubleshooting across your cluster.