View Log Files of Crashed Pods in Kubernetes

Learn how to view log files of crashed pods in Kubernetes. Our Kubernetes Support team is here to answer queries and concerns.

View Log Files of Crashed Pods in Kubernetes

Identifying the cause behind a pod crash can feel like solving a mystery, especially when the pod is no longer running.

Fortunately, Kubernetes offers several tools and strategies to determine what went wrong. In this blog, we’ll walk through how to retrieve logs from failed pods and dig deeper when the standard approaches fail.

Why Pod Logs Matter

Pod logs are invaluable when it comes to diagnosing:

• Application runtime errors
• Container startup failures
• Resource limit violations (e.g., memory or CPU)
• Kubernetes orchestration issues

Even if a pod is no longer running, Kubernetes briefly retains logs. This is long enough to investigate if we act quickly.

Also, understanding the different Kubernetes pod states can help you determine where things went wrong in the pod lifecycle.

Quick Checks with kubectl logs

1. If the pod is still active (even if it has restarted several times), we can view its logs with:

   kubectl logs pod-name

   To include the namespace (if it’s not in the default one):

   kubectl -n namespace logs pod-name
2. If a pod has crashed and restarted, use the `–previous` flag to see the logs from its last terminated state:

   kubectl logs pod-name –previous

   This is especially helpful in cases of CrashLoopBackOff errors, where the pod keeps restarting due to an unresolved issue.

   However, if the container is completely gone, we may see an error like:

   Error from server (BadRequest): previous terminated container “container-name” in pod “pod-name” not found

Digging Deeper When Logs Aren’t Available

If standard logging fails, we can proceed with checking pod events. They give us more insight into why a pod failed to start:

kubectl get events -n namespace –sort-by='.metadata.creationTimestamp'

Alternatively, describe the pod:

kubectl describe pod pod-name -n namespace

Look for the “Events” section, which usually contains reasons such as resource constraints or image pull failures.

For an even deeper dive into troubleshooting techniques, check out this step-by-step Kubernetes troubleshooting guide our support engineers use.

Common Failure Indicators in Pod Status

When describing a failed pod, we may see output like this:

Resource Exhaustion (OOMKilled)

State:          Waiting

Reason:       CrashLoopBackOff

Last State:     Terminated

Reason:       OOMKilled

Exit Code:    137

Application Error

Last State:     Terminated

Reason:       Error

Exit Code:    2

In either case, the “Reason” field and “Exit Code” will point us in the right direction. Sometimes, restarting the pod is all it takes to resolve a transient issue. Here’s how you can restart Kubernetes pods using kubectl.

Advanced Debugging: Node-Level Investigation

If we still can’t find the root cause:

1. Find the node where the pod was running with this command:

   kubectl get pod pod-name -n namespace -o=custom-columns=NAME:.metadata.name,STATUS:.status.phase,NODE:.spec.nodeName
2. Once we know which node the pod ran on, SSH into it and check the logs directly:

   
   
   cd /var/log/containers/
   
   ls | grep pod-name
   
   cat container-log-file.log
   
   

   Search for any errors or unusual messages.
3. Furthermore, use `journalctl` to check for Kubelet or system-level errors:

   sudo journalctl -u kubelet -n 100 –no-pager

Debugging AKS Nodes

If we are using AKS, we can SSH into a node with this command:

kubectl debug node/ -it --image=mcr.microsoft.com/dotnet/runtime-deps:6.0

chroot /host 

This gives us root-level access to the node to investigate logs as needed.

Bonus Tips

• Get logs for a specific container in a pod:

  kubectl logs pod-name --container container-name
• Tail logs in real-time:

  kubectl logs -f pod-name
• View logs from pods with a specific label:

  kubectl logs -l app=blue-green
• Watch events as they happen:

  kubectl get events -w | grep pod-name

[Need assistance with a different issue? Our team is available 24/7.]

Conclusion

When a pod crashes, we need quick access to logs. While `kubectl logs` is the go-to command, it may not give us the answer we seek. By checking events, digging into node-level logs, or even SSH-ing into the node, we can uncover the root cause behind the error.

In brief, our Support Experts demonstrated how to view log files of crashed pods in Kubernetes.