Learn how to fix Kubectl top Node “Metrics API Not Available” error in Amazon EKS. Our Kubernetes Support team is here to help you with your questions and concerns.

Kubectl top Node “Metrics API Not Available” Error in Amazon EKS

Recently, one of our customers ran into the “Metrics API not available” error after running these commands:

kubectl apply -f metrics-server.yaml
kubectl top node


According to our experts, this error occurs when the Kubernetes Metrics Server is either not running or is inaccessible.

The Kubernetes Metrics Server is critical for collecting and serving resource metrics.

The `kubectl top` command uses the Kubernetes Metrics API to display real-time CPU and memory usage statistics. However, in Amazon EKS clusters, the Metrics API may not be available by default or may not be configured properly.

An Overview:

• Common Causes of the Error
• Steps to Troubleshoot and Resolve
• Additional Tips

Common Causes of the Error

1. In many EKS clusters, the Metrics Server is not deployed by default, making it impossible for `kubectl top` to retrieve metrics.
2. Incorrect configurations can prevent the Metrics Server from collecting data or communicating with the Kubernetes API server.
3. Misconfigured RBAC can block the Metrics Server from accessing the required node and pod data.
4. Security groups, firewalls, or network policies may restrict communication between the Metrics Server and cluster components.
5. Older Kubernetes versions or custom EKS configurations might face compatibility challenges with the Metrics Server.

Steps to Troubleshoot and Resolve

1. If the Metrics Server is missing, install it manually:

   kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
   This command deploys the Metrics Server with its default configuration.
2. Amazon EKS often requires specific configurations for the Metrics Server. So, modify its deployment to include EKS-compatible flags by opening the Metrics Server deployment:

   kubectl edit deployment metrics-server -n kube-system

   Then, add the following flags under `spec.containers.args`:

   --kubelet-preferred-address-types=InternalIP
--kubelet-insecure-tls


   These flags ensure the Metrics Server uses internal IPs for communication and bypass TLS verification.
3. Next, ensure the Metrics Server is deployed and running:

   kubectl get deployment metrics-server -n kube-system

   Check that the deployment status shows as AVAILABLE. Additionally, confirm that no pods are in a `CrashLoopBackOff` or `Error` state:

   kubectl get pods -n kube-system | grep metrics-server
4. If the Metrics Server pods are not functioning as expected, review their logs for errors:

   kubectl logs -n kube-system -l k8s-app=metrics-server
5. Also, ensure the Metrics Server has the required RBAC permissions. Apply the following configuration if needed:

   
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: metrics-server:system:auth-delegator
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system

6. After verifying the Metrics Server setup, test its functionality:

   kubectl top nodes
kubectl top pods


   If configured correctly, these commands will display resource usage metrics. If the error persists, the server might still be initializing or require further adjustments.
7. Finally, verify that the network policies, security groups, or firewall rules allow communication between the Metrics Server and Kubernetes nodes and adjust these settings as necessary.

Additional Tips

1. After deploying or restarting the Metrics Server, give it a few minutes to initialize.
2. Use `kubectl logs` to analyze any underlying errors related to the Metrics Server.
3. If `kubectl top nodes` fails, use `kubectl top pods` to check resource usage for individual pods.

[Need assistance with a different issue? Our team is available 24/7.]

Conclusion

Following these steps, we can resolve the “Metrics API not available” error and ensure our EKS cluster provides accurate resource metrics for effective monitoring and management.

In brief, our Support Experts demonstrated how to fix the Kubectl top Node “Metrics API Not Available” error in Amazon EKS.