VPC CNI plugin fail to reach API Server | Resolved

VPC CNI plugin fail to reach API Server can be resolved with this handy guide from Bobcares. 

At Bobcares, we offer solutions for every query, big and small, as a part of our Server Management Service.

Let’s take a look at how our Support Team is ready to help customers when VPC CNI plugin fails to reach API Server.

How to resolve: VPC CNI plugin fail to reach API Server

If your VPC CNI plugin has been failing to reach the API server in Amazon EKS, you can stop worrying. Our Support Techs have come up with an innovative solution to help fix this specific issue.

In fact, this issue results when the ipamD daemon attempts to connect to the API Server before the kube-proxy adds the Kubernetes Service port. This results in the connection between the API Server and the ipamD times out. Our Support Engineer recommends troubleshooting the situation with these steps:

• Check the ipamD and kube-proxy logs
• Compare the timestamps between the ipamD and kube-proxy logs
• Add an init container

How to check the ipamD and kube-proxy logs

If the connection between the API Server and the ipamD times out, we will notice the following error message in the ipamD logs:

"Failed to create client: error communicating with apiserver:

The kubeproxy generates iptables routes on the worker node for Kubernetes API Server endpoints. Once the kube=proxy creates the route, we will come across the following error message in the kube-proxy logs:

Adding new service port \"default/kubernetes:https\"

How to compare the timestamps between ipamD and kube-proxy logs

When we compare the timestamps between ipamD and kube-proxy logs, we will notice the connection timed out and fail in case of this error. For instance, in this example, the ipamD daemon has attempted to connect to the API Server at 2021-12-22T10:40:49.735Z.

{"level":"error","ts":"2021-12-22T10:40:49.735Z","caller":"aws-k8s-agent/main.go:28",
"msg":"Failed to create client: error communicating with apiserver: 
Get https://10.77.0.1:443/version?timeout=32s: dial tcp 10.77.0.1:443: i/o timeout"}

Correspondingly, we will notice the following message in the kube-proxy logs:

{"log":"I0922 10:41:15.267648       1 service.go:379] Adding new service port 
\"default/kubernetes:https\" at 10.77.0.1:443/TCP\n","stream":"stderr","time":"2021-12-22T10:40:49.26766844Z"}

This indicates that the kube-proxy was added to the Kubernetes Service port 2021-12-22T10:41:15.26766844Z.

How to add an init container

1. First, we modify the aws-node specification in order with the following code to resolve the DNS for the Kubernetes Service name:

   $ kubectl -n kube-system edit daemonset/aws-node

   This results in the output:

   initContainers:
      - name: init-kubernetes-api
        image: busybox:1.28
        command: ['sh', '-c', "until nc -zv ${KUBERNETES_PORT_443_TCP_ADDR} 443; do echo waiting for kubernetes Service endpoint; sleep 2; done"]
2. Next, we will use the following code to verify the aws-node pods has created the init containers

   $ kubectl get pods -n kube-system  -w

   This results in the following output:

   ...
       kube-proxy-smvfl                          0/1     Pending             0          0s
       aws-node-v68bh                            0/1     Pending             0          0s
       kube-proxy-smvfl                          0/1     Pending             0          0s
       aws-node-v68bh                            0/1     Pending             0          0s
       aws-node-v68bh                            0/1     Init:0/1            0          0s
       kube-proxy-smvfl                          0/1     ContainerCreating   0          0s
       kube-proxy-smvfl                          1/1     Running             0          6s
       aws-node-v68bh                            0/1     PodInitializing     0          9s
       aws-node-v68bh                            0/1     Running             0          16s
       aws-node-v68bh                            1/1     Running             0          53s

[Looking for a solution to another query? We are just a click away.]

Conclusion

In essence, our skilled Support Engineers at Bobcares demonstrated how to proceed if VPC CNI plugin fails to reach API Server.