Connection timed out error during http-01 challenge propagation occurs due to incompatibilities with the Loadbalancer and the way Kubernetes works.

As a part of our Server Management Services, we help our Customers to fix similar Kubernetes related errors regularly.

Let us today discuss the possible causes and fixes for this error.

Connection timed out error during http-01 challenge propagation

Kubernetes Ingresses allow you to flexibly route traffic from outside your Kubernetes cluster to Services inside of your cluster.

Popular Ingress Controllers include Nginx, Contour, HAProxy, and Traefik. Ingresses provide a more efficient and flexible alternative to setting up multiple LoadBalancer services, each of which uses its own dedicated Load Balancer.

While trying to set up Nginx Ingress with Cert-Manager for the cluster on DigitalOcean, at times, the certificates may not get issued. The error message may show:

~~
Waiting for http-01 challenge propagation: failed to perform self-check GET request ‘http://cert-test.bobcares.com/.well-known/acme-challenge/kC9hDBr8qI898y1gqacU2BbytGBb21YfQHaCyUx1kEY’: Get http://cert-test.bobcares.com/.well-known/acme-challenge/kC9hDBr8qI898y1gqacU2BbytGBb21YfQHaCyUx1kEY: dial tcp 165.227.252.80:80: connect: connection timed out
~~

Cause for connection timed out error during http-01 challenge propagation

As we discussed earlier, incompatibility issues with the LoadBalancer and the way Kubernetes works could be a major factor that trigger the connection timeout error.

While using the LoadBalancer provided by DigitalOcean, Kubernetes may not route network through the LoadBalancer public interface.

Thus it does not add PROXY protocol header or SSL while setting it outside Kubernetes.

However, when the cert-manager tries to connect to the public domain name, it fails. It works from outside the cluster, but not within the cluster.

If we could add some switches where we can instruct the validator to add PROXY protocol or to disable validation for that domain it would help a lot.

How to fix connection timed out error during http-01 challenge propagation

A workaround to fix this error include the following steps:

Firstly, create a dedicated DNS entry pointing to the public IP address of the load balancer. we can check for EXTERNAL-IP via `kubectl get svc -n ingress-nginx command.

In the manifest used to create load balancer, add an annotation pointing to that newly created DNS entry. For instance, add the annotation as the one below:

service.beta.kubernetes.io/do-loadbalancer-hostname: “myhost.domain.com”

By default the Nginx Ingress LoadBalancer Service has service.spec.externalTrafficPolicy set to the value Local, which routes all load balancer traffic to nodes running Nginx Ingress Pods.

The other nodes will deliberately fail load balancer health checks so that Ingress traffic does not get routed to them.

Certificate-issuer will be on a different node than the load balancer, so it cannot talk to itself through the ingress. To fix this, in the manifest used to create load balancer, set externalTrafficPolicy to Cluster.

[Need any further assistance in fixing Kubernetes errors? – We’re available 24*7]

Conclusion

In short, connection timed out error during http-01 challenge propagation occurs due to incompatibilities with the Loadbalancer and the way Kubernetes works. Today, we saw how our Support Engineers fix this error.