Are you seeing proxy detected vanished client connection error in Proxmox logs?

Proxmox makes management of virtual machines easy.

Often server nodes show as online but may not be able to manage from the Proxmox interface.

And we help customers to fix Proxmox errors as part of our Server Management Services.

In this write-up, we’ll see how we made the server responding in GUI without any connection failure errors.

Where does the proxy detected vanished client connection error show up?

Let’s first get more information on the Proxmox error.

Our customer reported the problem as

I noticed that I am unable to manage VM’s and services in pve-xxx. The node shows online but not able to manage. Can you please have a look?

In Proxmox, the pveproxy is the REST API proxy server. From PVE 3.0+ onwards, it listens usually on port 8006. This service runs as the web user and forwards request to other nodes. And for some reason, if the communication fails, it results in errors. Moreover, the node becomes unmanageable from the graphical interface too.

How we fixed the proxy detected vanished client connection error

It’s now time to see how we fixed the Proxmox error for the customer.

As the first step, we check the Proxmox logs at /var/log/syslog and noticed this in the log.

Apr 6 23:52:50 pve-xxx pveproxy[15777]: proxy detected vanished client connection

We then verified the connection between the nodes through telnet and ping. They were working fine. Thus, it was clear that the proxy error happened due to some communication issues between the nodes.  Therefore, we restarted all services in all nodes and then the problem pve-xxx started responding in GUI without any connection failure errors.

Here, we simulated a reboot of all the machines without rebooting the individual VM on them. The exact commands included:

service corosync restart
service pvedaemon restart
service pvestatd restart
service pve-cluster restart
service pveproxy restart

Finally, the pveproxy status in the problem node showed up as:

root@pve-xxx:/var/log# service pveproxy status
● pveproxy.service - PVE API Proxy Server
Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2020-04-07 06:16:51 SAST; 4min 44s ago
Process: 26992 ExecStop=/usr/bin/pveproxy stop (code=exited, status=0/SUCCESS)
Process: 15721 ExecReload=/usr/bin/pveproxy restart (code=exited, status=0/SUCCESS)
Process: 27000 ExecStart=/usr/bin/pveproxy start (code=exited, status=0/SUCCESS)
Main PID: 27016 (pveproxy)
Tasks: 4 (limit: 7372)
Memory: 138.1M

The status of all nodes in the cluster appeared fine too.

[Tired of Proxmox errors? We can manage your cluster 24×7.]

Conclusion

In short, proxy detected vanished client connection error in Proxmox happens due to communication errors between the nodes. Today, we saw how our Support Engineers fixed it by restarting all PVE related services in all nodes.