Need help with troubleshooting Common HAProxy Errors? We can help you.
At Bobcares, we often receive requests to fix these errors as part of our Server Management Services.
Today, let’s discuss how our Support Engineers fix some of the most common errors easily for our customers.
Troubleshoot Common HAProxy Errors
There are three main commands and a common log location that we use to start troubleshooting HAProxy errors.
Generally, when we troubleshoot HAProxy, we use these commands in the given order. Then we examine the log file for specific diagnostic data.
systemctl– Used to control and interact with Linux services via the
journalctl– To query and view the logs that are generated by
haproxy– Used to check HAProxy’s configuration.
/var/log/haproxy.log– This file contains log entries from HAProxy itself detailing TCP and HTTP traffic handled by the server.
Let us now discuss how to use them and where we can find additional information about errors.
systemctl Commands for HAProxy
To troubleshoot common HAProxy errors using the
systemd service manager, the first step is to inspect the state of the HAProxy processes on the system.
systemctl commands will query
systemd for the state of HAProxy’s processes on most Linux distributions.
$ sudo systemctl status haproxy.service -l –no-pager
-l a flag will ensure that output is not truncated. The
--no-pager flag will make sure that output will go directly to our terminal.
If we omit the
--no-pager flag we will be able to scroll through the output using arrow keys. To quit from the pager use the
We will receive output like this:
● haproxy.service – HAProxy Load Balancer Loaded: loaded (/lib/systemd/system/haproxy.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2020-08-20 19:30:11 UTC; 5s ago Docs: man:haproxy(1) file:/usr/share/doc/haproxy/configuration.txt.gz Process: 487 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS (code=exited, status=0/SUCCESS) Main PID: 488 (haproxy) Tasks: 2 (limit: 2344) . . . Aug 19 21:31:46 d6cdd0c71489 systemd: Started HAProxy Load Balancer.
The output may be slightly different depending on the Linux distribution we use. But in any case, make a note of the
Active line in the output.
If the HAProxy server does not show active, there may be an error. Typically if there is a problem, we will have a line like the following in the output:
Active: failed (Result: exit-code) since Thu 2020-08-20 19:32:26 UTC; 6s ago
If there is a problem with the HAProxy configuration, our Support Engineers suggest using the
journalctl Commands for HAProxy
To inspect the
systemd logs for HAProxy, we can use the
journalctl command. It will usually indicate whether there is a problem with starting or managing the HAProxy process.
These logs are separate from HAProxy’s request and error logs.
journalctl displays logs from
systemd that describe the HAProxy service itself, from startup to shutdown. It will also describe any process errors that may encounter along the way.
$ sudo journalctl -u haproxy.service –since today –no-pager
--since today flag will limit the output of the command to log entries beginning at 00:00:00 of the current day. Using this option will help restrict the volume of log entries that we need to examine when checking for errors.
We should receive output like the following (there may be a few extra lines between the
Started lines depending on the Linux distribution):
Aug 20 19:37:08 d6cdd0c71489 systemd: Starting HAProxy Load Balancer… . . . Aug 20 19:37:08 d6cdd0c71489 systemd: Started HAProxy Load Balancer.
If there is an error, we will have a line in the output that is similar to the following:
Aug 20 19:32:25 yourhostname systemd: Failed to start HAProxy Load Balancer.
If our HAProxy server has errors in the
journalctl logs, the next step is to investigate HAProxy’s configuration using the
haproxy command-line tool.
Initially, to troubleshoot HAProxy configuration issues, we use the
haproxy -c command. The tool will parse HAProxy files and detect errors or missing settings before attempting to start the server.
We run the command like this on Ubuntu, Debian, CentOS, and Fedora-based distributions. We make sure to change the path to the configuration file if we use a different filename or location:
$ sudo haproxy -c -f /etc/haproxy/haproxy.cfg
A working HAProxy configuration will result in an output like the following:
Configuration file is valid
If there is an error in the HAProxy configuration,
haproxy -c will detect it and attempt to notify us about the problem.
For instance, attempting to use the
bind directive in
haproxy.cfg in the wrong location will result in:
[ALERT] 232/194354 (199) : parsing [/etc/haproxy/haproxy.cfg:13] : unknown keyword ‘bind’ in ‘global’ section [ALERT] 232/194354 (199) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg [ALERT] 232/194354 (199) : Fatal errors found in configuration.
In this example, the bind directive is inside a global configuration section. Hence the HAProxy unknown keyword error.
The message also mentions line 13. We can edit the file and fix or remove the erroneous line without having to search through the file.
Learning how to use
haproxy -c to detect and fix errors is useful while troubleshooting an existing error, or before you reload HAProxy with an edited configuration that may contain errors.
[Failed to track the error? We are available 24*7]
HAProxy Log Files
HAProxy log files are a very helpful resource for troubleshooting.
Generally, any error that we receive in a browser or HTTP client will have a corresponding entry in HAProxy’s logs. Sometimes HAProxy will also output errors related to configuration and other debugging information to its log files.
On Ubuntu and Debian based Linux distributions, the
haproxy package includes scripts that configure log output in
Similarly, on CentOS, Fedora, and other RedHat-derived Linux distributions,
haproxy does not output to a log file by default.
When we troubleshoot HAProxy using its log file, we examine
/var/log/haproxy.log for errors using a tool.
For example, to view the last two lines of the log using tail, we run:
$ sudo tail -n 2 /var/log/haproxy.log
An example error will resemble the following lines, regardless of which Linux distribution we are using to run your HAProxy server:
Aug 20 19:36:21 d6cdd0c71489 haproxy: [ALERT] 258/134605 (19202) : Proxy ‘app’, server ‘app1’ [/etc/haproxy/haproxy.cfg:88] verify is enabled by default but no CA file specified. If you’re running on a LAN where you’re certain to trust the server’s certificate, please set an explicit ‘verify none’ statement on the ‘server’ line, or use ‘ssl-server-verify none’ in the global section to disable server-side verifications by default. Aug 20 19:36:22 d6cdd0c71489 haproxy: 203.0.113.1:54428 [20/Aug/2020:19:36:22.288] main app/<NOSRV> 0/-1/-1/-1/1 503 212 – – SC– 1/1/0/0/0 0/0 “GET / HTTP/1.1”
These example lines are just for illustration purposes. If we diagnose errors with our HAProxy server, chances are the lines in logs have different contents than these. Some lines will include success responses and other non-critical diagnostic entries.
Regardless of our Linux distribution, the format of the lines in the HAProxy logs will include HTTP status codes, along with requesting IPs and the status of backend servers.
Once we have an idea of what might be causing problems with the HAProxy server we can continue researching and troubleshooting the issue.
The HTTP status code and text description are useful since they give us explicit and specific terms. We can use them to narrow down the range of possible causes of a problem.
[Stuck with troubleshooting? We’d be happy to assist]
To conclude, troubleshooting common HAProxy errors can range from diagnosing errors with the service itself to locating misconfigured options for modules. Today, we saw how our Support Engineers fix some of the most common errors.