Stuck with OnApp ‘Error unable to connect to node rabbit’? We can help you.

The OnApp vCD integration requires the usage of RabbitMQ to keep vCD and OnApp synchronized. At times we may get this error while trying to start the Rabbitmq-server.

Here at Bobcares, we often handle requests from our customers to fix similar OnApp errors. Today we will see how our support engineers fix this for our customers as a part of our Server Management Services.

What causes OnApp “Error unable to connect to node rabbit”

A typical error looks like the one given below:

Following are the reason which may cause this error:

1. This issue can happen during power outages.

2. The error can also be seen when we restart the Control Panel server by using the Reset button.

How to fix this error

Now we will see the steps that our Support Engineers follow to fix this issue for our customers:

The root cause can be checked in the default rabbitmq logging directory (/var/log/rabbitmq). Moreover, it can be found specifically in the “rabbit@cp.log” file.

Following are two examples of the most widespread root causes, which can be easily checked and fixed:

1. After checking the logs, from the following output we can understand that the rabbitmq queues have been corrupted:

[root@cp ~]# tail -n 25 /var/log/rabbitmq/rabbit\@cp.log
=INFO REPORT==== 4-Jan-2017::22:57:24 ===
Error description:
   {could_not_start,rabbit,
       {{badmatch,
            {error,
                {{{{case_clause,undefined},
                   [{rabbit_queue_index,add_segment_relseq_entry,3,
                        [{file,"src/rabbit_queue_index.erl"},{line,1091}]},
                    {rabbit_queue_index,parse_segment_entries,3,
                        [{file,"src/rabbit_queue_index.erl"},{line,1075}]},
                    {rabbit_queue_index,'-recover_journal/1-fun-0-',1,
                        [{file,"src/rabbit_queue_index.erl"},{line,863}]},
                    {lists,map,2,[{file,"lists.erl"},{line,1239}]},
                    {rabbit_queue_index,segment_map,2,
                        [{file,"src/rabbit_queue_index.erl"},{line,989}]},
                    {rabbit_queue_index,recover_journal,1,
                        [{file,"src/rabbit_queue_index.erl"},{line,856}]},
                    {rabbit_queue_index,scan_segments,3,
                        [{file,"src/rabbit_queue_index.erl"},{line,676}]},
                    {rabbit_queue_index,queue_index_walker_reader,2,
                        [{file,"src/rabbit_queue_index.erl"},{line,664}]}]},
                  {gen_server2,call,[<0.288.0>,out,infinity]}},

We can fix this issue by moving all the available queues off from the working rabbitmq directory.

This can be done using the following command:

[root@cp ~]# ls /var/lib/rabbitmq/mnesia/rabbit@cp/queues
17BD92SOBZHLASJ4UE5VT5EHQ 3UM4MIP6TJ0PEAHMBSCNUX731 621MT7APTTVO66OSRLB1R7FY5 9R728VE21EGPVSMK9AJ7YH67E A4SL081HP0B21UALPRHLK069R ACJVPWBZHI24WLV6QHBPNT5JE ETHYXYWKQVKD3Q2EYD84UWIS2
[root@cp ~]# mkdir -p /tmp/badrabbit/; mv /var/lib/rabbitmq/mnesia/rabbit@cp/queues/* /tmp/badrabbit/;

2. If we see the following output from the logs, we can assume that the dets table file has been corrupted:

[root@cp ~]# tail -n 15 /var/log/rabbitmq/rabbit\@cp.log
=INFO REPORT==== 16-Jan-2017::09:11:03 ===
Error description:
   {could_not_start,rabbit,
       {{badmatch,
            {error,
                {{{badmatch,
                      {error,
                          {not_a_dets_file,
                              "/var/lib/rabbitmq/mnesia/rabbit@cp/recovery.dets"}}},
                  [{rabbit_recovery_terms,open_table,0,
                       [{file,"src/rabbit_recovery_terms.erl"},{line,126}]},
                   {rabbit_recovery_terms,init,1,
                       [{file,"src/rabbit_recovery_terms.erl"},{line,107}]},
                   {gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},

We can fix these corrupt table files by moving dets file off from the working rabbitmq directory. For this we can use the following command:

[root@cp ~]# mkdir -p /tmp/badrabbit/; mv /var/lib/rabbitmq/mnesia/rabbit@cp/recovery.dets /tmp/badrabbit/;

[Need assistance? We can help you]

Conclusion

To conclude, today we saw the steps that our Support Engineers follow to fix OnApp ‘error unable to connect to node rabbit’ for our customers.