A fix to the error, “Solus agent restart invoke already executed tasks” comes in later updates. However, we can prevent them from occurring in a few steps.

As part of our Server Management Services, we assist our customers with several SolusIO queries.

Today, let us see how our techs go about this error.

Solus agent restart invoke already executed tasks

The most common symptom of this error includes the following:

• Solus agent restart on SolusIO Compute Resource makes rabbitmq push old already executed tasks.

In such a case, we can see the following message in /var/log/solus/agent.log:

INFO[2021-09-30 02:54:40]/usr/src/agent/pkg/queue/queue.go:253 agent/pkg/queue.(*queue).Consume() Consumer “cr-4-worker-vm-backup-restore-0” registered! processing messages…

• Connectivity issue from Compute Resource to RabbitMQ service on SolusIO Management node.

Similarly, we find the following error in /var/log/solus/agent.log on Compute Resource:

/usr/src/agent/pkg/queue/queue.go:96 agent/pkg/queue.(*queue).bindQueue() Queue “” declaration failed: Exception (504) Reason: “channel/connection is not open”

The major cause of this error is a SolusIO bug, the fix of which will be in future updates.

Moving ahead, let us see how our Support Techs work around these errors.

To begin with, we need to ensure that there aren’t any running tasks in SolusIO. Now, let us discuss the steps.

Initially, we connect to the Management node via SSH.

Then we need to back up the SolusIO Management node.

Eventually, we enter the rabbitmq container:

docker exec -ti $(docker ps –format ‘{{ .ID }}’ -f name=solus_rabbitmq) sh

And we fetch the list of all queues:

rabbitmqctl list_queues

After that, we remove the queues pushed to Compute Resource with the command:

rabbitmqctl delete_queue QUEUE_NAME

Here, we replace QUEUE_NAME with the corresponding queue name.

Later, we exit rabbitmq container:

exit

And again, we restart rabbitmq with the commands:

docker service scale solus_rabbitmq=0
docker service scale solus_rabbitmq=1

Then we connect to Compute Resource via SSH.

Finally, we restart the Solus agent with the command:

systemctl restart solus-agent

Conclusion

In short, we saw how our Support Techs troubleshoot this error for our customers.