Why OpenStack Instances Get Stuck?

OpenStack cloud stuck or slow? Identify control plane pressure causes and fixes with help from the Google Cloud Support team.

Stuck resources and control plane slowdowns can disrupt an OpenStack cloud. Understanding why these issues occur helps administrators keep the environment stable. This article explains what stuck resources mean, the role of the control plane, common reasons for control plane pressure, and how engineers detect and manage these problems. Read the article to learn more.

What Does Stuck Mean in an OpenStack Cloud

In an OpenStack cloud, a resource is called stuck when an operation starts but never finishes. The system keeps the resource in a temporary state instead of moving it to its final state. Because of this, users cannot continue working with that resource.

A stuck OpenStack cloud usually shows these symptoms

• Virtual machines remain in build, error, or deletion state
• Instance creation commands do not complete
• Volumes stay in the attaching or detaching state
• Network ports remain down
• The Horizon dashboard becomes slow or unresponsive

These problems usually occur when one of the OpenStack services stops responding or fails to update the resource status. As a result, the task remains incomplete, and the resource stays in that temporary state.

What is the OpenStack Control Plane

The OpenStack control plane is the part of the cloud that monitors and governs all operations. The control plane receives the request when a user creates a virtual machine, network, or volume. Permission checks and resource selections are performed to determine which services to provide and what to do. Main services in the control plane include neutron, which manages networking, placement that tracks available resources, keystone that manages user identity and access, glance that stores virtual machine images, and nova that monitors virtual machines and scheduling.

It has the supporting components that help these services communicate and store data.

• MariaDB stores cloud information, including users and instances.
• RabbitMQ passes messages between services
• The worker process handles background tasks

These are the services that generally run on control nodes and can be distributed among diverse systems for better reliability. In some modern deployments, they also run in containers or on Kubernetes platforms.

The control plane manages cloud operations, while the data plane handles the actual network traffic and user data.

Common Causes of Control Plane Backpressure

1. RabbitMQ Queue Saturation

RabbitMQ is crucial to the OpenStack service communication. When message traffic increases beyond hand, queues start to build up, thereby slowing down the system.

Common signs include

• Message queues are increasing continuously
• High memory or disk usage on RabbitMQ nodes
• Frequent heartbeat or connection timeout errors

Typical reasons include

• Too many API requests at the same time
• There are not enough message customers to handle the lines
• Delays in the network between services and RabbitMQ
• When memory becomes limited, Queues are moved to disk

When there are heavy workloads, RabbitMQ is often the first component that shows pressure in the majority of situations.

2. Database Contention and Galera Flow Control

The majority of the cloud state, including instances, projects, and users, is stored in the database. A heavy database load can cause the control plane to fall behind since so many services rely on it.

Common warning signs include

• Slow or blocked queries
• Long-running database transactions
• Flow control pauses in clustered database setups
• API requests waiting for database locks

Possible causes include

• Large tables that are not cleaned regularly
• Missing indexes in frequently used tables
• Database nodes that do not have enough resources
• Inefficient background cleanup tasks

Scheduling decisions and API replies slow down when the database is slow.

3. Scheduler Bottlenecks

Schedulers decide the location of new workloads, and the decisions about instance creation and placement take time if the scheduler is overloaded.

Common symptoms include

• High processor usage on scheduler nodes
• Decisions about workload placement are delayed.
• Instances that are still in the scheduling stage

Common reasons include

• There are too many scheduler filters active.
• Large computing infrastructures that aren’t properly tuned
• The placement service’s slow responses
• There are not enough scheduler replicas processing requests.

Schedulers need proper tuning to handle large-scale cloud environments.

4. Insufficient Worker Processes

Worker processes are used by OpenStack services to manage background tasks and incoming requests. Requests start to wait if there aren’t enough employees.

Common indicators include

• Even when CPU use appears regular, message queues are getting longer.
• Waiting for workers to respond to API calls
• Logs including timeout notifications for remote procedure calls

Default worker settings may not be enough for busy production environments.

5. Notification and Telemetry Overload

Monitoring and telemetry services use the numerous internal notifications that OpenStack produces. Inadequate scaling of these systems may result in an increase in the control plane’s load.

Problems often appear when

• Data is processed slowly by telemetry services.
• Production environments have debug logging enabled.
• Notifications are processed by external systems too slowly.

This can make the database and communications system work harder, which slows down the control plane even more.

Why The Cloud Appears Healthy But Is Not

Control plane backpressure can be difficult to notice in the beginning. The cloud may still look normal, even though internal processes are slowing down. Many basic checks show that services are running, so the problem is not always obvious.

From the outside, the system may appear healthy because

• Services are still running
• API requests still return responses
• Agents remain connected to the control system
• Monitoring tools may not report critical errors

However, inside the system, different issues start building up.

• Message queues slowly grow
• Database locks remain active for longer periods
• Services retry operations again and again

As these delays increase, tasks stop moving forward, and the cloud gradually becomes slower.

How Bobcares Engineers Identify Control Plane Backpressure

In an OpenStack cloud, engineers evaluate backpressure using more than just service status. Rather, they examine system behavior to identify areas within the control layer where delays are developing.

Typically, they verify

• RabbitMQ memory or disk utilization and queue size
• Lock activity and database query speed
• The behavior of the Galera flow control
• Error rates and response times for APIs
• Scheduler performance and logs

These tests assist in identifying the system-slowing component before it has an impact on the cloud as a whole.

Simple Steps for Mitigating Control Plane Pressure

When an OpenStack cloud slows down, engineers act quickly to maintain system stability by reducing the load on the control plane.

Typical actions consist of

• Pause non-critical operations
• Restart overloaded services carefully
• Add more control plane service instances
• Fix slow database queries
• Avoid restarting all services at once

These actions help reduce pressure until the main issue is resolved.

Eliminating Backpressure in Control Planes

The system must be appropriately sized and monitored to accept incoming requests without slowing down in order to maintain the stability of the control plane.

Beneficial techniques include

• Make sure RabbitMQ has enough memory.
• Execute several conductor and scheduler services
• Adjust employee procedures for actual workloads.
• Regularly maintain and clean the database
• Track response times and queue length.
• Examine the control plane while it is under load.

[Need assistance with a different issue? Our team is available 24/7.]

Conclusion 

Control plane pressure and stuck resources can slow down an OpenStack cloud and affect normal operations. Watching system activity, fixing delays early, and keeping key services properly tuned help maintain a stable environment. If your cloud shows similar issues, reach out to the Bobcares team for expert support and guidance.