Migrating etcd between cloud Kubernetes clusters made safe with key steps. Our Kubernetes Support team is ready to assist you. 

Migrating etcd Between Cloud Kubernetes Clusters

Etcd is the core data store of Kubernetes, holding cluster state and critical information. Migrating it to a new cluster requires careful steps to avoid downtime or data loss. This article explains how to prepare, move, and stabilize etcd during migration.

Migrating etcd to a New Cluster

Migrating etcd from one Kubernetes cluster to another requires careful preparation. Each step matters because an error can easily affect the stability of the cluster. By reducing database size, ensuring external access, configuring DNS, and moving nodes safely, you can complete the migration without disruption.

Step 1: Reducing the etcd database size

Before migration begins, check the size of the etcd database. A large database makes new nodes slower to bootstrap and increases the chance of issues. Start by taking a full backup so you can recover if needed. Then list all keys and identify the current revision number. Decide how many revisions to keep, usually about one thousand is enough, and subtract this number from the current revision.

Next, run compaction with the calculated value to remove unnecessary revisions. After compaction, defragment the database to reclaim physical storage. Run defrag on each node, one after another, beginning with the followers. Leave the leader until the end, and if necessary, transfer leadership so every node can be defragmented. Finally, compare the database size before and after. In many cases, it shrinks dramatically, which improves performance and reduces risk during migration.

Step 2: Making etcd nodes reachable from outside the cluster

With the database optimized, the next priority is access. In static clusters nodes are easy to reach, but in cloud environments their IP addresses often change. The best approach is to create individual load balancer services for each etcd node. Most cloud providers support annotations that determine whether the load balancer uses private addresses and which subnets it should attach to. After creating the services, test them by connecting from outside the cluster.

Step 3: Configuring DNS records

Now that nodes are reachable, you must ensure their names resolve properly. Etcd depends on the hostname of each pod, which means the name must be discoverable across the cluster. One method is to add static host entries directly inside pods, but this requires restarting them. A better solution is to configure DNS records at the cluster level. This way, each hostname maps consistently to the right IP. Test the configuration from inside the pods to confirm both short hostnames and full domain names resolve correctly.

Step 4: Adding new nodes to the cluster

When external access and DNS are in place, start adding new nodes. Add one node at a time so that the quorum is not affected. Use the member add command to register each new node, then update the StatefulSet with the correct cluster configuration. After each addition, verify that the node has successfully joined. Once confirmed, continue adding the remaining nodes using the same process.

Step 5: Rerouting client connections

After the new nodes are stable, update client systems to connect to them. Identify which applications use etcd, such as PostgreSQL clusters managed by Patroni, and adjust their configuration. Test each client thoroughly to make sure it communicates properly with the new endpoints.

Step 6: Removing the old nodes

When clients work with the new cluster, begin removing old nodes. Remove them gradually to prevent quorum loss. After deleting a node, check the health of the cluster and verify the remaining members are stable. It is wise to keep persistent volumes from the old nodes in case you need to roll back.

Step 7: Cleaning up old resources

The final step is cleanup. Once you are confident that the new cluster is healthy, delete the remaining resources from the old cluster. This includes persistent volumes, services, and configuration files that are no longer required. At this point the migration is complete and the etcd cluster is fully running in its new environment.

[Need assistance with a different issue? Our team is available 24/7.]

Conclusion 

Migrating etcd between cloud Kubernetes clusters is a delicate process, but with proper preparation it can be done safely. A methodical approach ensures stability and keeps your Kubernetes environment reliable in its new home.

In brief, our Support Experts demonstrated how to fix the “554 5.7.1 : Relay access denied” error.