Relocating a Non-Pinned Pod Across NSP Nodes in Kubernetes

Learn more about relocating a non-pinned pod across NSP nodes in Kubernetes. Take a minute and discover what our Kubernetes Support Team has to say.

Relocating a Non-Pinned Pod Across NSP Nodes in Kubernetes

Did you know that Kubernetes does not let us directly “move” a running pod from one node to another? Instead, we must evict or delete the pod so that the scheduler can recreate it on a different node. This works only for pods that are not pinned to a specific node (via node selectors, node affinity, or similar constraints).

Today, we will walk through moving a Kubernetes pod to another node and adding a new node to an NSP cluster.

Moving a Kubernetes Pod to a Different Node

Our Experts would like to point out that this procedure involves operations that may impact your NSP system.

1. First, log in as the root or NSP admin user.
2. Open a console window.
3. Then, list current pods with this command:

   kubectl get pods -A -o wide

   
   This displays all running pods with their node assignments. If you’re troubleshooting space issues before migration, you may want to check for problems like invalid capacity on image filesystem.
4. To list available nodes:

   kubectl get nodes
5. Then, identify the node currently hosting your pod:

   kubectl describe nodes node_name
6. Also, record memory requests and CPU requests for reference.
7. Next, check capacity on other nodes:

   kubectl describe nodes

   
   Ensure another node has enough resources (check the Allocated resources section).
8. To prevent new pods from being scheduled on a new node:

   kubectl cordon node_name

   
   With this command, existing pods remain unaffected.
9. To restart the pod, first list deployments:

   kubectl get deployments -A
10. Then, scale down the deployment to remove the pod:

    kubectl scale deployment  -n $(kubectl get pods -A | awk '// {print $1; exit}') –replicas=0
    
    
11. Next, scale it back up to create a new pod on another node:

    kubectl scale deployment  -n $(kubectl get pods -A | awk '// {print $1; exit}') –replicas=1
12. To verify the pod’s new location, run this command:

    kubectl get pods -A -o wide

    If you encounter issues during scheduling, check for unexpected admission errors that may block the pod from starting.
13. Similarly, we can uncordon the node with this command:

    kubectl uncordon node_name

    
    The node will return to the Running state.

In case the pod is stuck in Pending:

kubectl describe pod pod_name -n $(kubectl get pods -A | awk '/pod_name/ {print $1; exit}')

Then, review the output for insufficient resource issues.

Adding a New NSP Cluster Node

1. First, log in to the NSP deployer host as root.
2. Then, edit the configuration file:

   vi /opt/nsp/nsp-k8s-deployer-release-ID/config/k8s-deployer.yml
3. Next, add the new node under the hosts section:

   nodeName: node5
   
   nodeIp: 192.168.98.196
   
   accessIp: 203.0.113.5
   
   
4. Save the file and back it up to a secure remote location.
5. Now, it is time to create the Cluster configuration:

   cd /opt/nsp/nsp-k8s-deployer-release-ID/bin
   
   ./nspk8sctl config -c
   
   

   
   A confirmation will display the path to the generated hosts.yml file.
6. Then, generate an SSH key:

   ssh-keygen -N "" -f /home/user/.ssh/id_rsa -t rsa
7. Now, copy it to the new VM:

   ssh-copy-id -i /home/user/.ssh/id_rsa.pub user@ip_address
8. Now, it is time to prepare the new node. On the new VM:

   mkdir -p /opt/nsp/volumes/fluentd-posfiles
   
   chown -R 1000:1000 /opt/nsp/volumes
   
   
9. To install the updated cluster, run this command on the deployer host:

   ./nspk8sctl install

   
   If SSH keys are missing, use:

   ./nspk8sctl --ask-pass install
10. Then, it is time to verify node addition:

    kubectl get nodes

Now, we can deploy pods to the new node or enable additional NSP features. For better long-term management, consider setting up full Kubernetes monitoring with Prometheus and Grafana.

[Need assistance with a different issue? Our team is available 24/7.]

Conclusion

Moving pods between nodes is a manual process in Kubernetes, but it’s straightforward with the above steps. Similarly, adding nodes to the NSP cluster expands capacity and allows for better workload distribution.

In brief, our Support Experts demonstrated how to relocate a non-pinned pod across NSP nodes in Kubernetes.