In 2010, Virgin Blue Airlines reported a $20 million dip in profit due to an outage in its online reservation system. Hardware failure was identified as the reason for the outage, but it caused severe damage to Virgin Blue’s reputation, customer loyalty and profit.

Every online business is affected by a downtime at one time or another for reasons as varied as software bugs, hardware issues or human errors.

Mitigation of downtime due to hardware errors is perhaps the most researched topic in data centers. IaaS providers, VPS providers and Cloud providers use strategies such as redundant hardware, fail-over systems and scheduled downtimes to minimize impact to businesses.

For instance, in VPS hosting, it is common for servers to be equipped with RAID hard disks, dual network cards, etc. as a hedge against failure. But, even in such a system, hardware replacements are sometimes required, which can lead to business downtime.

In these situations, the solution is to transfer all virtual servers in a host to another system, and then power down the host for maintenance. The technology that enables such a transfer, without shutting down the virtual servers is called “live migration”.

All leading server virtualization systems support live migration, but each system have its own set of “quirks” or dependencies, that need to be satisfied before a successful live migration can be done.

Recently, we assisted a VPS provider setup live migration in their LXD/LXC server virtualization system. The VPS provider guaranteed 99.95% uptime, and used redundant hardware such as dual-power-supply, dual network card and RAID disks to mitigate the risk of hardware failure.

However, during scheduled maintenance, the VPSs had to be manually moved (shut-down, move, restart) to another server, which was affecting the SLA guarantee.

This is the story of how we setup live migration for an LXD/LXC VPS system.

LXC live migration setup overview

In LXC, there is no “single click” provision to migrate a container live from one server to another. For live migration to work, we fixed a few dependencies in LXC:

1. Clustering servers with identical processors – Live migration works in LXC only between servers with identical CPU architecture. So, we clustered servers into 2 clusters – one with AMD processors and the other with Intel processors.
2. Enabling CRIU support in containers – LXC used a Linux feature called CRIU (Checkpoint/Restore In Userspace) for live migration. So, we enabled each container with CRIU support, and resource settings were optimized to make sure the container worked fine in another server.
3. Configuring hosts to listen on TCP ports – By default, LXD servers listen on a unix socket, which cannot be accessed by external programs. For LXC live migration to work, LXD was linked to a public TCP port.

Once the dependencies were fixed, live migration worked flawlessly in the LXD servers. Here I’ll go through an example of how live migration was implemented to migrate a container named “VPS-silver-2752” from server “VPS-host-04” to “VPS-host-05”.

LXC live migration example – LXC container “VPS Silver 2752” is live migrated to VPS host 05

[Don’t lose your sleep over server stability! Our 24/7 server experts will maintain your servers secure and lean, at affordable pricing. ]

Configuring the container

To prepare a LXC container to be migrated live using the CRIU feature, a set of configuration settings were changed in it. These settings included security tweaking and enabling CRIU support. In our first step, we configured the container ‘VPS-Silver-2752’ to support the live migration feature.

root@host1:~# lxc profile apply VPS-Silver-2752 live-migration
 Profile live-migration applied to VPS-Silver-2752

Configuring the servers

Setting up the host (VPS host 04) and destination (VPS host 05) servers was the next step. By default, LXD service listened on a unix socket in the servers.

For LXC live migration to work, the lxd service in both the host and destination servers were configured to listen to a tcp port, say 8443. This was done to facilitate a tcp connection for the servers to communicate with each other. Both LXD servers were assigned a strong password.

root@host1:~# lxc config set core.https_address vps-host-04:8443

root@host1:~# lxc config set core.trust_password  password

The connectivity from the host server to the destination server was then configured using the ‘lxc remote add’ commands.

root@host1:~# lxc remote add host1 https://vps-host-04:8443
 Admin password for vps-host-04:

root@host1:~# lxc remote add host2 https://vps-host-05:8443
 Admin password for vps-host-05:

[ Looking for a stable server virtualization infrastructure setup? Our cloud management experts will help you setup the ideal solution. ]

LXC live migration process

To ensure that the VPS works fine in the migrated server, we copied over the container configuration from source to destination server.

root@host1:~# lxc profile copy VPS-Silver-2752 host2

The running VPS container was then migrated over from host to the destination server.

root@host1:~# lxc move  vps-host-04:VPS-Silver-2752 vps-host-05:VPS-Silver-2752

The above snippet shows that the VPS ‘VPS-Silver-2752’ was successfully migrated from VPS-host-04 to VPS-host-05 server in a few seconds.

Using live migration technique, whenever one server got overloaded or faced resource constraints, it was possible to easily move resource-intensive VPSs from that server to a less-loaded server.

Other benefits of live migration include – performing server maintenance, upgrade, scaling up resources or load balancing between servers – without affecting the accounts hosted in it. 

Here, we saw how a LXC container was moved from one server to another. Before performing live migration in a production environment, aspects such as security tweaking, user configuration settings, ownership of files and folders, etc. were taken care of.

Bobcares helps VPS providers, data centers and web hosts deliver industry-standard services through custom configuration and preventive maintenance of server virtualization solutions. Contact Us to know how we can help you.