Learn more about KVM Nested Hypervisors. Our KVM Support team is here to help you with your questions and concerns.

KVM Nested Hypervisor | An Overview

KVM is a virtualization technology integrated into the Linux kernel. It transforms a system into a type-1 hypervisor when enabled based on processor support. In fact, creating virtual machines using KVM offers versatility.

Here, QEMU harnesses KVM with libvirt and complements it with a user-friendly interface in the form of Virtual Machine Manager UI.

On compatible hardware, KVM extends its capabilities to help create nested virtual machines. This feature allows the inception of VMs inside VMs. This offers a solution for certain use cases.

For example, suppose a case where a virtualized development environment operates on our host system. It requires the execution of virtual device emulators nested two levels deep from the underlying bare metal hardware.

This demonstrates the flexibility and depth of virtualization possibilities achievable through KVM.

Here is an overview:

1. What is Nested Virtualization?
2. Key Concepts of Nested Virtualization
3. Benefits and Use Cases
4. Requirements and Considerations
5. How to Enable Nested Virtualization In KVM In Linux
6. Enable Nested Virtualization Permanently
7. How to Enable Nested Virtualization On KVM Guest Machines
8. Verification

What is Nested Virtualization?

Nested virtualization within KVM helps with the execution of virtual machines within other VMs. This capability supports the creation of a virtualized environment within an existing virtual machine, thereby allowing for the deployment and operation of additional VMs within this nested setup.

Nested virtualization comes in handy in testing and development contexts. It helps us avoid the need for physical hardware. This feature is handy when we need to generate multiple VMs with different configurations or operating systems.

For example consider a situation where compatibility testing of an application across various operating system versions is required—nested virtualization within KVM offers an efficient solution, streamlining the testing and development processes without the burden of additional hardware.

Key Concepts of Nested Virtualization

Let’s look at some of the key concepts regarding KVM nested virtualization:

• Primary Hypervisor (Host):

  The physical machine or server running the initial KVM hypervisor is referred to as the host or primary hypervisor.
• Guest VM (First Level):

  Within the primary hypervisor, we create a virtual machine (VM) that serves as a guest. This first-level VM is similar to any other VM but will have special configurations to enable nested virtualization.
• Nested Hypervisor (Second Level):

  Inside the first-level guest VM, we install and run another instance of the KVM hypervisor. This second-level KVM hypervisor functions just like the primary one, creating its own VMs and managing their resources.
• Nested VMs (Third Level and Beyond):

  Within the second-level KVM hypervisor, you can create additional VMs, forming additional levels of nesting. Each level of nesting represents a virtualized environment within another virtualized environment.

Benefits and Use Cases

• Testing and Development:

  Nested virtualization comes in handy for testing and development cases where we want a multi-tier virtualized infrastructure simulation.
• Training and Education:

  It offers educational purposes, thereby allowing users to simulate complex virtualization setups within a controlled environment.
• Cloud Environments:

  Additionally, in cloud environments, it can help run hypervisors like KVM within VMs, enabling various testing scenarios.

Requirements and Considerations

• Hardware Support:

  Nested virtualization needs hardware support from the underlying processor. In other words, the processor must support Intel VT-x or AMD-V virtualization extensions.
• Hypervisor Configuration:

  The primary KVM hypervisor has to be configured to expose virtualization extensions to the VMs. This includes setting specific options in the VM’s XML configuration file.
• Performance:

  Furthermore, we need to take performance into account, since each additional level of nesting adds to the overhead. It might not be suitable for production workloads where optimal performance is critical.

How to Enable Nested Virtualization In KVM In Linux

1. To begin with, we have to power off all running VMs.
2. Then, unload KVM modules.
3. Next, we can unload the KVM module as seen below:
   • On INTEL systems:
     $ sudo modprobe -r kvm_intel
   • On AMD systems:
     $ sudo modprobe -r kvm_amd
4. Then, reload the KVM module with the nested feature enabled as seen here:
   • On INTEL CPUs:
     $ sudo modprobe kvm_intel nested=1
   • On AMD CPUs:
     $ sudo modprobe kvm_amd nested=1

Enable Nested Virtualization Permanently

1. In order to permanently enable the nested feature we have to edit /etc/modprobe.d/kvm.conf file:

   $ sudo vi /etc/modprobe.d/kvm.conf

   In case the file does not exist, we have to create it.

2. Then, we have to add this line:
   • On INTEL hosts:
     options kvm_intel nested=1
   • On AMD hosts:
     options kvm_amd nested=1
3. Finally, save and close the file.

How to Enable Nested Virtualization On KVM Guest Machines

Additionally, we can enable nested features on a KVM VM with graphical KVM management applications or via the command line.

However, Guest VMs can only use nested virtualization if they are configured with a CPU mode that already supports it. So, the guest VM requires a CPU definition similar to the physical hardware on the host.

The majority of guests will operate when the CPU mode is configured as a host model by default. In other words, the guest gets a CPU similar to the host’s. In certain situations, we may need to use the host-passthrough mode. It replicates all the characteristics of the host CPU.

To check and modify a guest’s CPU type by accessing its manifest with the virsh tool. Then use the “list –all” to get a list of all the VMs.

After that, we can use the edit command to open the manifest of the VM we want.
In this file, we have to find the line that starts with “<cpu mode=”. Then change it to:

<cpu mode='host-model' check='partial' />
<cpu mode='host-passthrough' check='none' />

Now save the file and exit. At this point, the guest will be able to start its own nested guests. In case you run into trouble, our experts recommend changing modes.

We can check if we are successful by running this command in the VM to check if a Linux guest has access to virtualization:

cat /proc/cpuinfo | grep "svm|vmx"

If we get an output with svm or vmx highlighted in red, it means that virtualization is available. Here, SVM is displayed on AMD machines and VMX is displayed on Intel machines.

Then, install a virtualization technology in the guest. Now, we can start a new nested VM.

However, our experts would like to point out that there are a few limitations when it comes to nested guests.

Some KVM features are not available for guests that have started a nested VM. Furthermore, we cannot migrate, save, or load these VMs, unless the nested VM is stopped.

Verification

Now, we can check if Nested Virtualization has been enabled with this command if the processor is Intel:

$ cat /sys/module/kvm_intel/parameters/nested

If we get “Y” or “1”, it means that our system supports nested virtualization. On the other hand, if we get “N” or “0” as output, the system does not support nested virtualization.

However, if our processor is AMD, we have to check the contents of the “/sys/module/kvm_amd/parameters/nested” file.

We can also opt for this command to make sure the “kvm_intel” kernel module has nesting enabled:

• On INTEL CPU:
  $ modinfo kvm_intel | grep -i nested
• On AMD CPU:
  $ modinfo kvm_amd | grep -i nested

Alternatively, we can test the KVM nested virtualization by first checking the CPU mode of the VM with either Virt-Manager GUI or virsh edit command.

For the new VMs for which we want to use nested virtualization, we have to check the “Copy host CPU configuration” option. This option is available under the CPU settings in virt-manager.

Then, we have to the director VM and run the lscpu and lsmod commands.

Now, we can create a VM either from the virtual manager GUI or virt-install inside the director VM.

This will help us confirm whether nested virtualization has been enabled successfully as we will be able to create a VM inside another VM.

[Need assistance with a different issue? Our team is available 24/7.]

Conclusion

At the end of the day, our Support Experts gave us an overview of KVM nested hypervisors.