Enable experimental nested virtualization in Citrix XenServer 6.5 (SP1)

Simply put: Nested virtualization allows a virtual machine (VM) to be a host for further VMs. If you wanted to run VMware inside a VM, you would need this feature. But you would also do well to consider it if you were developing Android apps inside a VM and wanted to use the Android Emulator with hardware acceleration (= virtualization).

Yet, if you do not already know you need this feature, you probably should not be reading this post. Be warned: At the current state nested virtualization in Xen is considered “tech preview” requiring a recent processor and is not a production-ready feature. Citrix does not tell you about the feature and does not even document the parameter at all. I had to go through a bit of their open-sourced code to finally find the line.

Disclaimer done, lets start to play: To enable nested virtualization, go to the XenServer console (e.g. in XenCenter) and find out the UUID of the VM you want to permit being a virtualization host:

# xe vm-list

Go through the list and remember the first characters of your VM´s UUID. After making sure the VM is powered down, turn on nested hardware virtualization. Enter your VM’s UUID instead of <UUID>. After typing the first characters, you can hit TAB to auto-complete:

# xe vm-param-set uuid=<UUID> platform:exp-nested-hvm=true

Boot the VM and check if the virtual processor reports hardware virtualization (“vmx”) capabilities. In Linux this might look like this:

# cat /proc/cpuinfo 
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 60
model name : Intel(R) Core(TM) i5-4460 CPU @ 3.20GHz
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl pni pclmulqdq vmx ssse3 cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes rdrand hypervisor lahf_lm abm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 smep bmi2 erms invpcid

If you see the vmx flag, your VM should be able to be a virtualization host itself – at least in theory: Remember this feature is experimental.

For further reading check out these links:

11 thoughts on “Enable experimental nested virtualization in Citrix XenServer 6.5 (SP1)

  1. Wow, that’s an excellent find!

    I’ve been playing around with my home lab XenServer, but unfortunately I can’t get this to work on one of my Ubuntu VMs. Do you know if there are additional requirements with regards to the processor or VM?

    The CPU I’m using is an Intel Core i5 3550 @ 3.30 Ghz. I think it’s relatively new and I hope it supports it. The Ubuntu VM is in para-virtualized. The setting seems to imply hvm?

    Thanks for allowing me to pick your brain on this in advance.


  2. Just converted the VM to HVW and it seems to work now. According to /proc/cpuinfo the CPU in the VM has a VMX flag 🙂
    I’m a happy puppy!


  3. Pingback: How to get Jenkins to build and deploy an OSv Unikernel

  4. virt-host-validate returen following error for kvm

    FAIL (Check that the ‘kvm-intel’ or ‘kvm-amd’ modules are loaded & the BIOS has enabled virtualization)

  5. Was that error Message from a XenServer 6.5 or rather a newer version? Back when I played around with this feature, my server used Xen instead of KVM.

  6. Hi,

    I’m using virtual box on windows 10, i-5 7th gen.
    On virtual-box, I installed xenserver 7.3 vm.
    on xenserver vm, I installed an ubuntu appliance.
    It is throwing error: “HVM is required for this operation”.
    I have enable virtualization on xenserver using above commands but still getting same error.

    also i’m not getting VMX flag on running : # cat /proc/cpuinfo
    flags : fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush mmx fxsr sse sse2 syscall nx lm constant_tsc rep_good nopl nonstop_tsc pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx rdrand hypervisor lahf_lm abm 3dnowprefetch avx2 rdseed clflushopt

    Thanks in advance,

  7. Hi Ankit,

    For the use case you describe, VirtualBox would need to support nested virtualization. You want to run XenServer, which is a virtualization system, within VirtualBox, which is another virtualization system. The use case I had was essentially to run a virtualization within a Xen VM. There, Xen(Server) was the “top-most” / physical-hardware-level hypervisor.

    If you do succeed, please leave a note. I’d be highly interested if you managed to get it to run within VirtualBox.


  8. it does work for enabeling HAXM emulator on a Windows VM for Android Studio, thank you !
    Didn’t get to have my virtual device booting yet, but you’re help got me further ^^

  9. Pingback: How to get Jenkins to build and deploy an OSv Unikernel

  10. Pingback: Nested VM in XenServer - Boot Panic

Leave a Reply

Your email address will not be published. Required fields are marked *