Enable experimental nested virtualization in Citrix XenServer 6.5 (SP1)

Simply put: Nested virtualization allows a virtual machine (VM) to be a host for further VMs. If you wanted to run VMware inside a VM, you would need this feature. But you would also do well to consider it if you were developing Android apps inside a VM and wanted to use the Android Emulator with hardware acceleration (= virtualization).

Yet, if you do not already know you need this feature, you probably should not be reading this post. Be warned: At the current state nested virtualization in Xen is considered “tech preview” requiring a recent processor and is not a production-ready feature. Citrix does not tell you about the feature and does not even document the parameter at all. I had to go through a bit of their open-sourced code to finally find the line.

Disclaimer done, lets start to play: To enable nested virtualization, go to the XenServer console (e.g. in XenCenter) and find out the UUID of the VM you want to permit being a virtualization host:

# xe vm-list

Go through the list and remember the first characters of your VM´s UUID. After making sure the VM is powered down, turn on nested hardware virtualization. Enter your VM’s UUID instead of <UUID>. After typing the first characters, you can hit TAB to auto-complete:

# xe vm-param-set uuid=<UUID> platform:exp-nested-hvm=true

Boot the VM and check if the virtual processor reports hardware virtualization (“vmx”) capabilities. In Linux this might look like this:

# cat /proc/cpuinfo 
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 60
model name : Intel(R) Core(TM) i5-4460 CPU @ 3.20GHz
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl pni pclmulqdq vmx ssse3 cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes rdrand hypervisor lahf_lm abm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 smep bmi2 erms invpcid

If you see the vmx flag, your VM should be able to be a virtualization host itself – at least in theory: Remember this feature is experimental.

For further reading check out these links:

4 thoughts on “Enable experimental nested virtualization in Citrix XenServer 6.5 (SP1)

  1. Wow, that’s an excellent find!

    I’ve been playing around with my home lab XenServer, but unfortunately I can’t get this to work on one of my Ubuntu VMs. Do you know if there are additional requirements with regards to the processor or VM?

    The CPU I’m using is an Intel Core i5 3550 @ 3.30 Ghz. I think it’s relatively new and I hope it supports it. The Ubuntu VM is in para-virtualized. The setting seems to imply hvm?

    Thanks for allowing me to pick your brain on this in advance.


  2. Just converted the VM to HVW and it seems to work now. According to /proc/cpuinfo the CPU in the VM has a VMX flag 🙂
    I’m a happy puppy!


  3. Pingback: How to get Jenkins to build and deploy an OSv Unikernel

Leave a Reply

Your email address will not be published. Required fields are marked *