Re: [Qemu-discuss] Supported hypervisors running VMs in nested VM

[Rain Maker]

Unfortunately, no difference. WIth or without -hypervisor doesn't make
any difference to that flag.

But, experimenting on, I found something <very> odd.
The 0x3a register is 0 when the VM boots up.
Even when I start a L2 VM, 0x3a is still 0.

However, once I start WITH -enable-kvm, 0x3a is suddenly 5(!). See
this terminal session, which is executed within the L1 VM ("kvmtest").
http://storage4.static.itmages.com/i/15/1006/h_1444163503_6656916_6ffbfd2352.png

I was only executing mini.iso (an Ubuntu Netinstaller), and closing it
at the boot prompt. I did not do anything in the L2 VM. Both the Qemu
VM as the -enable-kvm VMs do boot.

Is this how the MSR is supposed to react?

AFAIK, the MSR can only be modified from kernelspace (which also
explains why Qemu would only reset it with -enable-kvm, there are no
kernelspace components used without it if I understand correctly)

Looking at this, I can imagine that Windows does not detect a correct
value. It will get 0. Would it make sense to cygwin KVM and see if
that changes the MSR register?

Sincerely,
Roel Brook.

2015-10-05 23:17 GMT+02:00 Bandan Das <address@hidden>:
> Rain Maker <address@hidden> writes:
>
>> Qemu on Linux works fine. I did not even have to explicitly set
>> -hypervisor. It simply works.
>
> Sorry, I meant running Linux with "-hypervisor" to see if specifying
> that is somehow messing with the feature flags.
>
>> As does VirtualBox FYI.
>>
>> Booting with UEFI didn't make any difference.
>>
>> After A LOT of Googling, I believe that Hyper-V actually checks bit
>> 0x3a of the MSR register (instead of, as the error would .
>> This is a 3 bit register (IA32_FEATURE_CONTROL). Within my VM, the
>> value returned by rdmsr is "0", while on my host it is "5". For
>
> That seems odd. Even Linux wouldn't work if that value is 0.
>
>> Hyper-V to work (from what I understand), it should be either 4 or 5.
>>
>> Googling that, funny enough, brought me back to this list:
>> https://lists.gnu.org/archive/html/qemu-devel/2015-01/msg01371.html.
>> I guess it really IS important to know what you're googling for to
>> find things fast...
>>
>> That thread simply says that the kernel is too old. Well, my host is
>> running 4.2, so should be new enough.
>> I'm a bit stuck. Any ideas?
>>
>> Sincerely,
>> Roel Brook
>>
>>
>>
>>
>> 2015-10-05 21:18 GMT+02:00 Bandan Das <address@hidden>:
>>> Rain Maker <address@hidden> writes:
>>>
>>>> Thanks Bandan.
>>>>
>>>> That helped a bit. It got me to the next hurdle, as you suspected.
>>>>
>>>> I modified the virsh XML so that -cpu host,+vmx,-hypervisor is passed,
>>>> and the installation now reports "Hyper-V cannot be installed because
>>>> virtualization support is not enabled in the BIOS.".
>>>
>>> Thanks for trying this out.
>>>
>>>> I am sure that vmx is passed. but "systeminfo" does report "Hyper-V
>>>> cannot be installed because virtualization support is not enabled in
>>>> the BIOS."
>>>>
>>>> Apparently, Microsoft queries the BIOS to verify that the
>>>
>>> When kvm is initialized, it checks for TXT and VMX both being enabled.
>>> That too, only if the feature control msr is locked. I don't think there
>>> are actually any specific "bios calls" to find this out. I would assume
>>> Hyper-V should be doing the same thing but your testing says otherwise.
>>>
>>> Can you please run linux as L1 with "-hypervisor" and see if it works ?
>>> If it doesn't, please check dmesg for relevant messages.
>>>
>>> Thanks,
>>> Bandan
>>>
>>>
>>>> virtualization bit is actually enabled, instead of simply relying on
>>>> the VMX flag.
>>>> Unfortunately, VMs are still not starting either. The seabios in Qemu
>>>> seems to be pretty difficult to modify. I'll check whether I can
>>>> reinstall on UEFI, maybe that is going to make a difference.
>>>>
>>>> The way VMWare does this is actually semi-documented (it hasn't always
>>>> been in the product, and a workaround involving manually editing the
>>>> configuration has been used for a long time). I'll see if I can
>>>> correlate these to Qemu options, to see whether we can use those
>>>> instructions to get this working on Qemu.
>>>>
>>>> 1. Set 'vhv.enable = "TRUE" on the VM
>>>>   It "enables virtual hardware virtualization". This seems equivalent
>>>> to the -hypervisor flag
>>>>
>>>> 2. Set 'monitor.virtual_exec = "hardware" on the VM.
>>>>   This option seems to force hardware virtualization for both CPU and
>>>> MMU. Unsure whether there's an equivalent Qemu configuration option.
>>>> Unsure whether it's needed on Qemu. Details at
>>>> http://www.vmware.com/files/pdf/perf-vsphere-monitor_modes.pdf
>>>>
>>>> 3. Set hypervisor.cpuid.v0 = “FALSE” in the VM configuration
>>>>   This seems synonymous to the +vmx flag
>>>>
>>>> 4. Enable the option to "Virtualize VT-x/EPT or AMD/RVI"
>>>>   I have not found any option to explicitly do this in Qemu. Looking
>>>> at my Ubuntu VM, the "ept" flag IS passed to the VM, so this should be
>>>> OK.
>>>>
>>>> 5. Add the following CPU mask Level ECX: ---- ---- ---- ---- ---- ---- 
>>>> --H- ----
>>>>   Not sure how to do that in Qemu or what it does. Looking at
>>>> https://en.wikipedia.org/wiki/CPUID, it seems to disable the XSAVE
>>>> instruction(?). For fun, I passed -cpu ...-xsave, but it did not seem
>>>> to make any difference whatsoever.
>>>>
>>>> Sincerely,
>>>> Roel Brook
>>>>
>>>>
>>>>
>>>> 2015-10-04 5:07 GMT+02:00 Bandan Das <address@hidden>:
>>>>> ...
>>>>>> Windows 2012 / 2016 technical preview 3
>>>>>> --------------------------------------------------------
>>>>>> The installation via the "default" method of Add/Remove Features does
>>>>>> not work. Hyper-V displays the error message "A hypervisor is already
>>>>>> running".
>>>>>>
>>>>>> This check can be skipped by using a different method of installation
>>>>>> (from PowerShell):
>>>>>> Enable-WindowsOptionalFeature –Online -FeatureName Microsoft-Hyper-V
>>>>>> –All -NoRestart
>>>>>>
>>>>>> This results in (again) the server booting up, but being unable to run
>>>>>> any guest VMs. The error message is less clear then that in 2008, just
>>>>>> "The Virtual Machine Management Service failed to start the virtual
>>>>>> machine 'New Virtual Machine' because one of the Hyper-V components is
>>>>>> not running (Virtual machine ID
>>>>>> 0C063B29-249A-41C8-8A5B-6D4D2E37EF7C)."
>>>>>> is what I could find.
>>>>>>
>>>>>> Other
>>>>>> --------
>>>>>> Just to verify that "nesting" is actually working, I've also installed
>>>>>> a Ubuntu 15.10 VM and installed Qemu on it.
>>>>>> This combination CAN successfully run a VM.
>>>>>>
>>>>>> I've also installed VirtualBox on one of the Windows VMs. This
>>>>>> VirtualBox instance is also capable of running virtual machines.
>>>>>> According to the icon in the bottom right, VirtualBox IS using the
>>>>>> hardware virtualization.
>>>>>>
>>>>>> Is this a problem specific to Hyper-V? Is there a method to get
>>>>>
>>>>> Nesting a Hyper-V L1 hypervisor is largely untested. But one of the 
>>>>> problems I recollect is that Hyper-V doesn’t like running in a 
>>>>> virtualized environment. It checks the “hypervisor” feature flag that 
>>>>> Qemu exports. You could try running qemu with “-cpu  host,-hypervisor” or 
>>>>> something similar and see if it makes any difference. I suspect there 
>>>>> would be other roadblocks though, this is just one of them.
>>>>>
>>>>>
>>>>>
>>>>>> Hyper-V working including running guests? I know for a fact that
>>>>>> VMWare Workstation / ESX is able to run Hyper-V fully, so it should
>>>>>
>>>>> Yes, IIRC one of the things ESX does is hide the hypervisor flag 
>>>>> specifically for Hyper-V.
>>>>>
>>>>> Bandan
>>>>>
>>>>>> not be completely impossible (but I dislike VMWare for different
>>>>>> reasons).
>>>>>>
>>>>>> My Qemu command line (generated by virt-manager). Except for disks and
>>>>>> domain names, all are identical:
>>>>>>
>>>>>> qemu-system-x86_64 -enable-kvm -name Windows_2008_R2 -S -machine
>>>>>> pc-i440fx-vivid,accel=kvm,usb=off -cpu
>>>>>> SandyBridge,+invtsc,+osxsave,+pcid,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme
>>>>>> -m 2048 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid
>>>>>> 54a8f3a3-66c2-45a5-a280-ecf7019a67fa -no-user-config -nodefaults
>>>>>> -chardev 
>>>>>> socket,id=charmonitor,path=/var/lib/libvirt/qemu/Windows_2008_R2.monitor,server,nowait
>>>>>> -mon chardev=charmonitor,id=monitor,mode=control -rtc
>>>>>> base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=discard
>>>>>> -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global
>>>>>> PIIX4_PM.disable_s4=1 -boot strict=on -device
>>>>>> ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device
>>>>>> ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6
>>>>>> -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1
>>>>>> -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2
>>>>>> -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive
>>>>>> file=/sub/kvm/Windows_2008_R2.qcow2,if=none,id=drive-ide0-0-0,format=qcow2,cache=unsafe,aio=threads
>>>>>> -device 
>>>>>> ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1
>>>>>> -drive 
>>>>>> file=/sub/ISO/en_windows_server_2008_r2_with_sp1_x64_dvd_617601.iso,if=none,id=drive-ide0-0-1,readonly=on,format=raw
>>>>>> -device ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1
>>>>>> -netdev tap,fd=24,id=hostnet0 -device
>>>>>> rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:7b:d7:d2,bus=pci.0,addr=0x3
>>>>>> -chardev pty,id=charserial0 -device
>>>>>> isa-serial,chardev=charserial0,id=serial0 -chardev
>>>>>> spicevmc,id=charchannel0,name=vdagent -device
>>>>>> virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0
>>>>>> -device usb-tablet,id=input0 -spice
>>>>>> port=5903,addr=127.0.0.1,disable-ticketing,seamless-migration=on
>>>>>> -device 
>>>>>> qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.0,addr=0x2
>>>>>> -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device
>>>>>> hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev
>>>>>> spicevmc,id=charredir0,name=usbredir -device
>>>>>> usb-redir,chardev=charredir0,id=redir0 -chardev
>>>>>> spicevmc,id=charredir1,name=usbredir -device
>>>>>> usb-redir,chardev=charredir1,id=redir1 -chardev
>>>>>> spicevmc,id=charredir2,name=usbredir -device
>>>>>> usb-redir,chardev=charredir2,id=redir2 -chardev
>>>>>> spicevmc,id=charredir3,name=usbredir -device
>>>>>> usb-redir,chardev=charredir3,id=redir3 -device
>>>>>> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on
>>>>>>
>>>>>> Thank you in advance for response.
>>>>>>
>>>>>> Sincerely,
>>>>>> Roel Brook
>>>>>>
>>>>>