Nested virtualization not working

[Michał Zegan]

Hello,

I have send an email related to a different problem with my vm, however 
I also have this one: nested virtualization does not work for a win11 guest.

What I mean is, nested=y is set in the kvm_intel module parameters, 
"host" is used as cpu model, and vtx flag is exposed. Everything works 
(baring the other problem I've reported related to vm randomly crashing) 
until I enable things like hyperv.

At that point, the virtual machine starts boot looping and dropping me 
off to the windows recovery. I do not have any evidence of what's 
actually wrong except that a boot loop is happening. I can stop the boot 
loop by disabling vtx or nested virtualization in kvm_intel. Of course 
if I uninstall hyperv it also stops from happening.

The same happened on my older skylake laptop, although there I was able 
to turn it on with vtx enabled, except it didn't work. Also on this 
older laptop it worked until some qemu update happened. I don't remember 
which qemu version it was and the current problem I have might be 
different, as nothing helps.


My configuration is:

MSI vector gp76 with intel core i7 12700h, 32gb ram as host laptop, 
running fedora 37 with kernel 6.1 (bug present since at least 6.0), 
microcode up to date, kvm_intel with nested=y.

Guest uses machine q35, uefi firmware with secureboot+tpm, cpu model set 
as host/migratable=off, operating system is win11 64 bit. Bugg happens 
when hyperv is enabled (my target is possibly enabling wsl2 for testing 
purposes, that's why I want nested virt).


Qemu command line: (note i was playing with settings of this vm so 
currently it has vmx disabled, the bug doesn't happen until I set vmx to 
on, with all other parameters left intact, things like rtm=off were 
because of some other hints that this might possibly help, but it did not)

/usr/bin/qemu-system-x86_64 \
-name guest=win11,debug-threads=on \
-S \
-object 
'{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-1-win11/master-key.aes"}' 
\
-blockdev 
'{"driver":"file","filename":"/usr/share/edk2/ovmf/OVMF_CODE.secboot.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' 
\
-blockdev 
'{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' 
\
-blockdev 
'{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/win11_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' 
\
-blockdev 
'{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' 
\
-machine 
pc-q35-7.0,usb=off,smm=on,dump-guest-core=off,kernel_irqchip=on,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format,memory-backend=pc.ram 
\
-accel kvm \
-cpu 
host,migratable=off,vmx=off,rtm=off,mpx=off,hv-time=on,kvm-pv-eoi=on,kvm-pv-unhalt=on,hv-relaxed=on,hv-vapic=on,hv-spinlocks=0x1000,hv-vpindex=on,hv-runtime=on,hv-synic=on,hv-stimer=on,hv-stimer-direct=on,hv-reset=on,hv-frequencies=on,hv-reenlightenment=on,hv-tlbflush=on,hv-ipi=on,hv-evmcs=on,hv-crash,kvm-poll-control=on,pmu=on,host-cache-info=on,l3-cache=off 
\
-global driver=cfi.pflash01,property=secure,value=on \
-m 8192 \
-object 
'{"qom-type":"memory-backend-memfd","id":"pc.ram","share":true,"x-use-canonical-path-for-ramblock-id":false,"size":8589934592}' 
\
-overcommit mem-lock=off \
-smp 8,sockets=1,dies=1,cores=8,threads=1 \
-uuid 589e17db-9ea9-49ac-8a66-c75bbc39ddd3 \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=29,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=localtime,clock=vm,driftfix=slew \
-no-shutdown \
-global ICH9-LPC.disable_s3=1 \
-global ICH9-LPC.disable_s4=1 \
-boot strict=on \
-device '{"driver":"intel-iommu","id":"iommu0","device-iotlb":true}' \
-device 
'{"driver":"pcie-root-port","port":16,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x2"}' 
\
-device 
'{"driver":"pcie-root-port","port":17,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x2.0x1"}' 
\
-device 
'{"driver":"pcie-root-port","port":18,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x2.0x2"}' 
\
-device 
'{"driver":"pcie-root-port","port":19,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x2.0x3"}' 
\
-device 
'{"driver":"pcie-root-port","port":20,"chassis":5,"id":"pci.5","bus":"pcie.0","addr":"0x2.0x4"}' 
\
-device 
'{"driver":"pcie-root-port","port":21,"chassis":6,"id":"pci.6","bus":"pcie.0","addr":"0x2.0x5"}' 
\
-device 
'{"driver":"pcie-root-port","port":22,"chassis":7,"id":"pci.7","bus":"pcie.0","addr":"0x2.0x6"}' 
\
-device 
'{"driver":"pcie-pci-bridge","id":"pci.8","bus":"pci.1","addr":"0x0"}' \
-device 
'{"driver":"pcie-root-port","port":23,"chassis":9,"id":"pci.9","bus":"pcie.0","addr":"0x2.0x7"}' 
\
-device 
'{"driver":"pcie-root-port","port":24,"chassis":10,"id":"pci.10","bus":"pcie.0","multifunction":true,"addr":"0x3"}' 
\
-device 
'{"driver":"pcie-root-port","port":25,"chassis":11,"id":"pci.11","bus":"pcie.0","addr":"0x3.0x1"}' 
\
-device '{"driver":"qemu-xhci","id":"usb","bus":"pci.5","addr":"0x0"}' \
-device 
'{"driver":"virtio-scsi-pci","iommu_platform":true,"packed":true,"id":"scsi0","num_queues":8,"bus":"pci.4","addr":"0x0"}' 
\
-device 
'{"driver":"virtio-serial-pci","iommu_platform":true,"packed":true,"id":"virtio-serial0","max_ports":16,"vectors":4,"bus":"pci.3","addr":"0x0"}' 
\
-blockdev 
'{"driver":"host_device","filename":"/dev/pool/win11","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' 
\
-blockdev 
'{"node-name":"libvirt-2-format","read-only":false,"driver":"raw","file":"libvirt-2-storage"}' 
\
-device 
'{"driver":"scsi-hd","bus":"scsi0.0","channel":0,"scsi-id":0,"lun":0,"device_id":"drive-scsi0-0-0-0","drive":"libvirt-2-format","id":"scsi0-0-0-0","bootindex":1}' 
\
-blockdev 
'{"driver":"file","filename":"/var/lib/libvirt/cdroms/virtio-win-0.1.225.iso","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' 
\
-blockdev 
'{"node-name":"libvirt-1-format","read-only":true,"driver":"raw","file":"libvirt-1-storage"}' 
\
-device 
'{"driver":"ide-cd","bus":"ide.1","drive":"libvirt-1-format","id":"sata0-0-1"}' 
\
-netdev 
tap,fds=30:32:33:34:35:36:37:38,vhost=on,vhostfds=39:40:41:42:43:44:45:46,id=hostnet0 
\
-device 
'{"driver":"virtio-net-pci","iommu_platform":true,"packed":true,"mq":true,"vectors":18,"netdev":"hostnet0","id":"net0","mac":"52:54:00:98:17:54","bus":"pci.2","addr":"0x0"}' 
\
-chardev pty,id=charserial0 \
-device 
'{"driver":"isa-serial","chardev":"charserial0","id":"serial0","index":0}' \
-chardev socket,id=charchannel0,fd=28,server=on,wait=off \
-device 
'{"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"}' 
\
-chardev spicevmc,id=charchannel1,name=vdagent \
-device 
'{"driver":"virtserialport","bus":"virtio-serial0.0","nr":2,"chardev":"charchannel1","id":"channel1","name":"com.redhat.spice.0"}' 
\
-chardev socket,id=chrtpm,path=/run/libvirt/qemu/swtpm/1-win11-swtpm.sock \
-tpmdev emulator,id=tpm-tpm0,chardev=chrtpm \
-device '{"driver":"tpm-crb","tpmdev":"tpm-tpm0","id":"tpm0"}' \
-device 
'{"driver":"virtio-keyboard-pci","id":"input0","bus":"pci.9","addr":"0x0"}' 
\
-device 
'{"driver":"virtio-tablet-pci","id":"input1","bus":"pci.10","addr":"0x0"}' \
-object 
'{"qom-type":"input-linux","id":"input2","evdev":"/dev/input/by-id/usb-MOSART_Semi._2.4G_INPUT_DEVICE-event-kbd","repeat":true,"grab_all":true,"grab-toggle":"ctrl-ctrl"}' 
\
-object 
'{"qom-type":"input-linux","id":"input3","evdev":"/dev/input/by-path/platform-i8042-serio-0-event-kbd","repeat":true,"grab_all":true,"grab-toggle":"ctrl-ctrl"}' 
\
-object 
'{"qom-type":"input-linux","id":"input4","evdev":"/dev/input/by-path/pci-0000:00:15.0-platform-i2c_designware.0-event-mouse"}' 
\
-object 
'{"qom-type":"input-linux","id":"input5","evdev":"/dev/input/by-path/platform-i8042-serio-1-event-mouse"}' 
\
-audiodev '{"id":"audio1","driver":"spice"}' \
-spice port=0,disable-ticketing=on,seamless-migration=on \
-device 
'{"driver":"qxl-vga","id":"video0","max_outputs":1,"ram_size":67108864,"vram_size":67108864,"vram64_size_mb":0,"vgamem_mb":16,"bus":"pcie.0","addr":"0x1"}' 
\
-device 
'{"driver":"ich9-intel-hda","id":"sound0","bus":"pcie.0","addr":"0x1b"}' \
-device 
'{"driver":"hda-duplex","id":"sound0-codec0","bus":"sound0.0","cad":0,"audiodev":"audio1"}' 
\
-device 
'{"driver":"i6300esb","id":"watchdog0","bus":"pci.8","addr":"0x1"}' \
-watchdog-action reset \
-chardev spicevmc,id=charredir0,name=usbredir \
-device 
'{"driver":"usb-redir","chardev":"charredir0","id":"redir0","bus":"usb.0","port":"1"}' 
\
-chardev spicevmc,id=charredir1,name=usbredir \
-device 
'{"driver":"usb-redir","chardev":"charredir1","id":"redir1","bus":"usb.0","port":"2"}' 
\
-device 
'{"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.6","addr":"0x0"}' 
\
-object '{"qom-type":"rng-builtin","id":"objrng0"}' \
-device 
'{"driver":"virtio-rng-pci","rng":"objrng0","id":"rng0","bus":"pci.7","addr":"0x0"}' 
\
-device '{"driver":"vmcoreinfo"}' \
-sandbox 
on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on