Re: [Qemu-discuss] Booting a QEMU instance within a QEMU instance using KVM doesn't work

[Peter Maydell]

On Tue, 5 Feb 2019 at 13:55, Kaushik Datta <address@hidden> wrote:
> I am trying to show that KVM is working by booting a guest QEMU instance 
> within a host QEMU instance using KVM.  To provide context, the host QEMU is 
> translating x86 to aarch64, while the guest QEMU is translating aarch 64 to 
> aarch64, so is a candidate for KVM acceleration.  I’m using QEMU 3.1.0 with 
> the xlnx-versal-virt machine model for both the host and the guest since this 
> is the only QEMU version + machine model combination that I’ve found that 
> successfully initializes the KVM hypervisor (shown below).

Personally I would recommend the "virt" board. That's the
most common one people use to run KVM for aarch64 setups
(and in general to run Linux in emulation too). I can't
help so much with the xlnx-versal-virt board, but I would
suggest not using that unless you specifically need to work
with an emulation of Xilinx hardware.

Emulation of the virtualization extensions in QEMU should work,
but it's possible you're running into bugs in it, since it's
not all that extensively tested.

> In addition, both the host and the guest are using a Debian 4.9.130-2 kernel 
> and a Yocto-generated rootfs.  Here is the command I use to launch the host 
> QEMU instance:
>
> qemu-system-aarch64 -M xlnx-versal-virt -m 2048 -cpu cortex-a53 -kernel 
> vmlinuz-4.9.0-8-arm64 -initrd new_rootfs.cpio.gz -nographic -D 
> "./qemu_debug.txt"
>
> After the host QEMU boots, here is the dmesg output regarding KVM:
>
> [   12.058077] kvm [1]: HYP VA range: 800000000000:ffffffffffff
> [   12.061797] kvm [1]: Hyp mode initialized successfully
>
> In addition, /dev/kvm exists and I’m operating as root, so that’s good.  
> However, when I do lsmod, I get an empty output, which is a concern.

I don't think KVM is a module for aarch64, so this is expected.

>  Anyway, this is the command I used to boot the guest QEMU instance with KVM 
> acceleration within the host QEMU:
>
> qemu-system-aarch64 -M xlnx-versal-virt -m 1024 -cpu cortex-a53 -kernel 
> vmlinuz-4.9.0-8-arm64 -initrd new_rootfs.cpio.gz -accel kvm -enable-kvm 
> -nographic -D "./qemu_debug.txt"
>
> However, this is the error that I get:
>
> qemu-system-aarch64: 
> /home/kdatta/hpsc-bsp/HEAD/work/poky_build/tmp/work/aarch64-poky-linux/qemu/3.1.0-r0/qemu-3.1.0/exec.c:892:
>  cpu_address_space_init: Assertion `asidx == 0 || !kvm_enabled()' failed.

The error handling here is not great, but the problem I think
that you're running into is that you're trying to run a setup
where the guest CPU has the EL3 (TrustZone) feature enabled:
this can't be emulated with KVM. The xlnx-versal-virt board
always has this feature enabled, because that's what the Xilinx
hardware setup is. If you switch to using the "virt" board instead
(for at least the inner guest, but switching for the outer guest
too would be sensible) this should go away, since 'virt' defaults
to not enabling EL3 for the guest CPU.

I would also suggest upgrading to a newer version of QEMU -- the
command line there suggests you're using 3.1.0-rc0, which is a
release candidate version of a version of QEMU that is not the
most recent one. Using an RC for anything other than "testing
before the release happens" isn't something I'd recommend.
I also did fix some bugs in emulation of the virtualization
extensions recently. Using 4.0 should be OK.

thanks
-- PMM