Re: GICv3 for MTTCG

[Andrey Shinkevich]

I built QEMU from the source files downloaded from
https://github.com/qemu/qemu
latest commit 3e9f48bcdabe57f8
I have applied the series "GICv3 LPI and ITS feature implementation".

When I tried to start QEMU back then with the '-kernel' option, the boot 
loader failed to locate the rootfs disk by its correct ID. Specifying 
'root=/dev/sda2' didn't help me also.
So, I used virt-manager successfully which runs QEMU with the following 
arguments:

/usr/local/bin/qemu-system-aarch64 -name 
guest=EulerOS-2.8-Rich,debug-threads=on -S -object 
secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-33-EulerOS-2.8-Rich/master-key.aes
 
-machine virt-6.1,accel=tcg,usb=off,dump-guest-core=off,gic-version=3 
-cpu max -drive 
file=/usr/share/AAVMF/AAVMF_CODE.fd,if=pflash,format=raw,unit=0,readonly=on 
-drive 
file=/var/lib/libvirt/qemu/nvram/EulerOS-2.8-Rich_VARS.fd,if=pflash,format=raw,unit=1
 
-m 4096 -smp 8,sockets=8,cores=1,threads=1 -uuid 
c95e0e92-011b-449a-8e3f-b5f0938aaaa7 -display none -no-user-config 
-nodefaults -chardev socket,id=charmonitor,fd=26,server,nowait -mon 
chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown 
-boot strict=on -device 
pcie-root-port,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 
-device 
pcie-root-port,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 
-device 
pcie-root-port,port=0xa,chassis=3,id=pci.3,bus=pcie.0,addr=0x1.0x2 
-device 
pcie-root-port,port=0xb,chassis=4,id=pci.4,bus=pcie.0,addr=0x1.0x3 
-device qemu-xhci,p2=8,p3=8,id=usb,bus=pci.2,addr=0x0 -device 
virtio-scsi-pci,id=scsi0,bus=pci.3,addr=0x0 -drive 
file=/var/lib/libvirt/images/EulerOS-2.8-Rich.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0
 
-device 
scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1
 
-drive if=none,id=drive-scsi0-0-0-1,readonly=on -device 
scsi-cd,bus=scsi0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi0-0-0-1,id=scsi0-0-0-1
 
-netdev tap,fd=28,id=hostnet0 -device 
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:f9:e0:69,bus=pci.1,addr=0x0 
-chardev pty,id=charserial0 -serial chardev:charserial0 -msg timestamp=on

Best regards,
Andrey.


On 5/13/21 8:20 PM, Alex Bennée wrote:
> 
> Andrey Shinkevich <andrey.shinkevich@huawei.com> writes:
> 
>> Dear colleagues,
>>
>> Thank you all very much for your responses. Let me reply with one message.
>>
>> I configured QEMU for AARCH64 guest:
>> $ ./configure --target-list=aarch64-softmmu
>>
>> When I start QEMU with GICv3 on an x86 host:
>> qemu-system-aarch64 -machine virt-6.0,accel=tcg,gic-version=3
> 
> Hmm are you sure you are running your built QEMU? For me the following
> works fine:
> 
>    ./aarch64-softmmu/qemu-system-aarch64 -machine 
> virt-6.0,gic-version=3,accel=tcg -cpu max -serial mon:stdio -nic 
> user,model=virtio-net-pci,hostfwd=tcp::2222-:22 -device virtio-scsi-pci 
> -device scsi-hd,drive=hd0 -blockdev 
> driver=raw,node-name=hd0,discard=unmap,file.driver=host_device,file.filename=/dev/zvol/hackpool-0/debian-buster-arm64
>  -kernel
> ~/lsrc/linux.git/builds/arm64.nopreempt/arch/arm64/boot/Image -append 
> "console=ttyAMA0 root=/dev/sda2" -display none -m 8G,maxmem=8G -smp 12
> 
> 
>>
>> QEMU reports this error from hw/pci/msix.c:
>> error_setg(errp, "MSI-X is not supported by interrupt controller");
>>
>> Probably, the variable 'msi_nonbroken' would be initialized in
>> hw/intc/arm_gicv3_its_common.c:
>> gicv3_its_init_mmio(..)
>>
>> I guess that it works with KVM acceleration only rather than with TCG.
>>
>> The error persists after applying the series:
>> https://lists.gnu.org/archive/html/qemu-arm/2021-04/msg00944.html
>> "GICv3 LPI and ITS feature implementation"
>> (special thanks for referring me to that)
>>
>> Please, make me clear and advise ideas how that error can be fixed?
>> Should the MSI-X support be implemented with GICv3 extra?
>>
>> When successful, I would like to test QEMU for a maximum number of cores
>> to get the best MTTCG performance.
>> Probably, we will get just some percentage of performance enhancement
>> with the BQL series applied, won't we? I will test it as well.
>>
>> Best regards,
>> Andrey Shinkevich
>>
>>
>> On 5/12/21 6:43 PM, Alex Bennée wrote:
>>>
>>> Andrey Shinkevich <andrey.shinkevich@huawei.com> writes:
>>>
>>>> Dear colleagues,
>>>>
>>>> I am looking for ways to accelerate the MTTCG for ARM guest on x86-64 host.
>>>> The maximum number of CPUs for MTTCG that uses GICv2 is limited by 8:
>>>>
>>>> include/hw/intc/arm_gic_common.h:#define GIC_NCPU 8
>>>>
>>>> The version 3 of the Generic Interrupt Controller (GICv3) is not
>>>> supported in QEMU for some reason unknown to me. It would allow to
>>>> increase the limit of CPUs and accelerate the MTTCG performance on a
>>>> multiple core hypervisor.
>>>
>>> It is supported, you just need to select it.
>>>
>>>> I have got an idea to implement the Interrupt Translation Service (ITS)
>>>> for using by MTTCG for ARM architecture.
>>>
>>> There is some work to support ITS under TCG already posted:
>>>
>>>     Subject: [PATCH v3 0/8] GICv3 LPI and ITS feature implementation
>>>     Date: Thu, 29 Apr 2021 19:41:53 -0400
>>>     Message-Id: <20210429234201.125565-1-shashi.mallela@linaro.org>
>>>
>>> please do review and test.
>>>
>>>> Do you find that idea useful and feasible?
>>>> If yes, how much time do you estimate for such a project to complete by
>>>> one developer?
>>>> If no, what are reasons for not implementing GICv3 for MTTCG in QEMU?
>>>
>>> As far as MTTCG performance is concerned there is a degree of
>>> diminishing returns to be expected as the synchronisation cost between
>>> threads will eventually outweigh the gains of additional threads.
>>>
>>> There are a number of parts that could improve this performance. The
>>> first would be picking up the BQL reduction series from your FutureWei
>>> colleges who worked on the problem when they were Linaro assignees:
>>>
>>>     Subject: [PATCH v2 0/7] accel/tcg: remove implied BQL from 
>>> cpu_handle_interrupt/exception path
>>>     Date: Wed, 19 Aug 2020 14:28:49 -0400
>>>     Message-Id: <20200819182856.4893-1-robert.foley@linaro.org>
>>>
>>> There was also a longer series moving towards per-CPU locks:
>>>
>>>     Subject: [PATCH v10 00/73] per-CPU locks
>>>     Date: Wed, 17 Jun 2020 17:01:18 -0400
>>>     Message-Id: <20200617210231.4393-1-robert.foley@linaro.org>
>>>
>>> I believe the initial measurements showed that the BQL cost started to
>>> edge up with GIC interactions. We did discuss approaches for this and I
>>> think one idea was use non-BQL locking for the GIC. You would need to
>>> revert:
>>>
>>>     Subject: [PATCH-for-5.2] exec: Remove MemoryRegion::global_locking field
>>>     Date: Thu,  6 Aug 2020 17:07:26 +0200
>>>     Message-Id: <20200806150726.962-1-philmd@redhat.com>
>>>
>>> and then implement a more fine tuned locking in the GIC emulation
>>> itself. However I think the BQL and per-CPU locks are lower hanging
>>> fruit to tackle first.
>>>
>>>>
>>>> Best regards,
>>>> Andrey Shinkevich
>>>
>>>
> 
>