Re: [Question] Reduce the msix_load cost for VFIO device

[Alex Williamson]

On Tue, 3 Aug 2021 16:43:07 +0800
"Longpeng (Mike, Cloud Infrastructure Service Product Dept.)"
<longpeng2@huawei.com> wrote:

> Hi Alex,
> 
> We found that the msix_load() will cost 40~50ms if the VF has 60+ interrupts,
> the following code cost too much for each interrupt:
> 
> msix_load:
>   for (vector = 0; vector < 60; vector++)
>     msix_handle_mask_update
>       vfio_msix_vector_do_use
>         vfio_add_kvm_msi_virq
>           kvm_irqchip_add_msi_route
>             kvm_irqchip_commit_routes <-- cost 0.8ms each time
> 
> In irq remapping mode, the VF interrupts are not routed through KVM irqchip

I'm not sure what this means.  Your MSIX interrupts are going through
QEMU anyway?  Why?

> in fact, so maybe we can reduce this cost by "x-no-kvm-msix=ture", right?
> Are there any risks if we do in this way ?

You're obviously free to configure your device this way, but the
expectation is that any sort of startup latency is more than offset by
improved runtime latency through the KVM route.  This option is usually
reserved for debugging, when we want to see all interaction with the
device in QEMU.

If there's a case where we're not detecting that a KVM route is
ineffective, then we should figure out how to detect that and skip this
code, but again the expectation is that the KVM route is worthwhile.

If this is specifically about kvm_irqchip_commit_routes(), maybe the
setup path needs a way to batch multiple routes and defer the commit,
if that's possible.  Thanks,

Alex