Re: [PATCH v2 10/20] vfio/common: Record DMA mapped IOVA ranges

[Joao Martins]

On 22/02/2023 22:10, Alex Williamson wrote:
> On Wed, 22 Feb 2023 19:49:05 +0200
> Avihai Horon <avihaih@nvidia.com> wrote:
>> From: Joao Martins <joao.m.martins@oracle.com>
>> @@ -612,6 +665,16 @@ static int vfio_dma_map(VFIOContainer *container, 
>> hwaddr iova,
>>          .iova = iova,
>>          .size = size,
>>      };
>> +    int ret;
>> +
>> +    ret = vfio_record_mapping(container, iova, size, readonly);
>> +    if (ret) {
>> +        error_report("vfio: Failed to record mapping, iova: 0x%" HWADDR_PRIx
>> +                     ", size: 0x" RAM_ADDR_FMT ", ret: %d (%s)",
>> +                     iova, size, ret, strerror(-ret));
>> +
>> +        return ret;
>> +    }
> 
> Is there no way to replay the mappings when a migration is started?
> This seems like a horrible latency and bloat trade-off for the
> possibility that the VM might migrate and the device might support
> these features.  Our performance with vIOMMU is already terrible, I
> can't help but believe this makes it worse.  Thanks,
> 

It is a nop if the vIOMMU is being used (entries in container->giommu_list) as
that uses a max-iova based IOVA range. So this is really for iommu identity
mapping and no-VIOMMU. We could replay them if they were tracked/stored 
anywhere.

I suppose we could move the vfio_devices_all_device_dirty_tracking() into this
patch and then conditionally call this vfio_{record,erase}_mapping() in case we
are passing through a device that doesn't have live-migration support? Would
that address the impact you're concerned wrt to non-live-migrateable devices?

On the other hand, the PCI device hotplug hypothetical even makes this a bit
complicated as we can still attempt to hotplug a device before migration is even
attempted. Meaning that we start with live-migrateable devices, and we added the
tracking, up to hotpluging a device without such support (adding a blocker)
leaving the mappings there with no further use. So it felt simpler to just track
always and avoid any mappings recording if the vIOMMU is in active use?