On Wed, 4 Nov 2020 13:25:40 +0530
Kirti Wankhede <kwankhede@nvidia.com> wrote:
On 11/4/2020 1:57 AM, Alex Williamson wrote:
On Wed, 4 Nov 2020 01:18:12 +0530
Kirti Wankhede <kwankhede@nvidia.com> wrote:
On 10/30/2020 12:35 AM, Alex Williamson wrote:
On Thu, 29 Oct 2020 23:11:16 +0530
Kirti Wankhede <kwankhede@nvidia.com> wrote:
<snip>
+System memory dirty pages tracking
+----------------------------------
+
+A ``log_sync`` memory listener callback is added to mark system memory pages
s/is added to mark/marks those/
+as dirty which are used for DMA by VFIO device. Dirty pages bitmap is queried
s/by/by the/
s/Dirty/The dirty/
+per container. All pages pinned by vendor driver through vfio_pin_pages()
s/by/by the/
+external API have to be marked as dirty during migration. When there are CPU
+writes, CPU dirty page tracking can identify dirtied pages, but any page pinned
+by vendor driver can also be written by device. There is currently no device
s/by/by the/ (x2)
+which has hardware support for dirty page tracking. So all pages which are
+pinned by vendor driver are considered as dirty.
+Dirty pages are tracked when device is in stop-and-copy phase because if pages
+are marked dirty during pre-copy phase and content is transfered from source to
+destination, there is no way to know newly dirtied pages from the point they
+were copied earlier until device stops. To avoid repeated copy of same content,
+pinned pages are marked dirty only during stop-and-copy phase.
Let me take a quick stab at rewriting this paragraph (not sure if I
understood it correctly):
"Dirty pages are tracked when the device is in the stop-and-copy phase.
During the pre-copy phase, it is not possible to distinguish a dirty
page that has been transferred from the source to the destination from
newly dirtied pages, which would lead to repeated copying of the same
content. Therefore, pinned pages are only marked dirty during the
stop-and-copy phase." ?
I think above rephrase only talks about repeated copying in pre-copy
phase. Used "copied earlier until device stops" to indicate both
pre-copy and stop-and-copy till device stops.
Now I'm confused, I thought we had abandoned the idea that we can only
report pinned pages during stop-and-copy. Doesn't the device needs to
expose its dirty memory footprint during the iterative phase regardless
of whether that causes repeat copies? If QEMU iterates and sees that
all memory is still dirty, it may have transferred more data, but it
can actually predict if it can achieve its downtime tolerances. Which
is more important, less data transfer or predictability? Thanks,
Even if QEMU copies and transfers content of all sys mem pages during
pre-copy (worst case with IOMMU backed mdev device when its vendor
driver is not smart to pin pages explicitly and all sys mem pages are
marked dirty), then also its prediction about downtime tolerance will
not be correct, because during stop-and-copy again all pages need to be
copied as device can write to any of those pinned pages.
I think you're only reiterating my point. If QEMU copies all of guest
memory during the iterative phase and each time it sees that all memory
is dirty, such as if CPUs or devices (including assigned devices) are
dirtying pages as fast as it copies them (or continuously marks them
dirty), then QEMU can predict that downtime will require copying all
pages.
But as of now there is no way to know if device has dirtied pages during
iterative phase.
This claim doesn't make any sense, pinned pages are considered
persistently dirtied, during the iterative phase and while stopped.
If instead devices don't mark dirty pages until the VM is
stopped, then QEMU might iterate through memory copy and predict a short
downtime because not much memory is dirty, only to be surprised that
all of memory is suddenly dirty. At that point it's too late, the VM
is already stopped, the predicted short downtime takes far longer than
expected. This is exactly why we made the kernel interface mark pinned
pages persistently dirty when it was proposed that we only report
pinned pages once. Thanks,
Since there is no way to know if device dirtied pages during iterative
phase, QEMU should query pinned pages in stop-and-copy phase.
As above, I don't believe this is true.
Whenever there will be hardware support or some software mechanism to
report pages dirtied by device then we will add a capability bit in
migration capability and based on that capability bit qemu/user space
app should decide to query dirty pages in iterative phase.
Yes, we could advertise support for fine granularity dirty page
tracking, but I completely disagree that we should consider pinned
pages clean until suddenly exposing them as dirty once the VM is
stopped. Thanks,