Re: [PATCH v2 0/6] migration/ram: Optimize for virtio-mem via RamDiscardManager

[David Hildenbrand]

On 22.07.21 13:29, Dr. David Alan Gilbert wrote:
> * David Hildenbrand (david@redhat.com) wrote:
> > virtio-mem exposes a dynamic amount of memory within RAMBlocks by
> > coordinating with the VM. Memory within a RAMBlock can either get
> > plugged and consequently used by the VM, or unplugged and consequently no
> > longer used by the VM. Logical unplug is realized by discarding the
> > physical memory backing for virtual memory ranges, similar to memory
> > ballooning.
> >
> > However, important difference to virtio-balloon are:
> >
> > a) A virtio-mem device only operates on its assigned memory region /
> >     RAMBlock ("device memory")
> > b) Initially, all device memory is logically unplugged
> > c) Virtual machines will never accidentally reuse memory that is currently
> >     logically unplugged. The spec defines most accesses to unplugged memory
> >     as "undefined behavior" -- except reading unplugged memory, which is
> >     currently expected to work, but that will change in the future.
> > d) The (un)plug granularity is in the range of megabytes -- "memory blocks"
> > e) The state (plugged/unplugged) of a memory block is always known and
> >     properly tracked.
> >
> > Whenever memory blocks within the RAMBlock get (un)plugged, changes are
> > communicated via the RamDiscardManager to other QEMU subsystems, most
> > prominently vfio which updates the DMA mapping accordingly. "Unplugging"
> > corresponds to "discarding" and "plugging" corresponds to "populating".
> >
> > While migrating (precopy/postcopy) that state of such memory blocks cannot
> > change.
>
> So no plugging/unplugging can happen during the migration?

Exactly:

static bool virtio_mem_is_busy(void)
{
    /*
     * Postcopy cannot handle concurrent discards and we don't want to migrate
     * pages on-demand with stale content when plugging new blocks.
     *
     * For precopy, we don't want unplugged blocks in our migration stream, and
     * when plugging new blocks, the page content might differ between source
     * and destination (observable by the guest when not initializing pages
     * after plugging them) until we're running on the destination (as we didn't
     * migrate these blocks when they were unplugged).
     */
    return migration_in_incoming_postcopy() || !migration_is_idle();
}

[...]

> > Let's reuse the RamDiscardManager infrastructure to essentially handle
> > precopy, postcopy and background snapshots cleanly, which means:
> >
> > a) In precopy code, always clearing all dirty bits from the bitmap that
> >     correspond to discarded range, whenever we update the dirty bitmap. This
> >     results in logically unplugged memory to never get migrated.
>
> Have you seen cases where discarded areas are being marked as dirty?
> That suggests something somewhere is writing to them and shouldn't be.

I have due to sub-optimal clear_bmap handling to be sorted out by

20210722083055.23352-1-wei.w.wang@intel.com">https://lkml.kernel.org/r/20210722083055.23352-1-wei.w.wang@intel.com

Whereby the issue is rather that initially dirty bits don't get cleared in
lower layers and keep popping up as dirty.

The issue with postcopy recovery code setting discarded ranges dirty in
the dirty bitmap, I did not try reproducing. But from looking at the
code, it's pretty clear that it would happen.

Apart from that, nothing should dirty that memory. Of course,
malicious guests could trigger it for now, in which case we wouldn't catch it
and migrate such pages with postcopy, because the final bitmap sync in
ram_postcopy_send_discard_bitmap() is performed without calling notifiers
right now.

--
Thanks,

David / dhildenb