Re: [PATCH v4 0/6] UFFD write-tracking migration/snapshots

[Peter Xu]

On Fri, Nov 27, 2020 at 01:00:48PM +0300, Andrey Gruzdev wrote:
> On 27.11.2020 12:49, Peter Krempa wrote:
> > On Fri, Nov 27, 2020 at 11:21:39 +0300, Andrey Gruzdev wrote:
> > > On 26.11.2020 18:47, Peter Krempa wrote:
> > > > On Thu, Nov 26, 2020 at 18:17:28 +0300, Andrey Gruzdev via wrote:
> > > > > This patch series is a kind of 'rethinking' of Denis Plotnikov's 
> > > > > ideas he's
> > > > > implemented in his series '[PATCH v0 0/4] migration: add background 
> > > > > snapshot'.
> > > > > 
> > > > > Currently the only way to make (external) live VM snapshot is using 
> > > > > existing
> > > > > dirty page logging migration mechanism. The main problem is that it 
> > > > > tends to
> > > > > produce a lot of page duplicates while running VM goes on updating 
> > > > > already
> > > > > saved pages. That leads to the fact that vmstate image size is 
> > > > > commonly several
> > > > > times bigger then non-zero part of virtual machine's RSS. Time 
> > > > > required to
> > > > > converge RAM migration and the size of snapshot image severely depend 
> > > > > on the
> > > > > guest memory write rate, sometimes resulting in unacceptably long 
> > > > > snapshot
> > > > > creation time and huge image size.
> > > > > 
> > > > > This series propose a way to solve the aforementioned problems. This 
> > > > > is done
> > > > > by using different RAM migration mechanism based on UFFD write 
> > > > > protection
> > > > > management introduced in v5.7 kernel. The migration strategy is to 
> > > > > 'freeze'
> > > > > guest RAM content using write-protection and iteratively release 
> > > > > protection
> > > > > for memory ranges that have already been saved to the migration 
> > > > > stream.
> > > > > At the same time we read in pending UFFD write fault events and save 
> > > > > those
> > > > > pages out-of-order with higher priority.
> > > > 
> > > > This sounds amazing! Based on your description I assume that the final
> > > > memory image contains state image from the beginning of the migration.
> > > > 
> > > > This would make it desirable for the 'migrate' qmp command to be used as
> > > > part of a 'transaction' qmp command so that we can do an instant disk
> > > > and memory snapshot without any extraneous pausing of the VM.
> > > > 
> > > > I'll have a look at using this mechanism in libvirt natively.
> > > > 
> > > 
> > > Correct, the final image contains state at the beginning of migration.
> > > 
> > > So far, if I'm not missing something about libvirt, for external snapshot
> > > creation it performs a sequence like that:
> > > migrate(fd)->transaction(blockdev-snapshot-all)->cont.
> > > 
> > > So, in case 'background-snapshot' capability is enabled, the sequence 
> > > would
> > > change to:
> > > stop->transaction(blockdev-snapshot-all)->migrate(fd).
> > > With background snapshot migration it will finish with VM running so 
> > > there's
> > > not need to 'cont' here.
> > 
> > Yes, that's correct.
> > 
> > The reason I've suggested that 'migrate' being part of a 'transaction'
> > is that it would remove the need to stop it for the disk snapshot part.
> > 
> 
> Hmm, I believe stopping VM for a short time is unavoidable to keep saved
> device state consistent with blockdev snapshot base.. May be I've missed
> something but it seems logical.

I guess PeterK meant an explicit stop vm command, rather than the stop in this
series that should be undetectable from the guest.  It would be definitely
great if PeterK could quickly follow up with this on libvirt. :)

One overall comment (before go into details) is that please remember to start
using "migration: " prefixes in patch subjects from the next version.

Thanks,

-- 
Peter Xu