Re: [Qemu-devel] [RFC][PATCH 0/3] Fix caching issues with live migration

[Anthony Liguori]

On 09/12/2010 05:46 AM, Avi Kivity wrote:
>  On 09/11/2010 05:04 PM, Anthony Liguori wrote:
> > Today, live migration only works when using shared storage that is fully
> > cache coherent using raw images.
> >
> > The failure case with weak coherent (i.e. NFS) is subtle but 
> > nontheless still
> > exists.  NFS only guarantees close-to-open coherence and when 
> > performing a live
> > migration, we do an open on the source and an open on the 
> > destination.  We
> > fsync() on the source before launching the destination but since we 
> > have two
> > simultaneous opens, we're not guaranteed coherence.
> >
> > This is not necessarily a problem except that we are a bit gratituous 
> > in reading
> > from the disk before launching a guest.  This means that as things 
> > stand today,
> > we're guaranteed to read the first 64k of the disk and as such, if a 
> > client
> > writes to that region during live migration, corruption will result.
> >
> > The second failure condition has to do with image files (such as 
> > qcow2).  Today,
> > we aggressively cache metadata in all image formats and that cache is 
> > definitely
> > not coherent even with fully coherent shared storage.
> >
> > In all image formats, we prefetch at least the L1 table in open() 
> > which means
> > that if there is a write operation that causes a modification to an 
> > L1 table,
> > corruption will ensue.
> >
> > This series attempts to address both of these issue.  Technically, if 
> > a NFS
> > client aggressively prefetches this solution is not enough but in 
> > practice,
> > Linux doesn't do that.
>
> I think it is unlikely that it will, but I prefer to be on the right 
> side of the standards.

I've been asking around about this and one thing that was suggested was 
acquiring a file lock as NFS requires that a lock acquisition drops any 
client cache for a file.  I need to understand this a bit more so it's 
step #2.

>   Why not delay image open until after migration completes?  I know 
> your concern about the image not being there, but we can verify that 
> with access().  If the image is deleted between access() and open() 
> then the user has much bigger problems.

3/3 would still be needed because if we delay the open we obviously can 
do a read until an open.

So it's only really a choice between invalidate_cache and delaying 
open.  It's a far less invasive change to just do invalidate_cache 
though and it has some nice properties.

Regards,

Anthony Liguori

> Note that on NFS, removing (and I think chmoding) a file after it is 
> opened will cause subsequent data access to fail, unlike posix.