[Qemu-devel] Re: Caching modes

[Anthony Liguori]

On 09/21/2010 09:26 AM, Christoph Hellwig wrote:
> On Mon, Sep 20, 2010 at 07:18:14PM -0500, Anthony Liguori wrote:
>    
> > O_DIRECT alone to a pre-allocated file on a normal file system should
> > result in the data being visible without any additional metadata
> > transactions.
> >      
> Anthony, for the third time: no.  O_DIRECT is a non-portable extension
> in Linux (taken from IRIX) and is defined as:
>
>
>         O_DIRECT (Since Linux 2.4.10)
>                Try  to minimize cache effects of the I/O to and from this file.
>                In general this will degrade performance, but it  is  useful  in
>                special  situations,  such  as  when  applications  do their own
>                caching.  File I/O is done directly to/from user space  buffers.
>                The O_DIRECT flag on its own makes at an effort to transfer data
>                synchronously, but does not give the guarantees  of  the  O_SYNC
>                that  data and necessary metadata are transferred.  To guarantee
>                synchronous I/O the O_SYNC must be used in addition to O_DIRECT.
>                See NOTES below for further discussion.
>
>                A  semantically  similar  (but  deprecated)  interface for block
>                devices is described in raw(8).
>
> O_DIRECT does not have any meaning for data integrity, it just tells the
> filesystem it *should* not use the pagecache.  Even if it should not
> various filesystem have fallbacks to buffered I/O for corner cases.
> It does *not* mean the actual disk cache gets flushed, and it *does*
> not guarantee anything about metadata which is very important.
>    

Yes, I understand all of this but I was trying to avoid accepting it.  
But after the call today, I'm convinced that this is fundamentally a 
filesystem problem.

I think what we need to do is:

1) make virtual WC guest controllable.  If a guest enables WC, &= 
~O_DSYNC.  If it disables WC, |= O_DSYNC.  Obviously, we can let a user 
specify the virtual WC mode but it has to be changable during live 
migration.

2) only let the user choose between using and not using the host page 
cache.  IOW, direct=on|off.  cache=XXX is deprecated.

3) make O_DIRECT | O_DSYNC not suck so badly on ext4.

> > > Barriers are a Linux-specific implementation details that is in the
> > > process of going away, probably in Linux 2.6.37.  But if you want
> > > O_DSYNC semantics with a volatile disk write cache there is no way
> > > around using a cache flush or the FUA bit on all I/O caused by it.
> > >        
> > If you have a volatile disk write cache, then we don't need O_DSYNC
> > semantics.
> >      
> If you present a volatile write cache to the guest you do indeed not
> need O_DSYNC and can rely on the guest sending fdatasync calls when it
> wants to flush the cache.  But for the statement above you can replace
> O_DSYC with fdatasync and it will still be correct.  O_DSYNC in current
> Linux kernels is nothing but an implicit range fdatasync after each
> write.
>    

Yes.  I was stuck on O_DSYNC being independent of the virtual WC but 
it's clear to me now that it cannot be.

> > > ext3 and ext4 have really bad fsync implementations.  Just use a better
> > > filesystem or bug one of it's developers if you want that fixed.  But
> > > except for disabling the disk cache there is no way to get data integrity
> > > without cache flushes (the FUA bit is nothing but an implicit flush).
> > >
> > >        
> > But why are we issuing more flushes than the guest is issuing if we
> > don't have to worry about filesystem metadata (i.e. preallocated storage
> > or physical devices)?
> >      
> Who is "we" and what is workload/filesystem/kernel combination?
> Specific details and numbers please.
>    

My concern is ext4.  With a preallocated file and cache=none as 
implemented today, performance is good even when barrier=1.  If we 
enable O_DSYNC, performance will plummet.  Ultimately, this is an ext4 
problem, not a QEMU problem.

Perhaps we can issue a warning if the WC is disabled and we do an fsstat 
and see that it's ext4 with barriers enabled.

I think it's more common for a user to want to disable a virtual WC 
because they have less faith in the hypervisor than they have in the 
underlying storage.

The scenarios I am concerned about:

1) User has enterprise storage, but has an image on ext4 with 
barrier=1.  User explicitly disables WC in guest because they have 
enterprise storage but not an UPS for the hypervisor.

2) User does not have enterprise storage, but has an image on ext4 with 
barrier=1.  User explicitly disables WC in guest because they don't know 
what they're doing.

In the case of (1), the answer may be "ext4 sucks, remount with 
barrier=0" but I think we need to at least warn the user of this.

For (2), again it's probably the user doing the wrong thing because if 
they don't have enterprise storage, then they shouldn't care about a 
virtual WC.  Practically though, I've seen a lot of this with users.

Regards,

Anthony Liguori