Re: [Qemu-discuss] Understanding QEMU Cache Modes and Emulated Disk Write Cache

[Andrew Martin]

----- Original Message -----
> From: "Jakob Bohm" <address@hidden>
> To: address@hidden
> Sent: Wednesday, August 17, 2016 9:34:32 AM
> Subject: Re: [Qemu-discuss] Understanding QEMU Cache Modes and Emulated Disk 
> Write Cache
> 
> On 17/08/2016 15:21, Andrew Martin wrote:
> >> From: "Fam Zheng" <address@hidden>
> >> To: "Andrew Martin" <address@hidden>
> >> Cc: address@hidden
> >> Sent: Wednesday, August 17, 2016 2:50:36 AM
> >> Subject: Re: [Qemu-discuss] Understanding QEMU Cache Modes and Emulated
> >> Disk Write Cache
> >>
> >> On Tue, 08/16 12:39, Andrew Martin wrote:
> >>> Hello,
> >>>
> >>> I am running QEMU 2.5 on Ubuntu 16.04. My guests are a mixture of Linux
> >>> (Ubuntu
> >>> 12.04 and newer with ext4) and Windows. The guest qcow2 images are stored
> >>> in a
> >>> ZFS filesystem with sync=standard (on top of a mirrored vdev of SSDs) on
> >>> the host
> >>> Ubuntu 16.04 OS. After having read through documentation and some
> >>> previous
> >>> threads on this list, I am still somewhat confused about caching.
> >>>
> >>> First, my understanding is that if I strace the qemu process and see it
> >>> regularly issuing fdatasync(2), this means WCE is enabled and it is safe
> >>> to
> >>> use
> >>> cache=writeback, correct?
> >>>
> >>> I see fdatasync(2) coming through about every 6 seconds, so I believe the
> >>> guest
> >>> is properly flushing to disk. Now in this setup, my understanding is
> >>> there
> >>> are 4
> >>> caches and permanent storage:
> >>> * guest page cache
> >>> * guest disk write cache (virtio)
> >>> * host page cache
> >>> * host disk write cache (SSD)
> >>> * permanent storage
> >>>
> >>> I understand that dirty pages get queued up in the guest page cache, and
> >>> that
> >>> when an ext4 barrier is hit an fdatasync(2) call causes them to get
> >>> flushed
> >>> to
> >>> disk. At this point, does fdatasync(2) force the data into the guest disk
> >>> write
> >>> cache, the host page cache, the host disk write cache, or permanent
> >>> storage?
> >>> Does this vary depending on if the virtio disk is set to cache=writeback
> >>> or
> >>> cache=writethrough? Can you explain how this works for both cases?
> >> With cache=writeback, a guest WRITE will put data to host page cache; a
> >> guest
> >> FLUSH will flush to permanent storage.
> >>
> >> With cache=writethrough, a guest WRITE will put data to host page cache,
> >> and
> >> then flushed to permanent storage with a host side fdatasync issued by
> >> QEMU.
> >> In
> >> this case the guest sees no write cache on the virtual device.
> > According to this page, cache=writeback enables the disk write cache:
> > http://www.ibm.com/support/knowledgecenter/linuxonibm/liaat/liaatbpkvmguestcache.htm
> >
> > Is this the write cache on the physical disk itself, or the write cache on
> > the
> > qcow2 image (does the qcow2 even have a disk write cache)? Does this cache
> > get
> > bypassed when fdatasync(2) calls come in from the guest?
> >
> > When is the guest's page cache ever used?
> Whenever the Guest OS feels like it, because the Guest OS doesn't (in
> principle) know the virtual disk is virtual.
Would this primarily be for reads then, since write(2) is written to the host
page cache?

Does QEMU create an emulated disk cache for the qcow2 image, or is that just
the host page cache?

> 
> > Let me see if I understand this correctly:
> >
> > cache=writeback - data stored in host page cache, guest responsible for
> > issuing
> > periodic fdatasync(2) calls; host disk cache enabled but never really used
> > due
> > to large amount of RAM available on host
> Guest issues virtual "virtual disk hardware cache flushes", which are
> turned into fdatasync(2) calls on host.
So if I strace the qemu process (irrespective of the OS the guest is running)
and I see fdatasync(2) calls coming through at regular intervals, 
cache=writeback
should be safe, correct?

The delay between when the data is written and when it is flushed to permanent
storage is just the period between subsequent fdatasync(2) calls, right?

> 
> Host disk cache enabled, but not very useful because most of the data
> in it is probably also in the host page cache.
> > cache=writethrough - data stored in host page cache, host issues
> > fdatasync(2)
> > immediately after write(2); host disk cache disabled
> Host disk cache enabled, but flushed (at least the relevant parts) by
> the fdatasync(2) calls.
> 
> > cache=none - data is written directly to the host disk cache, guest
> > responsible
> > for issuing periodic fdatasync(2) calls
> With the caveat above about what the the guest actually issues to cause
> host side fdatasync(2) calls.
> 

Thanks,

Andrew!