Re: [Qemu-discuss] Understanding QEMU Cache Modes and Emulated Disk Write Cache

[Andrew Martin]

----- Original Message -----
> From: "Fam Zheng" <address@hidden>
> To: "Andrew Martin" <address@hidden>
> Cc: address@hidden
> Sent: Wednesday, August 17, 2016 2:50:36 AM
> Subject: Re: [Qemu-discuss] Understanding QEMU Cache Modes and Emulated Disk 
> Write Cache
> 
> On Tue, 08/16 12:39, Andrew Martin wrote:
> > Hello,
> > 
> > I am running QEMU 2.5 on Ubuntu 16.04. My guests are a mixture of Linux
> > (Ubuntu
> > 12.04 and newer with ext4) and Windows. The guest qcow2 images are stored
> > in a
> > ZFS filesystem with sync=standard (on top of a mirrored vdev of SSDs) on
> > the host
> > Ubuntu 16.04 OS. After having read through documentation and some previous
> > threads on this list, I am still somewhat confused about caching.
> > 
> > First, my understanding is that if I strace the qemu process and see it
> > regularly issuing fdatasync(2), this means WCE is enabled and it is safe to
> > use
> > cache=writeback, correct?
> > 
> > I see fdatasync(2) coming through about every 6 seconds, so I believe the
> > guest
> > is properly flushing to disk. Now in this setup, my understanding is there
> > are 4
> > caches and permanent storage:
> > * guest page cache
> > * guest disk write cache (virtio)
> > * host page cache
> > * host disk write cache (SSD)
> > * permanent storage
> > 
> > I understand that dirty pages get queued up in the guest page cache, and
> > that
> > when an ext4 barrier is hit an fdatasync(2) call causes them to get flushed
> > to
> > disk. At this point, does fdatasync(2) force the data into the guest disk
> > write
> > cache, the host page cache, the host disk write cache, or permanent
> > storage?
> > Does this vary depending on if the virtio disk is set to cache=writeback or
> > cache=writethrough? Can you explain how this works for both cases?
> 
> With cache=writeback, a guest WRITE will put data to host page cache; a guest
> FLUSH will flush to permanent storage.
> 
> With cache=writethrough, a guest WRITE will put data to host page cache, and
> then flushed to permanent storage with a host side fdatasync issued by QEMU.
> In
> this case the guest sees no write cache on the virtual device.

According to this page, cache=writeback enables the disk write cache:
http://www.ibm.com/support/knowledgecenter/linuxonibm/liaat/liaatbpkvmguestcache.htm

Is this the write cache on the physical disk itself, or the write cache on the
qcow2 image (does the qcow2 even have a disk write cache)? Does this cache get 
bypassed when fdatasync(2) calls come in from the guest?

When is the guest's page cache ever used?

Let me see if I understand this correctly:

cache=writeback - data stored in host page cache, guest responsible for issuing
periodic fdatasync(2) calls; host disk cache enabled but never really used due
to large amount of RAM available on host

cache=writethrough - data stored in host page cache, host issues fdatasync(2)
immediately after write(2); host disk cache disabled

cache=none - data is written directly to the host disk cache, guest responsible
for issuing periodic fdatasync(2) calls

Thanks!

Andrew