Re: [Qemu-discuss] Understanding QEMU Cache Modes and Emulated Disk Write Cache

[Jakob Bohm]

On 17/08/2016 15:21, Andrew Martin wrote:
> > From: "Fam Zheng" <address@hidden>
> > To: "Andrew Martin" <address@hidden>
> > Cc: address@hidden
> > Sent: Wednesday, August 17, 2016 2:50:36 AM
> > Subject: Re: [Qemu-discuss] Understanding QEMU Cache Modes and Emulated Disk 
> > Write Cache
> >
> > On Tue, 08/16 12:39, Andrew Martin wrote:
> > > Hello,
> > >
> > > I am running QEMU 2.5 on Ubuntu 16.04. My guests are a mixture of Linux
> > > (Ubuntu
> > > 12.04 and newer with ext4) and Windows. The guest qcow2 images are stored
> > > in a
> > > ZFS filesystem with sync=standard (on top of a mirrored vdev of SSDs) on
> > > the host
> > > Ubuntu 16.04 OS. After having read through documentation and some previous
> > > threads on this list, I am still somewhat confused about caching.
> > >
> > > First, my understanding is that if I strace the qemu process and see it
> > > regularly issuing fdatasync(2), this means WCE is enabled and it is safe to
> > > use
> > > cache=writeback, correct?
> > >
> > > I see fdatasync(2) coming through about every 6 seconds, so I believe the
> > > guest
> > > is properly flushing to disk. Now in this setup, my understanding is there
> > > are 4
> > > caches and permanent storage:
> > > * guest page cache
> > > * guest disk write cache (virtio)
> > > * host page cache
> > > * host disk write cache (SSD)
> > > * permanent storage
> > >
> > > I understand that dirty pages get queued up in the guest page cache, and
> > > that
> > > when an ext4 barrier is hit an fdatasync(2) call causes them to get flushed
> > > to
> > > disk. At this point, does fdatasync(2) force the data into the guest disk
> > > write
> > > cache, the host page cache, the host disk write cache, or permanent
> > > storage?
> > > Does this vary depending on if the virtio disk is set to cache=writeback or
> > > cache=writethrough? Can you explain how this works for both cases?
> > With cache=writeback, a guest WRITE will put data to host page cache; a guest
> > FLUSH will flush to permanent storage.
> >
> > With cache=writethrough, a guest WRITE will put data to host page cache, and
> > then flushed to permanent storage with a host side fdatasync issued by QEMU.
> > In
> > this case the guest sees no write cache on the virtual device.
> According to this page, cache=writeback enables the disk write cache:
> http://www.ibm.com/support/knowledgecenter/linuxonibm/liaat/liaatbpkvmguestcache.htm
>
> Is this the write cache on the physical disk itself, or the write cache on the
> qcow2 image (does the qcow2 even have a disk write cache)? Does this cache get
> bypassed when fdatasync(2) calls come in from the guest?
>
> When is the guest's page cache ever used?
Whenever the Guest OS feels like it, because the Guest OS doesn't (in
principle) know the virtual disk is virtual.

> Let me see if I understand this correctly:
>
> cache=writeback - data stored in host page cache, guest responsible for issuing
> periodic fdatasync(2) calls; host disk cache enabled but never really used due
> to large amount of RAM available on host
Guest issues virtual "virtual disk hardware cache flushes", which are
turned into fdatasync(2) calls on host.

Host disk cache enabled, but not very useful because most of the data
in it is probably also in the host page cache.
> cache=writethrough - data stored in host page cache, host issues fdatasync(2)
> immediately after write(2); host disk cache disabled
Host disk cache enabled, but flushed (at least the relevant parts) by
the fdatasync(2) calls.

> cache=none - data is written directly to the host disk cache, guest responsible
> for issuing periodic fdatasync(2) calls
With the caveat above about what the the guest actually issues to cause
host side fdatasync(2) calls.

Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S.  https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark.  Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded