Re: [Qemu-discuss] drive cache=writethrough without flushing to physical disks OR drive cache=none with readonly host cache

[Gionatan Danti]

> Hello,
>
>As I understand so far:
>- cache=writethrough - uses host cache readonly and flushes all data 
to disks directly (avoid RAID cache)
>- cache=none - doesn't use host cache at all and written data goes to 
the RAID cache (if one exists)
>
>However there is no such a cache mode that lie in between:
>- uses host cache for reading and RAID cache for writing (without 
writing to host cache)
>
>Is there any reason not to have such a cache or am I missing something?
>
>Best regards,
>Stanislav

Hi,
from what I understand, you want something similar to a write-only 
O_DIRECT flag, ie: write directly to the device bypassing the host 
cache, but use the host-based cache for reading.

As far I know, this exact behavior can not be configured, nor in 
Qemu/KVM nor for any other Linux process. After all, think about what it 
means: even in a read-only caches, writing something to the device will 
immediately invalidate the read-only cache itself. The only option that 
I can imagine is to open the backing file two times, one with normal 
behavior and one with O_DIRECT behavior. At this point, you can use the 
first file descriptor for reads and reserve the second for writes. 
However, this will probably led to bad performance and maybe even to 
data corruption.

From the open(2) man page:
"Applications should avoid mixing O_DIRECT and normal I/O to the same 
file, and especially to overlapping byte regions in the same file. Even 
when the file system correctly handles the coherency issues in this 
situation, overall I/O throughput is likely to be slower than using 
either mode alone. Likewise, applications should avoid mixing mmap(2) of 
files with direct I/O to the same files."

Back to Qemu/KVM, you have to methods to obtain something similar to 
what you described (a read only cache):

1) simply use the writethrough cache and configure your battery 
backupped RAID card to ignore fsync requests (warning, you really _need_ 
a battery backupped RAID card or corruption / data loss will happen in 
case of a power failure). In this manner, you can use the OS-provided 
read cache, while passing all writes directly to the RAID card (wich 
will ignore the fsync requests attached with them). The problem is that 
the writethrough config will impose a big overhead on the OS (fsync are 
not cheap, even when the hardware ignore them), resulting in low 
performance anyway (probably).

2) with relatively recent Qemu versions (CentOS 6+), you can use the 
writeback cache, enable write barriers on the guest OS and configure 
your battery backupped RAID card to ignore fsync requests (the same 
warning apply: to safely configure your RAID cache as a writeback one, 
you _need_ a battery backupped card). With write barrier enabled on the 
guest side, you are sure that important writes will be synched to the 
RAID cache (as Qemu will pass the relative barriers to the underling 
OS), while the host will cache any read requests.

You can find some performance measurement on my site:
http://www.ilsistemista.net/index.php/virtualization/23-kvm-storage-performance-and-cache-settings-on-red-hat-enterprise-linux-62.html

Regards.