Re: [Qemu-devel] [RFC] Disk integrity in QEMU

[Anthony Liguori]

Avi Kivity wrote:
> Chris Wright wrote:
> > I think it's safe to say the perf folks are concerned w/ data integrity
> > first, stable/reproducible results second, and raw performance third.
> >
> > So seeing data cached in host was simply not what they expected.  I 
> > think
> > write through is sufficient.  However I think that uncached vs. wt will
> > show up on the radar under reproducible results (need to tune based on
> > cache size).  And in most overcommit scenarios memory is typically more
> > precious than cpu, it's unclear to me if the extra buffering is anything
> > other than memory overhead.  As long as it's configurable then it's
> > comparable and benchmarking and best practices can dictate best choice.
> >   
>
> Getting good performance because we have a huge amount of free memory 
> in the host is not a good benchmark.  Under most circumstances, the 
> free memory will be used either for more guests, or will be given to 
> the existing guests, which can utilize it more efficiently than the host.

There's two arguments for O_DIRECT.  The first is that you can avoid 
bringing in data into CPU cache.  This requires zero-copy in QEMU but 
ignoring that, the use of the page cache doesn't necessarily prevent us 
from achieving this.

In the future, most systems will have a DMA offload engine.  This is a 
pretty obvious thing to attempt to accelerate with such an engine which 
would prevent cache pollution.  Another possibility is to directly map 
the host's page cache into the guest's memory space.

The later is a bit tricky but is so much more interesting especially if 
you have a strong storage backend that is capable of deduplification 
(you get memory compaction for free).

I also have my doubts that the amount of memory saved by using O_DIRECT 
will have a noticable impact on performance considering that guest 
memory and page cache memory are entirely reclaimable.  An LRU should 
make the best decisions about whether memory is more valuable for the 
guests or for the host page cache.

Regards,

Anthony Liguori

> I can see two cases where this is not true:
>
> - using older, 32-bit guests which cannot utilize all of the cache.  I 
> think Windows XP is limited to 512MB of cache, and usually doesn't 
> utilize even that.  So if you have an application running on 32-bit 
> Windows (or on 32-bit Linux with pae disabled), and a huge host, you 
> will see a significant boost from cache=writethrough.  This is a case 
> where performance can exceed native, simply because native cannot 
> exploit all the resources of the host.
>
> - if cache requirements vary in time across the different guests, and 
> if some smart ballooning is not in place, having free memory on the 
> host means we utilize it for whichever guest has the greatest need, so 
> overall performance improves.