Re: [Qemu-devel] [PATCH] Inter-VM shared memory PCI device

[Jamie Lokier]

Paul Brook wrote:
> > > In a cross environment that becomes extremely hairy.  For example the x86
> > > architecture effectively has an implicit write barrier before every
> > > store, and an implicit read barrier before every load.
> > 
> > Btw, x86 doesn't have any implicit barriers due to ordinary loads.
> > Only stores and atomics have implicit barriers, afaik.
> 
> As of March 2009[1] Intel guarantees that memory reads occur in
> order (they may only be reordered relative to writes). It appears
> AMD do not provide this guarantee, which could be an interesting
> problem for heterogeneous migration..

(Summary: At least on AMD64, it does too, for normal accesses to
naturally aligned addresses in write-back cacheable memory.)

Oh, that's interesting.  Way back when I guess we knew writes were in
order and it wasn't explicit that reads were, hence smp_rmb() using a
locked atomic.

Here is a post by Nick Piggin from 2007 with links to Intel _and_ AMD
documents asserting that reads to cacheable memory are in program order:

    http://lkml.org/lkml/2007/9/28/212
    Subject: [patch] x86: improved memory barrier implementation

Links to documents:

    http://developer.intel.com/products/processor/manuals/318147.pdf
    
http://www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/24593.pdf

The Intel link doesn't work any more, but the AMD one does.

Nick asserts "both manufacturers are committed to in-order loads from
cacheable memory for the x86 architecture".

I have just read the AMD document, and it is in there (but not
completely obviously), in section 7.2.  The implicit load-load and
store-store barriers are only guaranteed for "normal cacheable
accesses on naturally aligned boundaries to WB [write-back cacheable]
memory".  There are also implicit load-store barriers but not
store-load.

Note that the document covers AMD64; it does not say anything about
their (now old) 32-bit processors.

> [*] The most recent docs I have handy. Up to and including Core-2 Duo.

Are you sure the read ordering applies to 32-bit Intel and AMD CPUs too?

Many years ago, before 64-bit x86 existed, I recall discussions on
LKML where it was made clear that stores were performed in program
order.  If it were known at the time that loads were performed in
program order on 32-bit x86s, I would have expected that to have been
mentioned by someone.

-- Jamie