Re: [Qemu-ppc] KVM and variable-endianness guest CPUs

[Benjamin Herrenschmidt]

> The point is simple, and Peter has made it over and over:
> Any consumer of a memory operation sees "value, len, address".
> 
> This is what KVM_EXIT_MMIO emulates.  So just by knowing the ABI
> definition and having a pointer to the structure you need to be able to
> tell me "value, len, address".

But that's useless because it doesn't tell you the byte order
of the value which is critical for emulation unless you *defined* in
your ABI the byte order of the value, and thus include an artificial
swap when the guest is in a different endian mode than the host.


My understanding is that ARM is byte-address invariant, as is powerpc,
so it makes a LOT more sense to carry a sequence of address ordered
bytes instead which will correspond to what the guest code thinks
it's writing, and have the device respond appropriately based on
the endianness of the bus it sits on or the device itself.


> > >  (2) the API between kernel and userspace needs to define
> > >      the semantics of mmio.data, ie how to map between
> > >      "x byte wide transaction with value v" and the array,
> > >      and that is primarily what this conversation is about
> > >  (3) the only choice which is both (a) sensible and (b)
> > >      not breaking existing usage is to say "the array is
> > >      in host-kernel-byte-order"
> > >  (4) PPC CPUs in BE mode and ARM CPUs in BE mode are not
> > >      the same, because in the ARM case it is doing an
> > >      internal-to-CPU byteswap, and in the PPC case it is not

I very much doubt that there is a difference here, I'm not sure about
that business with "internal byteswap".

The order in which the bytes of a word are presented on the bus changes
depending on the core endianness. If that's what you call a "byteswap"
then both ARM and PPC do it when they are in their "other" endian,
but that confusion comes from assuming that a data bus has an endianness
at all to begin with.

I was hoping that by 2014, such ideas were things of the past.

> > That is one of the key disconnects. I'll go find real examples
> > in ARM LE, ARM BE, and PPC BE Linux kernel. Just for
> > everybody sake's here is summary of the disconnect:
> > 
> > If we have the same h/w connected to memory bus in ARM
> > and PPC systems and we have the following three pieces
> > of code that work with r0 having same device same
> > register address:
> > 
> > 1. ARM LE word write of  0x04030201:
> > setend le
> > mov r1, #0x04030201
> > str r1, [r0]
> > 
> > 2. ARM BE word write of 0x01020304:
> > setend be
> > mov r1, #0x01020304
> > str r1, [r0]
> > 
> > 3. PPC BE word write of 0x01020304:
> > lis     r1,0x102
> > ori     r1,r1,0x304
> > stw    r1,0(r0)
> > 
> > I claim that h/w will see the same data on bus lines in all
> > three cases, and h/w would acts the same in all three
> > cases. Peter says that ARM BE and PPC BE case h/w
> > would act differently.
> > 
> > If anyone else can offer opinion on that while I am looking
> > for real examples that would be great.
> > 
> 
> I really don't think listing all these examples help.  You need to focus
> on the key points that Peter listed in his previous mail.
> 
> I tried in our chat to ask you this questions:
> 
> vcpu_data_host_to_guest() is handling a read from an emulated device.
> All the info you have is:
> (1) len of memory access
> (2) mmio.data pointer
> (3) destination register
> (4) host CPU endianness
> (5) guest CPU endianness
> 
> Based on this information alone, you need to decide whether you do a
> byteswap or not before loading the hardware register upon returning to
> the guest.
> 
> You will find it impossible to answer, because you don't know the layout
> of mmio.data, and that is the thing we are trying to solve.
> 
> If you cannot reply to this point in less than 50 lines or mention
> anything about devices being LE or BE or come with examples, I am
> probably not going to read your reply, sorry.
> 
> -Christoffer