Re: [Qemu-devel] KVM and variable-endianness guest CPUs

[Victor Kamensky]

On 23 January 2014 08:25, Victor Kamensky <address@hidden> wrote:
> On 23 January 2014 07:33, Peter Maydell <address@hidden> wrote:
>> On 23 January 2014 15:06, Victor Kamensky <address@hidden> wrote:
>>> In [1] I wrote
>>>
>>> "I don't see why you so attached to desire to describe
>>> data part of memory transaction as just one of int
>>> types. If we are talking about bunch of hypothetical
>>> cases imagine such bus that allow transaction with
>>> size of 6 bytes. How do you describe such data in
>>> your ints speak? What endianity you can assign to
>>> sequence of 6 bytes? While note that description of
>>> such transaction as set of 6 byte values at address
>>> $whatever makes perfect sense."
>>>
>>> But notice that in your next reply [2] you just dropped it
>>
>> Yes. This is because it was one of the places where
>> I would have just had to repeat "no, I'm afraid you're wrong
>> about how hardware works". I think in general it's going
>> to be better if I don't try to reply point by point to this
>> email; I think you should go back and reread the emails I've
>> sent. Key points:
>>  (1) hardware is not doing anything involving arrays
>>      of bytes
>
> Array of bytes or integers is just a way to describe data lines
> on the bus. Did you look at this document?
>
> http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0290g/ch06s05s01.html
>
> A0, A1, ,,, A7 byte values are the same for both LE and BE-8
> case (first two columns in the table) and they unambiguously
> describe data bus signals
>
>>  (2) the API between kernel and userspace needs to define
>>      the semantics of mmio.data, ie how to map between
>>      "x byte wide transaction with value v" and the array,
>>      and that is primarily what this conversation is about
>>  (3) the only choice which is both (a) sensible and (b)
>>      not breaking existing usage is to say "the array is
>>      in host-kernel-byte-order"
>>  (4) PPC CPUs in BE mode and ARM CPUs in BE mode are not
>>      the same, because in the ARM case it is doing an
>>      internal-to-CPU byteswap, and in the PPC case it is not
>
> That is one of the key disconnects. I'll go find real examples
> in ARM LE, ARM BE, and PPC BE Linux kernel. Just for
> everybody sake's here is summary of the disconnect:
>
> If we have the same h/w connected to memory bus in ARM
> and PPC systems and we have the following three pieces
> of code that work with r0 having same device same
> register address:
>
> 1. ARM LE word write of  0x04030201:
> setend le
> mov r1, #0x04030201
> str r1, [r0]
>
> 2. ARM BE word write of 0x01020304:
> setend be
> mov r1, #0x01020304
> str r1, [r0]
>
> 3. PPC BE word write of 0x01020304:
> lis     r1,0x102
> ori     r1,r1,0x304
> stw    r1,0(r0)
>
> I claim that h/w will see the same data on bus lines in all
> three cases, and h/w would acts the same in all three
> cases. Peter says that ARM BE and PPC BE case h/w
> would act differently.
>
> If anyone else can offer opinion on that while I am looking
> for real examples that would be great.

Here is my example:

Let's look at isp1760 USB host controller (effectively
one used by TC2). Source code is at
drivers/usb/host/isp1760-hcd.c file. The driver could
be enabled in kernel by adding CONFIG_USB_ISP1760_HCD=y
config I enabled it in ppc image build, arm TC2 already
have it

isp1760 USB host controller driver registers are
in LE format. isp1760 USB host controller driver uses
reg_write32 function to write memory mapped controler
registers. That in turns calls writel, which is LE
device word write:

void reg_write32(void __iomem *base, u32 reg, u32 val)
{
    writel(val, base + reg);
}

In C terms writel is LE word write function. It is
effective memory barrier, cpu_to_le32 and write.
cpu_to_le32 will do byteswap only in BE case.

LE ARM
------

00002e04 <reg_write32>:
    2e04:       e92d4070        push    {r4, r5, r6, lr}
    2e08:       e1a04000        mov     r4, r0
    2e0c:       e1a05001        mov     r5, r1
    2e10:       e1a06002        mov     r6, r2
    2e14:       f57ff04e        dsb     st
    2e18:       e59f3018        ldr     r3, [pc, #24]   ; 2e38
<reg_write32+0x34>
    2e1c:       e5933018        ldr     r3, [r3, #24]
    2e20:       e3530000        cmp     r3, #0
    2e24:       0a000000        beq     2e2c <reg_write32+0x28>
    2e28:       e12fff33        blx     r3
    2e2c:       e0841005        add     r1, r4, r5
    2e30:       e5816000        str     r6, [r1] @ <-------
    2e34:       e8bd8070        pop     {r4, r5, r6, pc}
    2e38:       00000000        .word   0x00000000

operates in LE, just write value to device register mem location

BE ARM
------

00000590 <reg_write32>:
     590:       e92d4070        push    {r4, r5, r6, lr}
     594:       e1a04000        mov     r4, r0
     598:       e1a05001        mov     r5, r1
     59c:       e1a06002        mov     r6, r2
     5a0:       f57ff04e        dsb     st
     5a4:       e59f301c        ldr     r3, [pc, #28]   ; 5c8 <reg_write32+0x38>
     5a8:       e5933018        ldr     r3, [r3, #24]
     5ac:       e3530000        cmp     r3, #0
     5b0:       0a000000        beq     5b8 <reg_write32+0x28>
     5b4:       e12fff33        blx     r3
     5b8:       e6bf2f36        rev     r2, r6   @ <-------
     5bc:       e0841005        add     r1, r4, r5
     5c0:       e5812000        str     r2, [r1] @ <-------
     5c4:       e8bd8070        pop     {r4, r5, r6, pc}
     5c8:       00000000        .word   0x00000000

operates in BE, byteswap first, then write value to device mem location

BE PPC
------

00003070 <reg_write32>:
    3070:       7c 08 02 a6     mflr    r0
    3074:       90 01 00 04     stw     r0,4(r1)
    3078:       48 00 00 01     bl      3078 <reg_write32+0x8>
    307c:       94 21 ff f0     stwu    r1,-16(r1)
    3080:       7c 08 02 a6     mflr    r0
    3084:       90 01 00 14     stw     r0,20(r1)
    3088:       7c 83 22 14     add     r4,r3,r4
    308c:       7c 00 04 ac     sync
    3090:       7c a0 25 2c     stwbrx  r5,0,r4 @ <-------
    3094:       80 01 00 14     lwz     r0,20(r1)
    3098:       38 21 00 10     addi    r1,r1,16
    309c:       7c 08 03 a6     mtlr    r0
    30a0:       4e 80 00 20     blr

stwbrx - is 'Store Word Byte-Reverse Indexed' instruction
bascially byteswap word and store it (r3 is base, r4
is reg, r5 is val).

operates in BE, byteswap and store value but now it is
done in one instruction. But it is exactly what ARM BE
code does.

So if one will call reg_write32(0x1000, 0, 0x04030201) it
would work the same way as earlier cited sequences - ARM
LE case would use 0x04030201 and write it out. BE ARM and
BE PPC would get byteswap value of (0x04030201)=0x01020304
and write it out.

Device see the same thing for all three cases, that is how
BE-8 memory bus works.

Thanks,
Victor

> Thanks,
> Victor
>
>> thanks
>> -- PMM