[lwip-users] Memory alignment and DMA controllers

[Will Wykeham]

Dear all,

I'm having an issue getting the stack producing data in a suitably aligned form 
for my DMA controller. I'm expecting this is something to do with configuration 
on my part rather than a stack limitation so I'd appreciate some help.

I'm running lwIP on a NIOS, currently just responding to ARP (although it will 
be doing more later). I'm using a scatter/gather DMA controller to talk to the 
Ethernet MAC. The receive side works perfectly, but I'm having some issues with 
the transmit side because the DMA controller requires alignment on a 32 bit 
boundary (I have MEM_ALIGNMENT set to 4).

After some digging this is what is going on:

etharp_raw is calling pbuf_alloc(PBUF_LINK, SIZEOF_ETHARP_HDR, PBUF_RAM);
This allocates a pbuf control structure, space for the eth header and then 
space for the arp packet. The allocation is aligned correctly, and the pbuf 
structure is 16 bytes so the start of the payload space is also aligned. The 14 
byte PBUF_LINK values is then rounded up to 16 due to the alignment, and so the 
start of the payload space is 16 bytes after the structure, also on an aligned 
boundary.
When the packet is handed from etharp down to ethernet_output, the payload 
pointer is wound back 14 bytes to make space for the header. As it started on 
an aligned boundary, we know that winding it back 14 bytes is guaranteed to 
make it not aligned, and that is what gets passed to link_output.

Some ASCII art showing the layout, with * indicate aligned 4 byte boundaries 
(and one column per byte):
*   *   *   *   *   *   *   *   *   *   *
+------------------------------------------------+
|              |--|            ||
|  PCB - 16    |--|  Eth - 14  ||   ARP ...
|              |--|            ||
+------------------------------------------------+
          2 bytes unused

The possible ways of fixing it that I have come up with are:

(1)
Change the argument in pbuf_alloc where it passes the pointer to 
pbuf_init_alloced_pbuf from:
LWIP_MEM_ALIGN((void *)((u8_t *)p + SIZEOF_STRUCT_PBUF + offset))
To:
LWIP_MEM_ALIGN((void *)((u8_t *)p + SIZEOF_STRUCT_PBUF)) + offset
This aligns the data space after the pbuf, and then adds on the offset. The 
payload pointer starts of unaligned, but becomes aligned when wound back by 14 
bytes.
This is fine as long as the 'layer' value is correct, but my understanding is 
that it is a conservative maximum and when more stack layers are involved it 
may be an over estimate, so we're not guaranteed the payload pointer will be 
wound back to where it needs to be. Also, that's me patching the stack which 
I'd rather not do.

(2)
Use the malloc allocator and set MEM_ALIGNMENT to 1. The allocation is aligned, 
so the start of the payload is aligned, and no padding is done because 
MEM_ALIGNMENT doesn't need it. This has the same effect as (1) - it ensures 
that we get an aligned pointer after the adjustment, but comes with the same 
risk of not using all the header space. Also, it's a horrible hack although one 
I can do entirely in the config headers.

(3)
Just do a memmove in my driver layer to move the data back to an aligned 
boundary. This is the most robust solution (and what I've got at the moment), 
it's just a bit sad it effectively involves a copy (although I can do it in the 
pbuf which at least avoids needing an extra buffer).

Am I missing something or is this just a limitation of how the stack works?

Thank you in advance for your help,
Will

____________________________________________________________________________________________________________________________________________________________________________________________
PA Consulting Services Limited is registered in England and Wales under 
registered number 414220. Its registered office is at 10 Bressenden Place, 
London, SW1E 5DN, United Kingdom.
If you are not the intended recipient of this email, please notify the sender 
as soon as possible and delete it from your systems. Please do not disclose the 
contents of this email to anyone else.