Re: [lwip-devel] again... Zero Copy Ethernet interface

[Jonathan Larmour]

Alain M. wrote:
> I will add 1cent more...
> 
> This buffer region is important in more ways: some ARM9 (Atmel for one)
> have some separate internal memory banks that if used for TX/TX can
> acoplish DMA at almost zero overhead using different access hardware. I
> understand that most DSP have memory banks with some kind of optimization...
> 
> So, I vote for this issue to be carefully considered to be integrated in
> future versions ;)

It certainly needs to be carefully considered. Different devices have
different properties:

- some must have all buffers (RX and TX) in a defined region of memory
- some must have RX in one region, TX in another
- some can have buffers in one region and use that preferentially for
speed, but can fall back to slower memory if needed
- some can tolerate the 'struct pbuf' header, some can't
- some can scatter gather, some can't
- Some allow entirely variably sized buffers.
- some have fixed size buffers, which may be fixed at the MTU, or may be
smaller; and may be hard-coded for that hardware (e.g. 128 bytes on
AT91SAM7X) or can be any size, but it has to be fixed for all buffers when
the device is initialised.
- some have fixed size buffers and require them to be filled for all but
the final one in the chain
- some have fixed size buffers and don't mind if they are not all filled
- some have associated buffer descriptors which need fiddling with when the
pbuf data becomes available again after having been freed.
- In any of the above, there can be alignment constraints above and beyond
the CPU architectural alignment; most frequently DMA and/or cache line.

And the real annoyance: you could have two different devices with different
(non-overlapping) sets of the above restrictions.

All the above requirements have an impact on how to implement zero copy
with lwIP. It's not difficult to solve naively, but requires careful
consideration for a properly lightweight solution.

I have implemented zero copy myself, with minimal changes required to the
lwIP core, but it does not allow for all the requirements above by any means.

In case it is useful the essence of what I did was add two hooks to pbuf.c:

One to memp_init() to allow a driver hook:
#ifdef PBUF_POOL_INIT
  PBUF_POOL_INIT();
#endif
...

this is done after the memp arrays are set up, and allows memp_tab for the
pbufs to be changed. memp_memory and memp_init() were also modded so that
the pbuf pool space was not included in the memp_memory array.

And a hook to pbuf_free(), which allowed lwip to call into the driver on free:
#ifndef PBUF_POOL_FREE_HOOK
#define PBUF_POOL_FREE_HOOK(p) do { } while (0)
#endif

These hooks allowed the pbufs to be allocated from a different region, and
allowed the pbufs to be released back and buffer descriptors updated. There
were a few other bits and pieces, but that was the core of it. It was
certainly sufficient for M68k Coldfire and Atmel SAM7/SAM9 MACs. But there
are plenty of MACs out there for which it isn't enough, so it's not the
basis of a proper solution.

Sorry I can't give out whole verbatim code - I'd have to ask for permission
from my employer for that.

Jifl
-- 
eCosCentric Limited      http://www.eCosCentric.com/     The eCos experts
Barnwell House, Barnwell Drive, Cambridge, UK.       Tel: +44 1223 245571
Registered in England and Wales: Reg No 4422071.
------["Si fractum non sit, noli id reficere"]------       Opinions==mine