Re: [lwip-users] Zero Copy Ethernet interface

[Jonathan Larmour]

Paul Black wrote:
> I'm trying to get to grips with lwip and one of the things I'm looking
> at is how data moves between the stack and the ethernet interface.
>
> For input:
> The examples ethernetif.c has a function called low_level_input which
> copies data from somewhere into a chain of pbufs that are then passed up.
>
> I'm thinking that I can do something like the following:
>  - Allocate several pbufs in advance for incoming packets: can I
>    allocate a single pbuf as I would need to dechain them otherwise. How
>    do I allocate a pbuf of maximum size or find the maximum space in a
>    single pbuf?
>  - When the packet comes in, find out how many pbufs were used - I'm
>    guessing I can then chain them together again with pbuf_cat()?
>  - Pass this to wherever.

Yes. I've implemented zero copy receives in a way similar to this (although 
I had to subvert the pbuf API, and fiddle the struct pbuf contents myself). 
 It's true you could preallocate your pool of pbufs each with the full MTU 
size, but I went with chains of pbufs of a smaller (but still fixed) size, 
as my hardware could cope with that; thus using far less space for the 
(very frequent) smaller packets.

My hardware[1] uses a circular list of buffer descriptors, so I also did 
the equivalent of pbuf_cat myself in the driver too. This also meant that 
when a packet is received, I get another pbuf from the pool and put its 
pbuf payload pointer in the hardware's buffer descriptor, thus ensuring the 
hardware buffers remain full of packet buffers.

Personally I did this with some modifications imposing extra constraints on 
the pbuf pool, which I did using a new override macro, along with a hook 
which I added to PBUF_POOL_FAST_FREE. This is because of both the buffer 
fiddling and extra alignment and positioning constraints on the payload.

The result, anyway, is zero copy receives. Which is nice, small and fast. 
Of course you start getting copies if you use the socket API, but netconn 
API (aka sequential API) or raw API users get the benefit. Well, in fact 
socket API does too as it reduces the number of copies by one.

> For output:
> I get passed a chain of pbufs:
>  - Because I want to hang on to these until an interrupt has fired, I
>    guess I need to call pbuf_ref()?

Yes and no. Yes, I have found that this works, but officially the answer is 
no because someone may modify a pbuf after it has been sent. This has 
resulted in task #6735 to provide a new pbuf type of PBUF_RAM_NOCOPY, in 
order to permit efficient driver operation because the alternative is 
having to copy the packet *every* *time* just in case.

For you, if you can impose a constraint on your applications that they are 
not allowed to modify pbufs after transmission (except to free them) then 
you can get away with just a pbuf_ref() (and consequent pbuf_free() when 
the transmission completes).

>  - It might make my code easier for tidying up after transmission if I
>    split the chain into its constituent parts - for this I would call
>    pbuf_dechain()?

Personally, I'd just iterate down the list using the struct pbuf internals. 
You don't care about consistent leftover pbufs on transmission, so 
pbuf_dechain is doing a lot of faff that would be unnecessary in a hardware 
driver. That's what I did anyway.

Jifl
[1] In fact I've implemented this approach now in two completely different 
ethernet drivers, both on-chip MACs using DMA, one coldfire, one AT91. 
Source code not available sorry. I know already I'd never get it past 
management.
--
eCosCentric Limited      http://www.eCosCentric.com/     The eCos experts
Barnwell House, Barnwell Drive, Cambridge, UK.       Tel: +44 1223 245571
Registered in England and Wales: Reg No 4422071.
 >>>> Visit us on stand 810 at The Embedded Systems Show 2007, NEC <<<<
 >>>> Oct 17-18 Birmingham, UK http://www.edaexhibitions.com/ess/  <<<<
------["The best things in life aren't things."]------      Opinions==mine