Re: [lwip-users] lwIP reentrancy issues and TCP

[FreeRTOS Info]

Hi Andy, et al.

>>  (although I would pay a small amount for somebody to provide one,
>> as per previous discussion).
> 
> You probably won't have to pay anyone. I've just completed a new "zero-copy" 
> emac/dma driver for the LPC1768. It's "tightly bound" to LWIP (in that it 
> uses 
> pbuf_pool for the descriptor target buffers).

Excellent.

> 
> There are two compile modes configured by macros.
> 
> 1) Classic Lwip: user responsible for calling lwip to process queues and 
> timers 
> in it's "single thread" user context process.
> 
> 2) FreeRTOS: uses just the technique you describe. The queues are handled by 
> queue messages from the ISR to the lwip task. Timers simply by the lwip task.
> 
> Still in beta at the moment but when I am happy I'll be publishing it all as 
> foss.

It is always apprecated by the FreeRTOS community if projects that that,
if they are using FreeRTOS, are posted to the FreeRTOS Interactive site:
http://interactive.freertos.org

Ultimately my aim to to have a reference that uses the sockets API and
runs in the Windows simulator.  Sockets because, although the weightiest
API, it is the most familiar to novices.  In the simulator because, you
can try it without needing any hardware or to purchase any tools.


> But while I have your attention I'd like to ask a question just so I can get 
> this spot on and usable as a good reference design for others.
> 
> First up, a brief description of "Classic Lwip" mode (no rtos) which was 
> where I 
> started out from (I decided to make it FreeRTOS friendly later which means a 
> re-factor, it's this re-factor I want to get right to make it truly useful 
> for 
> others).
> 
> In Classic Lwip mode, lwip pbuf_pool is setup to provide "many small pbufs". 
> In 
> my case at the moment, 38 of them all payload sized at 256 bytes. When the 
> emac 
> driver inits the rx descriptor arrays (there are 18 rx descriptors currently) 
> are each assigned a pbuf_pool pbuf. Once all is setup, the emac and isr is 
> enabled.
> 
> Now, here's where things get interesting. Although the emac driver is tightly 
> bound to lwip (in that is uses lwip memory manager) I wanted my driver to be 
> able to handle Ethernet protocols other than just IP and ARP. So my emac 
> driver 
> after init actually drops all packets. It's not until a "protocol handler" 
> registers itself as "I want these etherType packets" that the driver actually 
> starts doing anything useful (the register is done during init).
> 
> So, for example, during low_level_init() in the Lwip code it calls a function 
> to 
> register 0800 and 0806 etherType packets and provides a C function pointer to 
> callback to. The callback functions simply take a structpbuf * arg which 
> points 
> at the head pbuf in a pbuf chain. The callback then places that into a queue. 
> It's the "user process" that then empties this queue into lwip during it's 
> normal single user thread mode of operation. Likewise, other protocol 
> handlers 
> do the same (in my case I also have an LLDP protocol handler which plays nice 
> with lwip pbufs). If the driver has no etherType callbacks registered it 
> drops 
> the packet.

Sounds reasonable, so far, without the kernel coming into play.



> So this design means there is "some minimal intelligence" built into the 
> emacISR 
> handler.
> 
> So, moving onto FreeRTOS. Instead of registering a C function callback to 
> call 
> in ISR context, the protocol handler (lwip netif, or LLDP, etc) registers an 
> etherType number and a Queue into which it just injects the pbuf chain.
> 


So there is one queue per protocol being handled.

You could alternatively have a single queue, but post to the queue a
structure that has the protocol identifier as one member, and a pointer
to the data as another member.  Then you could have one task that
handles both protocols - the first thing it does is check the protocol
identifier to know which function to pass the data to to get processed
correctly.


> Additionally, when the emacISR forms pbuf chains from the descriptor array, 
> each 
> descriptor is now void of a payload taget. So the ISR makes an attempt to 
> replace this lost pbuf with a new one using pbuf_alloc() (using the usual 
> lwip 
> protection mechanisms in sys_arch). It may not succeed if all pbufs are in 
> use 
> in which case it attempts to "keep up" when other events trigger. 

Be careful of using critical section type protection mechanisms.  If
that can be done in FreeRTOS depends on the port being used.  I find
normally you can do the re-allocation of a buffer to the EMAC driver at
the task level too.


> Anyway, coming onto my question. You've said that in FreeRTOS the ISR should 
> "just send a message" and the task should then do the processing. However, 
> that 
> really doesn't fit my design as it assumes that the task is going to manage 
> the 
> emac descriptor array, forming pbuf_chains, etc etc. So I have two choices:-


As per my comment above - but why can that not be done in the task?
When you have data passed to the task, the task knows that the
descriptor from which the data originated needs to be allocated a new
buffer.  Then, when the processing of the data in the buffer has been
completed, the task releases the memory again.


> 1) Leave a minimal amount of intelligence in the ISR to manage things at the 
> hardware level and chain forming, queue injection, etc.

I'm not sure what you mean by "hardware level".  The hardware DMA can
normally manage looking for descriptors that point to memory buffers,
but giving a memory buffer to a descriptor in the first place would (in
my experience) have to be a software task for a zero-copy type
implementation.



> 2) Create a gatekeeper task that sits behind the ISR that is then responsible 
> managing the hardware, dispatching packets into queues, etc etc.

Maybe I'm not following your scenario correctly here.

I would have thought a gatekeeper task would be a possible design
solution if you have multiple tasks, or tasks and interrupts, both
accessing the same resource (in this case the EMAC descriptors and the
memory buffers).  It would not be required if there was only "TCP/IP
Task" that was the only thing that ever touched the descriptors and buffers.



> 
> Which is the "preferable design" given that the amount of work that's 
> actually 
> being done is quite minimal. Option 2 seems like the "most logical solution" 
> but 
> given the small amount of code involved option 1 seems like the "most 
> sensible 
> solution" given the tight constraints of the uCenv, memory, etc. In addition, 
> option 2 probably means more work for me in the Classic Lwip (non-rtos) 
> design. 
> Whereas keeping a small but efficient ISR means both designs become simpler 
> to 
> implement for both Classic and RTOS versions.
> 
> Although I'm fairly new to lwip-users mailing list I have noticed that the 
> work 
> I'm doing and going to publish addresses many of the questions I've seen on 
> the 
> list. 

Exactly.  I was pointed to the Windows example in the lwIP contrib
distribution as an example that may be good for you too.  That would
just then need converting to use FreeRTOS instead of the windows threads.


> So from a "reference" point of view (ie telling people "go look at this 
> for an example of how to do that") I'd like to get it spot on and an example 
> of 
> best practice. 

Agree again.

> As, I'm not just going to publish the code, I intend to to use 
> the Lwip Wiki to fully document from hardware to examples of Lwip use the 
> whole 
> process.
> 
> Sorry for the long and boring post, I'm just seeking to make sure I get the 
> best 
> possible implementation and thereby best possible reference for others to 
> look 
> at and pick apart to see how things can be done.
> 
> regards,
> --Andy


Regards,
Richard.

+ http://www.FreeRTOS.org
Designed for Microcontrollers.  More than 7000 downloads per month.

+ http://www.SafeRTOS.com
Certified by TÜV as meeting the requirements for safety related systems.