[lwip-users] lwip_send hang (probably due to my allocator)

[Stephen Cleary]

Hi folks –

 

I have an FTP server on lwIP that works for sending small files, but hangs when sending large files. The sending file code is quite simple – it just reads (up to) 2K and then sends it via lwip_send.

 

I’m running FreeRTOS on an AT91SAM7X512 (256K RAM).

 

When I say “hang”, I mean that all threads end up blocked. The ftp thread is waiting for a response from lwip_send (tcpip_apimsg -> sys_arch_sem_wait), and the tcpip_thread (lwip thread) is in tcpip_thread -> sys_mbox_fetch (it does time out and loop around, but does not get any messages).

 

I suspect the problem may be with the way I set up memory allocation. I wrote a TLSF-style allocator that can be called from an ISR or regular user code, and used that to replace both regular allocations (mem_malloc, etc) and pool allocations (MEMP_MEM_MALLOC == 1).

 

I have shown that when sending a large file, the memory is exhausted (mem_malloc does return 0). Also, I enabled link stats, and it shows drop and memerr non-zero (and equal).

 

I’m not a low-level TCP/IP guy, but I think what’s happening is that I’m filling up memory with outgoing buffers, and then there’s no memory for the eth driver to receive ACKs and pass them to the lwIP stack.

 

Does this sound correct?

 

If so, then I’m unsure what the best solution would be. I’m not fond of re-enabling the pools (our app usually does lots of small sends with multiple server and client connections, but occasionally needs to send or receive gigabytes; historically, tweaking the pool sizes has been a painful trial-and-error process).

 

Is it possible to statically allocate space for a pbuf chain large enough for any incoming packet? This doesn’t seem feasible since netif::input takes ownership of the pbuf chain.

 

One solution would be to limit the amount of data ftp passes to lwip_send. My TCP_SND_BUF is TCP_WND (== 4 * TCP_MSS, where TCP_MSS == 1460), so it’s only 5840 bytes, but I’m pretty sure lwip_send is allowing me to queue more data than that. Is there a procedure for detecting “how full” the send buffer is on a socket so that I can throttle in my FTP code?

 

I’d appreciate any advice on how to proceed.

 

        -Steve