Re: [lwip-users] endless loop in netconn_write

[Martin Gotink]

I'm working with Edwin on the project and i have been debugging some more on 
the problem.
 
First of all i enabled the debug ability of the stack and am now throwing all 
debug info over the UART.
I also wrote a little tool which creates a number of threads which each do one 
request on the webserver. This to be able to always do the same requests with 
the same interval.
Running the tool with 3 threads crashes the webserver after a random number of 
requests, there is not noticable pattern. Running the tool with 1 thread doesnt 
seem to crash the webserver, or it has to take a long time.
 
Checking the log the only strange thing i noticed is the 'mem_malloc: could not 
allocate 1528 bytes' errors, in the beginning they only occur once in a while, 
but the longer the webserver runs and requests are made, the more often this 
error occurs. My first opinion would be some kind of memory leak, but the 
lwip_stat->mem->used variable is telling me its being malloced and freed 
without any probs. Its 0x3C on boot, and 0x3C after a request has been handled.
 
The crash still starts in the netconn_write loop, where on one point its not 
able to allocate memory for the send buffer. But after about 1-2 minutes it 
breaks out of the loop because of a connection error, ECONNABORTED.
That memory should be freed by an ACK packet acking data which is already sent, 
which doesnt seem to happen. This sounds like the problem, but how do you 
explain the growing number of malloc errors with that? I didnt see any 
retransmits from the client in the ethereal logs (except on the point where the 
server stops sending data), and the memory is still being freed... The problem 
seems to be in some other file than the api_lib.c since this file doesnt 
directly free/fill the send buffer.
 
After it breaks out of that loop it doesnt want to send data anymore, if i 
start that tool i made with 3 threads, i only see one thread coming in through 
the debug log. But i don't see the stack doing anything with it. The same goes 
for ICMP ping packets. ARP packets are still handled as it seems in the debug 
log.
 
I attached the whole debug log of 1 session, if you search for mem_malloc you 
will notice the growing number of malloc errors. After the last malloc error i 
did wait some time till doing new requests. There are 4 new requests in the log 
('TCP connection request'), i started the tool with 3 threads, 4 times, it 
seems each time only 1 connection gets through.
The whole time i was pinging the webserver with default packet size and 10 
seconds timeout, you will notice them through the whole log, in and after the 
netconn_write loop it didnt respond to ping messages anymore either. The ping 
packets didnt seem to have any effect on the stability of the webserver.
 
I hope someone can shed some more light on this problem, after all the 
debugging we;ve done, we dont really know where to look next...
 
(just for your information: the lwip stack is v1.1.1, running freertos 4.0.4 on 
the AT91SAM7X256 controller from atmel on a AT91SAM7X-EK evaluation kit)

Martin

3_threads_debug.zip
Description: Zip compressed data