[lwip-users] Suggestions on how to attack this particular issue

[Micael]

Hi Guys.

I'm having issues with lwip, where it works for a while, but then, after some 
connects, it simply stops working, and I would appreciate suggestions on how 
to try to nail this particular problem.

I am running lwip with the tap interface under linux, using sockets, where I 
have one thread listening on a port for a connction, and another thread 
connecting to, and sending to the other server (which is a separate linux 
program using the linux tcpip stack).

I am using most, if not all stuff from unixsim; tap interface, and memory 
allocs, so it ought to be working somewhat.


So: 
Two threads, 
one for transmits, which connects, sends, closes all the time and one for 
reception which listens for incoming data; accept, read, close.
No direct interaction between the threads and their sockets.

The thing is, that when I enable a lot of debug output, system gets slow, and 
I get the feeling that this may give me other issues, than the one I am trying 
to find. If I set the LWIP_DBG_MIN_LEVEL to 1, I normally get no error 
outputs, and system is snappy until it breaks. At a few times, with this 
setting, I have got the error "tcp_enqueue: could not allocate memory for 
tcp_seg". But I have never seen it with high debug output. And the result are 
not consistent.

If I disable sendout*, it seams as if the stack survives, but I am not sure if 
this is because less work, less memory use or because there's only work going 
on, on one single thread.
The application works just fine, if I use linux native sockets.

I guess this indicates that I ought to hunt for problematic memory 
allocations and/or thread issues.


*I am not actually stopping the thread, just makes sure nothing is sent. 
There's also another thread, which sits in listen during my tests. I don't 
think that this changes anything as I am sure there's no connection attempts 
on this port during my tests.

For example, this is a debug output, where the stack is damaged:
..
tcpip: etharp_tmr()
tcpip: ip_reass_tmr()
tcpip: ip_reass_tmr()
tcpip: ip_reass_tmr()
tcpip: ip_reass_tmr()
tcpip: ip_reass_tmr()
tcpip_thread: PACKET 0x81a8088
ip_input: iphdr->dest 0xb0b010a netif->ip_addr 0xb0b010a (0xb010a, 0xb010a, 
0xb000000)
ip_input: packet accepted on interface tp
ip_input: 
IP header:
+-------------------------------+
| 4 | 5 |  0x00 |        60     | (v, hl, tos, len)
+-------------------------------+
|    59702      |010|       0   | (id, flags, offset)
+-------------------------------+
|   64  |    6  |    0x2778     | (ttl, proto, chksum)
+-------------------------------+
|   10  |    1  |   11  |    1  | (src)
+-------------------------------+
|   10  |    1  |   11  |   11  | (dest)
+-------------------------------+
ip_input: p->len 60 p->tot_len 60
TCP header:
+-------------------------------+
|    52408      |     3810      | (src port, dest port)
+-------------------------------+
|           3303448858          | (seq no)
+-------------------------------+
|           0000000000          | (ack no)
+-------------------------------+
| 10 |   |000010|      5840     | (hdrlen, flags (SYN ), win)
+-------------------------------+
|    0x6443     |         0     | (chksum, urgp)
+-------------------------------+
tcp_input: packed for LISTENing connection.
TCP connection request 52408 -> 3810.
tcpip: etharp_tmr()
tcpip: ip_reass_tmr()
tcpip: ip_reass_tmr()
tcpip: ip_reass_tmr()
tcpip: ip_reass_tmr()
tcpip: ip_reass_tmr()
tcpip: etharp_tmr()
tcpip: ip_reass_tmr()
tcpip: ip_reass_tmr()
tcpip: ip_reass_tmr()
tcpip: ip_reass_tmr()
tcpip_thread: PACKET 0x81a8088
ip_input: iphdr->dest 0xb0b010a netif->ip_addr 0xb0b010a (0xb010a, 0xb010a, 
0xb000000)
ip_input: packet accepted on interface tp
ip_input: 
IP header:
+-------------------------------+
| 4 | 5 |  0x00 |        60     | (v, hl, tos, len)
+-------------------------------+
|    17049      |010|       0   | (id, flags, offset)
+-------------------------------+
|   64  |    6  |    0xce15     | (ttl, proto, chksum)
+-------------------------------+
|   10  |    1  |   11  |    1  | (src)
+-------------------------------+
|   10  |    1  |   11  |   11  | (dest)
+-------------------------------+
ip_input: p->len 60 p->tot_len 60
TCP header:
+-------------------------------+
|    52409      |     3810      | (src port, dest port)
+-------------------------------+
|           3768527299          | (seq no)
+-------------------------------+
|           0000000000          | (ack no)
+-------------------------------+
| 10 |   |000010|      5840     | (hdrlen, flags (SYN ), win)
+-------------------------------+
|    0xb54b     |         0     | (chksum, urgp)
+-------------------------------+
tcp_input: packed for LISTENing connection.
TCP connection request 52409 -> 3810.
tcpip: ip_reass_tmr()
tcpip: etharp_tmr()
tcpip: ip_reass_tmr()
tcpip: ip_reass_tmr()
tcpip_thread: PACKET 0x81a8088
ip_input: iphdr->dest 0xb0b010a netif->ip_addr 0xb0b010a (0xb010a, 0xb010a, 
0xb000000)
ip_input: packet accepted on interface tp
ip_input: 
IP header:
+-------------------------------+
| 4 | 5 |  0x00 |        60     | (v, hl, tos, len)
+-------------------------------+
|    17050      |010|       0   | (id, flags, offset)
+-------------------------------+
|   64  |    6  |    0xce14     | (ttl, proto, chksum)
+-------------------------------+
|   10  |    1  |   11  |    1  | (src)
+-------------------------------+
|   10  |    1  |   11  |   11  | (dest)
+-------------------------------+
ip_input: p->len 60 p->tot_len 60
TCP header:
+-------------------------------+
|    52409      |     3810      | (src port, dest port)
+-------------------------------+
|           3768527299          | (seq no)
+-------------------------------+
|           0000000000          | (ack no)
+-------------------------------+
| 10 |   |000010|      5840     | (hdrlen, flags (SYN ), win)
+-------------------------------+
|    0xb1c7     |         0     | (chksum, urgp)
+-------------------------------+
tcp_input: packed for LISTENing connection.
TCP connection request 52409 -> 3810.
tcpip: ip_reass_tmr()
tcpip: ip_reass_tmr()
tcpip: ip_reass_tmr()
tcpip: etharp_tmr()
tcpip: ip_reass_tmr()
tcpip: ip_reass_tmr()
tcpip: ip_reass_tmr()
tcpip_thread: PACKET 0x81a8088
ip_input: iphdr->dest 0xb0b010a netif->ip_addr 0xb0b010a (0xb010a, 0xb010a, 
0xb000000)
ip_input: packet accepted on interface tp
ip_input: 
IP header:
+-------------------------------+
| 4 | 5 |  0x00 |        60     | (v, hl, tos, len)
+-------------------------------+
|    17051      |010|       0   | (id, flags, offset)
+-------------------------------+
|   64  |    6  |    0xce13     | (ttl, proto, chksum)
+-------------------------------+
|   10  |    1  |   11  |    1  | (src)
+-------------------------------+
|   10  |    1  |   11  |   11  | (dest)
+-------------------------------+
ip_input: p->len 60 p->tot_len 60
TCP header:
+-------------------------------+
|    52409      |     3810      | (src port, dest port)
+-------------------------------+
|           3768527299          | (seq no)
+-------------------------------+
|           0000000000          | (ack no)
+-------------------------------+
| 10 |   |000010|      5840     | (hdrlen, flags (SYN ), win)
+-------------------------------+
|    0xaabf     |         0     | (chksum, urgp)
+-------------------------------+
tcp_input: packed for LISTENing connection.
TCP connection request 52409 -> 3810.
tcpip: ip_reass_tmr()
tcpip: ip_reass_tmr()
tcpip: etharp_tmr()
tcpip: ip_reass_tmr()
tcpip: ip_reass_tmr()
tcpip: ip_reass_tmr()



Sorry for the long post,

  Micael