RE : [lwip-users] netconn_write blocking

[Frédéric BERNON]

I think that increase TCP_SND_QUEUELEN is not the solution in this case, since 
the problem is that application have to run when one of the peer is unplugged. 
So, a bigger TCP_SND_QUEUELEN should just increase the time between the 
blocking (if I don't do any mistake).
 
>> #define TCP_MAXRTX              4
>> #define TCP_SYNMAXRTX           4
>> #define TCP_TMR_INTERVAL        100
>> 
>> I was able to cut the amount of time for a timeout down to
>> like ~20 seconds.  This isn't great, as all of the other 
>> connections will be without data for that ~20 seconds, and   
>
>And that's the bad news! A better solution (which is not yet
>implemented)
>is to use send timeouts (like we have receive timeouts already). But as I 
>said, it remains a todo for lwIP...

You could continue to reduce TCP_MAXRTX and TCP_TMR_INTERVAL if you want to 
reduce the timeout (but, be careful, reduce TCP_TMR_INTERVAL will increase your 
cpu time used by the stack, and a too low TCP_MAXRTX could "drop" sometimes 
your tcp connections - 2 should be the lower value I think). This is already a 
"send timeout" (for packets already enqueued). Another one is the keepalive 
packets, but there are not more efficient in this case (default values drop a 
"dead" connection after several hours if I remember, but, it's mainly to drop a 
tcp connections when no data are exchanged all the times, a telnet server by 
example).

In all cases, a send timeout can't be the solution if you don't want to block 
(even some seconds). I think the solution is more in the design of your 
application: 

- you to use the raw api where there is no blocking functions (if the send 
buffer is full, you will got a error, so, you can handle the case). It's more 
difficult to use than the sequential api, but, you could handle all cases.

Or

- you can use one thread per socket and a "unblocking" queue between the "main 
sender" and "socket senders": like this, you use the standard lwIP code, and if 
a "peer" is unplugged, only the matching "socket sender" will be blocked.

Or

- you can check the send_queue available in a connection before calling 
netconn_write : "available = tcp_sndbuf((( struct netconn *) 
i->data)->pcb.tcp);" it's not very thread-safe (since you access to the 
pcb.tcp, which can be delete when err_tcp is called) and you have to implement 
your own "retry". If "available" is too long time too low, you can close the 
connection.

Or

- you can patch api_lib.c and api_msg.c to avoid a blocking when tcp_write 
failed to enqueue datas, and "return" only how many datas is really sent: it's 
more difficult, but safe-thread, and you still have to implement your own 
"retry" at application level.

I hope it give you some ideas...


====================================
Frédéric BERNON 
HYMATOM SA 
Chef de projet informatique 
Microsoft Certified Professional 
Tél. : +33 (0)4-67-87-61-10 
Fax. : +33 (0)4-67-70-85-44 
Email : address@hidden 
Web Site : http://www.hymatom.fr 
====================================
P Avant d'imprimer, penser à l'environnement
 


-----Message d'origine-----
De : address@hidden [mailto:address@hidden De la part de Goldschmidt Simon
Envoyé : mercredi 10 octobre 2007 08:21
À : Mailing list for lwIP users
Objet : RE: [lwip-users] netconn_write blocking




> [..]
> I realize that this probably isn't the best way to write to
> sockets, as if anything blocks, it stops sending data to all 
> of the sockets.

You can get problems this way, of course, but nevertheless, it should work... 
Until a client stops responding.

> This is what I'm pretty sure is currently
> happening with netconn_write
> 
> Ok, next, this is what I get for my debug output when i have
> the following in my lwipopts.h ...
> #define DBG_MIN_LEVEL                     DBG_LEVEL_SERIOUS
> #define DBG_TYPES_ON    ( API_LIB_DEBUG |\
>                                             API_MSG_DEBUG | \
>                                             )
> #define API_LIB_DEBUG                   LWIP_DBG_ON
> #define API_MSG_DEBUG                   LWIP_DBG_ON
> 
> Debug Output:
> ( I added the LWIP:\t to each line)
> ...
> LWIP:     tcp_write(pcb=0x207e78, data=0x20574c, len=18, copy=1)
> LWIP:     tcp_enqueue(pcb=0x207e78, arg=0x20574c, len=18, flags=0, co
> LWIP:     tcp_enqueue: queuelen: 32
> LWIP:     tcp_enqueue: too long queue 32 (max 32)

Oh, THAT queue! That's something different, of course! :-) There are two 
defines in opt.h to limit the amount of data being enqueued in one
tcp_pcb:

- TCP_SND_BUF: sender buffer space in bytes
- TCP_SND_QUEUELEN: number of pbufs allowed in the sender buffer

In your case, TCP_SND_QUEUELEN seems to be defined to 32, and 32 pbufs have 
already been enqueued. If you do all your writes using 18 bytes, you will have 
(18 bytes * 32 pbufs) 576 bytes in the queue, maybe that's a problem for the 
nagle algorithm or something else?

In any case, you should try to increase TCP_SND_QUEUELEN and see what happens!


>  Will netconn_write block until it can write the next segment
> in the queue? 

Yes.

> 
> Ok, now the good news....
> By setting the following:
> 
> #define TCP_MAXRTX              4
> #define TCP_SYNMAXRTX           4
> #define TCP_TMR_INTERVAL        100
> 
> I was able to cut the amount of time for a timeout down to
> like ~20 seconds.  This isn't great, as all of the other 
> connections will be without data for that ~20 seconds, and 

And that's the bad news! A better solution (which is not yet
implemented)
is to use send timeouts (like we have receive timeouts already). But as I said, 
it remains a todo for lwIP...

> also, it seems like I'm setting the number of times the
> packet can be retransmitted pretty low.  But once the 
> connection timed out, netconn_write returns, and the netconn 
> is deleted, and everything continues to operate normally. 
> 
> This is at best a flimsy fix, so if anyone has better ideas,
> please feel free to send them my way. 

Yes: increase TCP_SND_QUEUELEN and if it still locks up, use wireshark to see 
what's happening and post the packet trace here if you have questions about it!

Simon


_______________________________________________
lwip-users mailing list
address@hidden http://lists.nongnu.org/mailman/listinfo/lwip-users

Frédéric BERNON.vcf
Description: Frédéric BERNON.vcf