Re: RE : [lwip-devel] brain storming about "socket2"

[Piero 74]

Hi all
My few cents for this interesting discussion :O) :

> I used the zero copy API's because I liked the idea.
> It worked well. But over the years I have grown tired of not being
>
> able to reuse code, and every time re-learn how a particular zero copy
> API works. So now, I instead always try to remove the non-socket API's
> just because of this from old designs. I think that I increase the
>
>
>
>
>
>
> quality of the code this way, since it is less risk that I have
> misunderstood the API and so on.
> I guess, If you really need to squeeze the maximum bitrate out of a
> specific hardware, zc API's is the way to go. Things like routers I
>
>
>
>
>
>
> guess.
> But for most app's out there, this is not as important, at leas not in
> my area of business. 
> Yes, socket API is old, and not the most efficient. But it is quite
> good, and complete, and most of all it is available on many platforms.

In my opinion socket api are very useful: i like the power of select to manage synchronized IO, and i like to have (specially using tcp) to have a separation between application and tcpip stack.
(socket function send and recv, do a copy, so puts a "red line" between application code and stack code, consuming ram and loosing efficiency)
I know that socket reduces performance, but i think we should have a trade-off between performance and the ability to offer an easy way to write application code using the most powerful api like socket. We should have a "full" tcpip stack in each our application, which start from a very efficient emac driver, integrated in better way in lwip stack (see the discussion about zero-copy driver in lwip-dev mailing list), to socket interface, changing how socket are build above other layers (i.e. without netconn)
 

> netconn API because of this. But
> looking at the wiki, it seams there's no select() support for
>
>
> instance? How would you be waiting for many simultaneous connections?
> Because of differences like this, it does not *seam* to be a easy
>
>
>
> replacement for the socket API, at least not in a more complicated
> environment.

i think that netconn api could be useful for particular applications, which needs high performance. Socket are more general purpose, there is a lot of documentation about them, a lot of code based on them, offers supports for synchronization of many simultaneous connections. Netconn are oriented to "event on network", to have low response latency using callback
 

> I don't think Piero has been reporting flaws? Rather, wanting enhancements.

enhancements... absolutely!

Probably my misunderstanding, then. I was reading about the memmp issues
needing a stack restart. However, that is a different thread, and probably
no relevant to this discussion!

yes... is related to a strange problems i sw stressing my code... no relevant for this discussion

 

> Did you benchmark the two stacks?  Perhaps a definitive performance benchmark is the way to make this choice.
>
>
>
>
>
>
>
> he port is difficult because the hardware API is married to the Interniche stack (the HAL APIs use Interniche data structures) and Interniche is married to uC-OS/2 (delete uC-OS/2 in the IDE and it deletes Interniche too).

Before starting with lwip, i saw interniche stack (the light version, available from microcontroller vendor.
I saw a very difficult to debug code, no documentation and support are available until you decide to pay for commercial support, and it doesn't have real bsd socket as interface: in lwip you have a very well done code, a community of developers/users behind, and you CAN CHOICE between performance (netconn api or raw api) and BSD SOCKET
 

> So, as far as I can tell, we have two roads from here, where some of us
>
>
>
> prefers better socket compliance to POSIX, and others prefer higher
> efficiency, by accepting deviations from the standard.

in my opinion we can accept deviations from the standard posix, but some features must be compliance.
 

> I can tell I use it in my projects, and the code is stable. 1.3.0 add lots of functions to Socket API, but about performance, if the standard BSD socket API design cause lot of memcpy, another important problem is the "sequential api" (on which is based "netconn"): the fact that any operations cause a "task switching" to the "tcpip_thread" doesn't help to get high performance. That what I got when I did my tests, and when we introduced the "socket2.c" in "task #6994 : Redesign Socket Layer with LWIP_TCPIP_CORE_LOCKING".

i tried to study lwip code...
first, during my application performance analysis, i checked that some time is spent in recv function. (note: i'm using freertos as RTOS with lwip 130)
My understanding:
for each call to recv, a call to netconn_recv is performed.
this function calls:

[ ------ in context of application thread --------]
- sys_arch_mbox_fetch(conn->recvmbox, (void *)&p, 0);                              ----> a call to rtos kernel - wait msg on queue (if a checked socket with 'select()' before, a msg is already present)
- TCPIP_APIMSG(&msg);                                                                        ----> send a msg to tcpip task, calling:
                  - sys_mbox_post(mbox, &msg);                                             ----> a call to rtos kernel - send msq to queue (send requested operation to tcpip task, for safe execution)
                  - sys_arch_sem_wait(apimsg->msg.conn->op_completed, 0);   ----> a call to rtos kernel - wait on semaphore (wait that tcpip task executes operation)
                  [ --------- RTOS context switch -------- ]                                      ----> rtos kernel context switch - code execution continues in tcpip thread
[ ------ in context of tcpip thread --------]            
- sys_mbox_fetch(mbox, (void *)&msg);                                                     ----> a call to rtos kernel - wait msg on queue, requested operation is received in queue
- exec requested function                                                                         ----> CORE OF REQUESTED OPERATION (i.e. recv). At the end calls:
                  - TCPIP_APIMSG_ACK(msg);                                                ----> send msg to application task, calling:
                          - sys_sem_signal(m->conn->op_completed)                    ----> a call to rtos kernel - signal a semaphore (advise application thread: operation executed)
                  [ --------- RTOS context switch -------- ]                                      ----> rtos kernel context switch - code execution continues in application thread
[ ------ in context of application thread --------]

So, for recv operation, will be necessary:
- 5 calls to RTOS kernel
- 2 RTOS context switch


I saw that some piece of code in LWIP use LWIP_TCPIP_CORE_LOCKING, to protect access to lwip core, but only for send operation and INSIDE netconn api.
I suppose that the idea for socket2, build above raw api, could be to write socket code implementation which calls this low_level functions locking/unlocking core, using a general semaphore, without netconn api. In thsi case, a recv function should use:
- 2 calls to RTOS kernel (the first try to get semaphore before low_level call, the second release semaphore)

If i'm not in wrong, we should have better performance for socket api, and the tcpip thread should have less work to do.
Is it this the idea for new socket2 api?

> Of course, adding some extra-sockets API to avoid some memcpy can help. We have talk about that in task#6994. I changed is as "Private" to avoid to have lot of questions about it, since at this time, only Simon and me worked on it.

mmmm. socket without copy? it's not posix compliance, but could be very useful!!
 

> But I don't have lot of time to continue this task, so, if Simon is agree, we can change it as "Public". But remember it's just a first "draft" with lot of work to do. I'm not even sure it's always directly compatible with current CVS.

Can we explain how we can cooperate for this? starting with discussion and AFTER trying to change lwip code?



Bye
Piero