Re: [lwip-users] TCP Raw API questions about efficiency and threads

[Noam Weissman]

Hi,

What are you sending there ?

I think you should process all the data as it comes and not collect 3-4k and then process.

For example I have a TCP server (like Telnet but RAW TCP) I get a pBuf and process it. I copy

byte by byte to a temporary buffer until I find CR, line terminator. The data is then copied again

into a queue for later processing by the parser task.

RX function continues the copy of one by one until it finds another CR. If it does not it will

exit the RX call-back and continue on next pBuf...

I have no problems there. I can run it all day.

Sure you can load your code here. You can also send it privately as an attachment.

If you prefer to send it directly to me use:  address@hidden

BR,

Noam.

---

From: lwip-users <lwip-users-bounces+address@hidden> on behalf of Pîrvu Mihai <address@hidden>
Sent: Tuesday, June 7, 2016 8:26 PM
To: Mailing list for lwIP users
Subject: Re: [lwip-users] TCP Raw API questions about efficiency and threads

 

What you are describing is exactly what I did in my first implementation. Handle all the data in the pbuf in recv function and then call tcp_recved(pBuf->tot_len).  By handle I mean copy the data, cycle through all the possible packets, and send the data to the peers it is intended to.

> I meant that if you get a request (like in HTTP) and you need to send lots of data you start the send process it in the

recv and continue in the poll call-back, if needed.

This might be the part that I don't understand. When do I decide it's okay to stop the data transmission in recv callback, and save the rest of the packets for poll callback ? As far as I understand, if I stay too long in recv callback (tcp thread), I block incoming packets and if I stay too long in poll callback (main thread) I block the poll for other active pcbs.

I don't see how either way I can achieve good performance. I have a similar netconn code on UDP and my ping is 4-5ms between clients. It's just netconn has some transparency to what happens behind the scenes (raw udp) and I'd rather not dig (yet) and ask here.

I am aware of the contrib directory (only http-raw and smtp client are relevant tho). The more I read, the more I see that other programs don't copy the pbuf chains into a big continous char buffer (I memcpy all of them until I get to BUFSIZE or pbuf->tot_len) and then process the data inside this buffer. The problem is that I can receive multiple client packets in different pbufs (I think). For example Client A send 3 ICMP requests to Client B and I receive them as 111 -> 112222 -> 23333 -> 333. (4 nodes in pbuf list containing 3 different client packets, am I wrong on this?). I have a little delimitator at the beginning of clients' packets (just the size of the packet, actually), so what I do, is that I loop my contigous array, from packetSize to packetSize (based on those first 2 bytes at beginning of any packet), process it (see destination based on data link header) and send the packetSize bytes. Then go further in the buffer of packets (with an offset of packetSize) and do the same thing.

Is it okay if I upload some parts of the code (~130 lines) and analyze on it ? 

Mihai

On Tue, Jun 7, 2016 at 6:09 PM, Noam Weissman <address@hidden> wrote:

Dear,

 

I am sorry if I was not clear but you should not do what you did. No need to copy the data and process it

In the poll function.

 

I meant that if you get a request (like in HTTP) and you need to send lots of data you start the send process it in the

recv and continue in the poll call-back, if needed.

 

You should cycle all the data in the chain in one go (inside recv). Once you handled all the data you inform

the TCP stack that you have finished processing and exit.

 

Something like this (TCP):

 

  // take the first pBuf address and process the chain

  pBufChain = pBuf;

 

  do

  {    

 

      // process data…..

 

  } while((pBufChain = pBufChain->next) != NULL);

 

 

If data can be spread over several buffers you may need copy and concatenate data inside the loop.

 

Do remember that you have pBufChain->len and   pBuf->tot_len… the first is the chunk length and the

second is the total length of all the chained buffers, if any.

 

Once you are done processing the data you inform the TCP stack you are done:

 

// we need to free the pBuf

  if(pBuf != NULL)

  {

    // Inform TCP that we have taken the data.

    tcp_recved(pcb, pBuf->tot_len);       

    

    // free the buffer !

    pbuf_free(pBuf);

    pBuf = NULL;

  } 

 

 

If it is UDP you are not calling  tcp_recved  but data handling and freeing is similar.

 

Here are a few link that can give you more information and examples:

 

https://github.com/lsgunth

https://github.com/pabigot/lwip-contrib/tree/master/apps

 

 

Also take your time and read the rawapi.txt under the LwIP doc directory

 

 

Hope the above helped.

 

BR,

Noam.

 

 

 

 

 

 

From: lwip-users [mailto:lwip-users-bounces+noam=address@hidden] On Behalf Of P?rvu Mihai
Sent: Tuesday, June 07, 2016 3:44 PM
To: Mailing list for lwIP users
Subject: Re: [lwip-users] TCP Raw API questions about efficiency and threads

 

Alright, I've modified my code a little. What I've done is pretty much this:

 

- from recv callback, I just call tcp_recved on pbuf->tot_len and return ERR_OK

- also in recv callback, i save the pbuf in my client arg and process it in poll

- in accept callback, i set up the arg to my internal struct, and a poll function with argument 1 (as far as I understand that means polling every 500ms, can i change this? seems too slow for what I try to achieve)

- In poll callback, I process the packet(s) because there can be multiple packets arriving in my pbuf (packets that are concatenated together). I read all the data (pbuf->tot_len) in a separate buffer then loop the buffer for each individual inner packet. For each packet, I check the destination, if it's broadcast, I send it to all my active clients (note: that all connections are active until they disconnect from client-side, that means I don't have any keep-alive verification, if I get in recv callback with pbuf == NULL => client disconnected, otherwise I consider it active right now). If it's a unicast message, I send it just to the respective peer. I do my checking by MAC address (clients will send their MAC as the first message when they connect to server). I call tcp_write in order to send my messages (I saw something about tcp_output when I use tcp_write from outside recv callback, I'm not using that now).

 

After I've done this (before I processed the packet in recv callback, not poll callback), my ping between clients is even worse (700ms~) and only one client receives ICMP replies (so no concurrency).

 

I feel like I'm doing lots of stuff wrong, like copying the data from pbuf to my static buffer, then calling tcp_write with flag TCP_WRITE_FLAG_COPY seems like overkill when copying data.

 

Regarding of you saying stuff about examples online, I can't find any more "complex" example out there. All the examples have a really simple design, client sends message, receives message back and then the connection is closed. Is there any examples out there where connections are persistent and messages can come at any given time ?

 

Mihai

 

On Fri, Jun 3, 2016 at 10:21 AM, Noam Weissman <address@hidden> wrote:

 

In order to use RAW API efficiently you can use the poll call back to send more data. What do I 

mean by that ?

Lets assume you got some request for data and you are handling it inside the recv function.

If you send back all the data you are busy inside that recv instant and blocking other parallel

connections. One way to avoid this is to start sending (a portion) and get out of the recv. The rest of the send will be handled inside the poll function. the poll function is called periodically as long as the

current connection is a live... house keeping.

 

Once you no longer need that connection can gracefully close it from inside the poll function.

 

Another way to use the poll call back is to put some data in a queue (from main thread) while from

inside the poll call back (TCP own thread) read that data and send it to the open connection.

 

BR,

Noam.

 

 

 

 

 

 

---

From: lwip-users <lwip-users-bounces+noam=address@hidden> on behalf of Pîrvu Mihai <address@hidden>
Sent: Friday, June 3, 2016 12:15 AM
To: Mailing list for lwIP users
Subject: [lwip-users] TCP Raw API questions about efficiency and threads

 

Hello guys, I've just ported my application to tcp raw API and I was wondering a few things. 

 

I've noticed that the performance is a bit too bad (300-500 ms for pings, between local clients) and I was wondering if I'm doing some things wrong.

 

Firstly,  I'm doing all my packet processing + sending of replies in tcp_recv callback. Is this normal, or shall I save the data somehow and process it later. My idea is that this blocks the main thread and the next packets are delayed. But I don't know this for sure.

 

I don't understand how I can use multiple threads using this API. It's callback based, so where do I create the threads? Let's say I want to process the entire callback saved using tcp_recv on a separate thread. Is that possible or do I have to create a new thread inside the callback?

 

The application I'm using is a vpn server that has to inspect the packets (so I do that in tcp_recv), see where to forward one packet (from a list of connected clients to the server) and send that data to the client. I saw that each tcp_pcb has a unique memory address (obv.), so I save that in an array and if the client IP/MAC is the same as on in the array, I send my data to that pcb from the array. I can also send broadcasts (if mac is FF:FF:FF:FF:FF:FF). This is the "processing" that is done. 

 

One bad thing is that I copy the entire pbuf into a local buffer and free it (I saw that there's some idea to save it for later use, would love to see some examples on that).

 

_______________________________________________
lwip-users mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/lwip-users

 

_______________________________________________
lwip-users mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/lwip-users