Re: [lwip-users] More dropping packets

[Kieran Mansley]

On Sat, 2009-08-15 at 08:07 -0700, JM wrote:
> I'm still trying to fix this dropped packet issue.  When I have
> debugging on, it seems the only reason this is happening is bad TCP
> checksums.  Otherwise, I'm not seeing any problems.  The other issue
> with this is the delays caused by writing to my debug output device
> when enabled seems to somewhat alleviate the dropped packet issue.  I
> have used the debugger to break when it finds a bad TCP checksum, and
> indeed, the pbuf->payload contains data that I have no idea where it
> came from.  Right now, I'm simply throwing away all data I recieve
> just for testing.
> 
> So, now for a Wireshark trace that I am baffled by.  First in the
> trace is a connection to a Shoutcast streaming radio server.
> 192.168.0.169 is the lwIP board.  It starts to choke by the 11th
> packet.  Then, starting at packet 40 is a connection to a simple
> Windows app I wrote that waits for a connection from my lwIP board,
> then starts sending packets to lwIP.  My computer is 192.168.0.112.
> The data simply cycles from 0x0 to 0xFF.  This closely simulates the
> behavior of the streaming audio I believe, the main difference being
> that this is coming across a LAN; the streaming audio is coming over
> the Internet.  This stream is error-free with *no* dropped packets
> that I can see, yet is moving data *much* faster!
> 

There's nothing obvious in the packet capture to give clues as to what
is going wrong.  This smells to me like either a driver interrupt issue
or a scheduling problem of some sort.  The fact that when you keep the
board busier by sending more traffic or outputting debugging data you
get no problems makes me think that.  

My approach to this would be to avoid debug print statements as they are
making the problem go away.  Instead use counters to keep track of
different paths and problems, and periodically (or even in response to
some external trigger) output these counters via debug print statements
(or even just send a packet onto the network with these counters as the
payload and capture it with wireshark).  That way you can get some data
about the problem and then dump it later to view.  For example, look
through the stack for all the places where a packet can be dropped and
keep a count for each one if it drops the packet.  Or keep a count for
each function that is called on a packet's way up the stack (e.g.
ip_input called X times, tcp_receive called Y times, etc) and that might
help find where there is some discrepancy.  If these counters don't
reveal anything, then a total packets received counter compared to a
wireshark trace taken at the same time would help work out if the stack
even gets to see the missing frame or if it's dropped by the driver.

Finally, I know you think you have plenty of buffers, but you can check
lwIP agrees with you by looking at the LWIP stats after the problem has
occurred (again, without turning on all the debug statements as that
prevents the problem going away).  Just output and reset them once a
minute or something.

Hope that helps,

Kieran