Re: [PATCH] tcp: ack when we get an OOO/lost packet

[Andrei Borzenkov]

On Thu, Aug 13, 2015 at 4:59 PM, Josef Bacik <address@hidden> wrote:
> On 08/13/2015 04:19 AM, Andrei Borzenkov wrote:
>>
>> On Wed, Aug 12, 2015 at 6:16 PM, Josef Bacik <address@hidden> wrote:
>>>
>>> While adding tcp window scaling support I was finding that I'd get some
>>> packet
>>> loss or reordering when transferring from large distances and grub would
>>> just
>>> timeout.  This is because we weren't ack'ing when we got our OOO packet,
>>> so the
>>> sender didn't know it needed to retransmit anything, so eventually it
>>> would fill
>>> the window and stop transmitting, and we'd time out.  Fix this by ACK'ing
>>> when
>>> we don't find our next sequence numbered packet.  With this fix I no
>>> longer time
>>> out.  Thanks,
>>
>>
>> I have a feeling that your description is misleading. Patch simply
>> sends duplicated ACK, but partner does not know what has been received
>> and what has not, so it must wait for ACK timeout anyway before
>> retransmitting. What this patch may fix would be lost ACK packet
>> *from* GRUB, by increasing rate of ACK packets it sends. Do you have
>> packet trace for timeout case, ideally from both sides simultaneously?
>>
>
> The way linux works is that if you get <configurable amount> of DUP ack's it
> triggers a retransmit.  I only have traces from the server since tcpdump
> doesn't work in grub (or if it does I don't know how to do it).  The server
> is definitely getting all of the ACK's,

In packet trace you sent me there was almost certain ACK loss for the
segment 20801001- 20805881 (frame 19244). Note that here recovery was
rather fast - server started retransmission after ~0.5sec. It is
unlikely lost packet from server - next ACK from GRUB received by
server was for 20803441, which means it actually got at least initial
half of this segment. Unfortunately some packets are missing in
capture (even packets *from* server), which makes it harder to
interpret. After this server went down to 512 segment size and
everything went more or less well, until frame 19949. Here the server
behavior is rather interesting. It starts retransmission with initial
timeout ~6sec, even though it received quite a lot of DUP ACKs; and
doubling it every time until it hits GRUB timeout (~34 seconds).

Note the difference in behavior between the former and the latter. Did
you try to ask on Linux networking list why they are so different?

OTOH GRUB probably times out too early. Initial TCP RFC suggests 5
minutes general timeout and RFC1122 - at least 100 seconds. It would
be interesting to increase connection timeout to see if it recovers.
You could try bumping GRUB_NET_TRIES to 82  which result in timeout
slightly over 101 sec.

Also it seems that huge window may aggravate the issue. According to
trace, 10K is enough to fill pipe and you set it to 1M. It would be
interesting to see the same with default windows size.


>                                                     and from my printf()'ing 
> in grub we
> are either getting re-ordered packets (the most likely) or we are simply
> losing a packet here or there.  This is a pretty long distance and we have a
> lot of networking between Sweden and California so reordering or packet loss
> isn't out of the question.
>
> Regardless we definitely need to be ACK'ing packets that come in with the
> last seq we had as the spec says so the sender knows the last bit we had,
> otherwise we see timeouts once the window is full.
>
>> Did you consider implementing receive side SACK BTW? You have the
>> right environment to test it :)
>>
>
> So I found this bug while implementing SACK, and decided it was faster to
> just do this rather than add SACK.  This method is still exceedingly slow, I
> only get around 800kb/s over the entire transfer whereas I can sustain
> around 5.5 mb/s before we start losing stuff, so I'm definitely going to go
> back and try the timestamp echo stuff since the timeout stuff takes like 6
> seconds, and then if that doesn't work bite the bullet and add SACK.
>
> But first I want to get my ipv6 patches right ;).  Thanks,
>
> Josef
>