Re: lynx-dev RFC959 non-compliance in Lynx hangs the client

[Klaus Weide]

On Thu, 9 Sep 1999, Gregory A Lundberg wrote:

> We could simply do a close(), but that would not tell us if the client-side
> has seen it before we send the 2yz reply.
> 
> We could try to use SO_LINGER, but that will only tell us the packets have
> been sent when we do a close(); not that the client application has acted
> upon them.
>
> The intention is not to ensure the client's host has the traffic.  It is to
> ensure the CLIENT itself sees the remaining data BEFORE it sees the 2yz.

Even with your approach (below), you cannot ensure that the client
application has acted on your close() in any specific way.  The only
thing you know is that it has called close() (or shutdown()) too, you
*still* don't know that that means it won't act on the control
response prematurely.  For example the client "application" may have
delegated the socket handling to some lower-level subsystem which does
the closing without the application's knowledge.  Or the control and
data connections might be handled by different threads that aren't
properly synchronized.

You assume clients that "take receipt of the 2xx message literally"
exist (or could exist), but your attempt to help them assumes in
addition that their hypothetical brokenness is of a certain kind.
Of all hypothetical poorly written clients wou want to help, only a
subset would actually benefit.

> Why?  To support poorly written clients.  Do we know of any?  No.  We got
> the suggestion from an RFC author as part of his research into a
> clarification of this part of the protocol.

I would argue that such broken clients are extremely unlikely to exist
or be created; or if they exist, they won't last longer than a few days.
To qualify, a client (or its authors) would have to
 - ignore the clear specification of what constitutes an end-of-file
   indication (for stream mode, struct file).
 - watch the data and control connection at the same time; i.e.
   typically either a threaded or select-driven program.
 - violate common sense and common rules.  Authors of threaded
   applications are supposed to know about synchronization, locking,
   that events can occur in arbitrary order.  Authors of programs
   based on select() loops wouldn't even include a socket in a fdset
   for reading when the protocol state doesn't call for reading.
 - make invalid assumptions about behavior of the network.
   Such assumptions aren't likely to survive for long on the Internet,
   if anyone cares for integrity of the data transmitted.

Contrast that with the other fault discussed, found in lynx, libwww, squid:
 - make an assumption about server behavior, i.e. that the server won't
   sequence events as ftp.wu-ftpd.org now does.  An assumption that so
   far always has been valid when talking to existing servers (with a
   couple of specific rare server types which were handled specially,
   because the problem wasn't really properly understood, in the case of
   lynx).  Also nothing in the FTP specs (as far as I have seens) gives
   a hint that the "new" server behavior should be anticipated by clients,
   or that servers should act that way.
 - disregard RFC 959 wrt. "promptly closing", not stated as a clear
   requirement.

To summarize, the change in WU-FTPD
 - isn't required
 - breaks existing clients that make an understandable mistake
 - potentially helps some unknown clients, which make very hypothetical
   and unlikely mistakes, and which deserve to be broken at the earliest
   opportunity.

> That leaves us with shutdown(), which closes the socket on our end,
> followed by a read() to await an EOF (shutdown()/close()) from the client.
> This is what the wu-ftpd daemon is now doing.  

Doing a read() doesn't seem to be a clean thing, on principle.  A connection
is bidirectional after it has been established, with both directions
independent.  For the date connection used for RETR LIST etc., there is
otherwise no reason to read() at all.  In fact it should be valid for
the client to shutdown() its sending side immediately after establishment
since it's never going to write anything, and this shouldn't make a
difference to the server.  (I ought to test this...)

Actually RFC 959 has this cryptic sentence in 2.3:
                                            It ought to also be noted
      that the data connection may be used for simultaneous sending and
      receiving.

Now I have no idea how one would set it up for such a "simultaneous" use,
is there any actual application of this?  Anyway, imposing special
semantics on the read() as you now do (i.e. it's a confirmation of some
sort relevant for the *other* direction) would seem to preclude such use.

> In fact, if the user were
> not so impatient, things would eventually have worked.  The daemon has a
> timeout on the read() and would have proceeded eventually.  Unfortunately,
> that timeout is long (to allow for slow network connections .. 20 minutes
> by default), and the user impatient, so the session was terminated by
> resetting the client-side.

I assume (it's not clear) that what we're talking about is the same
thing that's referred to in
<ftp://ftp.wu-ftpd.org/private/prerelease/CHANGES> with

 o  An earlier change to the handling of the data connection breaks some
    clients.  The correct thing to do is get a client which is not broken.
    That may not be possible, so added a complie-time option which should
    allow broken clients to continue to work until they can be fixed and
    upgraded.  Note: this option is disabled by default; broken clients
    won't work.

You should revert to the old behavior as default, for reasons I tried
to give above.

If you insist on keeping the default as in the beta, at least make
the timeout 5 seconds instead of 20 minutes.

   Klaus