Re: lynx-dev wget and lynx, download problem

[Klaus Weide]

On Sun, 7 Mar 1999, Klaus Peter Wegge wrote:
[ much snipped ]
> 
> The number of bytes, downloaded by lynx, seems to depend on the
> current speed or usage of the connection.
> 
> About networking: My LINUX PC is connected to the local ethernet of
> our laboratory, which has a gateway to DFN (Deutsches Forschungsnetz).

Not just that - you are using a proxy:

> ================= Lynx.trace =========================
> 
>               Lynx Trace Log (2.8.2dev.17)
.....
> proxy server found: http://wwwproxy.uni-paderborn.de:8080/
> HTParse: 
> aName:http://wwwproxy.uni-paderborn.de:8080/http://www.flora.org/lynx-dev/mailbox/log99-02
>    relatedName:http:
> HTParse: result:http
> HTParse: 
> aName:http://wwwproxy.uni-paderborn.de:8080/http://www.flora.org/lynx-dev/mailbox/log99-02
>    relatedName:
> HTParse: result:wwwproxy.uni-paderborn.de:8080
> Looking up wwwproxy.uni-paderborn.de:8080.
> HTParseInet: parsing `wwwproxy.uni-paderborn.de:8080'.
> LYGetHostByName: parsing `wwwproxy.uni-paderborn.de'.
> End of LYGetHostByName: 0x400e998c { h_name = 0x400e9a7a 
> "stargate.uni-paderborn.de",
>        h_aliases = 0x400e99a0 {  0x400e9a46 "wwwproxy.uni-paderborn.de", 0x0 
> },
>        h_addrtype = 2, h_length = 4,
>        h_addr_list = 0x400e98fc { 0x400e9a98 "131.234.22.29", 0x0 } }
> LYGetHostByName: Resolved name to a hostent.
> HTParseInet: Parsed address as port 8080, IP address 131.234.22.29
> Making HTTP connection to wwwproxy.uni-paderborn.de:8080.
.....
> Read 1190 of 3164 KB of data, 4 KB/sec.
> Read 1191 of 3164 KB of data, 4 KB/sec.
> Data transfer complete
> LYCloseTempFP
> HTAccess:  status=200
> HTAccess: `http://www.flora.org/lynx-dev/mailbox/log99-02' has been accessed.
.....
As far as lynx is concerned, receiving data has finished normally.


> 
> ================= Lynx.strace =========================
> .....
> write(2, "Read 812 of 3164 KB of data, 2 K"..., 39) = 39
> oldselect(1024, [4], NULL, NULL, {0, 100000}) = 1 (in [4], left {0, 100000})
> read(4, "nx Developers <address@hidden>"..., 4096) = 1460

Last data received: a full packet of 1460 bytes (at least it's extremely
likely that this is not the end of the file, given the 1460).

.....
> write(2, "Read 814 of 3164 KB of data, 2 K"..., 39) = 39
> oldselect(1024, [4], NULL, NULL, {0, 100000}) = 0 (Timeout)
> oldselect(1024, [4], NULL, NULL, {0, 100000}) = 0 (Timeout)
> oldselect(1024, [4], NULL, NULL, {0, 100000}) = 0 (Timeout)
.....
A trace message, then starting to wait for more data, each
select waits for 100 ms.  There are 80 of those, which means
approx. 8 seconds.  Then:

> oldselect(1024, [4], NULL, NULL, {0, 100000}) = 1 (in [4], left {0, 40000})
> read(4, "", 4096)                       = 0
> write(2, "Data transfer complete\n", 23) = 23

That looks exactly like a normal close of the connection with FIN.
As far as lynx is concerned, it has received an end-of-file
indication.  (lynx doesn't compare the total bytes received to
the content-length from the header - that could be used to sometimes
detect an incomplete transmission, but it isn't always provided anyway
and might sometimes even be wrong.)

Since you are talking to a proxy, it must be the proxy who is responsible
for cutting off the transmission.

Connecting to http://wwwproxy.uni-paderborn.de:8080/ reveals that we are
dealing with an old version:
        <ADDRESS>
        Generated by squid/address@hidden
        </ADDRESS></BODY></HTML>

My only explanation of why you don't have the problem with wget is
that you didn't use the proxy with wget.

Possible solutions:

  - Use a different proxy (maybe a newer squid version is all it takes)
  - See to it that the proxy gets replaced / fixed / configured differently
  - Don't use a proxy

  Klaus