bug-wget
[Top][All Lists]
Advanced
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Problems downloading files on Windows with wget2 2.0.0

From: Tim Rühsen
Subject: Re: Problems downloading files on Windows with wget2 2.0.0
Date: Sun, 3 Oct 2021 11:21:58 +0200
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.13.0 
Hi Jernej,
while I try to reproduce here (have to build all the deps first), could you possibly do some more tests to narrow down the issue !? 

a) test with --max-threads=1 --no-http2
b) test with --max-threads=1 using a plain text (http://) URL
I think the message "failed to receive: 0" (the first one) should indicate a timeout - which means that the internally used select/poll behaves differently on Windows / MinGW. Hm, that possibly means that the file descriptors need some translation (or that it is wrong what we currently do)... 

Anyway, the above mentioned tests would help to narrow this down.

Regards, Tim

On 03.10.21 01:28, Jernej Simončič wrote:

I successfully built wget2 with mingw, both 32 and 64-bit version, but the resulting 
binary has problems downloading files larger than approximately 8 MB – it keeps showing 
"Failed to read 102400 bytes (24)", and often creates several truncated files 
with .1, .1.1 etc. extensions. Any ideas what's causing the problems?

I used the following libraries when compiling wget2:
brotli-1.0.9
bzip2-1.0.8
gpgme-1.16.0
libassuan-2.5.5
libgpg-error-1.42
libhsts-0.1.0
libidn2-2.3.2
libpsl-0.21.1
libunistring-0.9.8
lzlib-1.12
nghttp2-1.45.1
openssl-1.1.1l
pcre2-10.38
xz-5.2.5
zlib-1.2.11
zstd-1.5.0

D:\Stuff\wget\2.0.0\64>wget2 --version
GNU Wget2 2.0.0 - multithreaded metalink/file/website downloader

+digest +https +ssl/openssl +ipv6 +iri +large-file -nls -ntlm -opie +psl +hsts
+iconv +idn2 +zlib +lzma +brotlidec +zstd +bzip2 +lzip +http2 +gpgme

Copyright (C) 2012-2015 Tim Ruehsen
Copyright (C) 2015-2021 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later
<http://www.gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Please send bug reports and questions to <bug-wget@gnu.org>.

D:\Stuff\wget\2.0.0\64>wget2 -d https://eternallybored.org/misc/bigtest
03.010323.433 Local URI encoding = 'CP1250'
03.010323.439 Input URI encoding = 'CP1250'
03.010323.440 Fetched HSTS data from 
'W:\Users\ender\AppData\Local/wget/.wget-hsts'
03.010323.442 Fetched HPKP data from 
'W:\Users\ender\AppData\Local/wget/.wget-hpkp'
03.010323.443 Fetched OCSP hosts from 
'W:\Users\ender\AppData\Local/wget/.wget-ocsp_hosts'
03.010323.445 Fetched OCSP fingerprints from 
'W:\Users\ender\AppData\Local/wget/.wget-ocsp'
03.010323.447 set_exit_status(0)
03.010323.447 *url =
03.010323.448 *3 https://eternallybored.org/misc/bigtest
03.010323.449 transcoded 7 bytes from 'utf-8' to 'CP1250'
03.010323.450 local filename = 'bigtest'
03.010323.451 host_add_job: job fname bigtest
03.010323.451 host_add_job: 0x8a9a0 https://eternallybored.org/misc/bigtest
03.010323.452 host_add_job: qsize 1 host-qsize=1
03.010323.453 queue_size: qsize=1
03.010323.454 queue_size: qsize=1
03.010323.454 queue_size: qsize=1
03.010323.455 [0] action=1 pending=0 host=0x0
03.010323.457 dequeue job https://eternallybored.org/misc/bigtest
03.010323.458 resolving eternallybored.org:443...
03.010323.462 has 84.255.206.8:443
03.010323.462 trying 84.255.206.8:443...
03.010323.469 OpenSSL initialized
03.010323.470 Sending 'status_request' extension in handshake
03.010323.470 ALPN offering h2
03.010323.471 ALPN offering http/1.1
03.010323.471 No cached TLS session available. Will run a full handshake.
03.010323.478 No HPKP pinning found for host 'eternallybored.org'
03.010323.479 OCSP URI not given and not found in certificate. Skipping OCSP 
check for cert 0.
03.010323.480 OCSP URI not given and not found in certificate. Skipping OCSP 
check for cert 1.
03.010323.481 No HPKP pinning found for host 'eternallybored.org'
03.010323.482 No HPKP pinning found for host 'eternallybored.org'
03.010323.483 *** OCSP response status:
03.010323.485 successful
03.010323.485 No HPKP pinning found for host 'eternallybored.org'
03.010323.486 No HPKP pinning found for host 'eternallybored.org'
03.010323.486 *** OCSP cert status:
03.010323.487 good
03.010323.487 Got a stapled OCSP response. Length: 503. Status: OK
03.010323.488 Handshake completed (full handshake - not resumed)
03.010323.489 TLS session discarded
03.010323.490 ALPN: Server accepted protocol 'h2'
03.010323.491 established connection eternallybored.org
[0] Downloading 'https://eternallybored.org/misc/bigtest' ...
03.010323.496 cookie_create_request_header for host=eternallybored.org 
path=misc/bigtest
03.010323.497 HTTP2 stream id 1
03.010323.498 [0] action=1 pending=1 host=0x8a950
03.010323.499 [0] action=2 pending=1 host=0x8a950
03.010323.501   ##  pending_requests = 1
03.010323.502 [FRAME 0] > SETTINGS
03.010323.503 [FRAME 0] > WINDOW_UPDATE
03.010323.505 [FRAME 1] > HEADERS
03.010323.507 [FRAME 1] > :method: GET
03.010323.508 [FRAME 1] > :path: /misc/bigtest
03.010323.509 [FRAME 1] > :scheme: https
03.010323.510 [FRAME 1] > :authority: eternallybored.org
03.010323.511 [FRAME 1] > accept-encoding: gzip, deflate, bzip2, xz, lzma, br, 
zstd, lzip
03.010323.512 [FRAME 1] > accept: 
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
03.010323.516 [FRAME 1] > user-agent: wget2/2.0.0
03.010323.517 [FRAME 0] < SETTINGS
03.010323.518 [FRAME 0] < WINDOW_UPDATE
03.010323.519 [FRAME 0] > SETTINGS
03.010323.520 [FRAME 0] < SETTINGS
03.010323.521 :status: 200
03.010323.522 date: Sat, 02 Oct 2021 23:03:23 GMT
03.010323.523 server: Apache
03.010323.526 strict-transport-security: max-age=31536000; includeSubDomains
03.010323.527 last-modified: Sat, 14 Dec 2013 21:06:38 GMT
03.010323.529 etag: "ce5135eb-4ed84f4616c76"
03.010323.530 accept-ranges: bytes
03.010323.531 content-length: 3461428715
03.010323.532 [FRAME 1] < HEADERS
Saving 'bigtest'
Failed to read 102400 bytes (0)
03.010323.728 failed to receive: 0
03.010323.729 host_increase_failure: eternallybored.org failures=1
03.010323.731 [0] action=3 pending=1 host=0x8a950
03.010323.732 closing connection
03.010323.734 released job https://eternallybored.org/misc/bigtest
03.010323.735 [0] action=1 pending=0 host=0x0
03.010323.736 host eternallybored.org is paused 993ms
03.010323.740 main: wake up
03.010323.741 queue_size: qsize=1
03.010324.733 [0] action=1 pending=0 host=0x0
03.010324.734 dequeue job https://eternallybored.org/misc/bigtest
03.010324.736 Found dns cache entry eternallybored.org:443
03.010324.737 trying 84.255.206.8:443...
03.010324.738 Sending 'status_request' extension in handshake
03.010324.739 ALPN offering h2
03.010324.740 ALPN offering http/1.1
03.010324.741 No cached TLS session available. Will run a full handshake.
03.010324.746 No HPKP pinning found for host 'eternallybored.org'
03.010324.747 OCSP URI not given and not found in certificate. Skipping OCSP 
check for cert 0.
03.010324.749 OCSP URI not given and not found in certificate. Skipping OCSP 
check for cert 1.
03.010324.750 No HPKP pinning found for host 'eternallybored.org'
03.010324.753 No HPKP pinning found for host 'eternallybored.org'
03.010324.756 *** OCSP response status:
03.010324.757 successful
03.010324.758 No HPKP pinning found for host 'eternallybored.org'
03.010324.760 No HPKP pinning found for host 'eternallybored.org'
03.010324.761 *** OCSP cert status:
03.010324.762 good
03.010324.763 Got a stapled OCSP response. Length: 503. Status: OK
03.010324.766 Handshake completed (full handshake - not resumed)
03.010324.767 TLS session discarded
03.010324.768 ALPN: Server accepted protocol 'h2'
03.010324.770 established connection eternallybored.org
[0] Downloading 'https://eternallybored.org/misc/bigtest' ...
03.010324.772 cookie_create_request_header for host=eternallybored.org 
path=misc/bigtest
03.010324.773 HTTP2 stream id 1
03.010324.774 [0] action=1 pending=1 host=0x8a950
03.010324.777 [0] action=2 pending=1 host=0x8a950
03.010324.779   ##  pending_requests = 1
03.010324.780 [FRAME 0] > SETTINGS
03.010324.781 [FRAME 0] > WINDOW_UPDATE
03.010324.782 [FRAME 1] > HEADERS
03.010324.783 [FRAME 1] > :method: GET
03.010324.784 [FRAME 1] > :path: /misc/bigtest
03.010324.788 [FRAME 1] > :scheme: https
03.010324.789 [FRAME 1] > :authority: eternallybored.org
03.010324.790 [FRAME 1] > accept-encoding: gzip, deflate, bzip2, xz, lzma, br, 
zstd, lzip
03.010324.792 [FRAME 1] > accept: 
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
03.010324.793 [FRAME 1] > user-agent: wget2/2.0.0
03.010324.794 [FRAME 0] < SETTINGS
03.010324.795 [FRAME 0] < WINDOW_UPDATE
03.010324.798 [FRAME 0] > SETTINGS
03.010324.799 [FRAME 0] < SETTINGS
03.010324.800 :status: 200
03.010324.801 date: Sat, 02 Oct 2021 23:03:24 GMT
03.010324.802 server: Apache
03.010324.803 strict-transport-security: max-age=31536000; includeSubDomains
03.010324.805 last-modified: Sat, 14 Dec 2013 21:06:38 GMT
03.010324.807 etag: "ce5135eb-4ed84f4616c76"
03.010324.809 accept-ranges: bytes
03.010324.810 content-length: 3461428715
03.010324.811 [FRAME 1] < HEADERS
Saving 'bigtest.1'
03.010324.813 blacklist set filename: bigtest -> bigtest.1
Failed to read 102400 bytes (24)
03.010324.826 failed to receive: 24
03.010324.827 host_increase_failure: eternallybored.org failures=2
03.010324.829 [0] action=3 pending=1 host=0x8a950
03.010324.830 closing connection
03.010324.831 released job https://eternallybored.org/misc/bigtest
03.010324.833 [0] action=1 pending=0 host=0x0
03.010324.834 host eternallybored.org is paused 1993ms
03.010324.837 main: wake up
03.010324.838 queue_size: qsize=1
03.010326.831 [0] action=1 pending=0 host=0x0
03.010326.833 dequeue job https://eternallybored.org/misc/bigtest
03.010326.834 Found dns cache entry eternallybored.org:443
03.010326.835 trying 84.255.206.8:443...
03.010326.843 Sending 'status_request' extension in handshake
03.010326.844 ALPN offering h2
03.010326.845 ALPN offering http/1.1
03.010326.846 No cached TLS session available. Will run a full handshake.
03.010326.858 No HPKP pinning found for host 'eternallybored.org'
03.010326.860 OCSP URI not given and not found in certificate. Skipping OCSP 
check for cert 0.
03.010326.863 OCSP URI not given and not found in certificate. Skipping OCSP 
check for cert 1.
03.010326.865 No HPKP pinning found for host 'eternallybored.org'
03.010326.866 No HPKP pinning found for host 'eternallybored.org'
03.010326.868 *** OCSP response status:
03.010326.869 successful
03.010326.869 No HPKP pinning found for host 'eternallybored.org'
03.010326.872 No HPKP pinning found for host 'eternallybored.org'
03.010326.874 *** OCSP cert status:
03.010326.875 good
03.010326.876 Got a stapled OCSP response. Length: 503. Status: OK
03.010326.877 Handshake completed (full handshake - not resumed)
03.010326.878 TLS session discarded
03.010326.879 ALPN: Server accepted protocol 'h2'
03.010326.883 established connection eternallybored.org
[0] Downloading 'https://eternallybored.org/misc/bigtest' ...
03.010326.885 cookie_create_request_header for host=eternallybored.org 
path=misc/bigtest
03.010326.887 HTTP2 stream id 1
03.010326.888 [0] action=1 pending=1 host=0x8a950
03.010326.889 [0] action=2 pending=1 host=0x8a950
03.010326.890   ##  pending_requests = 1
Failed to read 102400 bytes (24)
03.010326.894 failed to receive: 24
03.010326.895 host_increase_failure: eternallybored.org failures=3
03.010326.897 [0] action=3 pending=1 host=0x8a950
03.010326.898 closing connection
03.010326.899 released job https://eternallybored.org/misc/bigtest
03.010326.901 [0] action=1 pending=0 host=0x0
03.010326.904 host eternallybored.org is paused 2991ms
03.010326.905 main: wake up
03.010326.906 queue_size: qsize=1
03.010329.908 [0] action=1 pending=0 host=0x0
03.010329.910 dequeue job https://eternallybored.org/misc/bigtest
03.010329.912 Found dns cache entry eternallybored.org:443
03.010329.913 trying 84.255.206.8:443...
03.010329.915 Sending 'status_request' extension in handshake
03.010329.916 ALPN offering h2
03.010329.917 ALPN offering http/1.1
03.010329.920 No cached TLS session available. Will run a full handshake.
03.010329.925 No HPKP pinning found for host 'eternallybored.org'
03.010329.927 OCSP URI not given and not found in certificate. Skipping OCSP 
check for cert 0.
03.010329.928 OCSP URI not given and not found in certificate. Skipping OCSP 
check for cert 1.
03.010329.932 No HPKP pinning found for host 'eternallybored.org'
03.010329.934 No HPKP pinning found for host 'eternallybored.org'
03.010329.936 *** OCSP response status:
03.010329.937 successful
03.010329.938 No HPKP pinning found for host 'eternallybored.org'
03.010329.941 No HPKP pinning found for host 'eternallybored.org'
03.010329.943 *** OCSP cert status:
03.010329.944 good
03.010329.945 Got a stapled OCSP response. Length: 503. Status: OK
03.010329.946 Handshake completed (full handshake - not resumed)
03.010329.948 TLS session discarded
03.010329.948 ALPN: Server accepted protocol 'h2'
03.010329.951 established connection eternallybored.org
[0] Downloading 'https://eternallybored.org/misc/bigtest' ...
03.010329.955 cookie_create_request_header for host=eternallybored.org 
path=misc/bigtest
03.010329.957 HTTP2 stream id 1
03.010329.958 [0] action=1 pending=1 host=0x8a950
03.010329.960 [0] action=2 pending=1 host=0x8a950
03.010329.962   ##  pending_requests = 1
Failed to read 102400 bytes (24)
03.010329.964 failed to receive: 24
03.010329.965 host_increase_failure: eternallybored.org failures=4
03.010329.966 [0] action=3 pending=1 host=0x8a950
03.010329.967 closing connection
03.010329.969 released job https://eternallybored.org/misc/bigtest
03.010329.972 [0] action=1 pending=0 host=0x0
03.010329.973 host eternallybored.org is paused 3992ms
03.010329.974 main: wake up
03.010329.975 queue_size: qsize=1
03.010333.967 [0] action=1 pending=0 host=0x0
03.010333.968 dequeue job https://eternallybored.org/misc/bigtest
03.010333.969 Found dns cache entry eternallybored.org:443
03.010333.971 trying 84.255.206.8:443...
03.010333.972 Sending 'status_request' extension in handshake
03.010333.973 ALPN offering h2
03.010333.974 ALPN offering http/1.1
03.010333.975 No cached TLS session available. Will run a full handshake.
03.010333.983 No HPKP pinning found for host 'eternallybored.org'
03.010333.984 OCSP URI not given and not found in certificate. Skipping OCSP 
check for cert 0.
03.010333.985 OCSP URI not given and not found in certificate. Skipping OCSP 
check for cert 1.
03.010333.989 No HPKP pinning found for host 'eternallybored.org'
03.010333.990 No HPKP pinning found for host 'eternallybored.org'
03.010333.992 *** OCSP response status:
03.010333.993 successful
03.010333.994 No HPKP pinning found for host 'eternallybored.org'
03.010333.995 No HPKP pinning found for host 'eternallybored.org'
03.010333.996 *** OCSP cert status:
03.010333.998 good
03.010334.000 Got a stapled OCSP response. Length: 503. Status: OK
03.010334.001 Handshake completed (full handshake - not resumed)
03.010334.003 TLS session discarded
03.010334.003 ALPN: Server accepted protocol 'h2'
03.010334.005 established connection eternallybored.org
[0] Downloading 'https://eternallybored.org/misc/bigtest' ...
03.010334.009 cookie_create_request_header for host=eternallybored.org 
path=misc/bigtest
03.010334.010 HTTP2 stream id 1
03.010334.011 [0] action=1 pending=1 host=0x8a950
03.010334.012 [0] action=2 pending=1 host=0x8a950
03.010334.014   ##  pending_requests = 1
Failed to read 102400 bytes (24)
03.010334.016 failed to receive: 24
03.010334.017 host_increase_failure: eternallybored.org failures=5
03.010334.020 [0] action=3 pending=1 host=0x8a950
03.010334.021 closing connection
03.010334.023 released job https://eternallybored.org/misc/bigtest
03.010334.026 [0] action=1 pending=0 host=0x0
03.010334.027 host eternallybored.org is paused 4990ms
03.010334.029 main: wake up
03.010334.030 queue_size: qsize=1

--
< Jernej Simončič ><><><><><><><><><><><>< https://eternallybored.org/ >

            Because 10 billion years' time is so fragile, so ephemeral...
it arouses such a bittersweet, almost heartbreaking fondness.

Attachment: OpenPGP_signature
Description: OpenPGP digital signature

reply via email to
[Prev in Thread] Current Thread [Next in Thread]