wget2 | Multithreaded download: Change server probing behavior to better match browsers (#626)

[Long Nguyen (@longnguyen2004)]

Long Nguyen created an issue: https://gitlab.com/gnuwget/wget2/-/issues/626



Original issue at https://github.com/rockdaboot/wget2/issues/278

Chromium-based browsers, on media content (video embedded with the <video> 
tag), send a GET request with `Range: bytes=0-` and discard the body, instead 
of a HEAD request. I've seen servers that take advantage of it as a form of 
anti-scraping. Specifically, they disallow the use of HEAD requests, and return 
404 for requests that don't have a Range header.

I propose that we add `Range: bytes=0-` to every initial GET request, and then 
check for 206 or the `Accept-Ranges: bytes` header to determine if segmented 
downloading is supported.

-- 
Reply to this email directly or view it on GitLab: 
https://gitlab.com/gnuwget/wget2/-/issues/626
You're receiving this email because of your account on gitlab.com.