Re: [Wget-dev] wget2 | Recursive downloads with HTTP/2 not uniformly divided (#397)

[Darshit Shah]

> It's not an overkill since your assumption is wrong. It doesn't mean '100 
> files simultaneously', it means 'send 100 requests and then wait for the 
> responses'. While technically the responses could come in multiplexed, you 
> will rarely see that (that would require massive parallelism on the server 
> side _and_ would possibly break cookie handling). The responses come one 
> after the other but without a pause (latency) between them. That is the good 
> thing with HTTP/2, the latency reduction. So the higher you set the request 
> window, the faster you are finished. Of course there might be some limits on 
> the server side - the server has to queue all the requests. 

My understanding is that when you send 100 multiplexed requests, the server can 
send as many multiplexed responses. I've definitely seen servers send 40 files 
in parallel as a result of the http2_request_window. Which is why I thought 
this isn't the best idea. Maybe, that's just a server configuration error then. 
If so, we should document it. With HTTP/2 being relatively new, I'm not 
surprised servers will be misconfigured, but we should document that somewhere 
so users know whom to blame.

> How then ? A value of 25 means what ... HTTP/2 window 1 and # of threads 25 ? 
> That is definitely not what anybody wants. So we need two values here (window 
> size and number of threads). 

No, that is not what I meant. But let me rephrase the entire thing for clarity:

## HTTP/1

- `--http2-request-window` is ignored (obviously).
- `--max-parallel-downloads` determines the number of threads (Default: 5)

## HTTP/2

- `--http2-request-window` determines the max number of multiplexed HTTP/2 
requests sent _per-thread_.
- `--max-parallel-downloads` determines the total number of simultaneous 
requests sent _across all threads_.

Hence, `nthreads = max_parallel_downloads / http2_request_window`. That is, if 
the user sets `http2_request_window = 30` and `max_parallel_downloads = 10` 
then only one thread is created and at most 10 multiplexed requests are sent 
effectively reducing the http2_request_window. However if, 
`max_parallel_downloads = 40`, then 2 threads are created.

I realise that this is already possible with the current `--max-threads` 
option. So, in hindsight, I guess my recommendation is to rename it. This is 
because `--max-threads` and `--http2-request-window` are more advanced options 
that requires the end user to have a deeper understanding of things. Very often 
they only know, "I want to download 10 files in parallel". Not, "I want to send 
30 multiplexed requests over a HTTP/2 connection but only 10 if my connection 
is HTTP/1". So, we must also make it easier for such users to do what they want.

-- 
Reply to this email directly or view it on GitLab: 
https://gitlab.com/gnuwget/wget2/issues/397#note_103453399
You're receiving this email because of your account on gitlab.com.