Re: [Bug-wget] Multi segment download

[Darshit Shah]

Thanking You,
Darshit Shah
Sent from mobile device. Please excuse my brevity
On 29-Aug-2015 1:13 pm, "Tim Rühsen" <address@hidden> wrote:
>
> Hi,
>
> normally it makes much more sense when having several download mirrors and
> checksums for each chunk. The perfect technique for such is called
'Metalink'
> (more on www.metalinker.org).
> Wget has it in branch 'master'. A GSOC project of Hubert Tarasiuk.
>
Sometimes the evil ISPs enforce a per connection bandwidth limit. In such a
case, multi segment downloads from a single server do make sense.

Since metalink already has support for downloading a file over multiple
connections, it should not be too difficult to reuse the code for use
outside of metalink.

I think it would be a good idea to do so. I'm not sure if all the possible
variations of the range headers are parsed by Wget.
> Additionally, Wget2 is under development, already having the option
--chunk-
> size (e.g. --chunk-size=1M) to start a multi-threaded download of a file.
>
> Regards, Tim
>
>
> Am Freitag, 28. August 2015, 15:41:27 schrieb Random Coder:
> > On Fri, Aug 28, 2015 at 3:06 PM, Ander Juaristi <address@hidden>
wrote:
> > > Hi,
> > >
> > > Would you point us to some potential use cases? How would a Wget user
> > > benefit from such a feature? One of the best regarded feature of
download
> > > managers is the ability to resume paused downloads, and that's already
> > > supported by Wget. Apart from that, I can't come across any other use
> > > case. But that's me, maybe you have a broader overview.
> > One possible feature, described in flowery language from a product
> > description: "... splits files into several sections and downloads
> > them simultaneously, allowing you to use any type of connection at the
> > maximum available speed. With FDM download speed increases, or even
> > more!"
> >
> > And, just show this can help, at least in some situations, here's an
> > example using curl (sorry, I don't know how to do a similar request in
> > wget).  First a normal download of the file:
> >
> > curl -o all http://mirror.internode.on.net/pub/test/100meg.test
> >
> > This command takes an average of 48.9 seconds to run on my current
> > network connection.  Now, if I split up the download as the download
> > manager will, and run these four commands at the same instant:
> >
> > curl -o part1 -r0-25000000
> > http://mirror.internode.on.net/pub/test/100meg.test curl -o part2
> > -r25000001-50000000
> > http://mirror.internode.on.net/pub/test/100meg.test
> > curl -o part3 -r50000001-75000000
> > http://mirror.internode.on.net/pub/test/100meg.test
> > curl -o part4 -r75000001-
> > http://mirror.internode.on.net/pub/test/100meg.test
> >
> > The union of time it takes all four commands to run ends up being an
> > average of 19.9 seconds over a few test runs on the same connection.
> > There's some penalty here because I need to spend time combining the
> > files afterwards, but if the command supported this logic internally,
> > no doubt much of that work could be done up front as the file is
> > downloaded.