[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Bug-wget] Help: Why wget wall clock time much higher than download

From: Tim Rühsen
Subject: Re: [Bug-wget] Help: Why wget wall clock time much higher than download time?
Date: Thu, 20 Jun 2019 09:55:38 +0200
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.7.1

On 6/17/19 10:32 PM, David Bodin wrote:
> *wget --page-requisites --span-hosts --convert-links --adjust-extension
> --execute robots=off --user-agent Mozilla
> --random-wait 
> https://www.invisionapp.com/inside-design/essential-steps-designing-empathy/
> <https://www.invisionapp.com/inside-design/essential-steps-designing-empathy/>*
> This command above provides the following stats:
> Total wall clock time: 35s
> Downloaded: 248 files, 39M in 4.2s (9.36 MB/s)
> This website takes about 5 seconds to download and display all files on a
> hard refresh in the browser.
> Why is the wall clock time *significantly longer* than the download time
> and is there a way to make it faster?

First of all, --random-wait waits 0.5 to 1.5 seconds after each
downloaded page. Don't use it - there have been times when web servers
blocked fast clients, but that shouldn't be the case today.

Wget uses just one connection for downloading, no compression by
default, no http/2.

You can try Wget2 which uses as many parallel connections as you like,
uses compression by default and http/2 if possible. Depending on the
HTTP server, Wget2 is often 10x faster then Wget just with it's default

You find the latest Wget2 tarball at

Instructions how to build at

Regards, Tim

Attachment: signature.asc
Description: OpenPGP digital signature

reply via email to

[Prev in Thread] Current Thread [Next in Thread]