[Bug-wget] [bug #50516] domain.com vs www.domain.com site duplication

[Ages Ayemtwo]

URL:
  <http://savannah.gnu.org/bugs/?50516>

                 Summary: domain.com vs www.domain.com site duplication
                 Project: GNU Wget
            Submitted by: ages2500
            Submitted on: Sat 11 Mar 2017 08:01:57 PM UTC
                Category: Feature Request
                Severity: 3 - Normal
                Priority: 5 - Normal
                  Status: None
                 Privacy: Public
             Assigned to: None
         Originator Name: 
        Originator Email: 
             Open/Closed: Open
         Discussion Lock: Any
                 Release: None
        Operating System: None
         Reproducibility: None
           Fixed Release: None
         Planned Release: None
              Regression: None
           Work Required: None
          Patch Included: No

    _______________________________________________________

Details:

When retrieving http://www.domain.com/, the site author may link a file to
domain.com, without the www. This also occurs when the opposite is true.

Either scenario results in the website being downloaded twice, creating a
hapazard mesh of file links between:

/domain.com/

and

/www.domain.com/

It also means that 404 pages will link to http://domain.com/ in the html of
files of one folder, and http://www.domain.com/ in the other.

If one were to overlook the local mess this creates, it still puts extra
strain on a large wget process by crawling and downloading near twice as much
data than it needs to.

Restricting the site to -D www.domain.com runs the risk of missing data. To
ensure I get all of the data from the domain in question, I use -D
domain.com.

It would be nice for an extra flag to treat domain.com and www.domain.com
content the same in wget, and store the content in the same folder without
content duplication.

I am not requesting that this feature be a default function, but rather an
additional flag/feature that treats www.domain.com and domain.com as coming
from the same domain.

The following URL will exhibit this behavior in wget:


wget -rkE -np -l inf -D runequake.com http://www.runequake.com/







    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?50516>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/