Re: [Bug-wget] language of the page depends on --compression={auto, none}?

[Tim Rühsen]

Hi Gabriele,

this seems to be a mis-configured proxy that serves different content
depending on the 'Accept-Encoding:' that the client sends.

It's not even possible to switch caching off with --no-cache. The proxy
ignores both 'Cache-Control: no-cache' and 'Pragma: no-cache' from the
client.

If this is important to you, you should contact the server's admin or
support.

Regards, Tim

On 13.10.19 13:42, address@hidden wrote:
> hello
> 
> I'm experiencing a problem related to --compression with wget.
> 
> I usually run with --compression=auto because some servers apparently
> serve gzip compressed pages unconditionally (eg
> http://llvm.org/releases/download.html).
> 
> Since few days (2019-10-11), for some reason, with --compression=auto
> wget will download the SPANISH version of this page:
> https://pypi.org/project/nose/, where with --compression=none it will
> download the (usual) ENGLISH version of the same page.
> 
> I never had this "spanish" problem before (the wget command is run in a
> daily procedure since many years); this makes me think that the
> problem might be related to a server upgrade on that site (I didn't
> record this: presently: Server: nginx/1.13.9)
> 
> I didn't notice any similar behavior with the many other sites I daily
> monitor using wget (and using --compression=auto)
> 
> In any case, I wouldn't expect the *language* of a downloaded page to be
> dependent on the --compression command line arg (!)
> 
> I don't know if this is a problem with wget or the http server...
> 
> I enclose a scriptlett that can be used to quickly check if the
> problem is reproducible by others
> 
> (I can add that with curl I have no problem)
> 
> thanks a lot
> ciao
> gabriele
> 
> ----8<----
> #!/bin/sh
> 
> WGET=wget
> 
> root_dir=/tmp/wget
> rm -rf ${root_dir}
> mkdir -p ${root_dir}/{llvm/{auto,none},pypi/{auto,none}}
> 
> echo
> echo "https://pypi.org/project/nose/";
> echo "=============================="
> cd ${root_dir}/pypi/auto
> rm -f ./auto.html && \
>     ${WGET} -d --compression=auto -O ./auto.html  \
>          https://pypi.org/project/nose/ >&./auto.log && \
>     egrep -qi fecha ./auto.html && \
>     echo "--compression=auto: spanish"
> 
> cd ${root_dir}/pypi/none
> rm -f ./none.html && \
>     ${WGET} -d --compression=none -O ./none.html  \
>          https://pypi.org/project/nose/ >&./none.log && \
>     egrep -qi 'Upload date' ./none.html && \
>     echo "--compression=none: english"
> 
> echo
> echo "http://llvm.org/releases/download.html";
> echo "======================================"
> cd ${root_dir}/llvm/auto
> rm -f ./auto.html && \
>     ${WGET} -d --compression=auto -O ./auto.html  \
>          http://llvm.org/releases/download.html >&./auto.log && \
>     echo "--compression=auto: $(file ./auto.html|sed -e's|^.*: ||' 
> -e's|,.*||')"
>     echo "                    $(egrep 'Accept-Encoding' ./auto.log|head -1)"
>     echo "                    $(egrep 'Content-Encoding' ./auto.log)"
> cd ${root_dir}/llvm/none
> rm -f ./none.html && \
>     ${WGET} -d --compression=none -O ./none.html  \
>          http://llvm.org/releases/download.html >&./none.log && \
>     echo "--compression=none: $(file ./none.html|sed -e's|^.*: ||' 
> -e's|,.*||')"
>     echo "                    $(egrep 'Accept-Encoding' ./none.log|head -1)"
>     echo "                    $(egrep 'Content-Encoding' ./none.log)"
> 
> exit
> ---->8----
>

signature.asc
Description: OpenPGP digital signature