[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Bug-wget] Inconsistent cookie handling between different machines
|
From: |
Casey, Sean |
|
Subject: |
Re: [Bug-wget] Inconsistent cookie handling between different machines |
|
Date: |
Sat, 18 Aug 2018 16:53:39 +0000 |
No Problem....see expanded output below. The only values that have been
modified are the document id query string value and the cookie values:
--------------------------------------------------------------------------------------------------------------------------------------------------
Wget Version Info Machine 1:
--------------------------------------------------------------------------------------------------------------------------------------------------
$ wget -V
GNU Wget 1.17.1 built on linux-gnu.
+digest -gpgme +https +ipv6 +iri +large-file -metalink +nls +ntlm
+opie -psl +ssl/openssl
Wgetrc:
/etc/wgetrc (system)
Locale:
/usr/share/locale
Compile:
gcc -DHAVE_CONFIG_H -DSYSTEM_WGETRC="/etc/wgetrc"
-DLOCALEDIR="/usr/share/locale" -I. -I../../src -I../lib
-I../../lib -Wdate-time -D_FORTIFY_SOURCE=2 -I/usr/include
-DHAVE_LIBSSL -DNDEBUG -g -O2 -fPIE -fstack-protector-strong
-Wformat -Werror=format-security -DNO_SSLv2 -D_FILE_OFFSET_BITS=64
-g -Wall
Link:
gcc -DHAVE_LIBSSL -DNDEBUG -g -O2 -fPIE -fstack-protector-strong
-Wformat -Werror=format-security -DNO_SSLv2 -D_FILE_OFFSET_BITS=64
-g -Wall -Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro
-Wl,-z,now -L/usr/lib -lpcre -luuid -lssl -lcrypto -lz -lidn
ftp-opie.o openssl.o http-ntlm.o ../lib/libgnu.a
--------------------------------------------------------------------------------------------------------------------------------------------------
Wget Version Info Machine 2:
--------------------------------------------------------------------------------------------------------------------------------------------------
$ wget -V
GNU Wget 1.17.1 built on linux-gnu.
+digest -gpgme +https +ipv6 +iri +large-file -metalink +nls +ntlm
+opie -psl +ssl/openssl
Wgetrc:
/etc/wgetrc (system)
Locale:
/usr/share/locale
Compile:
gcc -DHAVE_CONFIG_H -DSYSTEM_WGETRC="/etc/wgetrc"
-DLOCALEDIR="/usr/share/locale" -I. -I../../src -I../lib
-I../../lib -Wdate-time -D_FORTIFY_SOURCE=2 -I/usr/include
-DHAVE_LIBSSL -DNDEBUG -g -O2 -fPIE -fstack-protector-strong
-Wformat -Werror=format-security -DNO_SSLv2 -D_FILE_OFFSET_BITS=64
-g -Wall
Link:
gcc -DHAVE_LIBSSL -DNDEBUG -g -O2 -fPIE -fstack-protector-strong
-Wformat -Werror=format-security -DNO_SSLv2 -D_FILE_OFFSET_BITS=64
-g -Wall -Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro
-Wl,-z,now -L/usr/lib -lpcre -luuid -lssl -lcrypto -lz -lidn
ftp-opie.o openssl.o http-ntlm.o ../lib/libgnu.a
--------------------------------------------------------------------------------------------------------------------------------------------------
Full Debug Output from Machine 1
--------------------------------------------------------------------------------------------------------------------------------------------------
wget -d --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies
--no-check-certificate "https://docs.google.com/uc?export=download&id=$DOC_ID";
Setting --quiet (quiet) to 1
Setting --save-cookies (savecookies) to /tmp/cookies.txt
Setting --keep-session-cookies (keepsessioncookies) to 1
Setting --check-certificate (checkcertificate) to 0
DEBUG output created by Wget 1.17.1 on linux-gnu.
Reading HSTS entries from /home/seamuskc/.wget-hsts
URI encoding = ‘UTF-8’
Caching docs.google.com => 172.217.11.78 2607:f8b0:4007:802::200e
Created socket 3.
Releasing 0x00005633fd41fa20 (new refcount 1).
Initiating SSL handshake.
Handshake successful; connected socket 3 to SSL handle 0x00005633fd420eb0
certificate:
subject: CN=*.google.com,O=Google LLC,L=Mountain View,ST=California,C=US
issuer: CN=Google Internet Authority G3,O=Google Trust Services,C=US
X509 certificate successfully verified and matches host docs.google.com
---request begin---
GET /uc?export=download&id=someDocId HTTP/1.1
User-Agent: Wget/1.17.1 (linux-gnu)
Accept: */*
Accept-Encoding: identity
Host: docs.google.com
Connection: Keep-Alive
---request end---
---response begin---
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Date: Thu, 16 Aug 2018 23:18:03 GMT
P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Server: GSE
Set-Cookie:
download_warning_13058876669334088843_1OdHCtWyoM_ztgiULQBuoLZ6yBAqoVkaY=NFXg;
Domain=.docs.google.com; Expires=Thu, 16-Aug-2018 23:23:03 GMT; Path=/uc;
Secure; HttpOnly
Set-Cookie: NID=CV1;Domain=.google.com;Path=/;Expires=Fri, 15-Feb-2019 23:18:03
GMT;HttpOnly
Set-Cookie: NID=CV2;Domain=.google.com;Path=/;Expires=Fri, 15-Feb-2019 23:18:03
GMT;HttpOnly
Alt-Svc: quic=":443"; ma=2592000; v="44,43,39,35"
Accept-Ranges: none
Vary: Accept-Encoding
Transfer-Encoding: chunked
---response end---
cdm: 2
Stored cookie docs.google.com -1 (ANY) /uc <permanent> <secure> [expiry
2018-08-16 16:23:03]
download_warning_13058876669334088843_1OdHCtWyoM_ztgiULQBuoLZ6yBAqoVkaY NFXg
cdm: 2 3 4 5 6 7 8
Stored cookie google.com -1 (ANY) / <permanent> <insecure> [expiry 2019-02-15
15:18:03] NID CV1
cdm: 2 3 4 5 6 7 8Deleted old cookie (to be replaced.)
Stored cookie google.com -1 (ANY) / <permanent> <insecure> [expiry 2019-02-15
15:18:03] NID CV2
Registered socket 3 for persistent reuse.
URI content encoding = ‘utf-8’
Saving cookies to /tmp/cookies.txt.
Done saving cookies.
Saving HSTS entries to /home/seamuskc/.wget-hsts
--------------------------------------------------------------------------------------------------------------------------------------------------
Full Debug Output from Machine 2
--------------------------------------------------------------------------------------------------------------------------------------------------
wget -d --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies
--no-check-certificate "https://docs.google.com/uc?export=download&id=$DOC_ID";
Setting --quiet (quiet) to 1
Setting --save-cookies (savecookies) to /tmp/cookies.txt
Setting --keep-session-cookies (keepsessioncookies) to 1
Setting --check-certificate (checkcertificate) to 0
DEBUG output created by Wget 1.17.1 on linux-gnu.
Reading HSTS entries from /home/developer/.wget-hsts
URI encoding = ‘UTF-8’
Caching docs.google.com => 172.217.8.14 2607:f8b0:4004:801::200e
Created socket 3.
Releasing 0x0000560d40f6fe70 (new refcount 1).
Initiating SSL handshake.
Handshake successful; connected socket 3 to SSL handle 0x0000560d40f700f0
certificate:
subject: CN=*.google.com,O=Google LLC,L=Mountain View,ST=California,C=US
issuer: CN=Google Internet Authority G3,O=Google Trust Services,C=US
X509 certificate successfully verified and matches host docs.google.com
---request begin---
GET /uc?export=download&id=someDocId HTTP/1.1
User-Agent: Wget/1.17.1 (linux-gnu)
Accept: */*
Accept-Encoding: identity
Host: docs.google.com
Connection: Keep-Alive
---request end---
---response begin---
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Date: Fri, 17 Aug 2018 19:45:05 GMT
P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Server: GSE
Set-Cookie:
download_warning_13058876669334088843_1OdHCtWyoM_ztgiULQBuoLZ6yBAqoVkaY=HWSn;
Domain=.docs.google.com; Expires=Fri, 17-Aug-2018 19:50:05 GMT; Path=/uc;
Secure; HttpOnly
Set-Cookie: NID=CV1;Domain=.google.com;Path=/;Expires=Sat, 16-Feb-2019 19:45:05
GMT;HttpOnly
Set-Cookie: NID=CV2;Domain=.google.com;Path=/;Expires=Sat, 16-Feb-2019 19:45:05
GMT;HttpOnly
Alt-Svc: quic=":443"; ma=2592000; v="44,43,39,35"
Accept-Ranges: none
Vary: Accept-Encoding
Transfer-Encoding: chunked
---response end---
cdm: 2cdm: 2 3 4 5 6 7 8
Stored cookie google.com -1 (ANY) / <permanent> <insecure> [expiry 2019-02-16
19:45:05] NID CV1
cdm: 2 3 4 5 6 7 8Deleted old cookie (to be replaced.)
Stored cookie google.com -1 (ANY) / <permanent> <insecure> [expiry 2019-02-16
19:45:05] NID CV2
Registered socket 3 for persistent reuse.
URI content encoding = ‘utf-8’
Saving cookies to /tmp/cookies.txt.
Done saving cookies.
Saving HSTS entries to /home/developer/.wget-hsts+
-----Original Message-----
From: Darshit Shah <address@hidden>
Sent: Saturday, August 18, 2018 3:45 AM
To: Casey, Sean <address@hidden>
Cc: address@hidden
Subject: [External] Re: [Bug-wget] Inconsistent cookie handling between
different machines
Hi,
Thanks for the report and the analysis. However, could you please share the
entire debug output from both runs? Please don't cut out anything, very often
the contextual information around the problem area is about as important. You
may redact the actual Cookie data if you want.
Also, please do share the results of `wget -V` from both the machines. I know
you said, it's the same version, but the compilation options may be different.
* Casey, Sean <address@hidden> [180818 09:06]:
> Hello All,
>
> I'm not sure if this is the correct place to post this question but I've dug
> as deep as I can from my end and this seems like the best place to solicit
> help from the wget gurus.
>
> I'm running into some weird behavior where the same request sent from the
> same version of wget from different machines is handling cookie processing
> differently.
>
> This is the version that I'm running (on both machines):
>
> $ wget -V
> GNU Wget 1.17.1 built on linux-gnu.
>
> +digest -gpgme +https +ipv6 +iri +large-file -metalink +nls +ntlm opie
> +-psl +ssl/openssl
>
> Wgetrc:
> /etc/wgetrc (system)
> Locale:
> /usr/share/locale
> Compile:
> gcc -DHAVE_CONFIG_H -DSYSTEM_WGETRC="/etc/wgetrc"
> -DLOCALEDIR="/usr/share/locale" -I. -I../../src -I../lib
> -I../../lib -Wdate-time -D_FORTIFY_SOURCE=2 -I/usr/include
> -DHAVE_LIBSSL -DNDEBUG -g -O2 -fPIE -fstack-protector-strong
> -Wformat -Werror=format-security -DNO_SSLv2 -D_FILE_OFFSET_BITS=64
> -g -Wall
> Link:
> gcc -DHAVE_LIBSSL -DNDEBUG -g -O2 -fPIE -fstack-protector-strong
> -Wformat -Werror=format-security -DNO_SSLv2 -D_FILE_OFFSET_BITS=64
> -g -Wall -Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro
> -Wl,-z,now -L/usr/lib -lpcre -luuid -lssl -lcrypto -lz -lidn
> ftp-opie.o openssl.o http-ntlm.o ../lib/libgnu.a
>
>
> This is the command I'm running:
>
> $ wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies
> --no-check-certificate
> https://secure-web.cisco.com/1Qg-QKTt0hv7pFRt_RRzijalrUh5tHrPBOuos9EOK
> raSrKmDGMzeQ4TUM-r2fOQf_b8wrmSClxnrZHhKOIcctdcRunbtf5Kx85aMZrBld316nWu
> uYpJLUJ4TYBigwZrx0By68K_QG0KGyNdyfgh1eqmFAvYYSEzhbq9pbILmVYWWghm5p-75T
> 8rtrWQuLmpQLIQSaKbLsIs75j94bPtxibDk0xihVAEfum72N3IjmKP-0CP8R-FnghWwV8a
> 3kjKFohRJbwObb-cy7Ao0IJB2VvD2fi0lu3_QzZ7Gs2ed-u450cRheO8mh_t1DY0cP3N_j
> FjNrWQl1Nu0PpS19Gy50rQ/https%3A%2F%2Fdocs.google.com%2Fuc%3Fexport%3Dd
> ownload%26id%3D%24DOC_ID
>
> This is the response I'm getting on both machines (replaced cookie value with
> sample text for brevity). Note the three cookies being set:
>
> HTTP/1.1 200 OK
> Content-Type: text/html; charset=utf-8
> Cache-Control: no-cache, no-store, max-age=0, must-revalidate
> Pragma: no-cache
> Expires: Mon, 01 Jan 1990 00:00:00 GMT
> Date: Fri, 17 Aug 2018 19:45:05 GMT
> P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
> P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
> X-Content-Type-Options: nosniff
> X-Frame-Options: SAMEORIGIN
> X-XSS-Protection: 1; mode=block
> Server: GSE
> Set-Cookie: download_warning_random_text=SOME_COOKIE_VALUE;
> Domain=.docs.google.com; Expires=Fri, 17-Aug-2018 19:50:05 GMT;
> Path=/uc; Secure; HttpOnly
> Set-Cookie:
> NID=SOME_COOKIE_VALUE;Domain=.google.com;Path=/;Expires=Sat,
> 16-Feb-2019 19:45:05 GMT;HttpOnly
> Set-Cookie: NID=
> SOME_COOKIE_VALUE;Domain=.google.com;Path=/;Expires=Sat, 16-Feb-2019
> 19:45:05 GMT;HttpOnly
> Alt-Svc: quic=":443"; ma=2592000; v="44,43,39,35"
> Accept-Ranges: none
> Vary: Accept-Encoding
> Transfer-Encoding: chunked
>
> When running from machine 1 (Ubuntu - 16.04 LTS) I see the following debug
> statements (note all three cookies are stored):
>
> cdm: 2
> Stored cookie docs.google.com -1 (ANY) /uc <permanent> <secure>
> [expiry 2018-08-16 16:23:03] download_warning_random_text
> SOME_COOKIE_VALUE
> cdm: 2 3 4 5 6 7 8
> Stored cookie google.com -1 (ANY) / <permanent> <insecure> [expiry
> 2019-02-15 15:18:03] NID SOME_COOKIE_VALUE
> cdm: 2 3 4 5 6 7 8Deleted old cookie (to be replaced.)
>
> Stored cookie google.com -1 (ANY) / <permanent> <insecure> [expiry
> 2019-02-15 15:18:03] NID SOME_COOKIE_VALUE Registered socket 3 for persistent
> reuse.
> URI content encoding = 'utf-8'
> Saving cookies to /tmp/cookies.txt.
> Done saving cookies.
>
> When running from machine 2 (Ubuntu - 16.04.4 LTS) I see the following debug
> statements (note only two of the three cookies are stored):
>
> cdm: 2cdm: 2 3 4 5 6 7 8
> Stored cookie google.com -1 (ANY) / <permanent> <insecure> [expiry
> 2019-02-16 19:45:05] NID SOME_COOKIE_VALUE
> cdm: 2 3 4 5 6 7 8Deleted old cookie (to be replaced.)
>
> Stored cookie google.com -1 (ANY) / <permanent> <insecure> [expiry
> 2019-02-16 19:45:05] NID SOME_COOKIE_VALUE Registered socket 3 for persistent
> reuse.
> URI content encoding = 'utf-8'
> Saving cookies to /tmp/cookies.txt.
> Done saving cookies.
>
> For some reason the "download_warning" cookie is not being stored when
> running from machine 2.
>
> I'm not using a custom ~/.wgetrc file from either machine and both machines
> are using the stock /etc/wgetrc file (no difference between files).
>
> For the life of me, I can't figure out why that third cookie isn't being
> stored from machine 2? The only thing I noticed that is different about that
> cookie is that it's marked as "secure" while the other two are not.
>
> I looked through the wget man pages and didn't see any other options that
> impact cookie processing aside from the ones I've used. Any help would be
> greatly appreciated.
>
> Thanks
> Sean
> -This e-mail and any attachments may contain CONFIDENTIAL information,
> including PROTECTED HEALTH INFORMATION. If you are not the intended
> recipient, any use or disclosure of this information is STRICTLY PROHIBITED;
> you are requested to delete this e-mail and any attachments, notify the
> sender immediately, and notify the LabCorp Privacy Officer at address@hidden
> or call (877) 23-HIPAA / (877) 234-4722.
>
--
Thanking You,
Darshit Shah
PGP Fingerprint: 7845 120B 07CB D8D6 ECE5 FF2B 2A17 43ED A91A 35B6
-This e-mail and any attachments may contain CONFIDENTIAL information,
including PROTECTED HEALTH INFORMATION. If you are not the intended recipient,
any use or disclosure of this information is STRICTLY PROHIBITED; you are
requested to delete this e-mail and any attachments, notify the sender
immediately, and notify the LabCorp Privacy Officer at address@hidden or call
(877) 23-HIPAA / (877) 234-4722.