bug#32658: gnutls + non-blocking url-retrieve

[thomas]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: thomas@m3y3r.de
>> Date: Sun, 30 Sep 2018 23:33:10 +0200
>> 
>> 1.) I needed to revert to gnutls 3.5.19, the mingw64 build from the
>> gitlab ci build seems to have a working gnutls-cli tools on windows 10.
>> the gitlab builds for 3.6.3 and 3.6.4 seems to have another bug
>> (error code -53) in the gnutls-cli command.
>> 
>> so only gnutls 3.5.19 have a working gnutls-cli. i installed this version in 
>> emacs 26.1
>> 
>> 2.) testing gnutls stream
>> using open-gnutls-stream directly gives me a correct tls connection but
>> eww still fails to load the site.
>> 
>> when I change url-open-stream in url/url-gw.el to:
>>                        (open-network-stream
>>                         name buffer host service
>>                         :type gw-method
>>                         ;; Use non-blocking socket if we can.
>>                           :nowait nil))
>> 
>> I finally can open lwn.net in eww.
>> 
>> so something seems to be wrong possible with blocking/non-blocking
>> network access.
>> 
>> any ideas?
>

> Thanks for the info.

Hi, thanks for looking into this.

>
> First, I don't understand what does gnutls-cli have to do with this.

okay, thats an easy one to explain.
I did download emacs 26.1 from here:
http://mirror.netcologne.de/gnu/emacs/windows/emacs-26/emacs-26.1-x86_64.zip

in the bin directory there is the gnutls packaged. version is 3.6.0.

I wasn't sure where the bug in the TLS problems I see was, so I first
tried to use gnutls-cli to check if a connection can be made:

C:\Users\thomas\Downloads\emacs-26.1-x86_64\bin>gnutls-cli lwn.net
|<1>| There was a non-CA certificate in the trusted list: OU=Copyright (c) 1997 
Microsoft Corp.,OU=Microsoft Corporation,CN=Microsoft Root Authority.
|<1>| There was a non-CA certificate in the trusted list: 
C=US,O=MSFT,CN=Microsoft Authenticode(tm) Root Authority.
|<1>| There was a non-CA certificate in the trusted list: CN=Root Agency.
Processed 59 CA certificate(s).
Resolving 'lwn.net:443'...
Connecting to '2600:3c03::f03c:91ff:fe61:5c5b:443'...
*** Fatal error: Error in the push function.
Connecting to '45.33.94.129:443'...
*** Fatal error: Error in the push function.
Could not connect to 45.33.94.129:443: Bad file descriptor

This doesn't work, because of some error -53 (ERROR_BAD_NETPATH).

so this is why I did first try to upgrade to gnutls 3.6.3 from the
gnutls homepage which is a gitlab ci build, but with no success.

then i tried to downgrade the gnutls version to 3.5.19 and there the
gnutls-cli tool did work.

so gnutls is now able to create an TLS connection. now the question why
it doesn't work in emacs.

> Emacs on Windows doesn't support TLS connections that use gnutls-cli,
> because the way that works, it requires working support for signals,
> which cannot happen on Windows.  Are you saying that these problems
> happen when you use gnutls-cli?  If so, please move to the built-in
> GnuTLS support, because connections using gnutls-cli are deprecated,
> and I see no point in trying to support them on Windows.

see above explanation. hopefully this makes it clear.

>
> Second, I cannot reproduce the problem you are reporting.  Using stock
> Emacs 26.1 I built myself, with GnuTLS 3.4.15, I have no problems
> connecting to lwn.net via eww.  I see EAGAIN errors like you do, but
> they are non-fatal, so don't prevent the connection from continuing.
> It is strange that you are having these problems, but maybe these
> problems are specific to GnuTLS 3.6.x?  3.6.x is not a stable branch
> of GnuTLS, it could have bugs, in particular bugs specific to Windows.
> It is also possible that there are incompatibilities between GnuTLS
> 3.6.x and whatever version the Emacs binary you are using was built
> against.

I do use the emacs 26.1 zip file that is pre-build and linked to from
the emacs homepage, see above.

>
> In this message you say that you downgraded to GnuTLS 3.5.19, but you
> didn't show the gnutls.c log for that version -- does it mean you see
> an identical problem with EAGAIN there?
>
> Is it possible for you to downgrade GnuTLS to some version of the
> 3.4.x branch, and see if the problem persists?

will try out and report.

>
> Also, does this happen in "emacs -Q"?

yes, same error.

>
> Or maybe this is specific to your network connection?  Does any HTTPS
> connection cause these problems?

no, only some I think.

>
> Finally, what about other machines and/or Windows versions other than
> 10 -- do you have the same problem there with this Emacs version
> (assuming you can test that)?

I use emacs 26.1 on an linux x86 system and on android arm with termux,
both on the same network as the windows laptop, and everything works as
expected on these systems.

>
> Bottom line: I'm surprised that you have these problems, because I see
> none of that on my machines -- TLS connections "just work" for me,
> without any need to tinker with url-gw.el or elsewhere.  And judging
> by lack of similar bug reports, this also works for others.  So I
> wonder what causes this in your case.

I do also wonder!

here some more details, with vanilla emacs 26.1 + gnutls
3.5.19 and gnutls-log-level 1:

1.) eww lwn.net
Contacting host: lwn.net:80
gnutls.c: [1] (Emacs) connecting to host: lwn.net
gnutls.c: [1] (Emacs) allocating credentials
gnutls.c: [audit] There was a non-CA certificate in the trusted list:
OU=Copyright (c) 1997 Microsoft Corp.,OU=Microsoft
Corporation,CN=Microsoft Root Authority.

gnutls.c: [audit] There was a non-CA certificate in the trusted list: 
C=US,O=MSFT,CN=Microsoft Authenticode(tm) Root Authority.

gnutls.c: [audit] There was a non-CA certificate in the trusted list: CN=Root 
Agency.

gnutls.c: [1] (Emacs) setting the trustfile:  
C:\Users\thomas\emacs-26.1\etc\pki\ca-trust\extracted\pem\tls-ca-bundle.pem
gnutls.c: [1] (Emacs) setting the trustfile:  
C:\Users\thomas\emacs-26.1\ssl\certs\ca-bundle.crt
gnutls.c: [1] (Emacs) setting the trustfile:  
C:\Users\thomas\emacs-26.1\ssl\certs\ca-bundle.trust.crt
gnutls.c: [1] (Emacs) gnutls callbacks
gnutls.c: [1] (Emacs) gnutls_init
gnutls.c: [1] (Emacs) got non-default priority string: NORMAL:%DUMBFW
gnutls.c: [1] (Emacs) setting the priority string
gnutls.c: [audit] Note that the security level of the Diffie-Hellman
key exchange has been lowered to 128 bits and this may allow
decryption of the session data
gnutls.c: [1] (Emacs) non-fatal error: Resource temporarily unavailable, try 
again. [3 times]

After that eww shows:

Bad Request

Your browser sent a request that this server could not understand.
Reason: You're speaking plain HTTP to an SSL-enabled server port.
Instead use the HTTPS scheme to access this URL, please.

2.) open-gnutls-stream lwn.net

(open-gnutls-stream "test" (current-buffer) "lwn.net" "https")

gnutls.c: [1] (Emacs) connecting to host: lwn.net
gnutls.c: [1] (Emacs) allocating credentials
gnutls.c: [audit] There was a non-CA certificate in the trusted list:
OU=Copyright (c) 1997 Microsoft Corp.,OU=Microsoft
Corporation,CN=Microsoft Root Authority.

gnutls.c: [audit] There was a non-CA certificate in the trusted list: 
C=US,O=MSFT,CN=Microsoft Authenticode(tm) Root Authority.

gnutls.c: [audit] There was a non-CA certificate in the trusted list: CN=Root 
Agency.

gnutls.c: [1] (Emacs) setting the trustfile:  
C:\Users\thomas\emacs-26.1\etc\pki\ca-trust\extracted\pem\tls-ca-bundle.pem
gnutls.c: [1] (Emacs) setting the trustfile:  
C:\Users\thomas\emacs-26.1\ssl\certs\ca-bundle.crt
gnutls.c: [1] (Emacs) setting the trustfile:  
C:\Users\thomas\emacs-26.1\ssl\certs\ca-bundle.trust.crt
gnutls.c: [1] (Emacs) gnutls callbacks
gnutls.c: [1] (Emacs) gnutls_init
gnutls.c: [1] (Emacs) got non-default priority string: NORMAL:%DUMBFW
gnutls.c: [1] (Emacs) setting the priority string
gnutls.c: [audit] Note that the security level of the Diffie-Hellman
key exchange has been lowered to 128 bits and this may allow
decryption of the session data

gnutls.c: [1] (Emacs) non-fatal error: Resource temporarily unavailable, try 
again. [4905 times]
gnutls.c: [1] (Emacs) verification: the certificate was signed by an unknown 
and therefore untrusted authority
gnutls.c: [1] (Emacs) verification: certificate could not be verified
gnutls.c: [1] (Emacs) certificate validation failed: lwn.net

I think after that a TLS connection was successfully established and the
buffer prints:
#<process test>

what springs into the eye is the difference of number of re-tries that
are necessary to establish an TLS connection 4905 vs. 3.

Why does url-retrieve give up after 3 re-tries?

Here an debug-on-entry of open-network-stream of eww lwn.net:

* open-network-stream("lwn.net" #<buffer  *url-http-temp*> "lwn.net" 80 :type 
plain :nowait (:nowait t))
  url-open-stream("lwn.net" #<buffer  *url-http-temp*> "lwn.net" 80 nil)
  url-http-find-free-connection("lwn.net" 80 nil)
  url-http(#s(url :type "http" :user nil :password nil :host "lwn.net"
:portspec nil :filename "/" :target nil :attributes nil :fullness t
:silent nil :use-cookies t :asynchronous t) eww-render (nil
"http://lwn.net/"; nil #<buffer *eww*>))
  url-retrieve-internal("http://lwn.net/"; eww-render (nil "http://lwn.net/"; nil 
#<buffer *eww*>) nil nil)
  url-retrieve("http://lwn.net/"; eww-render ("http://lwn.net/"; nil #<buffer 
*eww*>))
  eww("lwn.net")
  funcall-interactively(eww "lwn.net")
  call-interactively(eww record nil)
  command-execute(eww record)
  execute-extended-command(nil "eww" "eww")
  funcall-interactively(execute-extended-command nil "eww" "eww")
  call-interactively(execute-extended-command nil nil)
  command-execute(execute-extended-command)

any ideas what going on here?

btw. emacs 25.3 with gnutls 3.5.19 works correctly on the same machine
when trying to connect to lwn.net with eww.