Re: [Savannah-hackers-public] new vcs server ssh host key fingerprint

[Mike Miller]

(I am subscribed now, no need to cc me anymore.)

On Fri, Jan 13, 2017 at 13:03:23 -0700, Bob Proulx wrote:
> Hmm...  This feels like something where the smallest of details can
> cause a lot of ... I usually say problems but in this case I will say
> noise.  And there are a lot of different ways to configure every
> possible part of the system including ssh.  It just isn't possible to
> avoid tripping over one snag or another.

Agree!

> 208.118.235.201 is the IP address of the new server.  208.118.235.72
> is the IP address of the old server.  Using crytographic keys to
> verify host systems is much stronger than using IP addresses.
> Therefore personally I don't track IP addresses and I personally
> configure ssh to track host keys only.  I know that I am in the
> minority since the default is to do both.  But there is no way to help
> it as the IP address must change.  All we can do is try to minimize
> the amount of thrash by limiting the changes as much as possible.
> 
> When I test switching from the previous IP address to the new IP
> address using default ssh options I see behavior like this:
> 
> Perform action using ssh adds this known_hosts entry:
> 
>   hg.sv.gnu.org,208.118.235.72 ssh-rsa 
> AAAAB3NzaC1yc2EAAAABIwAAAIEAzFQovi+67xa+wymRz9u3plx0ntQnELBoNU4SCl3RkwSFZkrZsRTC0fTpOKatQNs1r/BLFoVt21oVFwIXVevGQwB+Lf0Z+5w9qwVAQNu/YUAFHBPTqBze4wYK/gSWqQOLoj7rOhZk0xtAS6USqcfKdzMdRWgeuZ550P6gSzEHfv0=
> 
> Switch IP address of hg.sv.gnu.org to new server 208.118.235.201.
> Perform another ssh action.
> 
>   Warning: Permanently added the RSA host key for IP address 
> '208.118.235.201' to the list of known hosts.
> 
> Now the known_hosts entries looks like this (intentionally using 
> HashKnownHosts=no
> so as to show the entries, otherwise we would just have the hash):
> 
>   hg.sv.gnu.org,208.118.235.72 ssh-rsa 
> AAAAB3NzaC1yc2EAAAABIwAAAIEAzFQovi+67xa+wymRz9u3plx0ntQnELBoNU4SCl3RkwSFZkrZsRTC0fTpOKatQNs1r/BLFoVt21oVFwIXVevGQwB+Lf0Z+5w9qwVAQNu/YUAFHBPTqBze4wYK/gSWqQOLoj7rOhZk0xtAS6USqcfKdzMdRWgeuZ550P6gSzEHfv0=
>   208.118.235.201 ssh-rsa 
> AAAAB3NzaC1yc2EAAAABIwAAAIEAzFQovi+67xa+wymRz9u3plx0ntQnELBoNU4SCl3RkwSFZkrZsRTC0fTpOKatQNs1r/BLFoVt21oVFwIXVevGQwB+Lf0Z+5w9qwVAQNu/YUAFHBPTqBze4wYK/gSWqQOLoj7rOhZk0xtAS6USqcfKdzMdRWgeuZ550P6gSzEHfv0=
> 
> Switching the IP address back to the previous 208.118.235.72 causes no
> key warnings as the hg.sv.gnu.org,208.118.235.72 is a consistent state
> for that action.

Indeed, that's my experience with other server migrations where the keys
are preserved. Before I saw the thread about the old keys being cloned
to the new server, this looked to me like a case where it was a new
server with completely new keys.

> Well...  Just removing the old entry and trying again seems to defeat
> the anti-MITM checks here.  And once the entry has been deleted if it
> is a MITM attack then you have lost the original key to verify the
> previously known real host.  Let's say I am on a laptop at an airport
> and I really am being attacked by having a MITM try to spoof things.
> Let's say I stop doing anything at that point and get on the airplane
> and travel.  I maintain my known_hosts with the previously verified
> keys.  I travel elsewhere.  I connect again.  If I have avoided the
> attack then the verified connectivity now works without changing
> anything.

Of course. In this case I was on my regular network and a change was
fully expected. I could have kept the previous entry until I got
definitive confirmation of the new server keys. Either way, I did not
accept the host key until I got independent confirmation over a separate
(preferably signed in some way) channel, which is how I would have added
a new server in the first place.

> Since your behavior differs from my testing with the default options
> (and with my CheckHostIP=no case) I can only assume that once again
> the smallest of details is causing a lot of, in this case, noise.  I
> don't know why your case is different.  Perhaps you can debug your
> case to root cause?  It needs your access to your ssh version, config
> and known_hosts state.

Yes, I can reproduce this on a separate machine from a clean slate (with
CheckHostIP=yes).

First, assume I am a user who has only ever logged in to the old server:

    $ rm -rf ~/.ssh/known_hosts
    $ ssh -V
    OpenSSH_7.4p1 Debian-5, OpenSSL 1.0.2j  26 Sep 2016

  ## first connect to hg.savannah.gnu.org bound to the old server

    $ ssh -o FingerprintHash=md5 address@hidden true
    The authenticity of host 'hg.savannah.gnu.org (208.118.235.72)' can't be 
established.
    RSA key fingerprint is MD5:80:5a:b0:0c:ec:93:66:29:49:7e:04:2b:fd:ba:2c:d5.
    Are you sure you want to continue connecting (yes/no)? yes
    Warning: Permanently added 'hg.savannah.gnu.org,208.118.235.72' (RSA) to 
the list of known hosts.
    You tried to execute: true
    Sorry, you are not allowed to execute that command.
    ## I'll cut these lines from subsequent examples, assume success

  ## ok, time passes, dns now points at the new server

    $ ssh -o FingerprintHash=md5 address@hidden true
    Warning: Permanently added the RSA host key for IP address 
'208.118.235.201' to the list of known hosts.
    ...

Ok, this works as you described it should.

Now pretend that I have used the old server, but about a month ago I was
asked to test the new server under a different name:

    $ rm -rf ~/.ssh/known_hosts
    $ ssh -V
    OpenSSH_7.4p1 Debian-5, OpenSSL 1.0.2j  26 Sep 2016

  ## first connect to hg.savannah.gnu.org bound to the old server

    $ ssh -o FingerprintHash=md5 address@hidden true
    The authenticity of host 'hg.savannah.gnu.org (208.118.235.72)' can't be 
established.
    RSA key fingerprint is MD5:80:5a:b0:0c:ec:93:66:29:49:7e:04:2b:fd:ba:2c:d5.
    Are you sure you want to continue connecting (yes/no)? yes
    Warning: Permanently added 'hg.savannah.gnu.org,208.118.235.72' (RSA) to 
the list of known hosts.
    ...

  ## now connect to the new server, but not as the same name

    $ ssh -o FingerprintHash=md5 address@hidden true
    The authenticity of host 'hg0.savannah.gnu.org (208.118.235.201)' can't be 
established.
    ECDSA key fingerprint is 
MD5:65:b8:1c:2f:82:7c:0e:39:e1:4a:63:f2:13:10:e8:9c.
    Are you sure you want to continue connecting (yes/no)? yes
    Warning: Permanently added 'hg0.savannah.gnu.org,208.118.235.201' (ECDSA) 
to the list of known hosts.
    ...

  ## now with the old name pointing to the new server

    $ ssh -o FingerprintHash=md5 address@hidden true
    Warning: the RSA host key for 'hg.savannah.gnu.org' differs from the key 
for the IP address '208.118.235.201'
    Offending key for IP in /home/user/.ssh/known_hosts:4
    Matching host key in /home/user/.ssh/known_hosts:1
    Are you sure you want to continue connecting (yes/no)? no
    Host key verification failed.

A combination of (the default) CheckHostIP=yes, the fact that the new
server has keys that my client prefers over the old keys, and that I
must have connected to the new server before is what appears to be
triggering this. By the way, answering "yes" to this question does not
change known_hosts at all, the user will be asked the same question on
every connection from this point on, until one or more entries are
manually removed from known_hosts.

I can confirm that CheckHostIP=no eliminates this warning and prompt,
and connects to the new server using the original RSA host key.

Alternatively, if I start over and always connect with
HostKeyAlgorithms=ssh-rsa, the stronger key is ignored and there is also
no warning and no prompt.

I also observe that, in the case of a seamless migration where only the
warning about the new IP address is shown but no questions are asked,
~/.ssh/known_hosts only ever contains the original ssh-rsa key, until
the entry is completely removed from the file.

For my part, I have what I need, we don't need to investigate this any
further, but maybe this is helpful to you, let me know if I can test
anything else.

-- 
mike