[SOLVED] SSH Login to Karaf - Fails after update to 2.4.0 Build 1431

@mstormi

As you wrote

Check file access rights on your local private key ( ~/.ssh/identity.* ) and if the matching public key (still) is in /var/lib/openhab2/etc/keys.properties

I have nothing in ~/.ssh/identity beside a file in .ssh called known_hosts

Just rolled back to the previous VM snapshot to see what happens when I logged in to Karaf with the -v command.

I see that in the upgraded version I don’t get an SSH2_MSG_KEX_ECDH_REPLY it simply exit after the init.

debug1: sending SSH2_MSG_KEX_ECDH_INIT (Both version)
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY (Working version)

This is what I get: (This is the working version)

^[[A[root@openhab ~]# ssh -v -p 8101 openhab@localhost
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 58: Applying options for *
debug1: Connecting to localhost [::1] port 8101.
debug1: connect to address ::1 port 8101: Connection refused
debug1: Connecting to localhost [127.0.0.1] port 8101.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: Remote protocol version 2.0, remote software version SSHD-CORE-1.6.0
debug1: no match: SSHD-CORE-1.6.0
debug1: Authenticating to localhost:8101 as ‘openhab’
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: ecdh-sha2-nistp256
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha2-256 compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha2-256 compression: none
debug1: kex: ecdh-sha2-nistp256 need=32 dh_need=32
debug1: kex: ecdh-sha2-nistp256 need=32 dh_need=32
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ssh-rsa SHA256:eY73…JrqgoD400
debug1: Host ‘[localhost]:8101’ is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:2
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: keyboard-interactive,password,publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /root/.ssh/id_rsa
debug1: Trying private key: /root/.ssh/id_dsa
debug1: Trying private key: /root/.ssh/id_ecdsa
debug1: Trying private key: /root/.ssh/id_ed25519
debug1: Next authentication method: keyboard-interactive
Password authentication
Password:

can try to remove the existing entry from this file and try again ?

ssh-keygen -f "/root/.ssh/known_hosts" -R [localhost]:8101

It replaces the old known_host with an identical one…
And gives me the same result.

did you recently (not today) update OpenSSL and/or OpenSSH?

Maybe you have stumbled upon some kind of incompatibility… or the ssh keys on OH2 are “corrupted” ? not sure

did you check permissions in /var/lib/openhab2/ and subdirs?

This is my OH 2.4.0 Snapshot Build #1431 on a Debian Jessie:

root@homer:~# ssh -v openhab@localhost -p 8101
OpenSSH_6.7p1 Debian-5+deb8u7, OpenSSL 1.0.1t  3 May 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to localhost [::1] port 8101.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.7p1 Debian-5+deb8u7
debug1: Remote protocol version 2.0, remote software version SSHD-CORE-1.7.0
debug1: no match: SSHD-CORE-1.7.0
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-sha2-256 none
debug1: kex: client->server aes128-ctr hmac-sha2-256 none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: RSA xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx
debug1: Host '[localhost]:8101' is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:10
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: keyboard-interactive,password,publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /root/.ssh/id_rsa
debug1: Authentications that can continue: keyboard-interactive,password,publickey
debug1: Trying private key: /root/.ssh/id_dsa
debug1: Trying private key: /root/.ssh/id_ecdsa
debug1: Trying private key: /root/.ssh/id_ed25519
debug1: Next authentication method: keyboard-interactive
Password authentication
Password: 

That is the same I have running the previous working version.

I have tried to update the SSHD-Core-1.6.0 to the same you are running 1.7.0.

Still no luck.

You need to check the server side which is inside Karaf. Enable debug for org.apache.karaf and watch the outpu on SSH login attempts. Should give you a hint why the server side is dropping your attempt.

@mstormi

Ill try to do so.

wait… how did you try to update SSHD-Core?
If I am not completely wrong, this is the SSH Daemon used by Karaf.
OH 2.4.0 Build #1431 comes with SSHD-Core 1.7.0
You should be seeing the same

If not, it wouldn’t hurt to Clean cache & tmp

Yes the core was updated when I updated openhab.

Already cleaned Cache and tmp

@mstormi

Can you explain how to enable Karaf debug?

I wrote it in my post ?

I cannot find org.apache.karaf

You need to add lines to that file to contain org.apache.karaf. Come on it’s not that difficult.

1 Like

:slight_smile:

log4j2.logger.org_apache_karaf.level = DEBUG
log4j2.logger.org_apache_karaf.name = org.apache.karaf

I fixed the SSH login problem, by switching from the internal key format to a PEM format.

sed -i "s/hostKeyFormat = simple/hostKeyFormat = PEM/" /var/lib/openhab2/etc/org.apache.karaf.shell.cfg
openssl genrsa -out /var/lib/openhab2/etc/host.key 4096
ssh openhab@localhost -p 8101
RSA key fingerprint is xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx.

 sudo rm /var/lib/openhab2/etc/host.key

 sudo systemctl restart openhab2.service

$ ssh openhab@localhost -p 8101``

And now I can login with SSH

wait… this is an old story… (See: Karaf console)

so the simple (Karaf internal) key format does not work with OpenSSH_7.4p1, OpenSSL 1.0.2k-fips ?

If this is the case, we should capture this in an issue? wdyt @mstormi ?

The changing from internal format to PEM reissued a new key generation, I didn’t test if a new key generation would have fixed the issue.

1 Like

can you try this please? (if it’s not too much trouble)

Go back to internal key format in org.apache.karaf.shell.cfg, delete the existing PEM (OpenSSH) formated key in host.key and let OH2 generate a new one (with Karaf internal format) to see if you can still ssh to the console using the new one.