Need working example for openHAB MQTT over TLS

[tommycw10 (Tom)]

@edolis,

Thanks for this write up. Unfortunately I don’t think you are actually connecting via TLS.

• In your example, you create the CA certs, then you created server certs. (I think) This would allow the broker (mosquitto server) to send to the cert to the openHAB client and the openHAB client to remember that cert after the first connection. This allows the client to “verify” the host (this is what the cert pining stuff is in OH), but does not actually create a TLS connection.

• You never created a client cert for openHAB

• You never import any certs into openHAB

• You have a username/password in your openHAB client configuration. TLS connections do not use username/passwords at all, they verify the login via the certs that you never created.

For example: using the command line clients and connecting to a broker I have set to accept both plain and TLS connections:

My /etc/mosquitto/conf.d/mosquitto.conf

listener 1883
allow_anonymous false
password_file /etc/mosquitto/passwd

listener 8883
#ssl settings
cafile /etc/mosquitto/ca_certificates/ca.crt
keyfile /etc/mosquitto/certs/server.key
certfile /etc/mosquitto/certs/server.crt
#client certifcate settings
require_certificate true
use_identity_as_username true

First, I show that if I specify no credentials, the broker rejects the connection (note, in your example config of mosquitto, you don’t specify allow_anonymous false so I suspect anyone can connect to your broker on port 1883 with no credentials at all!)

mosquitto_pub -h 192.168.1.200 -p 1883 -i "Test-Client" -t "testtopic/test" -m 'foobbar'
Connection Refused: not authorised.
Error: The connection was refused.

Next, I show an unencrypted connection to my broker on port 1883 using uname/passwd authentication

mosquitto_pub -h 192.168.1.200 -p 1883 -i “Test-Client” -u foo -P bar -d -t “testtopic/test” -m ‘foobbar’
Client Test-Client sending CONNECT
Client Test-Client received CONNACK
Client Test-Client sending PUBLISH (d0, q0, r0, m1, ‘testtopic/test’, … (7 bytes))
Client Test-Client sending DISCONNECT

In my mosquitto log I see:

1609602216: New connection from 192.168.1.50 on port 1883.
1609602216: New client connected from 192.168.1.50 as Test-Client (c1, k60, u'openhabian').
1609602216: No will message specified.
1609602216: Sending CONNACK to Test-Client (0, 0)
1609602216: Received PUBLISH from Test-Client (d0, q0, r0, m0, 'testtopic/test', ... (7 bytes))
1609602216: Received DISCONNECT from Test-Client

Lastly is a TLS connection to port 8883 which uses no uname/passed at all

mosquitto_pub -h 192.168.1.200 -p 8883 -i "Test-Client" --cafile /home/tom/ssl/ca.crt --cert /home/tom/ssl/client.crt  --key /home/tom/ssl/client.key -d -t "testtopic/test" -m 'foobbar'

In my mosquitto log I see:

1609601153: New connection from 192.168.1.50 on port 8883.
1609601153: New client connected from 192.168.1.50 as Test-Client (c1, k60, u'Heyu').
1609601153: No will message specified.
1609601153: Sending CONNACK to Test-Client (0, 0)
1609601153: Received PUBLISH from Test-Client (d0, q0, r0, m0, 'testtopic/test', ... (7 bytes))
1609601153: Received DISCONNECT from Test-Client

BTW, to answer your questions, mosquitto logs are configured by you in

/etc/mosquitto/mosquitto.conf 

with the line:

log_dest file /var/log/mosquitto/mosquitto.log

So for me, to watch the log in real time, I use

tail -f /var/log/mosquitto/mosquitto.log

[edolis]

you are correct - I misunderstood SSL for TSL.Now I understand your issue way better.
Let me work on it and I’ll get back to you, I am also interested in TSL (now :D)
BTW just to understand, did you already try the instructions found here mosquitto-tls man page | Eclipse Mosquitto?

[tommycw10 (Tom)]

@edolis

BTW just to understand, did you already try the instructions found here mosquitto-tls man page | Eclipse Mosquitto?

This link provides instructions on how to generate certs and set up a mosquitto broker for TLS. Mine is already configured and accepting TLS connections for many clients.

What I need is instructions for configuring the openHAB client to connect via TLS.

Thanks for your help!

[edolis]

and working on the two parameters of the broker bridge
certificate: CAB1697C204DE83771A3511E2137347899D4D3A4AB494C2725A151795CBB17F0
publickey: AE9E2E08D4C0E2EFC0FB7C6E69A07F0415DA6913B0EA4C9CC4B9F8AE4934DCDC
did not help, right? Because those are the only places I see where you could work with certificates.

[tommycw10 (Tom)]

This cannot be configured via paperUI. You need to import the keys into a java keystore and then modify the openhab start up script to point to that keystore.

In my original post, I point to the tutorial that shows how to do this. Unfortunately is is out of date and no longer works, I cannot seem to figure out how to make it actually work:

MQTT Binding and SSL

MQTT Binding and SSL Purpose Until the MQTT binding 2.0 comes along, we have to do with the 1.9 version. Unfortunately this binding, despite having the ssl option is not able to implement it. I will show how to create your own ssl certificate and key, how to use keytool to create a keystore and a truststore for private and public keys for Java and finally how to modify the OpenHAB configuration to load these keys to enable ssl. Prerequisites The MQTT Binding is installed and running openHAB is …

[TRS-80]

I only set up certs (“real” ones, from Let’s Encrypt, for external facing services) however my understanding is that the certs only reside on the server?

In this scenario (I also use MQTT) isn’t OpenHAB the “client” and therefore should not need any certs?

Or maybe “real” PKI / TLS (i.e., httpS) work differently than “self-signed” certs. As I realize you are trying to do the latter here.

[tommycw10 (Tom)]

@TRS-80 [Love the uname BTW] - There are certs on the MQTT broker as well as client certs.

I have been under the belief that both client and server (broker) were needed for encrypted communication (TLS). What I think I may be coming to wonder is if the broker’s certs are actually used to encrypt the communication. Then, similar to ssh, you can log in via uname/passwd or use a cert on the client. I’m thinking that is what @edolis may have accomplished, encrypted communication but uname/passwd authentication allowed rather than using a cert for authentication, but to be honest I’m not sure.

If what I said above is true, I guess what I am trying to understand is how to import my certs into the openHAB MQTT client to use for authentication to the MQTT broker over TLS.

[edolis]

@ tommycw10, I think there must be a way to use no uname\pwd and have just client cert to take care of comm.
I have been trying to reproduce your environment for knowledge sake and at the moment I’m drained out of energies, everything is just too much at the moment and SSL things keep piling up the more I dig into that big pile. Most I accomplished is this for the time being

I tried to reproduce the set up and I still cannot make mosquitto TLS work (so I am at least a step behind you)

pi@raspi00:/etc/mosquitto/newSSL $ sudo mosquitto_pub -h 192.168.1.220 -p 8883 -i “Test-Client” --cafile /etc/mosquitto/newSSL/ca.crt --cert /etc/mosquitto/newSSL/client.crt --key /etc/mosquitto/newSSL/client.key -d -t “testtopic/test” -m ‘foobbar_TSL’
Enter PEM pass phrase:
Client Test-Client sending CONNECT
Error: host name verification failed.
OpenSSL Error[0]: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed
Error: A TLS error occurred.

which would mean the password of the cert is ok but the validation fails (why? dunno)

this goes in pair with the OH broker

io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

and on mosquitto side I get

1609616770: New connection from 127.0.0.1 on port 8883.
1609616770: OpenSSL Error[0]: error:1417C0C7:SSL routines:tls_process_client_certificate:peer did not return a certificate
1609616770: Socket error on client , disconnecting.

which sounds like yours.
Now I was reading here c++ - OpenSSl Error: peer did not return a certificate - Stack Overflow and the part which sounds interesting is

Part of requesting that the client send its certificate, via the CertificateRequest message, is including, in that request, a list of the CAs that the server trusts (i.e. that it will use for verifying any client-provided certificates). Servers may trust multiple different CAs, and a given client may have multiple different certificates to choose from. The CertificateRequest message thus contains a list of the CAs, and the client will then choose which of its client certificates matches up with those CAs.
Thus to configure OpenSSL with that list of CAs used for verifying client certs, you would use the SSL_CTX_load_verify_locations() function, and point it at a PEM file of concatenated certificates, and/or a directory of trusted certificates (hashed using the OpenSSL c_rehash utility). Without this, your server may be sending the CertificateRequest message, but with an empty list of CAs, and thus the client does not/cannot choose which of its client certs to send.

This to say that I think, let’s say I’ve got a gut feeling the the error does not lies with the sw implementation of Mosquitto or openhab, but might be an error raised at lower level by the functions of OpenSSL (I should to check the source code, the impression is that the log just tallies the response of openSSL which cannot handle the communication.
Now, the question would be why? As I have been reading around could be either this CA not being tranferret to the client who does not know what to reply or a silly misalignment in the parameters used ny openssl or something different in java11 maybe?
I’m too steamed now, but I would go and chech the doc mentioned in the original mosquitto.conf (the one huge, all commented out, it’s the default for the Win installation - to see if it gives some clue I mean this one

#-----------------------------------------------------------------
certificate based SSL/TLS support
#-----------------------------------------------------------------
#The following options can be used to enable certificate based SSL/TLS support
#for this listener. Note that the recommended port for MQTT over TLS is 8883,
#but this must be set manually.
#See also the mosquitto-tls man page and the “Pre-shared-key based SSL/TLS
support” section. Only one of certificate or PSK encryption support can be
#enabled for any listener.
#Both of certfile and keyfile must be defined to enable certificate based
tls encryption.
#Path to the PEM encoded server certificate.
#certfile
#Path to the PEM encoded keyfile.
#keyfile
if you wish to control which encryption ciphers are used, use the ciphers
and so on.

[tommycw10 (Tom)]

I think the client is not returning a certificate because you’ve never imported a certificate into openHAB for it to return.

[edolis]

I thought the keystore defined in the subfolder of hopenhab would have taken care of that. Maybe I’m wrong

-Dcom.ibm.ssl.trustManager=SunX509
-Dcom.ibm.ssl.keyManager=SunX509
-Dcom.ibm.ssl.contextProvider=SunJSSE
-Dcom.ibm.ssl.keyStore=/etc/keystore.jks
-Dcom.ibm.ssl.keyStorePassword=keystore_password
-Dcom.ibm.ssl.keyStoreType=JKS
-Dcom.ibm.ssl.keyStoreProvider=SUN
-Dcom.ibm.ssl.trustStore=/etc/truststore.jks
-Dcom.ibm.ssl.trustStorePassword=truststore_password
-Dcom.ibm.ssl.trustStoreType=JKS
-Dcom.ibm.ssl.trustStoreProvider=SUN

Next step would be looking at the certificate and public key fields I mentioned above in the text config of the broker. Need to see if I can extract the key and the cert and push it there as a string

Mh actually a good point. Iremember creating the keystore but not importing anything. I-ll have a look at that

[tommycw10 (Tom)]

Yes, you need to import your self-signed x509 keys into the java keystore in /etc/keystore.jks. I haven’t been able to successfully do this yet.

In paperUI, the Certificate Hash and Public Key Hash are not user writable fields, they are related to these:

It is a security mechanism to try to prevent MitM attacks. You get a hash from the broker the first time you connect and any changes to this hash would signal the broker you are trying to connect to has changed.

[tommycw10 (Tom)]

OK, so I just did a test here to try to prove out if you can connect to an MQTT broker with only username/passwd credentials.

So, first I changed my mosquitto config to remove the

require_certificate true

and add a pointer to the password file

 password_file /etc/mosquitto/passwd

so my conf now looks like this:

per_listener_settings true
  
listener 1883
allow_anonymous false
password_file /etc/mosquitto/passwd
log_type error
log_type notice
log_type information
log_type debug

listener 8883
#ssl settings
cafile /etc/mosquitto/ca_certificates/ca.crt
keyfile /etc/mosquitto/certs/server.key
certfile /etc/mosquitto/certs/server.crt
#client certifcate settings
#require_certificate true
password_file /etc/mosquitto/passwd
use_identity_as_username true

So, I attempted to connect with a uname/passwd but to port 8883 and saw the following:

mosquitto_pub -h 192.168.1.200 -p 8883 -i "test" -u uname -P "passwd'" -d -t "test/testing" -m 'foobar'

I got the following in my mosquitto log:

1609639233: Client connection from 192.168.1.50 failed: error:1408F10B:SSL routines:ssl3_get_record:wrong version number.

Googling for this error, I found this:

To get mosquitto_pub to attempt to start a SSL connection you need to provide either --cafile or --capath that points to the location of the CA certificates to verify the broker.

Without these options neither mosquitto_pub or mosquitto_sub will not attempt to start a SSL session and instead try and connect with a normal unencrypted MQTT connection

From here: openssl - mosquitto_pub gives the following error: 1408F10B: SSL routines: ssl3_get_record: wrong version number - Stack Overflow

So, since there is no global repo for self signed certs, the only way to communicate with the broker is to have a CA cert that was sued to sign the server’s cert. Makes sense.

OK, so then I tried only supplying the CA cert but still using name/passwd for authentication:

 mosquitto_pub -h 192.168.1.200 -p 8883 -i "test" --cafile /home/tom/ssl/ca.crt -u uname -P "passwd'" -d -t "test/testing" -m 'foobar'

Now I get the same error you are seeing in openHAB in my mosquitto log:

1609639323: OpenSSL Error: error:1417C0C7:SSL routines:tls_process_client_certificate:peer did not return a certificate.

Googling around, I don’t see much information about logging into TLS MQTT broker with a username/passwd, so I think my hypothesis was incorrect. Even if this was correct, you would still need to import at least the CA cert into the java keystore. So seems like we are back at the same step. We need to import rhe CA cert, the client cert and key into the java keystore in order to get openHAB to connect to a mosquitto broker over TLS.

[edolis]

just a quickie.
I can follow your steps and also I cannot get mosquitto_pub to work.

pi@raspi00:~ $ mosquitto_pub -h 192.168.1.220 -p 8883 -i “test” -u edolis --capath /home/pi/newSSL -P ****-d -t “test/testing” -m ‘foobar’
Client test sending CONNECT
OpenSSL Error[0]: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed
Error: A TLS error occurred.

but this to me seems a problem of mosquitto_pub. If I use a windows client things work for me:

and also OH fares good

So it looks like it’s a problem of mosquitto_pub. Frankly, not even sure I’ll spend time to try to understand why…
here is my mosquitto.conf

#listener 1883
#allow_anonymous false
log_type all
log_type error
log_type warning
#log_type notice
#log_type information
if set to true, client connection and disconnection messages will be included
#in the log.
connection_messages true
log_timestamp true
allow_anonymous false
password_file /etc/mosquitto/passwd
ssl certificates
listener 1883
listener 8883
certfile /etc/mosquitto/certs/server.crt
keyfile /etc/mosquitto/certs/server.key
cafile /home/pi/ca.crt
client certifcate settings
#require_certificate false
#use_identity_as_username true

EDIT:
I have investigated a little why mosquitto_sub fails

pi@raspi00:~ $ mosquitto_sub -h 192.168.1.220 -p 8883 -u edolis -P *** -t test --cafile /home/pi/newSSL/ca.crt
Error: A TLS error occurred.

in the log I read

1609669023: New connection from 192.168.1.220 on port 8883.
1609669023: OpenSSL Error[0]: error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
1609669023: Socket error on client , disconnecting.

so it’s a tlsv1 related error. Interesting

@tommycw10

We need to import rhe CA cert,

well, I did that using dpkg-reconfigure ca-certificates pi 3 - Entrusted Certificates installation - Raspberry Pi Stack Exchange
as for the keystore, I followed this Import an SSL certificate and private key
So for me it was

pi@raspi00:/etc $ sudo openssl pkcs12 -export -in edolis.client.crt -inkey edolis.client.key -name openHabClient -out edolis.client-PKCS-12.p12
Enter pass phrase for edolis.client.key:
Enter Export Password:
Verifying - Enter Export Password:
pi@raspi00:/etc $ sudo keytool -importkeystore -deststorepass xxxxx -destkeystore edolis.keystore.jks -srckeystore edolis.client-PKCS-12.p12 -srcstoretype PKCS12
Importing keystore edolis.client-PKCS-12.p12 to edolis.keystore.jks…
Enter source keystore password:
Entry for alias openhabclient successfully imported.
Import command completed: 1 entries successfully imported, 0 entries failed or cancelled

but still no joy

1609671308: OpenSSL Error[0]: error:1417C0C7:SSL routines:tls_process_client_certificate:peer did not return a certificate
1609671308: Socket error on client , disconnecting.

Now, thinking out of the box, I tried using the win client MQTT explorer, where I can manually load ca.crt, client.crt and client.key. So I can be sure certificates are loaded in the client properly.
Trying to connect I get, client side:

mosquitto side I get absolutely nothing, as if it did not get through something or the certificates have issues. and need to figure out what.
And if I cannot make a client external to mosquitto work TSL with certificates only, I think there is little point banging owns head against openhab.

[edolis]

looks like also the core example from Creating and Using Client Certificates with MQTT and Mosquitto does not work on raspberry for me, so I asked Steve who’s an authority on mosquitto and let’s see what he says

[edolis]

update for anyone interested
a) if your TSL certificate-only set-up does not seem to work, verify the DN of the server certificate (running Mosquitto) is matching the hostname of the server.
b) I think probably TSL certificate-only cannot be run on OH3 because there is no way to specify the protocol (ssl instead of http/https). Posted a bug/feature request for this
c) a workaround to make compatible the use of TSL certificate based MQTT server and OH3 with no TSL implemented could be opening a new listener on a specific port and using ACL to restrict access only to the user of the openHAB instance. And make the username/password arcane enough to reduce the risk to acceptable level. Need to try this.

[tommycw10 (Tom)]

I’m confused about this. According to Wikipedia, MQTT is a protocol that runs over TCP/IP, which can use TLS as an encryption layer if desired. I don’t see anything here about MQTT over http/https.

It also lists both MQTT and HTTP/HTTPS both as application layer protocols, so it seems there would be no such thing as MQTT over HTTP/HTTPS.

A quick google search for “mqtt over http” shows a lot of MQTT vs HTTP hits, but I don’t see anything about MQTT using HTTP. Maybe I’m misunderstanding. Can you explain a bit more?

[tommycw10 (Tom)]

@edolis

a workaround to make compatible the use

Another work around is to have two MQTT brokers - one is your main broker and the other is on the machine running OH3 that cannot connect to the main broker over MQTT/TLS. Make the OH3 instance connect to its localhost MQTT broker and set up a bridge between the two brokers that uses TLS. I haven’t dont it yet, but I’ve read about this bridging concept a few places. Essentially the two brokers keep each over in sync on all topics.

[edolis]

@tommycw10

Can you explain a bit more?

Honestly, not quite. I have not looked into it. I’m not going tofight the theoretical side too - I just see that the OH2 implementation had a “black box” ssl stamp which made it work, that’s enough for me. If in the underlying implementation they use really that ssl or any other protocol replaced during the coding chain, I don’t really mind (even if it’s messy) . Too many variables for me- coding, Java, openSSL.

@tommycw10

Another work around is to have two MQTT brokers

I considered that, using bridging. I already use bridging to a MQTT server over the internet.
But two brokers running on the same host? Seems a sure way to get a complicated life to me.
Unless you have two machines, which I haven’t got - as you seem t suggest. In any case, the broker communicating with OH has exactly the same exposure.
Yes, would be a safer approach for DOS attacks: the broker exposed to OH just dies, the other keeps going.
But as you know, you can restrict access to a specific listener to localhost. So having a OH connection with arcane username/encripted pwd secured with SSL using a CA certificate and accepted only from client localhost is safer than a sterile room to me. If your localhost is compromised, everything is lost.

[tommycw10 (Tom)]

Sorry - yes - this thread is about using TLS to encrypt MQTT communication over ethernet between two servers. There is little to no reason to use TLS to connect to a broker from a client on the same machine (localhost). You can easily configure your broker to accept connection from localhost with no / or password authentication, but require TLS for anything coming from outside of localhost.