Need working example for openHAB MQTT over TLS

@ tommycw10, I think there must be a way to use no uname\pwd and have just client cert to take care of comm.
I have been trying to reproduce your environment for knowledge sake and at the moment I’m drained out of energies, everything is just too much at the moment and SSL things keep piling up the more I dig into that big pile. Most I accomplished is this for the time being

I tried to reproduce the set up and I still cannot make mosquitto TLS work (so I am at least a step behind you)

pi@raspi00:/etc/mosquitto/newSSL $ sudo mosquitto_pub -h 192.168.1.220 -p 8883 -i “Test-Client” --cafile /etc/mosquitto/newSSL/ca.crt --cert /etc/mosquitto/newSSL/client.crt --key /etc/mosquitto/newSSL/client.key -d -t “testtopic/test” -m ‘foobbar_TSL’
Enter PEM pass phrase:
Client Test-Client sending CONNECT
Error: host name verification failed.
OpenSSL Error[0]: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed
Error: A TLS error occurred.

which would mean the password of the cert is ok but the validation fails (why? dunno)

this goes in pair with the OH broker

io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

and on mosquitto side I get

1609616770: New connection from 127.0.0.1 on port 8883.
1609616770: OpenSSL Error[0]: error:1417C0C7:SSL routines:tls_process_client_certificate:peer did not return a certificate
1609616770: Socket error on client , disconnecting.

which sounds like yours.
Now I was reading here c++ - OpenSSl Error: peer did not return a certificate - Stack Overflow and the part which sounds interesting is

Part of requesting that the client send its certificate, via the CertificateRequest message, is including, in that request, a list of the CAs that the server trusts (i.e. that it will use for verifying any client-provided certificates). Servers may trust multiple different CAs, and a given client may have multiple different certificates to choose from. The CertificateRequest message thus contains a list of the CAs, and the client will then choose which of its client certificates matches up with those CAs.
Thus to configure OpenSSL with that list of CAs used for verifying client certs, you would use the SSL_CTX_load_verify_locations() function, and point it at a PEM file of concatenated certificates, and/or a directory of trusted certificates (hashed using the OpenSSL c_rehash utility). Without this, your server may be sending the CertificateRequest message, but with an empty list of CAs, and thus the client does not/cannot choose which of its client certs to send.

This to say that I think, let’s say I’ve got a gut feeling the the error does not lies with the sw implementation of Mosquitto or openhab, but might be an error raised at lower level by the functions of OpenSSL (I should to check the source code, the impression is that the log just tallies the response of openSSL which cannot handle the communication.
Now, the question would be why? As I have been reading around could be either this CA not being tranferret to the client who does not know what to reply or a silly misalignment in the parameters used ny openssl or something different in java11 maybe?
I’m too steamed now, but I would go and chech the doc mentioned in the original mosquitto.conf (the one huge, all commented out, it’s the default for the Win installation - to see if it gives some clue I mean this one

#-----------------------------------------------------------------
certificate based SSL/TLS support
#-----------------------------------------------------------------
#The following options can be used to enable certificate based SSL/TLS support
#for this listener. Note that the recommended port for MQTT over TLS is 8883,
#but this must be set manually.
#See also the mosquitto-tls man page and the “Pre-shared-key based SSL/TLS
support” section. Only one of certificate or PSK encryption support can be
#enabled for any listener.
#Both of certfile and keyfile must be defined to enable certificate based
tls encryption.
#Path to the PEM encoded server certificate.
#certfile
#Path to the PEM encoded keyfile.
#keyfile
if you wish to control which encryption ciphers are used, use the ciphers
and so on.