To the best of my knowledge, openHAB cannot connect to the mosquitto server using a secure connection. What I have done is configure multiple ports on mosquitto, one secure and the other not secure. I host my mosquitto server in AWS so I lock the connection to the non-secure port down by only allowing traffic from my home IP but the TLS port is open to the world.
Since you have them all running on one device, you could just set up two ports and only expose the TLS port and not have to worry about limiting the non TLS port.
You can configure port 1883 to listen with no TLS, and port 8883 to listen with TLS. Only allow Internet access to port 8883. Something like this in /etc/mosquitto/conf.d/mosquitto.conf: