Need to subscribe to MQTT messages from an external server. Do I need a broker?

  • Platform information:
    • Hardware: Raspberry Pi Model 4b
    • OS: openHABian 1.6.5
    • Java Runtime Environment: Java v11
    • openHAB version: openHAB 3.4.2

I am trying to configure OpenHAB 3 (running on OpenHABian on RPi 4b) to subscribe to MQTT topics from an external AWS server. I was expecting to be able to do this by just installing the MQTT binding, and configuring the endpoint, client_id, cert hash, and public key hash. However, I have not been able to get this to work. I keep getting timeout exceptions or “java.nio.channels.closedChannelException” exceptions. I’ve tried creating a java keystore and truststore, as explained by the following thread:

But so far, no luck. I’ve also tried switching between using MQTT version 3 and version 5, as well as toggling between trying TCP and webSockets protocols, since I’m not 100% sure which of those AWS is using. But again, no luck. The openHAB log has not been particularly helpful, as it doesn’t list any meaningful error messages. It just keeps repeating:

2023-03-23 14:27:00.348 [INFO ] [.reconnect.PeriodicReconnectStrategy] - Try to restore connection to '[AWS_endpoint]'. Next attempt in 60000ms
2023-03-23 14:27:00.358 [INFO ] [.transport.mqtt.MqttBrokerConnection] - Starting MQTT broker connection to '[AWS_endpoint]' with clientid raspberry_pi_test

I’m trying to avoid having to install and run a separate MQTT broker just for this purpose. We’re going to have other services and functions installed on this pi, so we want to conserve as many of the limited resources as possible, and running a dedicated MQTT broker as a middle-man between the AWS broker and OpenHAB seems like unnecessary overkill.

Can I subscribe to MQTT topics without having to install a whole separate broker (such as Mosquitto)? If so, how do I configure the MQTT binding to allow that?
Thanks in advance.

Put the MQTT binding into debug or trace level logging to get more information about what the binding it doing.

There needs to be only one broker in the system.

You need to have details about the broker running on AWS and what it requires to connect to it. That is where the username, password, certificates, etc are dictated.

rlkoshak
You need to have details about the broker running on AWS and what it requires to connect to it. That is where the username, password, certificates, etc are dictated.

AWS provided me with an endpoint, client id, root certificate, certificate hash, and public key. AWS did not provide a username or password. I have input the endpoint, client id, certificate hash (prefixed with “SHA-256:”), and public key (also prefixed with “SHA-256:”) into the appropriate fields in the MQTT bridge config within the Paper UI.

As for the log info, I set everything to debug so that I would make sure I don’t miss anything. I get the following messages about trying to connect:

2023-03-23 16:18:11.646 [DEBUG] [onfig.core.ConfigDescriptionRegistry] - No config description found for 'thing:mqtt:broker:mqtt_broker_aws', using alias 'thing-type:mqtt:broker' instead
2023-03-23 16:18:11.652 [DEBUG] [nfig.core.status.ConfigStatusService] - There is no config status provider for entity mqtt:broker:mqtt_broker_aws available.
2023-03-23 16:18:19.104 [INFO ] [.reconnect.PeriodicReconnectStrategy] - Try to restore connection to '[AWS_endpoint]'. Next attempt in 60000ms
2023-03-23 16:18:19.115 [INFO ] [.transport.mqtt.MqttBrokerConnection] - Starting MQTT broker connection to '[AWS_endpoint]' with clientid rasberri_pi_test

There are also other log statements showing possible other MQTT services that are trying to subscribe to topics:

2023-03-23 16:16:10.748 [TRACE] [g.mqtt.handler.AbstractBrokerHandler] - Subscribed org.openhab.binding.mqtt.espmilighthub.internal.discovery.EspMilightHubDiscoveryService@2dbcdc to discovery topic milight/states/# on broker mqtt:broker:mqtt_broker_aws
2023-03-23 16:16:10.972 [TRACE] [g.mqtt.handler.AbstractBrokerHandler] - Subscribed org.openhab.binding.mqtt.homeassistant.internal.discovery.HomeAssistantDiscovery@16a2cd7 to discovery topic homeassistant/# on broker mqtt:broker:mqtt_broker_aws
2023-03-23 16:16:11.050 [TRACE] [g.mqtt.handler.AbstractBrokerHandler] - Subscribed org.openhab.binding.mqtt.homie.internal.discovery.Homie300Discovery@ac224b to discovery topic +/+/$homie on broker mqtt:broker:mqtt_broker_aws

But yet, when I bring up the Paper UI, it still shows that the MQTT bridge is OFFLINE with “java.nio.channels.ClosedChannelException” error. But I don’t find any references to that error in the log.

So I have no idea what is going on here…

It seems really odd that they would provide an MQTT broker without client password. I guess that client certificate hash is used for authentication? I don’t think that the MQTT binding supports certificate authentication for client login.

Note the description of “certificate hash” in the Thing config:

If certificatepin is set this hash is used to verify the connection. Clear to allow a new certificate pinning on the next connection attempt. If empty will be filled automatically by the next successful connection. An example input would be SHA-256:83F9171E06A313118889F7D79302BD1B7A2042EE0CFD029ABF8DD06FFA6CD9D3.

That’s used to verify the connection, not authenticate to the broker.

You are running OH 2.5? Or do you mean MainUI?

All I could suggest at this point is to forget OH for a moment. Work with MQTT Explorer or mqtt_subscribe to figure out how to properly connect to this AWS MQTT Broker. Once you figure out how to connect using those it should be more apparent how to configure the MQTT Broker Thing in OH to connect.

1 Like

This is perfect advice. There are several layers between your local network and whatever service you are running on AWS. Are you running AWS IoT? read Security in AWS IoT - AWS IoT Core

There may be several layers of access control between OH mqtt client and the AWS mqtt broker

In case you need a bridge … this may help:

rlkoshak
All I could suggest at this point is to forget OH for a moment. Work with MQTT Explorer or mqtt_subscribe to figure out how to properly connect to this AWS MQTT Broker. Once you figure out how to connect using those it should be more apparent how to configure the MQTT Broker Thing in OH to connect.

The only username and password that I have is the credentials that I would use to log into the AWS console, but I doubt that is what the MQTT client would use. I have been able to successfully post messages to a topic via python script, and see those messages on AWS by using the instructions provided here:

The above method only required providing the set of certificate files and endpoint. There was no username or password in the python script. The code that I’ve used to successfully publish a message is as follows:

import time as t
import json
import AWSIoTPythonSDK.MQTTLib as AWSIoTPyMQTT
import sys

# Define ENDPOINT, CLIENT_ID, PATH_TO_CERTIFICATE, PATH_TO_PRIVATE_KEY, PATH_TO_AMAZON_ROOT_CA_1, TOPIC
ENDPOINT = "[AWS_endpoint]"
CLIENT_ID = "rasberri_pi_test"
PATH_TO_CERTIFICATE = "/home/openhabian/certificates/aws/[cert_hash]-certificate.pem.crt"
PATH_TO_PRIVATE_KEY = "/home/openhabian/certificates/aws/[cert_hash]-private.pem.key"
PATH_TO_AMAZON_ROOT_CA_1 = "/home/openhabian/certificates/aws/AmazonRootCA1.pem"
TOPIC = "test/testing"


def pushMqttToAws(topic: str, did: str, key: str, value: int):
	#print('pushMqttToAws() invoked with topic=' + str(topic) + ', did=' + str(did) + ', key=' + str(key) + ', value=' + str(value))
	
	# Configure MQTT
	myAWSIoTMQTTClient = AWSIoTPyMQTT.AWSIoTMQTTClient(CLIENT_ID)
	myAWSIoTMQTTClient.configureEndpoint(ENDPOINT, 8883)
	myAWSIoTMQTTClient.configureCredentials(PATH_TO_AMAZON_ROOT_CA_1, PATH_TO_PRIVATE_KEY, PATH_TO_CERTIFICATE)

	# Post an MQTT message to AWS
	myAWSIoTMQTTClient.connect()
	#print('Begin MQTT Publish')
	message = { "DID": did, "Key": key, "Value": value }
	myAWSIoTMQTTClient.publish(TOPIC, json.dumps(message), 1) 
	#print("Published: '" + json.dumps(message) + "' to the topic: '" + topic + "'")
	t.sleep(0.1)
	#print('MQTT Publish End. Disconnecting.')
	myAWSIoTMQTTClient.disconnect()

As you can see, there is no username or password in the above code (unless it is stored within one of the cert files, or the AWSIoTPythonSDK.MQTTLib lib is using a default pw underneath the hood somewhere?).

rlkoshak
You are running OH 2.5? Or do you mean MainUI?

Oh is it not called “Paper UI” anymore? Didn’t realize the name had changed. I’m on OpenHAB 3.4.2 running on an OpenHABian install on an RPi 4b.

Anyway, does anyone know off-hand if the Mosquitto broker also requires a client username and password? I would hate to go to the trouble of installing and configuring Mosquitto, just to end up with the same problem. In the meantime, I guess I can also install and play around with MQTT Explorer and see if I find anything helpful there.

Note that I mentioned above that the MQTT binding does not support certificate based authentication. If that’s the only option, you’ll have to have some sort of bridge.

It wouldn’t be a bad thing to add certificate auth or even AWSIoT support to the binding but it’s not there now.

It’s not that the name changed, MainUI is a wholly new UI implemented from scratch with no lineage back to PaperUI. PaperUI only exited for OH 2. OH 3 has MainUI which is a combo of an end user interface and admin interface.

Requires? No, but even a sheltered Mosquitto instance on a LAN should have some sort of authentication.

That’s not how it works. It might help to go through an MQTT tutorial to really understand what the MQTT broker does. Even if Mosquitto did require and only support username/password authentication for it’s clients doesn’t matter. We know OH can connect to MQTT and do so with username and password. The problem is AWSIoT apparently doesn’t support username/password and only supports certificate login and OH does not support certificate based login.

But give the link @Wolfgang_S provided above Mosquitto does when configured as a bridge. So you’d set up Mosquitto as a bridge to AWSIoT using the tutorial above. Then you’d configure Mosquitto to listen and accept connection from openHAB, perhaps using username and password. The messages flow from OH → Mosquitto → AWSIoT → ??? and from AWSIoT → Mosquitto → OH.

rlkoshak
It wouldn’t be a bad thing to add certificate auth or even AWSIoT support to the binding but it’s not there now.

Yeah, especially if a platform as big as AWS is doing it this way. I imagine lots of people would use AWS for this sort of thing. It didn’t even occur to me to check the authentication methods prior to starting the project. I just saw that OpenHAB has an MQTT binding, so I assumed it would work. And AWS is the first platform that I’ve used for MQTT, so I had no idea that its certificate authentication method might be different than the standard practices.

Keep in mind that the “H” in “HAB” stands for “home”. A significant portion of OH users try to avoid the use of cloud services, not flock to try to use them nor demand their support. So there has not been a whole lot of demand for integration with paid cloud services like AWS IoT. And for home users, going through the trouble of creating and deploying client certificates for authentication is a huge pain.

Hi! Very insteresting issue. Did you found a way?

What is the standard practice …?
E.g. in your home you most probably use WLAN together with a user and password while in a company ( enterprise ) network again certificates on clients are used for authentication of your client.

@mpmainieri Sadly no. I could not get it to work with just the certificates. I’m going with the fallback plan of using Mosquitto as an in-between bridge, as suggested by @Wolfgang_S. I already have Mosquitto configured and talking to the cloud (via certs), so now I just have to figure out how to set it up so that it can forward those messages to OpenHAB.

Assuming that the AWS system is publishing data to your local Mosquitto broker, it should just be a case of subscribing to the same topics with openHAB, using the MQTT binding.

(As in, openHAB will effectively ‘pull’ that data from your Mosquitto broker because you’ve actively subscribed to a topic, rather than the broker ‘forwarding’ data to openHAB)

@hdr_jgrade

Out of curiousity, is there a specific reason that you’re using AWS for MQTT? I’m just wondering if there might be a path of less resistance, but that’s moot if you absolutely have to go with this approach.

@rpwong
I’m not married to AWS, specifically. It could be any cloud service, AWs is just the one I chose because I already have some professional experience with it. Basically, I’m playing around with the idea of aggregating data from multiple remote properties and doing some cloud-based analytics.

hafniumzinc
Assuming that the AWS system is publishing data to your local Mosquitto broker, it should just be a case of subscribing to the same topics with openHAB, using the MQTT binding.

Yep. I just had to look up how to set up Mosquitto as a bridge service (instead of as a server), and now it’s working!

4 Likes

Gotcha. I’ve been interested in Beebotte, but haven’t found a good reason to actually do anything with it.

https://beebotte.com/