MQTT StateTopic Syntax

galileo · October 12, 2021, 7:01am

I have an existing OH3.1.0 install with Tasmota devices & Zigbee2MQTT Zigbee devices using MQTT OK.
I am trying to decode the MQTT output from a Meraki AP which has BLE & Wifi scanning radios. I have got the output from the Meraki arriving at OpenHAB, and I have confirmed the MQTT structure with an MQTT debugger.
I am struggling to come up with correct StateTopic as the Meraki appears to only support MQTT Topics in the two following formats:

Simple One Word Topic: eg meraki
Complex Topic String as follows: meraki/v1/mr/N_601793500207383877/E0:CB:BC:8A:A6:1B/ble/24:F1:28:01:8B:D7

I can decode the data with the “One Word” topic (but it not very useful as it is not device specific), but I have tried a number of attempts to get the StateTopic correct for the Complex Topic with no luck. I am assuming this is due to the non-alpha characters it contains.

I have the Meraki set to the Complex Topic string.

Here are my Thing/Channel examples:
Working currently but not specific enough to be useful:

Thing topic UnknownBLETag "Unknown BLE Tag"  @ "Meraki MQTT" {
	Channels:
		Type string : bletype "BLE Type"		[ stateTopic="meraki", transformationPattern="JSONPATH:$.bleType"]
		Type string : clientmac "Client MAC"	[ stateTopic="meraki", transformationPattern="JSONPATH:$.clientMac"]
		Type string : timestamp "Timestamp" 	[ stateTopic="meraki", transformationPattern="JSONPATH:$.timestamp"]
		Type string : radio "Radio"          	[ stateTopic="meraki", transformationPattern="JSONPATH:$.radio"]
		Type number : rssi "RSSI"               [ stateTopic="meraki", transformationPattern="JSONPATH:$.rssi"]
	}

Not Working Example:
(Note my attempt on RSSI channel to use Unicode escape code for “_” & “:” with no luck)

	Thing topic UnknownBLETagOld "Unknown BLE Tag"  @ "Meraki MQTT" {
	Channels:
		Type string : bletype "BLE Type"		[ stateTopic="meraki/v1/mr/N_601793500207383877/E0:CB:BC:8A:A6:1B/ble/24:F1:28:01:8B:D7", transformationPattern="JSONPATH:$.bleType"]
		Type string : clientmac "Client MAC"	[ stateTopic="meraki/v1/mr/N_601793500207383877/E0:CB:BC:8A:A6:1B/ble/24:F1:28:01:8B:D7", transformationPattern="JSONPATH:$.clientMac"]
		Type string : timestamp "Timestamp" 	[ stateTopic="meraki/v1/mr/N_601793500207383877/E0:CB:BC:8A:A6:1B/ble/24:F1:28:01:8B:D7", transformationPattern="JSONPATH:$.timestamp"]
		Type string : radio "Radio"          	[ stateTopic="meraki/v1/mr/N_601793500207383877/E0:CB:BC:8A:A6:1B/ble/24:F1:28:01:8B:D7", transformationPattern="JSONPATH:$.radio"]
		Type number : rssi "RSSI"               [ stateTopic="meraki/v1/mr/N\u005F601793500207383877/E0\u003ACB\u003ABC\u003A8A\u003AA6\u003A1B/ble/24\u003AF1\u003A28\u003A01\u003A8B\u003AD7", transformationPattern="JSONPATH:$.rssi" ]
	}

I would appreciate any thoughts on how to handle an MQTT topic containing these characters.

With Thanks
Rod

rossko57 · October 12, 2021, 10:02am

colons are active separators in xxx.things files, so I would imagine they are the problem here.
I’m surprised your openhab.log doesn’t have something to say about it, at file load time?
This probably doesn’t arise when configuring via GUI.

If I recall right, you’d need to use double backslash to escape each colon.
\\:

Or cheat and use a wildcard maybe
stateTopic="meraki/v1/mr/N_601793500207383877/#",

denominator · October 13, 2021, 2:10am

@rossko57 Is one way the other may be to filter your incoming json messages to allow devise specific input.

Here is a working example

    Thing topic Kettle "Kettle in Kitchen" @ "Kitchen" [ availabilityTopic ="tele/kettle/LWT", payloadAvailable ="Online", payloadNotAvailable ="Offline"] {
    Channels:
        Type switch : PowerSwitch  [ stateTopic ="stat/kettle/POWER", commandTopic ="cmnd/kettle/POWER", on="ON", off="OFF"]
        Type string : WarmMode [ commandTopic ="cmnd/kettle/TuyaSend4"]
        Type number : temperature "Temperature [%.0f °C]" [ stateTopic ="tele/kettle/TUYARECEIVED", transformationPattern ="REGEX:(.*DpId\":5.*)∩JSONPATH:$.TuyaReceived.DpIdData∩JS:HEXtoDEC.js"  ]
      }

Looking a the transformationPattern if the message has DpId":5 then extract message in JSONPATH transformation. Then because its in HEX JS transform is used to display Kettle temp in deg c

∩ is not an n its an intersection

galileo · October 13, 2021, 3:45am

@rossko57 Thanks for your suggestions

I have tested with:

stateTopic="meraki/v1/mr/N_601793500207383877/#"

And this works fine, proving the “_” is not the problem and my issue is the “:” as you suggested, but it still doesn’t give me a way to track a unique device without complex transformation.

I tried double slash in front of colon as you suggested in the static config, and also tried creating the thing via GUI (with/without the extra colons) with no luck:

Thing topic OtherBLETag "Other BLE Tag"  @ "Meraki MQTT" {
	Channels:
		Type string : bletype "BLE Type"		[ stateTopic="meraki/v1/mr/N_601793500207383877/E0\\:CB\\:BC\\:8A\\:A6\\:1B/ble/24\\:F1\\:28\\:05\\:E7\\:F7", transformationPattern="JSONPATH:$.bleType"]
		Type string : clientmac "Client MAC"	[ stateTopic="meraki/v1/mr/N_601793500207383877/E0\\:CB\\:BC\\:8A\\:A6\\:1B/ble/24\\:F1\\:28\\:05\\:E7\\:F7", transformationPattern="JSONPATH:$.clientMac"]
		Type string : timestamp "Timestamp" 	[ stateTopic="meraki/v1/mr/N_601793500207383877/E0\\:CB\\:BC\\:8A\\:A6\\:1B/ble/24\\:F1\\:28\\:05\\:E7\\:F7", transformationPattern="JSONPATH:$.timestamp"]
		Type string : radio "Radio"          	[ stateTopic="meraki/v1/mr/N_601793500207383877/E0\\:CB\\:BC\\:8A\\:A6\\:1B/ble/24\\:F1\\:28\\:05\\:E7\\:F7", transformationPattern="JSONPATH:$.radio"]
		Type number : rssi "RSSI"               [ stateTopic="meraki/v1/mr/N_601793500207383877/E0\\:CB\\:BC\\:8A\\:A6\\:1B/ble/24\\:F1\\:28\\:05\\:E7\\:F7", transformationPattern="JSONPATH:$.rssi" ]
	}

I have the MQTT binding set for TRACE level debugging and I see no useful logs, just the lack of updates to the Thing channels.

The last part of the MQTT Topic is the unique device MAC, so if the “:” can be accepted in some way that will be the simplest.

Thanks Rod

galileo · October 13, 2021, 4:40am

@denominator Thanks for your response.

Is this something like you were suggesting ? (as I am not a REGEX expert)

		Type string : radio "Radio"          	[ stateTopic="meraki/v1/mr/N_601793500207383877/+/ble", transformationPattern="REGEX:(.*24:F1:28:05:E7:F7.*)∩JSONPATH:$.radio"]
		Type number : rssi "RSSI"               [ stateTopic="meraki/v1/mr/N_601793500207383877/+/ble", transformationPattern="REGEX:(.*24:F1:28:05:E7:F7.*)∩JSONPATH:$.rssi" ]

I have put a “+” as an MQTT wildcard to mask out the Wireless AP MAC which is a fixed value containing colons but have left enough of the Topic to discriminate BLE & WiFi devices which have slightly different data. The value in the REGEX string is the MAC of the end device.

Thanks
Rod

rossko57 · October 13, 2021, 10:08am

That’s the general idea … but I think you’ll still have to escape the colons in the REGEX expression \\

More like
stateTopic="meraki/v1/mr/N_601793500207383877/+/ble/+
?

rossko57 · October 13, 2021, 10:39am

I tried this out

    Thing topic testing "test play" {
    Channels:
        Type switch : test1  [ stateTopic="test/apple:orange" ]
        Type switch : test2  [ stateTopic="test/lemon\\:banana" ]
      }

No complaints in openhab.log for this, but note that it is not possible to use single backslash here like lemon\:banana because that will break the Things file parser and complain.

Okay, so both those two stateTopics seem to be accepted … what does the MQTT Broker think?
I looked at my Mosquitto log to see the subscriptions.

Received SUBSCRIBE from OpenHAB
   test/apple:orange (QoS 0)
Received SUBSCRIBE from OpenHAB
   test/lemon\:banana (QoS 0)

Deduction : the escape backslash has operated only on the next backslash and ‘escaped’ it to arrive in our topic.
The colon doesn’t need escaping at all in this usage, and you’ve got some other problem going on here.

(It would still need escaping within the transformationPattern because there’s an extra layer of parsing there and colon is separator)

denominator · October 13, 2021, 10:33pm

Yes that is exactly what I suggested. I am no expert either

So I just looked at the api docs and the default structure of the topic is

> Meraki/v1/mr/<NetworkID>/<AP MAC>/<ble/wifi>/<Client MAC>/

Looking at @rossko57 testing your setup looks good and should work. Maybe your “Not Working Example” thing file didn’t load correctly.

Can you answer this question

Which MQTT debugger do you use, I like MQTT Explorer?

How do you edit your things files?

Can your try setting up a new MQTT thing using the GUI

or try

stateTopic="meraki/v1/mr/N_601793500207383877/+/ble/+", transformationPattern="REGEX:(.*24\\:F1\\:28\\:05\\:E7\\:F7.*)∩JSONPATH:$.radio"]

galileo · October 14, 2021, 1:04pm

@denominator @rossko57
Guys, Thanks for all your help, it is now working !

Here is a working BLE and WiFi example for the Meraki AP MQTT:

Thing topic SpareKeysBLETag "Spare Keys BLE Tag"  @ "Meraki MQTT" {
	Channels:
		Type string : bletype "BLE Type"		[ stateTopic="meraki/v1/mr/+/+/ble/#",transformationPattern="REGEX:(.*24\\:F1\\:28\\:06\\:25\\:5B.*)∩JSONPATH:$.bleType"]
		Type string : clientmac "Client MAC"	[ stateTopic="meraki/v1/mr/+/+/ble/#",transformationPattern="REGEX:(.*24\\:F1\\:28\\:06\\:25\\:5B.*)∩JSONPATH:$.clientMac"]
		Type string : timestamp "Timestamp" 	[ stateTopic="meraki/v1/mr/+/+/ble/#",transformationPattern="REGEX:(.*24\\:F1\\:28\\:06\\:25\\:5B.*)∩JSONPATH:$.timestamp"]
		Type string : radio "Radio"          	[ stateTopic="meraki/v1/mr/+/+/ble/#",transformationPattern="REGEX:(.*24\\:F1\\:28\\:06\\:25\\:5B.*)∩JSONPATH:$.radio"]
		Type number : rssi "RSSI"               [ stateTopic="meraki/v1/mr/+/+/ble/#",transformationPattern="REGEX:(.*24\\:F1\\:28\\:06\\:25\\:5B.*)∩JSONPATH:$.rssi"]
	}
Thing topic TestDeviceWifiDev "Test WiFi Device"  @ "Meraki MQTT" {
	Channels:
		Type string : clienttype "Client Type"	[ stateTopic="meraki/v1/mr/+/+/wifi/#", transformationPattern="REGEX:(.*38\\:8B\\:59\\:24\\:AE\\:36.*)∩JSONPATH:$.clientType"]
		Type string : clientmac "Client MAC"	[ stateTopic="meraki/v1/mr/+/+/wifi/#", transformationPattern="REGEX:(.*38\\:8B\\:59\\:24\\:AE\\:36.*)∩JSONPATH:$.clientMac"]
		Type string : timestamp "Timestamp" 	[ stateTopic="meraki/v1/mr/+/+/wifi/#", transformationPattern="REGEX:(.*38\\:8B\\:59\\:24\\:AE\\:36.*)∩JSONPATH:$.timestamp"]
		Type string : radio "Radio"          	[ stateTopic="meraki/v1/mr/+/+/wifi/#", transformationPattern="REGEX:(.*38\\:8B\\:59\\:24\\:AE\\:36.*)∩JSONPATH:$.radio"]
		Type number : rssi "RSSI"               [ stateTopic="meraki/v1/mr/+/+/wifi/#", transformationPattern="REGEX:(.*38\\:8B\\:59\\:24\\:AE\\:36.*)∩JSONPATH:$.rssi" ]
	}

Key Learnings for me:

MQTT “#” multi-level wildcard is used to match any Topic value after it
MQTT “+” single-level wildcard is good for shortening the MQTT Topic or masking out one level, or where the # is not suitable (but is not suitable to end the Topic)
Topics containing “:” don’t seem to match correctly
Attempts to “escape” the “:” using “\\” in front of the “:” in the Topic didn’t seem to work, but was essential in the RegEx expression.
MQTT debugging gives most detailed results at boot time
When you start using RegEx expressions for the first time don’t forget to install the RegEx Transformation Addon
MQTT Explorer as recommended by denominator is best MQTT debugger I have seen as it collates all Topics receiving messages in a very easy to use fashion.

Once I got it working I tried again to get it to work without the RegEx transformation, just using the “\\” before the “:” values in the MAC address in the MQTT Topic with no luck.

I always try to learn from previous forum posts to come up with a solution, but this one had me stuck. Thanks again guys, you are both a credit to the OpenHAB community and your help has been awesome !!!

Regards
Rod

rossko57 · October 14, 2021, 5:15pm

Not me, this is to @denominator credit.
(I get by for my small needs with built in Mosquitto tools)

Out of interest, I did what I should have done before with

Type switch : test1  [ stateTopic="test/apple:orange" ]

The full end-to-end test.
I can subscribe to the topic with colon successfully.
Using Mosquitto_pub, I can publish to that topic and openHAB picks it up just fine and updates the linked Item.
No problem with colons in topics.