Best way to pack/unpack multiple items in a JSON MQTT payload

Setup, Configuration and Use Beginners
1

I’ve built some custom light controllers controlled via MQTT messages, with a few different functions, eg on/off switch, dimmer values, colour temp values etc.

I searched around, but info is mainly related to incoming MQTT JSON payloads rather than a bi-directional system.

Currently each associated item within OpenHAB is linked to a separate channel with a separate MQTT topic. For various optimisation reasons, I’m experimenting with packing all the data into a single MQTT payload as a JSON.

This thread is more of a brainstorm rather than a troubleshooting exercise, as I have come up with a workable solution, but I’m not sure it’s the default or best way to do it. Since there are other off the shelf products that operate in a similar manner, there must be a few other solutions out there.

To create an outgoing MQTT message, I created a single string item which is linked to a string/text channel which has an associated MQTT command topic. I then wrote a script which is triggered by commands sent to any of the associated switches/dimmer etc. The script creates a JSON object, and converts it to a string, which is then sent to the aforementioned string item, which is then sent to the linked channel and MQTT topic.

Basically the reverse happens for incoming MQTT messages. There’s a string item which receives the incoming MQTT JSON payload, then a script parses the incoming payload into a JSON object, then all the required items are updated from the name/data pairs in the JSON.

The first attempt used the JSONPATH addon in order to update the items directly by extracting the name/data pairs from the MQTT message directly. However when a channel has an update MQTT topic, but no command topic, the item becomes a read-only item which can’t be controlled through the UI.

Is this input/output string method the simplest solution? It’s quite a “native” solution, as it only relies on the Javascript scripting addon which is pretty integral to OH anyways.

Anyways, what are your approaches?

2

One of the driving design goals of MQTT is to not impose work on the clients. I fine it a little maddening when I see JSON used in combination with MQTT because use of JSON imposes work on the end points (i.e the message must be parsed and a JSON library must be included which can be a burden on microcontrollers with limited memory).

MQTT best practice works be to have separate topics instead. Each property world have two topics, one to publish state changes and another to accept commands.

This requires the heart amount of work on the end points as there is limited to no parading required to interpret the message.

So why isn’t there a command topic? how are you communicating these devices in the first place? There really is nothing wrong with this approach. You just need to add the command side to the channels and your are good to go. You can use a Jinja or JS transformation on the commands if the JSON that needs to be published is a little complicated. Otherwise you could get away with the output format property

But, like I said, MQTT best practice works be to have separate topics instead of building everything into one JSON message on fewer topics.

It’s worth noting however that many systems do not follow the MQTT philosophy and bundle stuff up using JSON.

3

To be fair :wink: subscribing to many topics may rise workload at clients as well as the broker, so json payload may not be as bad as it seems… but I totally agree that separate topics are more convenient for the user. :slight_smile:

4

Not really for the clients. The libraries are pretty darned effecient in that regard. There’s a few extra bytes and cycles to iterate over which topics to subscribe to (maybe, if you can’t use wild card subscriptions) but the code that processes the messages is the same no matter how many topics you’ve subscribed to. And processing one value received in 10 messages is less than processing one message but needing to parse and extract 10 values from it.

It does increase the load of the broker, but the broker isn’t a little SCADA microcontroller 500 miles away on an oil pipeline powered by battery (don’t forget what MQTT was invented for). It’s a running on a full ARM or x86 machine with adequate resources. It’s far less of a concern to optimize for the broker than the end clients.

1 Like
5

Good opportunity to repeat that in zigbee2mqtt you can use the option output: attribute and all attributes are sent nicely with separate messages without the need to decode json.

6

Point taken in regards to MQTT. I was under the impression that JSON was a fairly standardised approach to the payloads.

In my case the client processing isn’t really an issue, as the client is actually a python program running on the same RPi as OpenHAB. It handles the communication between the microcontrollers over an RS485 network.

But say for arguments sake, if you need to interface bi-direcionally with a commercial product that only uses JSON as the payload, my non functioning example looked something like this:

Each item receives the same topic, but extracts different keys from the JSON payload

(Incoming MQTT message) -----> (JSONPATH) --> (Item1)
                        |----> (JSONPATH) --> (Item2)
                        |----> (JSONPATH) --> (Item3)

However they send messages though the output string item:

(Item1) ---->|Rule which|
(Item2) ---->| creates  | ----> (Output string Item) ----> (Publish MQTT)
(Item3) ---->|   JSON   |

In this case, the items become read-only, hence the other solution where the individual items neither publish nor subscribe, but instead are updated or commanded, or send commands through a rule and output item.

I’ve begun doing a complete redesign of the python program though, and as such there’s going to be less of a benefit using JSON as the payloads. Still, python does make it fairly convenient to use JSONs as there’s more built in error checking, where as I have to parse all the topics manually to see if they’re valid.

7

I’m not sure what extra error checking you’d get from JSON. You can send primitive numbers and booleans as MQTT messages so there’s no need to parse. And JSON doesn’t really provide. a whole lot more error checking than that beyond ensuring that the message is properly formatted which doesn’t matter if the message is already what it needs to be. And in cases where you are using a String enum, you still need to test the part you get from JSON.

But that’s kind of my point. You’ve added the work of encoding the message as JSON, then you have to decode it but you still have to validate the properties are valid.

Anyway, to not have read only Things you would have:

(Incoming Item Command) ------> commandTransformation which creates JSON-------> MQTT message published

If the outgoing JSON involves more than one Item you’d use a transformation Script (e.g. JS) to buils the outgoing JSON. Otherwise you could use Jinja or even just the outgoing value format property.

image
image746×153 7.54 KB

For example using the outgoing value format.,

{ mySwitch: { command: "%s" }}

A live example where I use Jinja (this is on an HTTP binding though where JSON is not going against the philosophy of the transport)

JINJA:{"protection_enabled": {{value}}}

image
image779×177 4.88 KB

8

I use both! :wink:
there are devices, which provide a JSON as a payload (e.g. Tasmota-devices) and there’s a “kinda” JSON source, which I simply encoded as JSON and send to openHAB.
But my preference is having more topics and less work at the clients. and I use MQTT heavily, I have about 900 topics and I guess a mere 50 or so of them are JSON, the rest is “single” payload like numbers, boolean or text.

If you have “single-value” topics, they are easier human readable. Für JSON you’d have to put the payload in a JSON formatter to see the contents, especially if the JSON is a bit more complex.