MQTT 2.5 M1+ How to implement the equivalent to MQTT1 REGEX filters

Problem

In the MQTT1 binding there was a feature that let us filter messages published to a given topic and only pass those that match a given REGEX to the transformation and/or on to the Item. This was useful to handle devices that publish messages to a single well known topic for all their devices.

For example, the official firmware from the vendor of Shelly devices have a request/response protocol over MQTT to get meta data about the devices (IP address, MAC address, firmware version, whether there is a firmware update, etc.). Each individual device has their own topic you can send “announce” to. In response, the device will publish a JSON message to shellies/announce. ?! :angry:

OK, well we have to deal with it if we don’t want to have to install alternative firmware.

Each Shelly device has a unique name. For example shelly1-25A289. The JSON published to shellies/announcement includes this unique name.

Approach

The MQTT2 binding lacks direct support for a REGEX filter as a separate parameter. But it does support chaining transformations on incoming messages. So we can chain a REGEX transformation that matches and returns the entire message but only if it contains the unique name. Next we chain that to a JSONPATH transformation to extract the desired field. Thus, the Channel will ignore the messages posted for some other device and only pass the messages for this device to the JSONPATH.

Based on my tests there is no error in the log when the REGEX does not match the message, which is exactly what we want.

When we are done, the Thing will look something like:

Request Channel

MQTT command topic: shellies/<shelly name>/command
Custom On/Open value: announce
Custom Off/Closed value: announce

Replace <shelly name> with the name of your specific device which you can find on config web page for the device. You need to know this anyway to command the device. For example, shellies/shelly1-25A289/command.

Any time the Switch linked to this Channel receives any sort of command, it will publish “announce” to the device’s command topic.

IP Address

MQTT state topic: shellies/announce
Incoming value transformations: REGEX:(.*<shelly name>.*)∩JSONPATH:$.ip

The trick is the REGEX needs to match the full message, hence the parens around the full expression. But we only want to pass on messages that include this device’s name. As above, replace <shelly name> with the real name. For example REGEX:(.*shelly1-25A289.*)∩JSONPATH:$.ip.

The rest of the fields in the JSON can be extracted in the same way using a different JSONPATH expression.

Commentary

Obviously, this approach can be used in different and more powerful ways. For example, the REGEX can both filter and select just a portion of the message, not just filter the message. Any transformation can follow the REGEX. And more than two transformations can be chained. This should give us the same flexibility, if not more than the MQTT 1 REGEX filter.

10 Likes

I try this setup but it doesn’t work or I didn’t understand how to do.
I have a json string like "{"fp2":"E"}" or "{"fp1":"H","fp2":"H","fp5":"H"}"
My declaration :
Type string : Bureau_FilPilote [ stateTopic="remora/fp", transformationPattern="REGEX:(.*fp5.*)∩JSONPATH:$.fp5" ]
With the long sjson string it works but with the short string I have this error :

22:01:04.848 [WARN ] [tt.generic.ChannelStateTransformation] - Executing the JSONPATH-transformation failed: Invalid path '$.fp5' in '{"fp2":"E"}'
22:01:04.849 [INFO ] [smarthome.event.ItemStateChangedEvent] - Bureau_FilPilote changed from H to {"fp2":"E"}

Any idea ? I don’t want to use JS script … (I’m using OH 2.5M3)

You need a certain binding version minimum to support chained transforms

I have the same problems. My version binding is:

202 ? Active   ?  80 ? 0.11.0.oh250M1         ? Eclipse SmartHome MQTT Binding

and my channel have this transform:

Incoming value transformations 
REGEX:(.*nrf1t1t*)∩JSONPATH:$.sensors.nrf1t1t
Applies transformations to an incoming MQTT topic value. A transformation example for a received JSON would be "JSONPATH:$.device.status.temperature" for a json {device: {status: { temperature: 23.2 }}}. You can chain transformations by separating them with the intersection character ∩.

My incoming incoming JSON does not have everytime json value - nrf1t1t and i get warnings in log if incoming json does not have this value. I want to reduce warnings using RegEx but my transformation does not work correctly. The warning apears in log. Wha i do wrong?

You have both REGEX and JSONPATH transforms installed?

1 Like

Yes! Can i debug my transformation expression? I have no idea why he does not work.
image

Post two examples of your JSON, one with the string and one with out.

I don’t really have any good advice. It appears to work for me so I’ve never had to debug the any problems of it not working.

@rlkoshak This is without my nrf1t1 field:

{
  "system": {
    "hostname": "Bar_NODE",
    "uptime": 284100,
    "rssi": -80,
    "freemem": 32656
  },
  "sensors": {},
  "pwm": {
    "0": 0,
    "1": 0,
    "2": 0
  },
  "gpio": {
    "16": 0
  }
}

and thi is with

{
  "system": {
    "hostname": "Bar_NODE",
    "uptime": 284100,
    "rssi": -80,
    "freemem": 32656
  },
  "sensors": { 
    "nrf1t1t" : 20,
     "nrf1h1" : 33
    
  },
  "pwm": {
    "0": 0,
    "1": 0,
    "2": 0
  },
  "gpio": {
    "16": 0
  }

}

I just tried to set it up the same way and it looks like the behavior of the REGEX transform may have changed since the OP was posted. It looks like previously the REGEX transform would return nothing if there was no match. Now it appears to return everything if there is no match.

I created an issue:

4 Likes

@rlkoshak thank you very much! Hope to fast resolving this issue

Good spot. Be interesting to see the outcome - while it was certainly convenient the way it worked before, it felt a bit out of character for a transform to “fail” silently.

Maybe there needs to be some way to distinguish -
“this is a filter, null output is tolerated”
from
“this is an ordinary transform, null output is unexpected”

channel profiles seem like a good place to do this kind of thing, perhaps benefiting other bindings too. Obviously they don’t support chaining yet.
But that is probably too far down the processing.

I agree, but on the other hand, it also makes sense that the we may expect a transform to return nothing on occasion. And if it does return null, it seems equally correct to pass on nothing rather than passing on the full message to the next transformation.

I do like any idea that involves Profiles over changes to bindings. I’d like to see transforms completely removed from the bindings so that all bindings can benefit. IMHO, they should have been separate from the bindings from the start. I’ll never get my wish though and that’s OK too. :wink:

But in the case of this particular binding, there is one capability of the MQTT 1 binding that requires this ability to filter in order to have feature parity, the ability to filter.

When you look at the use case used in the OP, the only way to handle that is to filter the messages.

you create an issue for MQTT but I think this bug is related to REGEX il you try with http binding you will have the same behavior.

I ran a test. When you use the REGEX transform in a Rule and there is no match it returns null. So this does appear to be a change in how the MQTT binding handles the fact that the transform returns null. If the change was to the REGEX transform itself then I would have received the full message back instead of null in the Rule.

I also looked at the code for the REGEX transformation and there is nothing there to cause it to return the full message when there is no match.

The HTTP binding doesn’t support chaining transformations. But if the binding uses the full message if the transform returns null than that too is the choice of the binding on how to handle the null and not the result of the transformation code itself.

1 Like

Does this problem resolved in 2.5 m4 New version? I did not see this new commit in New OH release.

The change to fix this was merged yesterday so it probably did not make the cut for 2.5 M4. You will need to run today or tomorrow’s snapshot.

Thanks @rlkoshak for this solution and for following up on the “bug”. I just discovered this (thanks to you) while on 2.5M5 and it seems to be working, i.e. the REGEX transformation will not return the full string upon non-match.

I wish that chaining the REGEX in front isn’t necessary and that only using JSONPATH alone would give us the same result (i.e. suppressing the warnings). Would that not be a good idea?

Given that the JSONPATH transform has worked the way it does pretty much forever and it is a very important transformation for many bindings and even used internally by some bindings I think, and change like this would have a huge impact on both the code and users and almost certainly be a breaking change that impacts thousands of users.

Works fine now with 2.5.M5
Thank @all

1 Like

thanks to this topic i thought i could solve my warnings in the log but at the moment i’m struggling with the implementation

My Input over MQTT looks like that:
{“lux”:306} or {“tem”:11.5}

I would like to filter to lux or temp in the MQTT Channel

this does not work:
transformationPattern=“REGEX:(.*lux.*)∩JSONPATH:$.lux” ]

=> Transformation ‘{“lux”:306}’ returned null on ‘REGEX’, discarding message

doing the same in a rule with two steps seems to work:

1.) transform(“REGEX”, “(.*lux.*)”
2.) transform(“JSONPATH”, “$.lux”
{“lux”:306} => {“lux”:306} => 306
{“tem”:11.5} = null

I’m on 2.5.0
266 │ Active │ 81 │ 2.5.0 │ openHAB Add-ons :: Bundles :: MQTT Things and Channels

I’m absolutely no familiar with REGEX, but I thougt that should work ?

thanks for any hints…

Michael