MQTT 2.5 M1+ How to implement the equivalent to MQTT1 REGEX filters

Now working just fine. I copied this symbol after restarting OH with -Dfile.encoding=UTF8 option.
The symbol did not change after restart OH.
Thank you. @rlkoshak

1 Like

I’m also trying to move my OpenHAB 2.4 configuration with MQTT v1 binding towards a test config with OpenHAB 2.5.1 and the new MQTT v2.5.1 binding. My config is completely text based for 2 reasons: I generate all things, items, rules and sitemaps from a small Python script; secondly: it enables me to easily copy parts of the configuration from one OpenHAB instance to another. I really need thorough testing to avoid issues with our heating system and big trouble with my wife :open_mouth:
I tried to use same combination of transformations, but I can’t get the special “∩” saved: it’s converted into the letter “n”. I also tried to add this line in the /etc/default/openhab2 file:
EXTRA_JAVA_OPTS="-Dfile.encoding=UTF8"
but no luck.

Here’s what I’ve got in the old items file (MQTT v1 binding):

Number Temp_Wemos4_CV_Ketel “Wemos4_CV_Ketel_T [%.2f C]” (gTemperature) {mqtt="<[mozquitoin:MBroadCast:state:JSONPATH($.TempValue):.“Temp_Wemos4_CV_Ketel”.]"}

And I added this in the new things file (MQTT v2.5.1 binding):

Thing topic MBroadcaster "MBroadcastR1" @ "MBroadcast" {
Channels:
    Type number : Temp_Wemos4_CV_Ketel "Wemos4_CV_Ketel_T"    [ stateTopic="MBroadCast", transformationPattern="REGEX:(.*Temp_Wemos4_CV_Ketel.*)∩JSONPATH:$.TempValue" ]
}

When looking in PaperUI I see this:
image

No errors in the logs… All help is welcome.

There is debate in java circles about UTF8 or UTF-8

How you can put this symbol? I put it by COPYING ctrl c ctrlv in to paperui. But in my situation all cyrillic symbols was ‘?’. Adding UTF8 option resolve this problem in my case.

It’s working fine now. I first started to implement it via PaperUI and it worked immediately. After that I came to the conclusion that my text editor didn’t really save the file in UTF8 format, even though I explicitely set the option. I had to change the “default encoding” of the editor to have it right.
Thanks for this post !

This ∩ character really complexifies everything as it forces us to use UTF-8 format, only for this. Isn’t there an alternative way, can it be replaced by some other character / combination of characters that allows us to keep using ASCII instead of UTF-8? If not possible, is there a platform where we can raise the question?

Important

All text files must be created with UTF-8 encoding.

From

You can file an issue on the openhab2-addons github, but as rossko57 points out, UTF-8 is the requirement anyway. So the issue is moot.

Hi @rlkoshak,

I’ve this:

        Type string : name      "Name"                  [ stateTopic="home/OpenMQTTGateway_ESP32_BLE/BTtoMQTT/E72E014260FF", transformationPattern="REGEXP:(.*name.*)∩JSONPATH:$.name"]
        Type number : rssi      "WiFi Signal Strength"  [ stateTopic="home/OpenMQTTGateway_ESP32_BLE/BTtoMQTT/E72E014260FF", transformationPattern="REGEXP:(.*rssi.*)∩JSONPATH:$.rssi"]
        Type number : distance  "WiFi Distance"         [ stateTopic="home/OpenMQTTGateway_ESP32_BLE/BTtoMQTT/E72E014260FF", transformationPattern="REGEXP:(.*distance.*)∩JSONPATH:$.distance"]
        Type number : temperature   "Temperature"       [ stateTopic="home/OpenMQTTGateway_ESP32_BLE/BTtoMQTT/E72E014260FF", transformationPattern="REGEXP:(.*tem.*)∩JSONPATH:$.tem"]
        Type number : humidty   "Humidity"              [ stateTopic="home/OpenMQTTGateway_ESP32_BLE/BTtoMQTT/E72E014260FF", transformationPattern="REGEXP:(.*hum.*)∩JSONPATH:$.hum"]
        Type number : battery   "Battery"               [ stateTopic="home/OpenMQTTGateway_ESP32_BLE/BTtoMQTT/E72E014260FF", transformationPattern="REGEXP:(.*batt.*)∩JSONPATH:$.batt"]

But now I’ve more warnings :slight_smile:

 20-08-02 18:08:29.252 [WARN ] [t.generic.ChannelStateTransformation] - Transformation service REGEXP for pattern (.*name.*) not found!
2020-08-02 18:08:29.253 [WARN ] [t.generic.ChannelStateTransformation] - Executing the JSONPATH-transformation failed: Invalid path '$.name' in '{"tem":29.6}'

Can you point my error?

Doesn’t look like you have the REGEX transformation service installed?

PaperUI → Add-ons → Transformations → RegEx Transformation → Install

@hafniumzinc Yes I’ve it:

openhab> list | grep -i reg
277 │ Active │  75 │ 2.5.7                   │ openHAB Add-ons :: Bundles :: Transformation Service :: RegEx

And I use it in anothers places, like this:

{ udp="<[192.168.1.65:*:'REGEX(.*<chan id=\'0\'>.*<day units=\'wh\'>(.*)</day></chan><chan id='1'>.*)']" }

But you have to spell it right. You asked for REGEXP, it told you you haven’t got one.

1 Like

The most difficult mistake is always the most obvious :wink:

I’m am still having trouble with this…

Functionally, it all seems to work, but logs are spammed out :confused:

REGEX:(.*D3F9EE.*)∩JSONPATH:$.RfReceived.Data

as in;

image

But still get log errors for channels getting;

tele/rf_bridge/RESULT

when just one message is sent;

2020-09-29 21:10:45.008 [WARN ] [t.generic.ChannelStateTransformation] - Transformation service REGEX for pattern (.*4065D4.*) not found!
2020-09-29 21:10:45.009 [WARN ] [ab.binding.mqtt.generic.ChannelState] - Command '4065D8' not supported by type 'OnOffValue': No enum constant org.eclipse.smarthome.core.library.types.OnOffType.4065D8
2020-09-29 21:10:45.010 [WARN ] [t.generic.ChannelStateTransformation] - Transformation service REGEX for pattern (.*4065D1.*) not found!
2020-09-29 21:10:45.010 [WARN ] [ab.binding.mqtt.generic.ChannelState] - Command '4065D8' not supported by type 'OnOffValue': No enum constant org.eclipse.smarthome.core.library.types.OnOffType.4065D8
2020-09-29 21:10:45.011 [WARN ] [t.generic.ChannelStateTransformation] - Transformation service REGEX for pattern (.*D3F9EE.*) not found!
2020-09-29 21:10:45.011 [WARN ] [ab.binding.mqtt.generic.ChannelState] - Command '4065D8' not supported by type 'OnOffValue': No enum constant org.eclipse.smarthome.core.library.types.OnOffType.4065D8
2020-09-29 21:10:45.012 [WARN ] [t.generic.ChannelStateTransformation] - Transformation service REGEX for pattern (.*E4CF0[AE].*) not found!
2020-09-29 21:10:45.012 [WARN ] [ab.binding.mqtt.generic.ChannelState] - Command '4065D8' not supported by type 'OpenCloseValue': No enum constant org.eclipse.smarthome.core.library.types.OpenClosedType.4065D8
2020-09-29 21:10:45.013 [WARN ] [t.generic.ChannelStateTransformation] - Transformation service REGEX for pattern (.*D3F9E6.*) not found!
2020-09-29 21:10:45.013 [WARN ] [ab.binding.mqtt.generic.ChannelState] - Command '4065D8' not supported by type 'OnOffValue': No enum constant org.eclipse.smarthome.core.library.types.OnOffType.4065D8
2020-09-29 21:10:45.014 [WARN ] [t.generic.ChannelStateTransformation] - Transformation service REGEX for pattern (.*E46E0[AE].*) not found!
2020-09-29 21:10:45.015 [WARN ] [ab.binding.mqtt.generic.ChannelState] - Command '4065D8' not supported by type 'OpenCloseValue': No enum constant org.eclipse.smarthome.core.library.types.OpenClosedType.4065D8
2020-09-29 21:10:45.015 [WARN ] [t.generic.ChannelStateTransformation] - Transformation service REGEX for pattern (.*E3BA0[AE].*) not found!
2020-09-29 21:10:45.015 [WARN ] [ab.binding.mqtt.generic.ChannelState] - Command '4065D8' not supported by type 'OpenCloseValue': No enum constant org.eclipse.smarthome.core.library.types.OpenClosedType.4065D8
Transformation service REGEX for pattern (.*4065D4.*) not found!

You need to install the REGEX transformation service! Add-ons -> Transformation

1 Like

Thank you, that sorted it. :slight_smile:

Just a quick note that this works in the UI with openHAB 3.1.0 Release version.
I was seeing warnings in the log for an item linked to the Brightness channel of an MQTT Dimmer Thing (Tasmota firmware on the dimmer). The state topic only returned the dimmer brightness when it was changed, so I was getting warnings when the power was turned on and off. I added the REGEX transformation and then chained the REGEX to the JSONPATH in the Incoming Value Transformation field (you have to select advanced options in the UI to see that field) in the channel configuration for the Brightness channel…

REGEX:(.*Dimmer.*)∩JSONPATH:$.Dimmer

Thanks for the tip!

Sorry to bring that old discussion up again, but I am facing a problem here (most probably it is my lack of knowledge).

(Running OH 3.3.0 as a VM with Openhabian / Debian on Proxmox)

I am using a MQTT Smartmeter Device that reads the power consumption from the grid. Unfortunately, the meter does provide only a single value per packet, still they are all using the same format:

{"Time":"2022-08-29T18:44:01","Z1":{"76_7_0":-4.04}}

The value after Z1 changes, so 76_7_0 could be 1_8_1 or 2_8_0 f.e., hence I needed to implement the REGEX function.

Now, when defining the MQTT Thing in a text file, the expression looks like that:

Thing topic Zaehler @ "MQTT" {
    Channels:
        Type number : z1Zaehlerstand       "Zählerstand"                [stateTopic="tele/Zaehler/SENSOR", transformationPattern="REGEX:(.*1_8_0.*)∩JSONPATH:$.Z1.1_8_0"] 
        Type number : z1BezugTarif1        "Bezug Tarif 1"              [stateTopic="tele/Zaehler/SENSOR", transformationPattern="REGEX:(.*1_8_1.*)∩JSONPATH:$.Z1.1_8_1"]
        Type number : z1BezugTarif2        "Bezug Tarif 2"              [stateTopic="tele/Zaehler/SENSOR", transformationPattern="REGEX:(.*1_8_2.*)∩JSONPATH:$.Z1.1_8_2"]
        Type number : z1Einspeisung        "Einspeisung"                [stateTopic="tele/Zaehler/SENSOR", transformationPattern="REGEX:(.*2_8_0.*)∩JSONPATH:$.Z1.2_8_0"]        
        Type number : z1Leistung           "Aktuelle Gesamtleistung"    [stateTopic="tele/Zaehler/SENSOR", transformationPattern="REGEX:(.*16_7_0.*)∩JSONPATH:$.Z1.16_7_0"]        
        Type number : z1LeistungL1         "Aktuelle Leistung L1"       [stateTopic="tele/Zaehler/SENSOR", transformationPattern="REGEX:(.*36_7_0.*)∩JSONPATH:$.Z1.36_7_0"]        
        Type number : z1LeistungL2         "Aktuelle Leistung L2"       [stateTopic="tele/Zaehler/SENSOR", transformationPattern="REGEX:(.*56_7_0.*)∩JSONPATH:$.Z1.56_7_0"]        
        Type number : z1LeistungL3         "Aktuelle Leistung L3"       [stateTopic="tele/Zaehler/SENSOR", transformationPattern="REGEX:(.*76_7_0.*)∩JSONPATH:$.Z1.76_7_0"]        
    }

Without the REGEX, the well know error is to be seen in the logs. With the REGEX, ALL outcome is filtered and no packed is passed through at all.

So I defined the things in the UI, using the same parameters:

UID: mqtt:topic:6296256670:Zaehler
label: Generic MQTT Thing
thingTypeUID: mqtt:topic
configuration: {}
bridgeUID: mqtt:broker:jarvis
channels:
  - id: z1Leistung
    channelTypeUID: mqtt:number
    label: Einspeisung / Bezug (Zähler)
    description: ""
    configuration:
      stateTopic: tele/Zaehler/SENSOR
      transformationPattern: REGEX:(.*16_7_0.*)∩JSONPATH:$.Z1.16_7_0
      unit: W
  - id: z1Zaehlerstand
    channelTypeUID: mqtt:number
    label: Zählerstand
    description: ""
    configuration:
      stateTopic: tele/Zaehler/SENSOR
      transformationPattern: REGEX:(.*1_8_0.*)∩JSONPATH:$.Z1.1_8_0
      unit: Wh
  - id: z1Einspeisung
    channelTypeUID: mqtt:number
    label: Einspeisung
    description: ""
    configuration:
      stateTopic: tele/Zaehler/SENSOR
      transformationPattern: REGEX:(.*2_8_0.*)∩JSONPATH:$.Z1.2_8_0
      unit: Wh

Which works like a charm. Now, call me old fashioned, but I would like to stay with the text files. Any idea why it does work only with the UI version ?

Looking at the docs for the binding, the REGEX should work just fine (unless I made a mistake with the syntax, that is…)

Query for the JSON using the REST API for the UI defined Things.
Replace the UI defined things with the text defined Things.
Query for those and compare the JSON.

If there are differences you have your answer and you might have to file an issue but you’ll at least have something to go on. If not, you’ll have to file an issue on the binding anyway, only there will be less to go on.

Hm, what do you mean by “query for…” ? If I take away the REGEX from the text files, the output is the same as with the UI defined things, e.g. the values are read correct. So to me the JSONPATH seems to work just fine in both situations. It is just that with the REGEX Expressions in the text definition, no packed at all seems to pass. But I am not sure if the text-based syntax is fully correct in regards to the REGEX part. Unfortunately, the documentation is quite slim here…