Strange behavior of REGEX:∩JSONPATH in transformationPattern of .things

Setup, Configuration and Use Items & Sitemaps
1

openHabian on a raspi3b+, latest update of everything, RegEx and JSONPath transformation 2.5.7

This is about integrating 4 (MagicHome) Led controller on Tasmota into my system. Ok, I lied. They are integrated and work as expected but now my otherwise rather clean log gets spammed with ‘Executing the JSONPATH failed’ warnings.

I followed MQTT 2.5 M1+ How to implement the equivalent to MQTT1 REGEX filters on this. I’ve seen some people report success while others had no such luck. I seemingly belong to both groups.

4 identical controllers, flashed the same day with the same tasmota, configured identically in the things file and with 3 of them the regex filtering fails, but it works for one. That’s puzzling me.

Here are the details:

 Bridge mqtt:broker:mqBroker [ host="192.168.50.1",secure=false, username="oh", password="secret", qos=1 ]
{
    
Thing topic ullu1 "ullu1"  {

    Channels:
        Type switch : power "power" 
                                                [stateTopic="tasmota/ullu1/stat/RESULT", 
                                                transformationPattern="REGEX:(.*POWER.*)∩JSONPATH:$.POWER",
                                                commandTopic="tasmota/ullu1/cmnd/POWER"]
        Type dimmer : brightness "brightness" 
                                                [stateTopic="tasmota/ullu1/stat/RESULT", 
                                                transformationPattern="REGEX:(.*Dimmer.*)∩JSONPATH:$.Dimmer", 
                                                commandTopic="tasmota/ullu1/cmnd/Dimmer", min=0, max=100, step=1]
        Type switch : fade "fade" 
                                                [stateTopic="tasmota/ullu1/stat/RESULT", 
                                                transformationPattern="REGEX:(.*Fade.*)∩JSONPATH:$.Fade",
                                                commandTopic="tasmota/ullu1/cmnd/Fade"]
        Type dimmer : fadeTime "fadeTime" 
                                                [stateTopic="tasmota/ullu1/stat/RESULT", 
                                                transformationPattern="REGEX:(.*Speed.*)∩JSONPATH:$.Speed", 
                                                commandTopic="tasmota/ullu1/cmnd/Speed", min=0, max=6, step=1]
        Type colorHSB : color "color" 
                                                [stateTopic="tasmota/ullu1/RESULT", 
                                                transformationPattern="REGEX:(.*HSBColor.*)∩JSONPATH:$.HSBColor",
                                                commandTopic="tasmota/ullu1/cmnd/HSBColor" ]

    }

   
Thing topic ullu2 "ullu2"  {

    Channels:
        Type switch : power "power" 
                                                [stateTopic="tasmota/ullu2/stat/RESULT", 
                                                transformationPattern="REGEX:(.*POWER.*)∩JSONPATH:$.POWER",
                                                commandTopic="tasmota/ullu2/cmnd/POWER"]
        Type dimmer : brightness "brightness" 
                                                [stateTopic="tasmota/ullu2/stat/RESULT", 
                                                transformationPattern="REGEX:(.*Dimmer.*)∩JSONPATH:$.Dimmer", 
                                                commandTopic="tasmota/ullu2/cmnd/Dimmer", min=0, max=100, step=1]
        Type switch : fade "fade" 
                                                [stateTopic="tasmota/ullu2/stat/RESULT", 
                                                transformationPattern="REGEX:(.*Fade.*)∩JSONPATH:$.Fade",
                                                commandTopic="tasmota/ullu2/cmnd/Fade"]
        Type dimmer : fadeTime "fadeTime" 
                                                [stateTopic="tasmota/ullu2/stat/RESULT", 
                                                transformationPattern="REGEX:(.*Speed.*)∩JSONPATH:$.Speed", 
                                                commandTopic="tasmota/ullu2/cmnd/Speed", min=0, max=6, step=1]
        Type colorHSB : color "color" 
                                                [stateTopic="tasmota/ullu2/RESULT", 
                                                transformationPattern="REGEX:(.*HSBColor.*)∩JSONPATH:$.HSBColor",
                                                commandTopic="tasmota/ullu2/cmnd/HSBColor" ]

    }

   
Thing topic ullu3 "ullu3"  {

    Channels:
        Type switch : power "power" 
                                                [stateTopic="tasmota/ullu3/stat/RESULT", 
                                                transformationPattern="REGEX:(.*POWER.*)∩JSONPATH:$.POWER",
                                                commandTopic="tasmota/ullu3/cmnd/POWER"]
        Type dimmer : brightness "brightness" 
                                                [stateTopic="tasmota/ullu3/stat/RESULT", 
                                                transformationPattern="REGEX:(.*Dimmer.*)∩JSONPATH:$.Dimmer", 
                                                commandTopic="tasmota/ullu3/cmnd/Dimmer", min=0, max=100, step=1]
        Type switch : fade "fade" 
                                                [stateTopic="tasmota/ullu3/stat/RESULT", 
                                                transformationPattern="REGEX:(.*Fade.*)∩JSONPATH:$.Fade",
                                                commandTopic="tasmota/ullu3/cmnd/Fade"]
        Type dimmer : fadeTime "fadeTime" 
                                                [stateTopic="tasmota/ullu3/stat/RESULT", 
                                                transformationPattern="REGEX:(.*Speed.*)∩JSONPATH:$.Speed", 
                                                commandTopic="tasmota/ullu3/cmnd/Speed", min=0, max=6, step=1]
        Type colorHSB : color "color" 
                                                [stateTopic="tasmota/ullu3/RESULT", 
                                                transformationPattern="REGEX:(.*HSBColor.*)∩JSONPATH:$.HSBColor",
                                                commandTopic="tasmota/ullu3/cmnd/HSBColor" ]

    }

   
Thing topic ullu4 "ullu4"  {

    Channels:
        Type switch : power "power" 
                                                [stateTopic="tasmota/ullu4/stat/RESULT", 
                                                transformationPattern="REGEX:(.*POWER.*)∩JSONPATH:$.POWER",
                                                commandTopic="tasmota/ullu4/cmnd/POWER"]
        Type dimmer : brightness "brightness" 
                                                [stateTopic="tasmota/ullu4/stat/RESULT", 
                                                transformationPattern="REGEX:(.*Dimmer.*)∩JSONPATH:$.Dimmer", 
                                                commandTopic="tasmota/ullu4/cmnd/Dimmer", min=0, max=100, step=1]
        Type switch : fade "fade" 
                                                [stateTopic="tasmota/ullu4/stat/RESULT", 
                                                transformationPattern="REGEX:(.*Fade.*)∩JSONPATH:$.Fade",
                                                commandTopic="tasmota/ullu4/cmnd/Fade"]
        Type dimmer : fadeTime "fadeTime" 
                                                [stateTopic="tasmota/ullu4/stat/RESULT", 
                                                transformationPattern="REGEX:(.*Speed.*)∩JSONPATH:$.Speed", 
                                                commandTopic="tasmota/ullu4/cmnd/Speed", min=0, max=6, step=1]
        Type colorHSB : color "color" 
                                                [stateTopic="tasmota/ullu4/RESULT", 
                                                transformationPattern="REGEX:(.*HSBColor.*)∩JSONPATH:$.HSBColor",
                                                commandTopic="tasmota/ullu4/cmnd/HSBColor" ]

    }


}

The 4 thing definitions are copies of each other with only the name changed.

So now I do the ‘critical interaction’ for all four of them, one by one from the command line:


[dp@star ~]$ mosquitto_pub  -h kranich.intern -u dp -P secret -t 'tasmota/ullu1/cmnd/Speed' -m '1'
[dp@star ~]$ mosquitto_pub  -h kranich.intern -u dp -P secret -t 'tasmota/ullu2/cmnd/Speed' -m '2'
[dp@star ~]$ mosquitto_pub  -h kranich.intern -u dp -P secret -t 'tasmota/ullu3/cmnd/Speed' -m '3'
[dp@star ~]$ mosquitto_pub  -h kranich.intern -u dp -P secret -t 'tasmota/ullu4/cmnd/Speed' -m '4'

You will notice how I use the value of the parameter to match the surname of the thing in order to help clarify which action gives which results. Valid values for that parameter go from 0 to 40.

Now, how does that chat look on the mqtt bus?


30.07.2020 19:55:16 tasmota/ullu1/cmnd/Speed 1
30.07.2020 19:55:16 tasmota/ullu1/stat/RESULT {"Speed":1}

30.07.2020 19:55:28 tasmota/ullu2/cmnd/Speed 2
30.07.2020 19:55:28 tasmota/ullu2/stat/RESULT {"Speed":2}

30.07.2020 19:55:38 tasmota/ullu3/cmnd/Speed 3
30.07.2020 19:55:38 tasmota/ullu3/stat/RESULT {"Speed":3}

30.07.2020 19:55:46 tasmota/ullu4/cmnd/Speed 4
30.07.2020 19:55:46 tasmota/ullu4/stat/RESULT {"Speed":4}

4 very similar pairs of lines, the first line the message I sent, each of the tasmotae receives it, sets the new value and finally reports success with the new setting on the RESULT path.
That’s the naughty bit cause tasmota reports all sorts of name/value pairs and lists of them or singles under the same path. Speed and Fade are reported as singles while POWER, Dimmer and HSBColor are reported in a list next to each other.

But that is well known and for that the regex and jsonpath expression are chained together in the things file above.

So how does all that reflect on the openhab.log?


'
2020-07-30 19:55:16.162 [WARN ] [t.generic.ChannelStateTransformation] - Executing the JSONPATH-transformation failed: Invalid path '$.Dimmer' in '{"Speed":1}'
2020-07-30 19:55:16.166 [WARN ] [t.generic.ChannelStateTransformation] - Executing the JSONPATH-transformation failed: Invalid path '$.POWER' in '{"Speed":1}'
2020-07-30 19:55:28.709 [WARN ] [t.generic.ChannelStateTransformation] - Executing the JSONPATH-transformation failed: Invalid path '$.POWER' in '{"Speed":2}'
2020-07-30 19:55:28.713 [WARN ] [t.generic.ChannelStateTransformation] - Executing the JSONPATH-transformation failed: Invalid path '$.Dimmer' in '{"Speed":2}'
2020-07-30 19:55:46.458 [WARN ] [t.generic.ChannelStateTransformation] - Executing the JSONPATH-transformation failed: Invalid path '$.POWER' in '{"Speed":4}'
2020-07-30 19:55:46.460 [WARN ] [t.generic.ChannelStateTransformation] - Executing the JSONPATH-transformation failed: Invalid path '$.Dimmer' in '{"Speed":4}'

4 commands. 1, 2, 4 lead to an error that’s reported in the log. What happened to #3 and made her fail to fail?
That’s sort of unfair, isn’t it - the others found a way to drop out and she has to work?
And another bit: Why no warnings from fade and color?

Glad for any pointers,
best regards

2

Have you installed the REGEX add-on?

Have you restarted the binding since editing your Things? Many bindings are not so good at catching small edits. I expect you were copy-pasting some stuff before this?

3

Thanks for your input. I did a bundle:restart for both of them and repeated my test and the result was just like before.

4

It’s likely to be a configuration issue, not a sometimes-there, sometimes-not bug.
Do you have the expected load/refresh message for your xxx.things file in your openhab.log?
Is your edited things file in UTF-8 format?
Sometimes invisible control characters creep in. Have a close look at your problem lines, or type them afresh (without copy-paste).

1 Like
5

things file loaded successfully after each edit,
If the file wasn’t utf-8 the ∩-symbol wouldn’t show, but it does.
I thought of invisible rot myself and had thrown all the thing definitions out except the one that worked, and copied the other ones from that. No change though. That #3 that works had been the first and model for all the other ones.

  • …
    There’s one thing that can be really frustrating when working with complex systems. You have that issue. Reproducible (on your box) and really odd. You’ve circumfenced it, bare bones example, steps by step, bizarre outcome, on every test.
    Before you shout the ‘b’-word, out of caution and because you feel that’s the next thing your debugging partner would suggest, you just restart the whole thing.
    Wait patiently until the system found it’s equilibrium. Do your tests again. And? nothing. nada. gone. war wat?
    Thanks @rossko57 for your input and suggestions and questions.
    Looks like a restart solved it (for now)
6

I share your pain! Now, if something doesn’t work as I expect, and I’m certain I haven’t made a mistake (…), I do the following:

  1. Deliberately malform (just type in nonsense, or remove a needed ") the particular configuration file I’m working on (especially things files), and save it. openHAB will complain in the logs, and for things files it will unload all the things contained within it. I then fix the file and re-save, prompting openHAB to reload all the things back in, with the extra edits included. This works for 99% of any wierd issues I encounter.
  2. If that doesn’t work: Restart openHAB.
1 Like
7

that’s a clever one! I’ll keep it
thanks