Getting shelly / shellies announce data via MQTT 2.4

You really don’t need the Timer for the 5A4CE4 Shelly. It will run immediately anyway so putting it into a timer doesn’t really do anything for you.

But, let’s see if we can adjust this Rule so it scales. As written, you would have to add code to this Rule every time you add a Shelly.

I can see several ways to do this:

  1. Add an announce Item for each Shelly, add those to a Group, and loop through the Group to send the announce command to each in turn.

  2. Add a tag to one of the Items associated with this device, let’s say the Online Item, with the unique address for that given shelly. Put the Online Items into a Group and loop through the Group, constructing the topic to send the announce command to using the tag.

  3. Create a Map in the Rule that maps an Item instead of a tag.

  4. Use a .map file in the Rule and trasnform Action instead of a tag.

I’m sure there are more.

The “proper” way to do this would be 1. This is a separate capability for each device and therefore it should be represented as a Channel on the Generic MQTT Thing that represents this Thing and linked to an Item, so let’s show that. I won’t show the Thing or Item definition, you can figure that out with the examples above.

We will call the Group of Announce Items “Shelly_Announces”.

rule "ask for firmware version"
when
        Channel 'astro:sun:local:noon#event' triggered START
then
    // Loop through the Shelly Announce Items and send the "announce" command to each, 5 seconds apart
    Shelly_Announces.members.forEach[ shelly, index | 
        createTimer(now.plusSeconds(5*index), [ | shelly.sendCommand("announce") ] )
    ]
end

The Rule to extract the data would also need to be changed for every new Shelly one adds to the system. Let’s see if we can make it more generic. The only specific part is that switch statement in the middle so I’ll focus there.

Again, there are different ways we can deal with this. This is one of those cases where the REGEX filter we can apply using the MQTT 1.x binding would come in handy as you could just have your FWVer et al Items subscribe to the same topic but only process those that contain the “shelly name” that the Item represents. Perhaps there is already a way we con do this by chaining incoming transformations, which is a feature that the MQTT 2 binding supports.

I’ve only one shelly right now so I can’t fully test it. But it appears to work. And this approach would eliminate the need for the second Rule.

  1. Create a Switch Channel on the Thing for each Shelly that sends “announce” to the command topic for that Shelly device.

  2. Create a Text Channel on the Thing for each Shelly that represents the three pieces of information you care about (IP, FW Version, Firmware Update). These Channels subscribe to shellies/announce and we use a chained transformation to filter on the Shelly name and then extract the value from the JSON.

I don’t use .things files but the Thing would look something like in PaperUI:

The Request Announce Channel is just a switch with a mapping configured so whenever it receives ON or OFF it publishes “announce” to the command topic.

MQTT Command Topic: shellies/shelly1-25A289/command
Custom On/Open value: announce
Custom Off/Closed value: announce

Now let’s look at the IP Channel.

MQTT state topic: shellies/announce
Incoming value transformations: REGEX:(.*shelly1-25A289.*)∩JSONPATH:$.ip

The transformation will match the full message but only if it has the “shelly name”, in this case shelly1-25A289. It appears that there is no error in the logs when the REGEX doesn’t match anything. If it does match then it passes the full JSON to the JSONPATH where the IP field is extracted.

I only have one Shelly set up right now so can’t fully test this, but when I “break” the REGEX on this Thing nothing happens (which is what we want) and when I fix it so it’s correct I get the IP address.

So I recommend implementing all of your second Rule in your Things and link the Items to the Channels.This way the “shelly name” for the device is wholly contained within the Thing freeing you to not know nor care what Shelly decided to name this device in your Rules or your sitemap nor the need to provide some external mapping between that and your chosen Item names.

2 Likes

awesome reply!!! I will test it soon

@rlkoshak
unfortunately I have a problem with the thing configuration mentioned above. The REGEX part works, but JSONPATH is ignored. What kind of special character did you use after the closing bracket, just in front of the J for JSONPATH?

Did you install the JSONPATH transform?

The character is what the binding developer chose to separate transforms. It’s the mathematical set intersection operator I think. You can just copy the character from the text under the filed in PaperUI.

1 Like

EDIT: Could it be that I require a later snapshot of the binding??? I don’t even see this sentence about the special character in my paperui

thanks, this explains a lot, but still doesn’t solve my issue. when I did copy/paste of the character via ssh session, it got converted to a bullet point. so I changed the character encoding in putty to UTF 8 and now it looks good.

Automatic addition in PaperUI:

But it still doesn’t work (just REGEX)

Shelly1_01_IP changed from UNDEF to {“id”:“shelly1-55521D”,“mac”:“B6E62D55521D”,“ip”:“192.168.2.32”,“new_fw”:false, “fw_ver”:“20190214-074430/1.4.7-revertwifi@0f3372b3”}

That could be the case. When I moved this to it’s own tutorial I did indicated that you need 2.5 M1 or later.

I’m pretty sure that chaining transformations didn’t get implemented until after the 2.4 release.

1 Like

ouch, thanks. at least we know why it’s not working. any idea which of different mqtt bindings I need to upgrade? like mqtt or mqtt.generic? and do you know if I will succeed by just updating the binding and staying at OH 2.4 stable? Or will I need to upgrade everything?

You don’t need to change the Things at all. You just need to upgrade the binding.

It will be way easier to upgrade your whole OH. 2.5 M1 was release a month after 2.4 Release. It’s very stable and I’ve been running on it since it was released with no problem.

OH is a rolling release. Honestly, there really isn’t any major difference between a milestone release and a release except the release has received just a tab more testing. But both are really just snapshots in time where there are no known show stopper bugs in OH core that get packaged up and frozen in place.

To just upgrade the binding you will need to uninstall it (and the broker add-on if you are using it too) then download the 2.5 M1 jar file and drop it into the addons folder. It will help to keep the console up and list the bundles to make sure the new binding is picked up and you don’t accidentally end up with both running at the same time.

1 Like

hey, me again. I updated to M1 and applied all your steps mentioned above.

When I was about to update my tutorial with your instructions, I noticed that the REGEX followed by JSON transformation doesn’t work as expected.

Actually the regex part is ignored and just the json part will be used. This way I have the same values (like IP address) assigned to all items. It’s exactly the other way round like with OH 2.4 where the regex worked, but json transform was ignored.

I re-installed the regex transformation add-on, but this didn’t solve the problem.
Actually even while the add-on WAS NOT installed, the json string was still transformed fine (but still ingoring the regex part).

Any idea what’s wrong?
things

 Thing topic Shelly1-01 "Shelly1-01" {
        Channels:
        Type string : online            "Online"                [ stateTopic="shellies/shelly1-55521D/online" ]
        Type switch : applyfwupdate     "ApplyFwUpdate"         [ commandTopic="shellies/shelly1-55521D/command", on="update_fw", off=""]
        Type switch : power             "Power"                 [ stateTopic="shellies/shelly1-55521D/relay/0", on="on", off="off", commandTopic="shellies/shelly1-55521D/relay/0/command", on="on", off="off"]
        Type switch : forceannounce     "ForceAnnounce"         [ commandTopic="shellies/shelly1-55521D/command", on="announce", off="announce" ]
        Type string : ip                "Ip Address"            [ stateTopic="shellies/announce", transformationPattern="REGEX:(.*shelly1-55521D.*)∩JSONPATH:$.ip" ]
        Type string : newfwflag         "New FW Available"      [ stateTopic="shellies/announce", transformationPattern="REGEX:(.*shelly1-55521D.*)∩JSONPATH:$.new_fw" ]
        }
        Thing topic Shelly2-01 "Shelly2-01" {
        Channels:
        Type string : online            "Online"                [ stateTopic="shellies/shellyswitch-5A4CE4/online" ]
        Type switch : applyfwupdate     "ApplyFwUpdate"         [ commandTopic="shellies/shellyswitch-5A4CE4/command", on="update_fw", off=""]
        Type switch : power1            "Power1"                [ stateTopic="shellies/shellyswitch-5A4CE4/relay/0", on="on", off="off", commandTopic="shellies/shellyswitch-5A4CE4/relay/0/command", on="on", off="off"]
        Type switch : power2            "Power2"                [ stateTopic="shellies/shellyswitch-5A4CE4/relay/1", on="on", off="off", commandTopic="shellies/shellyswitch-5A4CE4/relay/1/command", on="on", off="off"]
        Type switch : forceannounce     "ForceAnnounce"         [ commandTopic="shellies/shellyswitch-5A4CE4/command", on="announce", off="announce" ]
        Type string : ip                "Ip Address"            [ stateTopic="shellies/announce", transformationPattern="REGEX:(.*shellyswitch-5A4CE4.*)∩JSONPATH:$.ip" ]
        Type string : newfwflag         "New FW Available"      [ stateTopic="shellies/announce", transformationPattern="REGEX:(.*shellyswitch-5A4CE4.*)∩JSONPATH:$.new_fw" ]
        }

items

String Shellies_AnnounceData_String     "Shellies AnnounceData" { channel="mqtt:topic:mosquitto:ShellyAnnounceData:announcedata" }

Switch Shelly1_01_Switch        "HomeOffice Licht[]"            [ "Switchable" ]                { channel="mqtt:topic:mosquitto:Shelly1-01:power" }
String Shelly1_01_Online        "Shelly1_01 Online [%s]"        ( gShellyOnlineStatus )         { channel="mqtt:topic:mosquitto:Shelly1-01:online" }
String Shelly1_01_NewFWFlag     "Shelly1_01 New FW [%s]"        ( gShellyNewFWFlag )            { channel="mqtt:topic:mosquitto:Shelly1-01:newfwflag" }
Switch Shelly1_01_Ready2Update
String Shelly1_01_IP            "HomeOffice Licht IP [%s]"                                      { channel="mqtt:topic:mosquitto:Shelly1-01:ip" }
Switch Shelly1_01_ApplyFWUpdate "HomeOffice Licht FWUpdate"                                     { channel="mqtt:topic:mosquitto:Shelly1-01:applyfwupdate", autoupdate="false"}
Switch Shelly1_01_ForceAnnounce "HomeOffice Licht ForceAnnounce [%s]"   (gShellyForceAnnounce)  { channel="mqtt:topic:mosquitto:Shelly1-01:forceannounce" }

Switch Shelly2_01_Switch1       "Shelly2_01 Power1"             [ "Switchable" ]        { channel="mqtt:topic:mosquitto:Shelly2-01:power1" }
Switch Shelly2_01_Switch2       "Shelly2_01 Power2"             [ "Switchable" ]        { channel="mqtt:topic:mosquitto:Shelly2-01:power2" }
String Shelly2_01_Online        "Shelly2_01 Online [%s]"        ( gShellyOnlineStatus ) { channel="mqtt:topic:mosquitto:Shelly2-01:online" }
String Shelly2_01_NewFWFlag     "Shelly2_01 New FW [%s]"        ( gShellyNewFWFlag )    { channel="mqtt:topic:mosquitto:Shelly2-01:newfwflag" }
Switch Shelly2_01_Ready2Update
String Shelly2_01_IP            "Shelly2_01 IP [%s]"                                    { channel="mqtt:topic:mosquitto:Shelly2-01:ip" }
Switch Shelly2_01_ApplyFWUpdate "Shelly2_01 FWUpdate"                                   { channel="mqtt:topic:mosquitto:Shelly2-01:applyfwupdate", autoupdate="false"}
Switch Shelly2_01_ForceAnnounce "Shelly2_01 ForceAnnounce [%s]" (gShellyForceAnnounce)  { channel="mqtt:topic:mosquitto:Shelly2-01:forceannounce" }

then I manually publish a fake mqtt topic:

{“id”:“shelly1-55521D”,“mac”:“B6E62D55521D”,“ip”:“justfake”,“new_fw”:true, “fw_ver”:“20190214-074430/1.4.7-revertwifi@0f3372b3”}

log

2019-03-27 21:26:19.575 [vent.ItemStateChangedEvent] - Shellies_AnnounceData_String changed from {id:shelly1-55521D,mac:B6E62D55521D,ip:192.168.2.132,new_fw:true, fw_ver:20190214-074430/1.4.7-revertwifi@0f3372b3} to {id:shelly1-55521D,mac:B6E62D55521D,ip:justfake,new_fw:true, fw_ver:20190214-074430/1.4.7-revertwifi@0f3372b3}

2019-03-27 21:26:19.607 [vent.ItemStateChangedEvent] - Shelly1_01_IP changed from 192.168.2.132 to justfake

2019-03-27 21:26:19.613 [vent.ItemStateChangedEvent] - Shelly2_01_IP changed from 192.168.2.132 to justfake

Thanks in advance

I don’t use .things files for this so I can’t say if there is something wrong there. All I can say is it works for me. If you have both transforms installed and don’t see any errors in the logs I don’t see anything that looks wrong.

that’s indeed interesting. maybe it’s a bug? I re-created a Generic MQTT Thing via PaperUI, create an TestItem in PaperUI and then linked the item to the channel via PaperUI, same issue. If I enter something like “BLA” into the transformation field I do see an error in the log. But as soon as I turn this into REGEX:BLA it will be accepted and ignored.

Just in case something shows up as obvious, I wrote a tutorial on how I did this: MQTT 2.5 M1+ How to implement the equivalent to MQTT1 REGEX filters. That contains my current configuration.

1 Like

REGEX:bla should accept all incoming mqtt values that are exactly “bla”.

If you use regex first, you must make sure that a valid JSON is produced out of that transformation otherwise the jsonpath cannot work successfully.

thanks for trying to help. it’s driving me crazy. I’ve read your tutorial, but it seems all good to me. Here some screenshots:

and the log shows:
2019-03-29 10:03:11.299 [vent.ItemStateChangedEvent] - MQTTTestItem changed from UNDEF to this just doesn't make sense

Finally, if I change the transformation value from “REGEX:BLA” to just “BLA” I do see an expected error in the logs:

2019-03-29 10:33:24.572 [WARN ] [internal.handler.GenericThingHandler] - Channel configuration error

java.lang.IllegalArgumentException: The transformation pattern must consist of the type and the pattern separated by a colon

But then again, as soon as I enter “BLA:BLA” the thing configuration will be accepted and the thing goes online. Again the transformation will just be ignored and the item gets assigned the full announce string.

So I’d say, either my installation is somehow corrupted, or it’s a bug.
Any idea how to generate debug logs for the transformation process?

Ah ok. So you are saying that there should be an error log if a transformation cannot be found instead of just passing the value on.

Please create an issue on GitHub to track this.

But wouldn’t that break this use case?

The use case (i.e. using REGEX to filter out messages published to a single topic so only certain ones get passed to the JSONPATH) depends on REGEX failing silently and passing the full message on if there is no match.

If the transform is malformed it does makes sense that there is an error.

I’m not sure I understand everything going on here.

no, this is not what I’m saying. my problem is that the transformation just does not work. I had set:

REGEX:bla

But the value that was assigned to the item is != bla
This must not be the case, right?

I don’t know if this was already suggested, but you could simplify the whole “command: announce” loop by sending the MQTT message to “shellies/command” with the payload “announce”. This will force all shellies on the MQTT server to send their announce message.

Only if the full message is literally “bla” would the regex match. But because you didn’t provide any parens the REGEX doesn’t actually return anything.

If you want “bla” assigned to the Item, like that your message needs to be just “bla” and the REGEX needs to be

REGEX:(bla)

If you want the Item to be assigned “bla” when any message that contains bla in it the REGEX would be

REGEX:.*(bla).*

If you want the Item to be assigned the full message for any message that contains bla the REGEX needs to be

REGEX:(.*bla.*)

The expression must match the full message. And you must provide parens to indicate the part of the message you want to pass on.

1 Like

You cannot imagine how thankful I am, but at the same time I’m really sorry because I don’t understand.

What I did:

  1. REGEX:(bla)
  2. MQTT message ‘foobar’ on topic ‘shellies/announce’

Second test with:

  1. REGEX:.*(bla).*
  2. MQTT message ‘foobar’ on topic ‘shellies/announce’

What I see in both cases:
MQTTTestItem changed from […] to foobar

It shouldn’t behave like this, right? I’d expect the item value either to be empty or unchanged.