MQTT: How to set UNDEF when invalid data returned from Zigbee2MQTT?

Background:

I have several Keen Home vents configured via Zigbee2MQTT. They mostly work great, but my original Kickstarter vents have a tendency to go offline randomly or otherwise return bad (blank) data. So my logs start having entries like:

2021-06-05 10:04:35.878 [WARN ] [ab.binding.mqtt.generic.ChannelState] - Incoming payload '' not supported by type 'NumberValue'
2021-06-05 10:10:51.900 [WARN ] [ab.binding.mqtt.generic.ChannelState] - Incoming payload '' not supported by type 'NumberValue'
2021-06-05 10:10:54.448 [WARN ] [ab.binding.mqtt.generic.ChannelState] - Command '' not supported by type 'OnOffValue': No enum constant org.openhab.core.library.types.OnOffType.

This means my Item state does not match reality anymore.

How do I get it to set my Item state to UNDEF when this happens? I’m on openHAB 3.1.0.M5

My MQTT things config:

  Thing topic test_keenhome { //Vent
    Type number : battery [ stateTopic="zigbee2mqtt/test_keenhome/battery"]
    Type number : pressure [ stateTopic="zigbee2mqtt/test_keenhome/pressure"]
    Type number : temperature [ stateTopic="zigbee2mqtt/test_keenhome/temperature", transformationPattern="JS:c2f.js"]
    Type dimmer : position [ stateTopic="zigbee2mqtt/test_keenhome/position", commandTopic="zigbee2mqtt/test_keenhome/set/position"]
    Type switch : state [ stateTopic="zigbee2mqtt/test_keenhome/state", commandTopic="zigbee2mqtt/test_keenhome/set/state", on="OPEN", off="CLOSE"]
  }

You probably want to chain a regex transform to filter out the empty messages. See MQTT 2.5 M1+ How to implement the equivalent to MQTT1 REGEX filters

Then create a new channel that filters for the messages that are blank and sets the Item to OFF when a blank is received.

When it’s in an unknown state it should go to UNDEF, but I can’t find a way to do that with a transform. I’ve tried returning UNDEF, Undefined, undef, undefined, -1 (for a Dimmer), and many other variants from a JS transform, none seem to work.

I can set an Item to UNDEF in RulesDSL code, but is there a reason it can’t be set that way by the MQTT Binding itself or done with a Transform? It looks like the enhancement (bug fix?) request from last year ([Javascript Transformation] support NULL/null/UNDEF as NumberValue · Issue #7622 · openhab/openhab-addons · GitHub) is not being addressed.

All transforms always return a String. Depending on the Channel/Item type it should parse that String to a proper state so if you return “UNDEF” it should set the Item to UNDEF. However there are two places where that won’t work:

  • the Item is a String type Item where the Item will get set to “UNDEF” and not UNDEF
  • the message is sent as a command; UNDEF cannot be sent as a command to an Item

There might be other cases and situations where returning “UNDEF” from a transform might not work as well.

But my suggestion was to have two Channels, one that ignores the blank messages and updates your Number Item. The other one matches blank messages and commands a Switch to OFF.

If you want to have your Number Item update to UNDEF when the Switch receives an OFF command you’d use a simple rule for that which triggers when the Switch receives an OFF command and updates the Item with UNDEF.

Ideally and a more MQTT correct way would be for the source of these messages to have a status topic and it would publish ONLINE (or something like that) when it’s online and OFFLINE when it is not. Then the MQTT Thing can be configured to look for those messages on those topics and it will set all the Channels to UNDEF when it sees the OFFLINE message. But the number of MQTT services/devices that do not use MQTT correctly in this way sometimes makes that impossible.

  • the message is sent as a command; UNDEF cannot be sent as a command to an Item

Hmmm… when the state change comes from Z2mqtt, it comes in as a stateTopic (zigbee2mqtt/test_keenhome/position). Are you saying that even when that Channel also has a different commandTopic defined, it’s still receiving it as a command even though it’s a state change?

I’m not sure I understand your recommendation. Wouldn’t I have to also create multiple Items then to act on each other along with extra logic to set states and handle UNDEF properly?

With 20+ vents, this won’t scale well.

I’ll have to find a block of time to experiment.

My .things definition:

  Thing topic test_keenhome { //Vent
    Type number : battery [ stateTopic="zigbee2mqtt/test_keenhome/battery"]
    Type number : pressure [ stateTopic="zigbee2mqtt/test_keenhome/pressure"]
    Type number : temperature [ stateTopic="zigbee2mqtt/test_keenhome/temperature", transformationPattern="JS:c2f.js"]
    Type dimmer : position [ stateTopic="zigbee2mqtt/test_keenhome/position", commandTopic="zigbee2mqtt/test_keenhome/set/position", transformationPattern="JS:NULLfilter.js"]
  }

My .items:

Number  keen1_battery       "Keen1 Battery [%d %%]" (gTest,gSensors,gBattery) ["Battery"] {channel="mqtt:topic:Mosq:test_keenhome:battery", expire="4h,state=0"}
Dimmer  keen1_position      "Keen1 Position [%d]" (gTest) {channel="mqtt:topic:Mosq:test_keenhome:position", autoupdate="false",expire="1h"}
Number  keen1_temp          "Keen1 Temp [%.1f %unit%]" (gVent_temp,gTest) ["Temperature"] {channel="mqtt:topic:Mosq:test_keenhome:temperature",expire="1h"}
Number  keen1_pressure      "Keen1 Pressure [%.1f %unit%]" (gTest) {channel="mqtt:topic:Mosq:test_keenhome:pressure",expire="1h"}

Here’s my NULLfilter.js:

(function(i) {

    var debug = 1

    if (debug>0) {
        var logger = Java.type("org.slf4j.LoggerFactory").getLogger("NULLfilter.js");
        // logger.warn("DEBUG: NULLfilter.js received: " + i );
    }

    if (i=="") {
        if (debug>0) logger.warn("WARN: ***NULLfilter.js received invalid null, returning undefined")
        return undefined;
    } 

    if (debug>1) logger.warn("DEBUG: NULLfilter.js received valid value, returning it: " + i)
    return i;

})(input)

No, but there is a flag to set that will treat incoming messages as a command to the linked Item instead of the usual update.

Yes.

See OH 3 Tips and Tricks, in particular the “Buying in Bulk” section.

There should only need to be one Rule to handle all the Items. Most of the rest is going to be copy/paste/edit.

It still sounds like an ugly workaround for a deficiency in the MQTT Binding. If you can set an Item to UNDEF in code, then the binding should be able to do it, too when it receives a null value. So I definitely support the bug report/enhancement request created over a year ago by @radoslavv.

Thanks for your tips, I think I’ve learned some new things already. Now to delve in and see what works best for me.

Well, that’s pointless. There’s only one UNDEF. What actually happens when your JS returns an “UNDEF” string?

It is possible this binding isn’t coded to handle UndefType through channel updates.

EDIT - yes, eventually the bell rang for me, MQTT binding is deficient in this respect…

Despite the issue title, this is an MQTT channel limitation, not JS transform.

I can think of a cheating workaround.
Have a simple JS on the MQTT channel, that passes “good” readings as-is, but substitutes some silly but distinguishable value for “bad” readings - say 9999.
That should work, as you always get a valid number.
Then apply a transform profile to the channel-Item link. This time we substitute UNDEF for 9999, and pass other numbers.
(The transform profile should not have the same limitation.)

Interesting reading. I was searching for a way to change my invalid values from something like 9999 towards UNDEF and came across this topic. Turns out that you recommend the 9999 so the circle is round :frowning:
The reason why I prefer UNDEF is to get decent graphs in Grafana/Influxdb. The 9999 reading are killing the graphs. Not sure which issue to solve now: go further with UNDEF (if anyone has found a better solution: please tell me) or dive deeper into exception handling of Grafan( still need some time to figure things out).

Well golly, I see no recommendation, just a suggestion of approach and an example plucked out of the air.
You may choose any magic value you like in a fairly large range - say, -1 or 9999999999 or 987654321

For your graphing, you may need to take into account that openHAB persistence never stores UNDEF values. (so you generally get no stored data while Item has UNDEF state)

Indeed, my magic value is 3084 but the principle behind remains the same.
My aim is indeed that no values are stored in the persistance, that’s why I prefer UNDEF instead of 3084 or 9999 because that breaks the graph and makes autoscaling unusable.
Another workaround would be to work with dummy items and copy them to the right item with a rule (when item changed…). This allows to sort out the 9999 values and use a postUpdate(UNDEF).This doubles the number of those items, but I have a script that can handle that :wink:

A better work around is to:

  1. create a .persist file and exclude these Items from being saved.
  2. create a rule that triggers when this Item changes
  3. if and only if it’s a valid state, call .persist() on the Item to save the value

That will only store valid values to the database.

Thanks (again) @rlkoshak! It seems indeed the most clean way to handle this. I’ll first go through the documentation as I haven’t created any persist file until now.