[MQTT 2.5] Problem with receiving status updates on some things

[JustLuke (Luke)]

Hi,

I have a problem with the mqtt 2.5 binding:
for some things the receiving status updating is working only the first time and after that just nothing happens when there is an update on the corresponding mqtt topic.
But for other items it is working without any problem.

Here is what I did:
manual publish: zigbee2mqtt/hue_white_1/set with {“brightness”:“150”}
observed the status update from the device (checked in a mqtt client)
log:

2020-01-10 15:50:16.116 [TRACE] [ansport.mqtt.internal.ClientCallback] - Received message on topic 'zigbee2mqtt/hue_color_wohnzimmer_rechts' : {"state":"ON","linkquality":21,"brightness":150,"color":{"x":0.45497869131350777,"y":0.1453366652166652,"saturation":100,"hue":46.2},"color_temp":153,"color_mode":1}
2020-01-10 15:50:16.118 [TRACE] [ansport.mqtt.internal.ClientCallback] - Topic match for 'zigbee2mqtt/hue_color_wohnzimmer_rechts' using regex zigbee2mqtt/hue_color_wohnzimmer_rechts

2020-01-10 15:50:16.121 [TRACE] [ansport.mqtt.internal.ClientCallback] - No topic match for 'zigbee2mqtt/hue_color_wohnzimmer_rechts' using regex zigbee2mqtt/hue_white_1

2020-01-10 15:50:16.123 [TRACE] [ansport.mqtt.internal.ClientCallback] - No topic match for 'zigbee2mqtt/hue_color_wohnzimmer_rechts' using regex zigbee2mqtt/osram_plug_3

2020-01-10 15:50:16.126 [TRACE] [ansport.mqtt.internal.ClientCallback] - No topic match for 'zigbee2mqtt/hue_color_wohnzimmer_rechts' using regex zigbee2mqtt/osram_plug_2

2020-01-10 15:50:16.128 [TRACE] [ansport.mqtt.internal.ClientCallback] - No topic match for 'zigbee2mqtt/hue_color_wohnzimmer_rechts' using regex homeassistant/.*

2020-01-10 15:50:16.130 [TRACE] [ansport.mqtt.internal.ClientCallback] - No topic match for 'zigbee2mqtt/hue_color_wohnzimmer_rechts' using regex zigbee2mqtt/bridge/log

2020-01-10 15:50:16.133 [TRACE] [ansport.mqtt.internal.ClientCallback] - No topic match for 'zigbee2mqtt/hue_color_wohnzimmer_rechts' using regex zigbee2mqtt/osram_plug_1

2020-01-10 15:50:16.135 [TRACE] [ansport.mqtt.internal.ClientCallback] - No topic match for 'zigbee2mqtt/hue_color_wohnzimmer_rechts' using regex zigbee2mqtt/hue_color_wohnzimmer_links

2020-01-10 15:50:16.137 [TRACE] [ansport.mqtt.internal.ClientCallback] - No topic match for 'zigbee2mqtt/hue_color_wohnzimmer_rechts' using regex zigbee2mqtt/bridge/state

2020-01-10 15:50:16.140 [TRACE] [ansport.mqtt.internal.ClientCallback] - No topic match for 'zigbee2mqtt/hue_color_wohnzimmer_rechts' using regex valetudo/rockrobo/state

2020-01-10 15:50:16.142 [TRACE] [ansport.mqtt.internal.ClientCallback] - No topic match for 'zigbee2mqtt/hue_color_wohnzimmer_rechts' using regex [^/]*/[^/]*/\$homie

-> Looks like JS:mapHueToPercent.js is not triggered but at least we get the mqtt status update

manual publish: zigbee2mqtt/hue_white_1/set with {“brightness”:“50”}
observed the status update from the device (checked in a mqtt client)
log: nothing

waited some time

manual publish: zigbee2mqtt/hue_white_1/set with {“brightness”:“150”}
observed the status update from the device (checked in a mqtt client)
log: nothing

[…]

I have an other device (zigbee2mqtt/hue_white_1) where I use JSONPATH:$.brightness + a rule for conversion and for this device it’s working without a problem: I get the “Received message on topic” + “Topic match for” message for every update.

Also double checked mapHueToPercent.js script (BTW: transformationPatternOut is working without a problem).
Furthermore I pasted the mapHueToPercent.js to a browsers javascript console to check if it’s valid js (with a small change: var logger = console)

Note: I’m also experiencing this bug: https://github.com/openhab/openhab-addons/issues/6243 - working around it with a “sudo service openhab2 restart” after each change. The next update after a restart is always catched.

Environment:
2.5.0 Release Build

openhab> version
4.2.7
openhab> feature:list | grep mqtt
openhab-misc-mqttbroker                           x 2.5.0            x          x Uninstalled x openhab-addons-2.5.0     x MQTT Broker Moquette
openhab-binding-mqtt                              x 2.5.0            x x        x Started     x openhab-addons-2.5.0     x MQTT Binding
openhab-persistence-mqtt                          x 1.14.0           x          x Uninstalled x openhab-addons-2.5.0     x MQTT Persistence
openhab-core-io-transport-mqtt                    x 2.5.0            x          x Started     x distro-2.5.0             x
openhab-transport-mqtt                            x 2.5.0            x          x Started     x distro-2.5.0             x MQTT Transport

Things:

Bridge mqtt:broker:MQTTBroker [ host="127.0.0.1", secure=false, username="openhabian", password="asdqwe" , clientID="openhab" ]

{

    Thing topic hue_white_1 "Hue_white_1" @ "Wohnzimmer" {

        Type switch : power [ stateTopic="zigbee2mqtt/hue_white_1", transformationPattern="JSONPATH:$.state", on="ON", off="OFF", commandTopic="zigbee2mqtt/hue_white_1/set", formatBeforePublish="{\"state\":\"%s\"}"]

        Type number : brightness [ stateTopic="zigbee2mqtt/hue_white_1", transformationPattern="JSONPATH:$.brightness", min=0, max=255, commandTopic="zigbee2mqtt/hue_white_1/set", formatBeforePublish="{\"brightness\":\"%s\"}"]        

    }

    Thing topic hue_color_wohnzimmer_links "Hue_color_wohnzimmer_links" @ "Wohnzimmer" {

        Type switch : power "An/Aus" [ stateTopic="zigbee2mqtt/hue_color_wohnzimmer_links", transformationPattern="JSONPATH:$.state", on="ON", off="OFF", commandTopic="zigbee2mqtt/hue_color_wohnzimmer_links/set", formatBeforePublish="{\"state\":\"%s\"}"]

        Type number : brightness "Helligkeit" [ stateTopic="zigbee2mqtt/hue_color_wohnzimmer_links", transformationPattern="JS:mapHueToPercent.js", min=0, max=255, commandTopic="zigbee2mqtt/hue_color_wohnzimmer_links/set", transformationPatternOut="JS:mapPercentToHue.js"]

        Type colorHSB : colorHSB "Farbe" [stateTopic="zigbee2mqtt/hue_color_wohnzimmer_links", transformationPattern="JS:xyz2hsv.js", commandTopic="zigbee2mqtt/hue_color_wohnzimmer_links/set", transformationPatternOut="JS:hsv2xyz.js"]

        Type number : temperature "Farbtemperatur" [commandTopic="zigbee2mqtt/hue_color_wohnzimmer_links/set", formatBeforePublish="{\"color_temp\":\"%s\"}", min=153, max=500]

    }

    Thing topic hue_color_wohnzimmer_rechts "Hue_color_wohnzimmer_rechts" @ "Wohnzimmer" {

        Type switch : power "An/Aus" [ stateTopic="zigbee2mqtt/hue_color_wohnzimmer_rechts", transformationPattern="JSONPATH:$.state", on="ON", off="OFF", commandTopic="zigbee2mqtt/hue_color_wohnzimmer_rechts/set", formatBeforePublish="{\"state\":\"%s\"}"]

        Type number : brightness "Helligkeit" [ stateTopic="zigbee2mqtt/hue_color_wohnzimmer_rechts", transformationPattern="JS:mapHueToPercent.js", min=0, max=255, commandTopic="zigbee2mqtt/hue_color_wohnzimmer_rechts/set", transformationPatternOut="JS:mapPercentToHue.js"]

        Type colorHSB : colorHSB "Farbe" [stateTopic="zigbee2mqtt/hue_color_wohnzimmer_rechts", transformationPattern="JS:xyz2hsv.js", commandTopic="zigbee2mqtt/hue_color_wohnzimmer_rechts/set", transformationPatternOut="JS:hsv2xyz.js"]

        Type number : temperature "Farbtemperatur" [commandTopic="zigbee2mqtt/hue_color_wohnzimmer_rechts/set", formatBeforePublish="{\"color_temp\":\"%s\"}", min=153, max=500]

    }

}

mapHueToPercent.js:

(function(value){

    var logger = Java.type("org.slf4j.LoggerFactory").getLogger("mapHueToPercent");

    logger.warn("mapHueToPercent");

    var data = JSON.parse(value);

    var percentValue = parseInt(data.brightness)/255.0

    logger.warn(data.brightness + "->" + percentValue);

    return percentValue;

})(input)

mapPercentToHue.js:

(function(value){

    var logger = Java.type("org.slf4j.LoggerFactory").getLogger("mapPercentToHue");

    logger.warn("mapPercentToHue");

    var hueValue = Math.round(value*255.0)

    logger.warn(value + "->" + hueValue);   

    return '{"brightness":"' + hueValue + '"}';

})(input)

I’m investigating on this problem for multiple days now and getting out of ideas…

[H102 (CM6.5 H102)]

Have you read this topic:

Zigbee2mqtt revisited: No more ugly transformations

I’ve been using zigbee2mqtt for quite some time, for details look here: and here: Recently I discovered a new option of zigbee2mqtt: attribute output. This option, in company with the mqtt2 binding, provides a nice solution. This option simply gets rid of the json-formatted output and transmits all attributes separately. You enable it via configuration.yaml. experimental: output: attribute The benefits are: no more json-encoded output => no jsonpath-transformation no more missing …

For brightness I use:

getZigbeeBrightness.js

(function(x){

    var result;
 
    var json = JSON.parse(x);  
    result = json.brightness * 100 / 255;

    return result;
    
})(input)

setZigbeeBrightness.js

(function(x){

    var brightness = x * 255/100;

    var result = new Object();
    result.brightness = brightness;
 
    return JSON.stringify(result);
    
})(input)

My things file:

Thing topic zigbee2mqtt "Bedroom Light" @ "Bedroom" {
    Channels:
        Type switch : power  "Power"               [ stateTopic="zigbee/0xb0ce1814030ac279/state",
                                                    commandTopic="zigbee/0xb0ce1814030ac279/set/state", on="ON", off="OFF" ]
        Type dimmer : dimmer "Dimmer"              [ stateTopic="zigbee/0xb0ce1814030ac279",
                                                    commandTopic="zigbee/0xb0ce1814030ac279/set", transformationPattern="JSONPATH:$.brightness", formatBeforePublish="{\"brightness\":%s}" ]

[JustLuke (Luke)]

Hi H102,

thanks for you answer!

@H102

Have you read this topic:
Zigbee2mqtt revisited: No more ugly transformations

Yes - but I decided against it as it is a experimental feature.
But maybe I should try it - I like the idea of not longer doing manual serialization.
Do you know if I call fallback to json cases I want to send multiple values at once (e.g. transition + color).

But in the end it does not solve my problem: I can’t trust that the mqtt biding is subscribed to the state topics (as long as I do not know the reason).

[H102 (CM6.5 H102)]

@JustLuke

Yes - but I decided against it as it is a experimental feature.

It’s been working good for me so far.

@JustLuke

Do you know if I call fallback to json cases I want to send multiple values at once (e.g. transition + color).

Can’t say as I do not have color bulbs to test with.

[JustLuke (Luke)]

@H102

Can’t say as I do not have color bulbs to test with.

I don’t think the color bulbs are the problem - I use zigbee2mqtt and when I attach an other mqtt client to the broker I can see that the bulbs are always instantly publishing the new status (correct json).
So the problem is most likely somewhere in the mqtt binding.

But I already enabled tracing for everything with mqtt in the name the the logs above are all I get.

[H102 (CM6.5 H102)]

You are using mosquitto broker and not the embedded on correct? I see above it’s listed but not installed. Have you stopped OH and cleaned the cache since making all the changes?

[JustLuke (Luke)]

@H102

You are using mosquitto broker and not the embedded on correct?

Yes (and I can connect to it and view the messages with mqtt spy)

@H102

Have you stopped OH and cleaned the cache since making all the changes?

It tried resetting the cache but not after the last changes - I will try it now…

[H102 (CM6.5 H102)]

I would also check in /var/lib/openhab2/config/org/openhab/addons.config to see if the embedded broker is listed. If so stop OH and delete it then make sure it’s not in /etc/openhab2/services/addons.cfg as well and restart. If you did have to delete it I would do a clean cache and tmp again.

[JustLuke (Luke)]

@H102

Have you stopped OH and cleaned the cache since making all the changes?

same problem - first update is shown in the log, all others nothing…

@H102

/var/lib/openhab2/config/org/openhab/addons.config

Should be ok:
binding="yamahareceiver,miio,mqtt,expire1,amazonechocontrol"

@H102

/etc/openhab2/services/addons.cfg

Also not there…

[H102 (CM6.5 H102)]

I found this transformation:

(function(i) {

  var rgb = i.split(",");
  var color = {"r":rgb[0],"g":rgb[1],"b":rgb[2]}
  return JSON.stringify({color});

})(input)

It’s for RGB not HSB but could be worth a try. Maybe even see if the item will work as RGB??

Kinda wish I had a color bulb to play with…

EDIT: @JustLuke just found this as well:

JS Transformation file HSBtoRGB.js

(function (x) {
    if (x=="ON" || x=="INCREASE") {
        return '{"state":"ON"}'
    } else if (x=="OFF" || x=="DECREASE") {
        return '{"state":"OFF"}'
    } else {
        var tex = x.split(',');
        s = parseInt(tex[1]);
        v = parseInt(tex[2]);
        h = parseInt(tex[0]);
    
        h=(!h ? 0 : h/360.0);
        s=(!s ? 0 : s/100.0);
        v=(!v ? 0 : v/100.0);

        i = Math.floor(h * 6);
        f = h * 6 - i;
        p = v * (1 - s);
        q = v * (1 - f * s);
        t = v * (1 - (1 - f) * s);
        switch (i % 6) {
            case 0: r = v, g = t, b = p; break;
            case 1: r = q, g = v, b = p; break;
            case 2: r = p, g = v, b = t; break;
            case 3: r = p, g = q, b = v; break;
            case 4: r = t, g = p, b = v; break;
            case 5: r = v, g = p, b = q; break;
        }
        return '{"brightness":'+ Math.round(v * 255)+',"color":{"rgb":"'+ Math.round(r * 255)+','+ Math.round(g * 255)+','+Math.round(b * 255)+'"}}'
    }
})(input)

From this topic

Zigbee2mqtt and Zigbee bulbs small tutorial

Thanks. I’m still on MQTT 1.x, but with MQTT 2.x it should be something like this: Things file: Bridge mqtt:broker:b51_1 "WRT"@"B51" [ host="192.168.x.x", secure=false] Thing mqtt:topic:lg_ikea_color_1 "Ikea Color 1" (mqtt:broker:b51_1) { Channels: Type colorHSB:color "Ikea Color 1" [ stateTopic="zigbee2mqtt/ikea_led_color_1/dumb", commandTopic="zigbee2mqtt/ikea_led_color_1/set", transformationPatternOut="JS:HSBtoRGB.js" ] } Item: Color LG_IKEA_Color_1_M24 …

[JustLuke (Luke)]

Hi @H102,

I finally found the root cause of my problem:
My JS tranformation xyz2hsv.js used a function wich was definierend in lambda style.
It seems that openhab does not like this and fail somewhere silently.
After I rewrote that function to a classical function definition the mqtt subscription is working without any problems multiple times in a row.

thanks for your support!