Zigbee2mqtt revisited: No more ugly transformations

Hi Andre, thnx for the time your spending on this.
Here is a Screenshot from the device:

I also changed the MQTT output type.

I made 2 other channels with this bulb and everything is working correct:


And i made a String item:


The result shown is the payload:

{"brightness":153,"color":"hue":308,"saturation":100,"x":0.4066,"y":0.1643},"color_mode":"xy","color_temp":153,"color_temp_startup":341,"linkquality":47,"power_on_behavior":"off","state":"ON","update":{"state":"available"}} 

The only thing i can think of is the .js Scripts. I checked them 10 times and they are just as yours are.
The example you give in the zigbeeColorIN.js :

example input: {"brightness":50,"color":{"hue":359,"saturation":100,"x":0.6942,"y":0.2963},"color_mode":"xy","color_temp":158}

differs from the text i read in Licht Büro Text, thats the only difference i see.
I Have the Java Script, Regex and Json Path transformation installed.

The Channel:

  • id: MQQTTZ2MColorCh1Test
    channelTypeUID: mqtt:color
    label: LEDKochinselXY
    description: null
    configuration:
    commandTopic: zigbee2mqtt/0x0017880106788d39/set
    colorMode: XYY
    transformationPatternOut: JS:zigbeeColorOUT.js
    formatBeforePublish: “%”
    stateTopic: zigbee2mqtt/0x0017880106788d39
    transformationPattern: JS:zigbeeColorIN.js

I created an item with no profile:

Anything i missed ? I am going crazy :slight_smile:

MUHAHAHAHA I got it working :slight_smile:

Well at least i can control the color etc. from the Thing/item in the UI.
I still have to try to get it going in the Openhab UI. But that is another adventure.

After cleaning up all the Channels and items i have tried with and adding the “attribute and json” Output type it works.
I will test if it really works with the “attribute and json” option or if i just messed up everything, even with checking 100 times there must be something that was still wrong.
At the end i learned a lot.

Thank you Andre for helping me out here and spending your time on my issue.!! Big Thank You!!!
If you ever happen to be near Frankfurt/Germany give me a buzz and i will pay you a coffee or beer.
If my investigation turns out that i really missed something even though i checked many times, it will be more than 1 beer :slight_smile:

1 Like

Sure - your welcome. I just spotted the missing “{” at your example input. But glad you could figure this out :clap:t3:

And if I in that region in the near future I may be getting back to you :slight_smile:

Also, there are some pre made widgets for color and stuff. Just search for it - should also be fairly easy to implement :+1:

:+1:

I recently bought a GIDEALED ZC05M RGBW LED strip and use it with Z2M (in attribute output mode). After reading this thread, I managed to send HSB values through openHAB (via the colorpicker) to Z2M. This works so far. However, I can’t get openHAB to receive external state updates originating e.g. from the Z2M Web UI. The strip changes colors, but the openHAB HSB color item still keeps the value set via openHAB. Here’s the relevant config:

Things:

Thing mqtt:topic:LightbandStairs "Lichtband Treppe" (mqtt:broker:mosquitto) @ "Treppe" {
  Channels:
    Type number   : hue         [ stateTopic="zigbee2mqtt/Lightband_Stairs/color-hue", commandTopic="zigbee2mqtt/Lightband_Stairs/set/color-hue", min=0, max=360 ]
    Type dimmer   : saturation  [ stateTopic="zigbee2mqtt/Lightband_Stairs/color-saturation", commandTopic="zigbee2mqtt/Lightband_Stairs/set/color-saturation", min=0, max=100 ]
    Type number   : brightness  [ stateTopic="zigbee2mqtt/Lightband_Stairs/brightness", commandTopic="zigbee2mqtt/Lightband_Stairs/set/brightness", min=0, max=254 ]
    Type color    : color_hsb   [ stateTopic="zigbee2mqtt/Lightband_Stairs/color-hsb", commandTopic="zigbee2mqtt/Lightband_Stairs/set", formatBeforePublish="{\"color\":{\"hue\":%s,\"saturation\":%s},\"brightness\":%s}", onBrightness=50 ]
    Type dimmer   : color_temp  [ stateTopic="zigbee2mqtt/Lightband_Stairs/color_temp", commandTopic="zigbee2mqtt/Lightband_Stairs/set/color_temp", min=153, max=500 ]
    Type switch   : state       [ stateTopic="zigbee2mqtt/Lightband_Stairs/state", commandTopic="zigbee2mqtt/Lightband_Stairs/set/state" ]
}

Items:

Number LightbandStairsHue        "Licht Treppe (Hue)"                              { channel="mqtt:topic:LightbandStairs:hue" }
Dimmer LightbandStairsSaturation "Licht Treppe (Saturation)"                       { channel="mqtt:topic:LightbandStairs:saturation" }
Number LightbandStairsBrightness "Licht Treppe (Brightness)"                       { channel="mqtt:topic:LightbandStairs:brightness" }
Switch LightbandStairs           "Licht Treppe [MAP(de.map):%s]"                   { channel="mqtt:topic:LightbandStairs:state" }
Dimmer LightbandStairsColorTemp  "Licht Treppe Farbtemperatur [%s]"                { channel="mqtt:topic:LightbandStairs:color_temp" }
Color  LightbandStairsColorHSB   "Licht Treppe Farbe (HSB) [%s]"                   { channel="mqtt:topic:LightbandStairs:color_hsb" }

Strip state example from (Z2M Web UI):

{
  "brightness": 100,
  "color": {
      "hue": 359,
      "saturation": 100,
      "x": 0.6942,
      "y": 0.2963
  },
  "color_mode": "xy",
  "color_temp": 158,
  "color_temp_startup": 65535,
  "last_seen": "2023-01-24T16:19:34+01:00",
  "linkquality": 29,
  "power_on_behavior": "off",
  "state": "OFF"
}

As I said, sending HSB color updates through MQTT works. However, I have to use the above formatBeforePublish. If I use the proposed/default {“color”: “hsb”: “%s,%s,%s”}, the brightness won’t be updated. As a fix for the “openHAB doesn’t get external changes” problem, I introduced a rule to sync hue, saturation and brightness to the openHAB HSB color item, if the three attributes are changed externally, because these (attribute) changes are picked up by openHAB:

rule "Compensate for broken MQTT color state topic updates"
when
  Item LightbandStairsHue changed or
  Item LightbandStairsSaturation changed or
  Item LightbandStairsBrightness changed
then
  LightbandStairsColorHSB.postUpdate(LightbandStairsHue.state + "," + LightbandStairsSaturation.state + "," + LightbandStairsBrightness.state)
end

Now, I can change colors using openHAB, Z2M or MQTT and everyone is happy - i.e. the states in sync. But is there a way to get rid of the rule fix? I’d even go the “output attribute_and_json route”, if the solution is more elegant. For example, I don’t know if the stateTopic for color_hsb is right. No matter what I’ve tried so far (no postfix, color, color-hsb), nothing changes. Maybe there’s some magic hidden somewhere that I’m not aware of?

IIUC, there are two problems:

  1. Setting brightness doesn’t work. Try {"color":{"hsb":"360,100,100"}} instead of {"color":{"hue":360,"saturation":100},"brightness":100}
  2. openHAB gets out of sync. Use a transformationPattern to transform the state message from Zigbee2MQTT into H,S,B, see Zigbee2mqtt revisited: No more ugly transformations - #201 by AndreHimSelf .
1 Like

Thanks for the hint, @anon71759204! #1 (setting brightness) wasn’t a problem, that already worked. The link in #2, however, did it. I changed Z2M’s output type to “attribute_and_json” and wrote javascript transformations for reading and writing hsb color values. Now I don’t need to manually sync hue, saturation and brightness with the HSB color.

1 Like

so what’s the best settings too use now?

output attribute&json
home assistant on?
autodiscovery

output attribute
home assistant off
add things with files

Hi,
For me, it was a mix of the above things:

  • enable Attribute & JSON in ZigBee2Mqtt
  • HomeAssist off in ZigBee2Mqtt (don’t know if nessesarry)
  • Add thing manually in openHAB
  • Use JS For Conversion of Values in openHAB Thing
1 Like

Hello, i’m trying to get the effect channel working that zigbee2mqtt uses for lightbulbs such as philips hue, tradfri etc…

z2mqtt mode:

advanced:
   output: attribute

From the zigbee2mqtt docs
Philips Hue White bulb:

To write (/set) a value publish a message to topic zigbee2mqtt/FRIENDLY_NAME/set with payload {"effect": NEW_VALUE}. The possible values are: blink, breathe, okay, channel_change, finish_effect, stop_effect

I’m using attribute mode no json transform should be needed.
things:

// Philips White 1
Thing topic Test_BULB_1 "Ligthbulb (Philips)"{
    Channels:
        Type string : effect "Effekt" [stateTopic="zigbee2mqtt/Philips_WHITE_BULB_1/effect", commandTopic="zigbee2mqtt/Philips_WHITE_BULB_1/set/effect"]
}

items:

String Philips_White_BULB_1_EFFECT "Lightbulb Effekt [%s]" {channel="mqtt:topic:mosquitto:Test_BULB_1:effect"}

rules:

....
Philips_White_BULB_1_EFFECT.sendCommand("okay")

Openhab log looks good:

2023-04-29 19:52:42.556 [INFO ] [openhab.event.ItemCommandEvent      ] - Item 'Philips_White_BULB_1_EFFECT' received command okay
2023-04-29 19:52:42.562 [INFO ] [penhab.event.ItemStatePredictedEvent] - Item 'Philips_White_BULB_1_EFFECT' predicted to become okay

However the bulb does nothing and zigbee2mqtt log shows:

error 2023-04-29 19:54:32: Invalid message 'null', skipping...

The manual way via terminal:

mosquitto_pub -h localhost -t zigbee2mqtt/Philips_WHITE_BULB_1/set/effect -m okay

leads to success and the bulb is blinking as it should.

Is the DSL .sendCommand function formatting the string in some other way than raw as zigbee2mqtt log shows ‘null’ ? An example would be nice in case someone has as working setup using attribute mode only in zigbee2mqtt because i quite wasted some time today :grinning:.

Thank you, winux

Your item name in the items definition does not match the item name in your rule.

This does not solve the issue though, as you can see in the logs. The item in the log outputs also exists. Sorry, that was only a copy paste error on my side =). I copied the wrong line while experimenting and just fixed the original post.

edit:
This is really strange… Is there something missing in the things file? Something like formatBeforePublish or allowedStates=“blink,okay”? Makes no sense though. I guess it has something to do with the formatting of the string value. zigbee2mqtt expects a raw value and it’s getting ‘null’ from either the sendCommand function or the things channel mqtt commandTopic.
I’d appreciate if someone who has one of these bulbs could give it a try and report back, because i guess not many people are using this feature in that attribute mode scenario ^^.

edit2:
Ok i solved it. The solution was the restart of openhab while having the correct setup as in the post above… It did not work for me because i made changes to the files without restarting the appropriate services. So basicly it did not write to the correct topic all the time. DUUUHHH… half a day wasted but happy in the end :rofl:

This will throw an error if homeassistant is enabled.
It took me a while to capture this :upside_down_face:

As this thread is currently the best source for zigbee2mqtt-examples here is my solution to connect Lidl bulbs (but should work with other manufacturers too).
One problem is that the range of the brightness is hardware specific and not a percentage value like in OH. And there is a bug in z2m with hsb-payloads.

Solution:
For the brightness item there is a (not very well documented) feature (brightness_percent) to let z2m do the conversion.
Then for the color we need a transformation and we need the attribute_and_json option within the z2m-config enabled.

Here the example:
First the item-definition in file-format:

Thing mqtt:topic:Lidl1 "Lidl1" (mqtt:broker:MosquittoMqttBroker) { Channels:
  Type switch : state "State"           [ stateTopic = "zigbee2mqtt/Lidl1/state",      commandTopic = "zigbee2mqtt/Lidl1/set",             on="ON", off="OFF" ]
  Type color : color "Color"            [ stateTopic = "zigbee2mqtt/Lidl1/color-hsb",  commandTopic = "zigbee2mqtt/Lidl1/set" , transformationPatternOut="JS:color_hsb_fix.js"]
  Type dimmer : brightness "brightness" [ stateTopic = "zigbee2mqtt/Lidl1/brightness_percent", commandTopic = "zigbee2mqtt/Lidl1/set/brightness_percent" ]
  Type datetime : last_seen "last_seen" [ stateTopic = "zigbee2mqtt/Lidl1/last_seen" ]    
} 

Then the file transformation color_hsb_fix.js which has to be placed in the transform-folder:

(function(color){
  //var log = Java.type("org.slf4j.LoggerFactory").getLogger("org.openhab.model.script.fix_hsb");
  //log.info("color_in: " + color) ;
  var bright = Number(color.split(/[,:]/)[2]) * 2.54;

  var returnVal = '{"brightness":' + bright.toFixed(0) + ',"color":{"h":' + Number(color.split(/[,:]/)[0]) + ',"s":' + Number(color.split(/[,:]/)[1]) + '}}';    
  return returnVal;
})(input)

The point here is to not send hsb (not working in z2m) but to send the brightness and for the color only “h” (Hue) and “s” (Saturation) in one json-payload.
Took me a while to figure this out, so I hope this may help somebody else.

3 Likes

brightness_percent not updated towards openhab for a dimmer

I am on the way to integrate some zigbee-devices.
I have setup mosquitto and zigbee2mqtt with openhabian.
After the installtion of the mqtt-binding i have create the brigde and a thing for a dimmer:

UID: mqtt:topic:mqtt_ohshp:Bad_Dimmer
label: MQTT Bad_Dimmer
thingTypeUID: mqtt:topic
configuration: {}
bridgeUID: mqtt:broker:mqtt_ohshp
channels:
  - id: ch_Bad_Dimmer_Dim
    channelTypeUID: mqtt:dimmer
    label: ch_Bad_Dimmer_Dim
    description: ""
    configuration:
      commandTopic: zigbee2mqtt/OG_Bad_Dimmer/set/brightness_percent
      min: 0
      qos: 1
      stateTopic: zigbee2mqtt/OG_Bad_Dimmer/brightness_percent
      max: 100
  - id: ch_Bad_Dimmer_Switch
    channelTypeUID: mqtt:switch
    label: ch_Bad_Dimmer_Switch
    description: ""
    configuration:
      commandTopic: zigbee2mqtt/OG_Bad_Dimmer/set/state
      stateTopic: zigbee2mqtt/OG_Bad_Dimmer/state

the device in zigbee2mqtt:

Gerätename
    OG_Bad_Dimmer
Beschreibung

Zuletzt gesehen
    N/A
Verfügbarkeit
    Deaktiviert
Geräte-Typ
    Router
Zigbee-Modell
    HK-LN-DIM-A
Zigbee-Hersteller
    Sunricher
Beschreibung
    ZigBee AC phase-cut dimmer
Unterstützungsstatus

    Unterstützt
IEEE-Adresse
    0x943469fffea79e75
Netzwerk-Adresse
    0x03BA
Firmware-Datum
    NULL
Firmware-Version
    2.9.2_r54
Hersteller
    LED-Trading
Modell
    HK-LN-DIM-A
Spannungsversorgung
Interview erfolgreich
    Trifft zu

Except one case everything is running fine.
The olny issue i am facing is that the brightness_percent value is not updated towards openhab in cases where i change the brightness with the manuell switch wired to the dimmer or in the dashboard from zigbee2mqtt.
With the MQTT Explorer this can been seen:

zigbee2mqtt
▼bridge
state = {"state":"online"}
info = {"commit":"3c962042","config":{"advanced":{"adapter_concurrent":null,"adapter_delay":null,"availability_blacklist":[],"availability_blocklist":[],"availability_passlist":[],"availability_whitelist":[],"cache_state":true,"cache_state_persistent":true,"cache_state_send_on_startup":true,"channel":25,"elapsed":false,"ext_pan_id":[221,221,221,221,221,221,221,221],"homeassistant_legacy_entity_attributes…
devices = [{"definition":null,"disabled":false,"endpoints":{"1":{"bindings":[],"clusters":{"input":["genBasic","genIdentify","genOnOff","genTime","genOta","26","lightingColorCtrl"],"output":["genBasic","genIdentify","genGroups","genScenes","genOnOff","genLevelCtrl","genPollCtrl","lightingColorCtrl","msIlluminanceMeasurement","msTemperatureMeasurement","msRelativeHumidity","msOccupancySensing","ssIasZone","h…
groups = []
extensions = []
logging = {"level":"debug","message":"Responded to OTA request of 'OG_Bad_Dimmer' with 'NO_IMAGE_AVAILABLE'"}
▼OG_Bad_Dimmer = {"brightness":66,"linkquality":255,"state":"ON"}
▼set
state = OFF
brightness_percent = 40
state = ON
brightness = 66
linkquality = 255

**changed brightness, brightness_percent not updated**
OG_Bad_Dimmer = {"brightness":101,"linkquality":255,"state":"ON"}
▼set
state = OFF
brightness_percent = 40
state = ON
brightness = 101
linkquality = 255

Is this a bug in zigbee2mqtt or is it based on the zigbee device or a configuration error?

any hint is wellcome, thanks in advance

I think zigbee2mqtt does not send brightness-percent values. So you would need an incoming transformation for that.
Should be something like (haven’t tested)

(function (x) {
  var tmp = JSON.parse(x);
  var bri = Math.round(tmp.brightness / 2.54);
  return bri
})(input)

Try to adjust the min/max configuration in your MQTT thing for a dimmer channel to 0-254. OpenHAB will then automatically convert it to percents when setting it to Item.

See my example dimmer channel config:

  - id: brightness
    channelTypeUID: mqtt:dimmer
    label: Jas
    configuration:
      commandTopic: zigbee2mqtt/ikea-lamp/set
      min: 0
      formatBeforePublish: '{"brightness": "%s"}'
      stateTopic: zigbee2mqtt/ikea-lamp
      transformationPattern: JSONPATH:$.brightness
      max: 254

Fillally it works!
@djal: i have tried your proposal, but was not sucessfull. Error in zigbee2mqtt with …NULL… don#t remember exactly the message.

@Larsen
This was the solution even i have not expected that brightness_percent is a oneway toward zigbee2mqtt.
here is my configuaration in case someelse has a similar problem:

Thing+Channel:
exceprt from UI:

UID: mqtt:topic:mqtt_ohshp:Bad_Dimmer
label: MQTT Bad_Dimmer
thingTypeUID: mqtt:topic
configuration: {}
bridgeUID: mqtt:broker:mqtt_ohshp
channels:
  - id: ch_Bad_Dimmer_Dim
    channelTypeUID: mqtt:dimmer
    label: ch_Bad_Dimmer_Dim
    description: ""
    configuration:
      commandTopic: zigbee2mqtt/OG_Bad_Dimmer/set/brightness_percent
      min: 0
      qos: 1
      stateTopic: zigbee2mqtt/OG_Bad_Dimmer
      transformationPattern: JS:SR_Dimm_brightness_in.js
      max: 100
  - id: ch_Bad_Dimmer_Switch
    channelTypeUID: mqtt:switch
    label: ch_Bad_Dimmer_Switch
    description: ""
    configuration:
      commandTopic: zigbee2mqtt/OG_Bad_Dimmer/set/state
      stateTopic: zigbee2mqtt/OG_Bad_Dimmer/state

the JS-Script:

(function (x) {
    var tmp = JSON.parse(x);
    // The brightness value is received as integer 1-254 but needed as percentage
    var bri = Math.round(tmp.brightness / 2.54);
    return bri
  })(input)

:slightly_smiling_face: and thanks to all

hi,
was not sure should I start a new thread or necroing this one…

I got a tuya zigbee fingerbot
TuYa TS0001_fingerbot control via MQTT | Zigbee2MQTT

its paired to my zigbee2mqtt which has some other aqara xiaomi sensors and ikea buttons paired as well. but with these devices I’m just reading the state, not sending anything to devices.
I cannot make that work (to send ON command to the switch so it moves the “finger”)
I can read signal strength etc…
I configured everything same as examples above in this topic. As I understand no other ugly transformations are needed (nor am I using them for my other zigbee stuff)

thing

    Thing topic Fingerbot "Fingerbot" @ "Living Room"
    {
        Channels:
            Type switch : state "state" [ stateTopic = "zigbee2mqtt/Fingerbot/state", commandTopic = "zigbee2mqtt/Fingerbot/state/set", on="ON", off="OFF" ]   
            Type number : battery "battery" [ stateTopic = "zigbee2mqtt/Fingerbot/battery" ]     
            Type number : linkquality "linkquality" [ stateTopic = "zigbee2mqtt/Fingerbot/linkquality" ]    
            Type string : mode "mode" [ stateTopic = "zigbee2mqtt/Fingerbot/mode" ]             
    }
}

item

Switch Fingerbot          "Fingerbot [%s]"  <button>   {channel="mqtt:topic:mosquitto:Fingerbot:state"}

log from z2mqtt

info  2024-05-15 19:05:04: MQTT publish: topic 'zigbee2mqtt/bridge/event', payload '{"data":{"friendly_name":"Fingerbot","ieee_address":"0xa4c1381e308cb0d6"},"type":"device_announce"}'
info  2024-05-15 19:05:04: MQTT publish: topic 'zigbee2mqtt/bridge/log', payload '{"message":"announce","meta":{"friendly_name":"Fingerbot"},"type":"device_announced"}'
info  2024-05-15 19:05:07: MQTT publish: topic 'zigbee2mqtt/Fingerbot/state', payload 'ON'
info  2024-05-15 19:05:07: MQTT publish: topic 'zigbee2mqtt/Fingerbot/linkquality', payload '18'
info  2024-05-15 19:05:07: MQTT publish: topic 'zigbee2mqtt/Fingerbot/state', payload 'ON'
info  2024-05-15 19:05:07: MQTT publish: topic 'zigbee2mqtt/Fingerbot/linkquality', payload '18'

log when i try to send ON command to the switch

info  2024-05-15 19:03:03: MQTT publish: topic 'zigbee2mqtt/bridge/log', payload '{"message":{"friendly_name":"Fingerbot/state"},"type":"entity_not_found"}'
error 2024-05-15 19:03:03: Entity 'Fingerbot/state' is unknown

any help in getting this working would be appreciated

thx

your commandTopic is wrong, must be something like this:

commandTopic = "zigbee2mqtt/Fingerbot/set"