Zigbee2mqtt revisited: No more ugly transformations

AndreHimSelf · December 31, 2022, 6:36am

I’ve got the first part working!

So here is the channel with the regex transformation

configuration: {}
bridgeUID: mqtt:broker:cf94322e0a
channels:
  - id: LEDKochinselHSB
    channelTypeUID: mqtt:color
    label: LEDKochinselHSB
    description: ""
    configuration:
      commandTopic: zigbee2mqtt/LEDKochinsel/set
      colorMode: XYY
      formatBeforePublish: "%s"
      stateTopic: zigbee2mqtt/LEDKochinsel
      transformationPattern: REGEX:s/\{"brightness":(.*?),"color":\{(?<=)"hue":(.*?),"saturation":(.*?)(?=),"x":(.*?),"y":(.*?)\},.*\}/$4,$5,$1/g

I did need to extend the regex but extract only brightness ($1) x($4) and y ($5).

Next I need to get the outgoing stuff working and add the brightness conversion … Man is that difficult to get this stuff working.

I will create an issue on the zigbee2mqtt git as I think they expose the wrong values or at least mix of xyY and HSB…

Update: DONE!
So here are the working channels:

channels:
  - id: LEDKochinselXY
    channelTypeUID: mqtt:color
    label: LEDKochinselXY
    description: null
    configuration:
      commandTopic: zigbee2mqtt/LEDKochinsel/set
      colorMode: XYY
      formatBeforePublish: '{"color":{"x":"%1$s","y":"%2$s"}}'
      stateTopic: zigbee2mqtt/LEDKochinsel
      transformationPattern: REGEX:s/\{"brightness":(.*?),"color":\{(?<=)"hue":(.*?),"saturation":(.*?)(?=),"x":(.*?),"y":(.*?)\},.*\}/$4,$5,$1/g
  - id: LEDKochinselBrightness
    channelTypeUID: mqtt:dimmer
    label: LEDKochinselBrightness
    description: ""
    configuration:
      commandTopic: zigbee2mqtt/LEDKochinsel/set
      min: 0
      formatBeforePublish: '{"brightness":"%s"}'
      stateTopic: zigbee2mqtt/LEDKochinsel
      transformationPattern: JSONPATH:$.brightness
      max: 254
  - id: LEDKochinselColorTemp
    channelTypeUID: mqtt:dimmer
    label: LEDKochinselColorTemp
    description: null
    configuration:
      commandTopic: zigbee2mqtt/LEDKochinsel/set
      min: 153
      formatBeforePublish: '{"color_temp":"%s"}'
      stateTopic: zigbee2mqtt/LEDKochinsel
      transformationPattern: JSONPATH:$.color_temp
      max: 500

Works for my liking. The only thing that is bothering me is that i cant get the brightness contol in the color picker working as it is a diffrent channel - but thats for future me i guess

AndreHimSelf · January 3, 2023, 12:43pm

Well my previous “solution” was bad as it does not convert the brightness part to the correct values as @anon71759204 pointed out earlier So here is what i came up with which now everything is working as expected: (Also most parts are c + p from other community members which i’m really glad that they share thier findings with us.)

openHAB channel config:

channels:
  - id: LEDKochinselXY
    channelTypeUID: mqtt:color
    label: LEDKochinselXY
    description: null
    configuration:
      commandTopic: zigbee2mqtt/LEDKochinsel/set
      colorMode: XYY
      transformationPatternOut: JS:zigbeeColorOUT.js
      formatBeforePublish: "%"
      stateTopic: zigbee2mqtt/LEDKochinsel
      transformationPattern: JS:zigbeeColorIN.js

transformation scripts:

zigbeeColorOUT.js

(function(color){
	// convert 100% to 254 Steps of brightness
	var bright = Number(color.split(/[,:]/)[2]) * 2.54;
	// example output to mqtt: {"brightness":239,"color":{"x":0.601297,"y":0.311017}}
	return '{"brightness":' + bright.toFixed(0) + ',"color":{"x":' + color.split(/[,:]/)[0] + ',"y":' + color.split(/[,:]/)[1] + '}}';
})(input)

zigbeeColorIN.js

// example input: {"brightness":50,"color":{"hue":359,"saturation":100,"x":0.6942,"y":0.2963},"color_mode":"xy","color_temp":158}
(function(color){
	// convert 254 Steps to 100% of brightness
	var bright = Number(color.split(/[,:]/)[1]) / 2.54;
	// example output to openHAB: x,y,brightness - f.e.: 0.6942,0.2963,50
	return color.split(/[,:]/)[8] + "," + color.split(/[,:}]/)[10] + "," + bright.toFixed(0);
})(input)

Airmr · January 5, 2023, 2:33pm

Hi,
i tried to integrate a Hue color bulb connected with Zigbee2Mqtt into OH3 and had no luck so far.
I have the channel config and the .js scripts from the link in this post. The .js files are in the conf/transform folder. I still dont get any color information in my Channel/item.
Through the UI i created the channel and the item.
channels:

id: MQQTTZ2MColorCh1
channelTypeUID: mqtt:color
label: MQQTTZ2MColorCh1
description: “”
configuration:
commandTopic: zigbee2mqtt/0x0017880106788d39/set
colorMode: XYY
transformationPatternOut: JS:zigbeeColorOUT.js
formatBeforePublish: “%”
stateTopic: zigbee2mqtt/0x0017880106788d39
transformationPattern: JS:zigbeeColorIN.js

I created an item type: color.
I dont get any Info on that channel/item:

I also tried to text config the channel and the item:

    Type color : farbe "MQTT Hue WhiteColor1 Color" [ stateTopic="zigbee2mqtt/0x0017880106788d39/color", commandTopic="zigbee2mqtt/0x0017880106788d39/set/color", transformationPattern="JS:zigbeeColorIN.js", transformationPatternOut="JS:zigbeeColorOUT.js"]

Color Farbe "Mqtt hue white color1 color" {channel="mqtt:topic:MyMQTTBroker:MQTTHueWhiteColor1:farbe"}

With the Text File configuration i get this result:

Does anyone have a tip for me? I think there can only be something small i am missing.
Thnx in advance

AndreHimSelf · January 5, 2023, 5:41pm

Hey @Airmr,

We need a bit more Infos to try to figure out what’s wrong. So here are some steps your cloud do:

make sure zgbee2mqtt is sending messages to openhab (i.e. broker config). Use the WebUI from zigbee2Mqtt and then use the expose feature (device > expose) to change the color or turn on the bulb.
Use f.e. MQTT Explorer to see the messages that are received and send via your broker
check your syntax like 10 times. And then again It could be just a space or so to render everything not working
check openhab.log : maybe enable debug log for mqtt
check if all add-ons and services you want to use are installed and working
post the context of your JavaScript transform files
also check the device support site on zigbee2mqtt - there might be some useful hints on how everything needs to be formatted, which color type use have to use and so on.
also check if you enabled the attribute_json output in zigbee2mqtt. Not sure if really needed but I have it turned on

Good luck

Airmr · January 5, 2023, 8:25pm

Hi Andre, thnx for the quick answer.
Z2M is sending messages out.

MQTT publish: topic 'zigbee2mqtt/0x0017880106788d39', payload '{"brightness":53,"color":{"hue":359,"saturation":100,"x":0.6942,"y":0.2963},"color_mode":"xy","color_temp":158,"color_temp_startup":341,"linkquality":61,"power_on_behavior":"off","state":"ON","update":{"state":"available"}}'

MQTT Explorer:

{"brightness":53,"color":{"hue":359,"saturation":100,"x":0.6942,"y":0.2963},"color_mode":"xy","color_temp":158,"color_temp_startup":341,"linkquality":61,"power_on_behavior":"off","state":"ON","update":{"state":"available"}}

Syntax is just copied from your post with the friendly name changed. Checked the Syntax many times.
I dont really know how to enable a debug log for mqtt
The JavaScript transform files are the ones you posted saved in the transform directory.

I couldnt find any working examples on the Z2M device site.
And i couldnt find the attribute_json output in Z2M. I use the Frontend and am not editing the yaml file.

I think there is something not right in the way i created the Thing in the .things file…but i cant figure it out…
Shouldn’t the UI created Thing at least work when using your code you posted with my friendly name?

Thnx

AndreHimSelf · January 6, 2023, 6:46am

Hm, that wired. You could enable debug by following How to debug MQTT2 Binding issues? f.e.

Could you post the manufacturer and model name of the device?
Also the device should show up as supported and there should be a link behind the model like this:

Also the Output Setting is configurable via the webGUI:

Well, I created everything in openHAB via the Webinterface (I don’t like Textfiles anymore ) so this should be fine.

Maybe try to give the device in z2m a friendly name which is shorter…

Airmr · January 6, 2023, 5:01pm

Hi Andre, thnx for the time your spending on this.
Here is a Screenshot from the device:

I also changed the MQTT output type.

I made 2 other channels with this bulb and everything is working correct:

And i made a String item:

The result shown is the payload:

{"brightness":153,"color":"hue":308,"saturation":100,"x":0.4066,"y":0.1643},"color_mode":"xy","color_temp":153,"color_temp_startup":341,"linkquality":47,"power_on_behavior":"off","state":"ON","update":{"state":"available"}}

The only thing i can think of is the .js Scripts. I checked them 10 times and they are just as yours are.
The example you give in the zigbeeColorIN.js :

example input: {"brightness":50,"color":{"hue":359,"saturation":100,"x":0.6942,"y":0.2963},"color_mode":"xy","color_temp":158}

differs from the text i read in Licht Büro Text, thats the only difference i see.
I Have the Java Script, Regex and Json Path transformation installed.

The Channel:

id: MQQTTZ2MColorCh1Test
channelTypeUID: mqtt:color
label: LEDKochinselXY
description: null
configuration:
commandTopic: zigbee2mqtt/0x0017880106788d39/set
colorMode: XYY
transformationPatternOut: JS:zigbeeColorOUT.js
formatBeforePublish: “%”
stateTopic: zigbee2mqtt/0x0017880106788d39
transformationPattern: JS:zigbeeColorIN.js

I created an item with no profile:

Anything i missed ? I am going crazy

Airmr · January 6, 2023, 5:22pm

MUHAHAHAHA I got it working

Well at least i can control the color etc. from the Thing/item in the UI.
I still have to try to get it going in the Openhab UI. But that is another adventure.

After cleaning up all the Channels and items i have tried with and adding the “attribute and json” Output type it works.
I will test if it really works with the “attribute and json” option or if i just messed up everything, even with checking 100 times there must be something that was still wrong.
At the end i learned a lot.

Thank you Andre for helping me out here and spending your time on my issue.!! Big Thank You!!!
If you ever happen to be near Frankfurt/Germany give me a buzz and i will pay you a coffee or beer.
If my investigation turns out that i really missed something even though i checked many times, it will be more than 1 beer

AndreHimSelf · January 6, 2023, 5:36pm

Sure - your welcome. I just spotted the missing “{” at your example input. But glad you could figure this out

And if I in that region in the near future I may be getting back to you

Also, there are some pre made widgets for color and stuff. Just search for it - should also be fairly easy to implement

Airmr · January 6, 2023, 9:52pm

Nyarly · January 24, 2023, 3:46pm

I recently bought a GIDEALED ZC05M RGBW LED strip and use it with Z2M (in attribute output mode). After reading this thread, I managed to send HSB values through openHAB (via the colorpicker) to Z2M. This works so far. However, I can’t get openHAB to receive external state updates originating e.g. from the Z2M Web UI. The strip changes colors, but the openHAB HSB color item still keeps the value set via openHAB. Here’s the relevant config:

Things:

Thing mqtt:topic:LightbandStairs "Lichtband Treppe" (mqtt:broker:mosquitto) @ "Treppe" {
  Channels:
    Type number   : hue         [ stateTopic="zigbee2mqtt/Lightband_Stairs/color-hue", commandTopic="zigbee2mqtt/Lightband_Stairs/set/color-hue", min=0, max=360 ]
    Type dimmer   : saturation  [ stateTopic="zigbee2mqtt/Lightband_Stairs/color-saturation", commandTopic="zigbee2mqtt/Lightband_Stairs/set/color-saturation", min=0, max=100 ]
    Type number   : brightness  [ stateTopic="zigbee2mqtt/Lightband_Stairs/brightness", commandTopic="zigbee2mqtt/Lightband_Stairs/set/brightness", min=0, max=254 ]
    Type color    : color_hsb   [ stateTopic="zigbee2mqtt/Lightband_Stairs/color-hsb", commandTopic="zigbee2mqtt/Lightband_Stairs/set", formatBeforePublish="{\"color\":{\"hue\":%s,\"saturation\":%s},\"brightness\":%s}", onBrightness=50 ]
    Type dimmer   : color_temp  [ stateTopic="zigbee2mqtt/Lightband_Stairs/color_temp", commandTopic="zigbee2mqtt/Lightband_Stairs/set/color_temp", min=153, max=500 ]
    Type switch   : state       [ stateTopic="zigbee2mqtt/Lightband_Stairs/state", commandTopic="zigbee2mqtt/Lightband_Stairs/set/state" ]
}

Items:

Number LightbandStairsHue        "Licht Treppe (Hue)"                              { channel="mqtt:topic:LightbandStairs:hue" }
Dimmer LightbandStairsSaturation "Licht Treppe (Saturation)"                       { channel="mqtt:topic:LightbandStairs:saturation" }
Number LightbandStairsBrightness "Licht Treppe (Brightness)"                       { channel="mqtt:topic:LightbandStairs:brightness" }
Switch LightbandStairs           "Licht Treppe [MAP(de.map):%s]"                   { channel="mqtt:topic:LightbandStairs:state" }
Dimmer LightbandStairsColorTemp  "Licht Treppe Farbtemperatur [%s]"                { channel="mqtt:topic:LightbandStairs:color_temp" }
Color  LightbandStairsColorHSB   "Licht Treppe Farbe (HSB) [%s]"                   { channel="mqtt:topic:LightbandStairs:color_hsb" }

Strip state example from (Z2M Web UI):

{
  "brightness": 100,
  "color": {
      "hue": 359,
      "saturation": 100,
      "x": 0.6942,
      "y": 0.2963
  },
  "color_mode": "xy",
  "color_temp": 158,
  "color_temp_startup": 65535,
  "last_seen": "2023-01-24T16:19:34+01:00",
  "linkquality": 29,
  "power_on_behavior": "off",
  "state": "OFF"
}

As I said, sending HSB color updates through MQTT works. However, I have to use the above formatBeforePublish. If I use the proposed/default {“color”: “hsb”: “%s,%s,%s”}, the brightness won’t be updated. As a fix for the “openHAB doesn’t get external changes” problem, I introduced a rule to sync hue, saturation and brightness to the openHAB HSB color item, if the three attributes are changed externally, because these (attribute) changes are picked up by openHAB:

rule "Compensate for broken MQTT color state topic updates"
when
  Item LightbandStairsHue changed or
  Item LightbandStairsSaturation changed or
  Item LightbandStairsBrightness changed
then
  LightbandStairsColorHSB.postUpdate(LightbandStairsHue.state + "," + LightbandStairsSaturation.state + "," + LightbandStairsBrightness.state)
end

Now, I can change colors using openHAB, Z2M or MQTT and everyone is happy - i.e. the states in sync. But is there a way to get rid of the rule fix? I’d even go the “output attribute_and_json route”, if the solution is more elegant. For example, I don’t know if the stateTopic for color_hsb is right. No matter what I’ve tried so far (no postfix, color, color-hsb), nothing changes. Maybe there’s some magic hidden somewhere that I’m not aware of?

anon71759204 · January 24, 2023, 9:37pm

IIUC, there are two problems:

Setting brightness doesn’t work. Try {"color":{"hsb":"360,100,100"}} instead of {"color":{"hue":360,"saturation":100},"brightness":100}
openHAB gets out of sync. Use a transformationPattern to transform the state message from Zigbee2MQTT into H,S,B, see Zigbee2mqtt revisited: No more ugly transformations - #201 by AndreHimSelf .

Nyarly · January 27, 2023, 7:59am

Thanks for the hint, @anon71759204! #1 (setting brightness) wasn’t a problem, that already worked. The link in #2, however, did it. I changed Z2M’s output type to “attribute_and_json” and wrote javascript transformations for reading and writing hsb color values. Now I don’t need to manually sync hue, saturation and brightness with the HSB color.

Sharpy · February 15, 2023, 10:07pm

so what’s the best settings too use now?

output attribute&json
home assistant on?
autodiscovery

output attribute
home assistant off
add things with files

AndreHimSelf · February 26, 2023, 6:11am

Hi,
For me, it was a mix of the above things:

enable Attribute & JSON in ZigBee2Mqtt
HomeAssist off in ZigBee2Mqtt (don’t know if nessesarry)
Add thing manually in openHAB
Use JS For Conversion of Values in openHAB Thing

winux · April 29, 2023, 6:27pm

Hello, i’m trying to get the effect channel working that zigbee2mqtt uses for lightbulbs such as philips hue, tradfri etc…

z2mqtt mode:

advanced:
   output: attribute

From the zigbee2mqtt docs
Philips Hue White bulb:

To write (/set) a value publish a message to topic zigbee2mqtt/FRIENDLY_NAME/set with payload {"effect": NEW_VALUE}. The possible values are: blink, breathe, okay, channel_change, finish_effect, stop_effect

I’m using attribute mode no json transform should be needed.
things:

// Philips White 1
Thing topic Test_BULB_1 "Ligthbulb (Philips)"{
    Channels:
        Type string : effect "Effekt" [stateTopic="zigbee2mqtt/Philips_WHITE_BULB_1/effect", commandTopic="zigbee2mqtt/Philips_WHITE_BULB_1/set/effect"]
}

items:

String Philips_White_BULB_1_EFFECT "Lightbulb Effekt [%s]" {channel="mqtt:topic:mosquitto:Test_BULB_1:effect"}

rules:

....
Philips_White_BULB_1_EFFECT.sendCommand("okay")

Openhab log looks good:

2023-04-29 19:52:42.556 [INFO ] [openhab.event.ItemCommandEvent      ] - Item 'Philips_White_BULB_1_EFFECT' received command okay
2023-04-29 19:52:42.562 [INFO ] [penhab.event.ItemStatePredictedEvent] - Item 'Philips_White_BULB_1_EFFECT' predicted to become okay

However the bulb does nothing and zigbee2mqtt log shows:

error 2023-04-29 19:54:32: Invalid message 'null', skipping...

The manual way via terminal:

mosquitto_pub -h localhost -t zigbee2mqtt/Philips_WHITE_BULB_1/set/effect -m okay

leads to success and the bulb is blinking as it should.

Is the DSL .sendCommand function formatting the string in some other way than raw as zigbee2mqtt log shows ‘null’ ? An example would be nice in case someone has as working setup using attribute mode only in zigbee2mqtt because i quite wasted some time today .

Thank you, winux

sihui · April 29, 2023, 6:45pm

Your item name in the items definition does not match the item name in your rule.

winux · April 29, 2023, 7:22pm

This does not solve the issue though, as you can see in the logs. The item in the log outputs also exists. Sorry, that was only a copy paste error on my side =). I copied the wrong line while experimenting and just fixed the original post.

edit:
This is really strange… Is there something missing in the things file? Something like formatBeforePublish or allowedStates=“blink,okay”? Makes no sense though. I guess it has something to do with the formatting of the string value. zigbee2mqtt expects a raw value and it’s getting ‘null’ from either the sendCommand function or the things channel mqtt commandTopic.
I’d appreciate if someone who has one of these bulbs could give it a try and report back, because i guess not many people are using this feature in that attribute mode scenario ^^.

edit2:
Ok i solved it. The solution was the restart of openhab while having the correct setup as in the post above… It did not work for me because i made changes to the files without restarting the appropriate services. So basicly it did not write to the correct topic all the time. DUUUHHH… half a day wasted but happy in the end

Waleed_Alrasheed · November 28, 2023, 10:57pm

This will throw an error if homeassistant is enabled.
It took me a while to capture this

Larsen · January 2, 2024, 10:54am

As this thread is currently the best source for zigbee2mqtt-examples here is my solution to connect Lidl bulbs (but should work with other manufacturers too).
One problem is that the range of the brightness is hardware specific and not a percentage value like in OH. And there is a bug in z2m with hsb-payloads.

Solution:
For the brightness item there is a (not very well documented) feature (brightness_percent) to let z2m do the conversion.
Then for the color we need a transformation and we need the attribute_and_json option within the z2m-config enabled.

Here the example:
First the item-definition in file-format:

Thing mqtt:topic:Lidl1 "Lidl1" (mqtt:broker:MosquittoMqttBroker) { Channels:
  Type switch : state "State"           [ stateTopic = "zigbee2mqtt/Lidl1/state",      commandTopic = "zigbee2mqtt/Lidl1/set",             on="ON", off="OFF" ]
  Type color : color "Color"            [ stateTopic = "zigbee2mqtt/Lidl1/color-hsb",  commandTopic = "zigbee2mqtt/Lidl1/set" , transformationPatternOut="JS:color_hsb_fix.js"]
  Type dimmer : brightness "brightness" [ stateTopic = "zigbee2mqtt/Lidl1/brightness_percent", commandTopic = "zigbee2mqtt/Lidl1/set/brightness_percent" ]
  Type datetime : last_seen "last_seen" [ stateTopic = "zigbee2mqtt/Lidl1/last_seen" ]    
}

Then the file transformation color_hsb_fix.js which has to be placed in the transform-folder:

(function(color){
  //var log = Java.type("org.slf4j.LoggerFactory").getLogger("org.openhab.model.script.fix_hsb");
  //log.info("color_in: " + color) ;
  var bright = Number(color.split(/[,:]/)[2]) * 2.54;

  var returnVal = '{"brightness":' + bright.toFixed(0) + ',"color":{"h":' + Number(color.split(/[,:]/)[0]) + ',"s":' + Number(color.split(/[,:]/)[1]) + '}}';    
  return returnVal;
})(input)

The point here is to not send hsb (not working in z2m) but to send the brightness and for the color only “h” (Hue) and “s” (Saturation) in one json-payload.
Took me a while to figure this out, so I hope this may help somebody else.