Jinja transformation failed - wallplug with auto discover homeassistan

Jacek_Kaczmarczyk · September 28, 2022, 9:16pm

Hi, see this i think it comes from homeassistant autodiscover from zigbe2mqqt but not sure which device is posting such json (in zigbee2mqtt i don’t see such device how can i trace it ?

Exception: Unknown token found: value_json.energy
2022-09-28 23:08:59.252 [WARN ] [t.generic.ChannelStateTransformation] - Executing the JINJA-transformation failed: An error occurred while transformation. UnknownTokenException: Unknown token found: value_json[‘update’][‘state’]
2022-09-28 23:08:59.662 [WARN ] [t.generic.ChannelStateTransformation] - Executing the JINJA-transformation failed: An error occurred while transformation. UnknownTokenException: Unknown token found: value_json.energy
2022-09-28 23:08:59.667 [WARN ] [t.generic.ChannelStateTransformation] - Executing the JINJA-transformation failed: An error occurred while transformation. UnknownTokenException: Unknown token found: value_json[‘update’][‘state’]
2022-09-28 23:09:05.576 [WARN ] [t.generic.ChannelStateTransformation] - Executing the JINJA-transformation failed: An error occurred while transformation. UnknownTokenException: Unknown token found: value_json.energy
2022-09-28 23:09:05.583 [WARN ] [t.generic.ChannelStateTransformation] - Executing the JINJA-transformation failed: An error occurred while transformation. UnknownTokenException: Unknown token found: value_json[‘update’][‘state’]
2022-09-28 23:09:07.941 [WARN ] [t.generic.ChannelStateTransformation] - Executing the JINJA-transformation failed: An error occurred while transformation. UnknownTokenException: Unknown token found: value_json.energy
2022-09-28 23:09:07.950 [WARN ] [t.generic.ChannelStateTransformation] - Executing the JINJA-transformation failed: An error occurred while transformation. UnknownTokenException: Unknown token found: value_json[‘update’][‘state’]
2022-09-28 23:09:19.977 [WARN ] [t.generic.ChannelStateTransformation] - Executing the JINJA-transformation failed: An error occurred while transformation. UnknownTokenException: Unknown token found: value_json.energy
2022-09-28 23:09:19.988 [WARN ] [t.generic.ChannelStateTransformation] - Executing the JINJA-transformation failed: An error occurred while transformation. UnknownTokenException: Unknown token found: value_json[‘update’][‘state’]

rlkoshak · September 28, 2022, 9:20pm

I believe you have to separately install the Jinja Transformation add-on to do HA discovery with the MQTT binding.

Jacek_Kaczmarczyk · September 28, 2022, 9:24pm

i did, it is installed. but i can’t see any device posting something like update state so don’t know what is triggering this automation to run

rlkoshak · September 28, 2022, 9:30pm

Something must be posting. The MQTT binding doesn’t get triggered unless it receives a message somehow. The only time it’ll receive a message that wasn’t just posted by the device is if the retained bit was set, in which case OH will get the message when the binding first connects to the broker no matter when the message was originally sent.

The warning is coming from a ChannelStateTransformation so I’m pretty sure this is the binding attempting to apply the transformation to some message (as opposed to the transform profile or something like that).

I do not fully know and understand the HA spec for MQTT but I suspect it might be kind of chatty. For certain it publishes a bunch of retained messages to special topics which is what allows for the auto discovery in the first place.

If you put the binding into debug mode, you might see what messages on what topics it’s trying to process to generate these warnings.

Jacek_Kaczmarczyk · September 28, 2022, 10:18pm

ok found it.
Transformation ‘{“child_lock”:“UNLOCK”,“current”:0,“energy”:null,“indicator_mode”:“off/on”,“last_seen”:“2022-09-29T00:10:51+02:00”,“linkquality”:0,“power”:0,“power_outage_memory”:“restore”,“state”:“OFF”,“update”:{“state”:null},“update_available”:null,“voltage”:231}’ returned null on ‘JINJA’, discarding message
device json :

{
    "child_lock": "UNLOCK",
    "current": 0,
    "indicator_mode": "off/on",
    "last_seen": "2022-09-29T00:10:51+02:00",
    "linkquality": 0,
    "power": 0,
    "power_outage_memory": "restore",
    "state": "OFF",
    "voltage": 231,
    "energy": null,
    "update": {
        "state": null
    },
    "update_available": null
}

but how to solve this issue? this thing is auto-generated by discovery :

{“availability”:[{“topic”:“zigbee2mqtt/bridge/state”,“value_template”:“{{ value_json.state }}”},{“topic”:“zigbee2mqtt/WallPlug2/availability”,“value_template”:“{{ value_json.state }}”}],“availability_mode”:“all”,“device”:{“identifiers”:[“zigbee2mqtt_0xa4c138764bf197d1”],“manufacturer”:“TuYa”,“model”:“Smart plug (with power monitoring) (TS011F_plug_1)”,“name”:“WallPlug2”},“enabled_by_default”:false,“entity_category”:“diagnostic”,“icon”:“mdi:update”,“json_attributes_topic”:“zigbee2mqtt/WallPlug2”,“name”:“WallPlug2 update state”,“state_topic”:“zigbee2mqtt/WallPlug2”,“unique_id”:“0xa4c138764bf197d1_update_state_zigbee2mqtt”,“value_template”:“{{ value_json[‘update’][‘state’] }}”}

rossko57 · September 29, 2022, 1:39am

Probably choking on the null values? Seems to correspond with the complaints.

First identify if nulls are allowed in Homeassistant discovery protocol - if not, the device needs fixing.
If nulls are permitted, looks like the HA section of the openHAB MQTT binding needs fixing.

Jacek_Kaczmarczyk · September 29, 2022, 7:28am

github.com/home-assistant/core

Jinja templates inconsistencies with None/null value

opened 03:42PM - 06 Feb 22 UTC

closed 04:11PM - 15 Mar 22 UTC

energywave

stale

### The problem I've found several inconsistencies handling None/null value. I …list the issues here and I demonstrate them later. 1. None value, if alone in developers tools models, is shown as **null** 2. None value, if alone in developers tools models, is shown as type **dict** 3. **default** filter is not working with None/null value (the result is None/null) 4. **int** and **float** filters are converting the value to 0 (is that right?) but if you provide a default value it will be used I hope to get better consistency as when I develop complex scripts these things are a huge loss of time, often more than the script itself. I think that a complete check of special values and how they're handled by jinja templates would be great. Here the text to use in Developers tools -> Models: ``` {# I set value with a not existant attribute at the moment. Same to assign it with None #} {% set value = state_attr("media_center.google_mansarda", "volume_level") %} Value: {{ value }} Value is None: {{ value == None }} Default value: {{ value | default(0.6) }} As float: {{ value | float }} As float with default: {{ value | float(0.6) }} As int: {{ value | int }} As int with default: {{ value | int(0.6) }} ``` Here is the result: ![image](https://user-images.githubusercontent.com/2051567/152688549-b4990fa5-5dad-4871-bb48-2beccc6f283d.png) To demonstrate point 1 and 2 use this: ``` {# I set value with a not existant attribute at the moment. Same to assign it with None #} {% set value = state_attr("media_center.google_mansarda", "volume_level") %} {{ value }} ``` And here is the result: ![image](https://user-images.githubusercontent.com/2051567/152688610-3a0cf53b-6f3a-4796-8636-07cbff47c95f.png) ### What version of Home Assistant Core has the issue? core-2021.12.10 ### What was the last working version of Home Assistant Core? _No response_ ### What type of installation are you running? Home Assistant OS ### Integration causing the issue _No response_ ### Link to integration documentation on our website _No response_ ### Diagnostics information _No response_ ### Example YAML snippet _No response_ ### Anything in the logs that might be useful for us? _No response_ ### Additional information I suspect there could be other inconsistencies that I've never came across other than that. I suggest a review of how that special values are handled, especially with default values so that we have a predictable result when scripting :) That said, thank you for your work, I really **LOVE** it! PS: I'm a programmer but Python is not in my skill set, otherwise I would have been happy to do it by myself.

rossko57 · September 29, 2022, 12:02pm

While it’s interesting to note that Homeassistant has troubles with JINJA handling null values, it doesn’t seem to move you forward.
This is openHAB.
JINJA is just used as a tool here to interpret the MQTT payload.

Let’s try step one;

The HA Discovery protocol is expressed in JSON.
So it must be valid JSON.
Does JSON allow null values?
Yes it does.

So turning to HA Discovery protocol proper.
Does that allow null values?

I can’t see anywhere it is expressly forbidden.

Neither can I see any reason that you’d want to send null values.
If your device has no ‘energy’ sensing for example, you simply would not send an ‘energy’ keyword in the discovery announcement.
Sending "energy": null is announcing that you don’t know anything about “energy”, you are not able to describe it in any way, the same as saying nothing at all.

So to me, it appears to be a mad thing to do, and may or may not be actually “allowed” in HA discovery.

I’d missed out before, that it would be sensible to find out if zigbee2mqtt is supposed to be sending this null stuff at all.

Either way, it seems to me that openHAB ‘HA Discovery’ is doing exactly the right thing - generating a WARN that a nonsense payload has been received.
I am assuming here that your device/Thing etc. is all working properly otherwise (now that you know which one it is) - you didn’t say?

I suppose the WARN could be enhanced to give the suspect MQTT topic as well, to aid diagnosis.

rossko57 · September 29, 2022, 3:02pm

Ah wait - I’ve misinterpreted.

The payload causing the WARN is not the discovery part all. Whether that described e.g. ‘energy’ or not is irrelevant.

This a working payload, a “normal update”. It’s just JSON, so null is valid.

The channel (that was previously configured by discovery) cannot handle the null.
People encounter this issue in regular non-HA MQTT usage as well, and have to configure their channels to work around it - there’s a choice, because you might want to treat as an error and report it; or you might want to just ignore and not update the linked Item, or you might want to treat it as meaningful and update the Item with UNDEF or similar.

The problem here is that the channel was auto-configured so you don’t get to manipulate how it responds.

I don’t think there’s any HA rule about how to treat null values, which would be sensible. If you’ve nothing to say, don’t say it.
So I still arrive at the conclusion the binding is doing the correct thing - reporting WARN about nonsense received (and I guess issuing no update at all to linked Item?)
And would still question if zigbee2mqtt is doing the wrong thing sending nulls in the first place.

Gargam3L0 · February 15, 2023, 8:10pm

I am too getting these warn log messages but seems for a different reason.
Any solution other but not use HA auto discovery with Z2M ?

[WARN ] [t.generic.ChannelStateTransformation] - Executing the JINJA-transformation failed: An error occurred while transformation. UnknownTokenException: Unknown token found: value_json.power_on_behavior

{
  "availability": [
    {
      "topic": "zigbee2mqtt/bridge/state"
    }
  ],
  "device": {
    "identifiers": [
      "zigbee2mqtt_0x60a423fffed2e99f"
    ],
    "manufacturer": "TuYa",
    "model": "Socket module (TS011F_socket_module)",
    "name": "wallplug16A"
  },
  "enabled_by_default": false,
  "entity_category": "config",
  "icon": "mdi:power-settings",
  "json_attributes_topic": "zigbee2mqtt/wallplug16A",
  "name": "wallplug16A power on behavior",
  "state_topic": "zigbee2mqtt/wallplug16A",
  "unique_id": "0x60a423fffed2e99f_power_on_behavior_zigbee2mqtt",
  "value_template": "{{ value_json.power_on_behavior }}"
}

Gargam3L0 · April 23, 2023, 6:39pm

Does anyone have a clue how to resolve this warning?

miwok · April 24, 2023, 6:01am

I have the same messages in openhab.log. I opened an issue on github (zigbee2mqtt) because I don’t know if the error is coming from zigbee2mqtt or from OH.
The following 3 devices provide the warning message:

Xiaomi MiJia light intensity sensor (GZCGQ01LM)
Aqara door & window contact sensor (MCCGQ11LM)
Aqara human body movement and illuminance sensor (RTCGQ11LM)

Not only NULL values are affected. Here is an example of the data for RTCGQ11LM:

firmwareVersion 3000-0001
modelId 		Aqara human body movement and illuminance sensor (RTCGQ11LM)
deviceId 		zigbee2mqtt_5F0x00158d00054752c0
vendor 			Xiaomi

binary_sensor/0x00158d00054752c0/occupancy
sensor/0x00158d00054752c0/battery
sensor/0x00158d00054752c0/device_temperature
sensor/0x00158d00054752c0/illuminance
sensor/0x00158d00054752c0/illuminance_lux
sensor/0x00158d00054752c0/linkquality
sensor/0x00158d00054752c0/power_outage_count
sensor/0x00158d00054752c0/voltage

homeassistant/sensor/0x00158d00054752c0/power_outage_count/config

{
  "availability": [
    {
      "topic": "zigbee2mqttpi/bridge/state"
    },
    {
      "topic": "zigbee2mqttpi/Xiaomi_RTCGQ11LM_BWM_01/availability"
    }
  ],
  "availability_mode": "all",
  "device": {
    "identifiers": [
      "zigbee2mqtt_0x00158d00054752c0"
    ],
    "manufacturer": "Xiaomi",
    "model": "Aqara human body movement and illuminance sensor (RTCGQ11LM)",
    "name": "Xiaomi_RTCGQ11LM_BWM_01",
    "sw_version": "3000-0001"
  },
  "enabled_by_default": true,
  "name": "Xiaomi_RTCGQ11LM_BWM_01_power_outage_count",
  "state_topic": "zigbee2mqttpi/Xiaomi_RTCGQ11LM_BWM_01",
  "unique_id": "0x00158d00054752c0_power_outage_count_zigbee2mqttpi",
  "value_template": "{{ value_json.power_outage_count }}"
}

zigbee2mqttpi/Xiaomi_RTCGQ11LM_BWM_01

{
  "battery": 100,
  "device_temperature": 25,
  "illuminance": 29,
  "illuminance_lux": 29,
  "linkquality": 123,
  "occupancy": true,
  "power_outage_count": 36,
  "temperature": 28,
  "voltage": 3025
}

2023-04-06 18:07:56.201 [WARN ] [t.generic.ChannelStateTransformation] - Executing the JINJA-transformation failed: An error occurred while transformation. UnknownTokenException: Unknown token found: value_json.power_outage_count

Gargam3L0 · May 13, 2023, 10:23am

Hi @miwok ,
Could you please check maybe the proerty power_outage_count is not supported by the node so that the zigbee2mqtt recipe for this model needs a correction?
I have see that in zigbee2mqtt latest releases there is some cleanup going on like “Disable unsupported power_on_behavior for more devices ([@Koenkk]”. For example when I try changing this parameter with the zigbee2zqtt web control I get an error on unsupported setting.

miwok · July 16, 2023, 7:02pm

Here is a DEBUG log for the Xiaomi_GZCGQ01LM

2023-07-16 20:47:11.356 [DEBUG] [.internal.JinjaTransformationService] - about to transform '{"battery":100,"illuminance":0,"illuminance_lux":0,"linkquality":87,"power_outage_count":null,"voltage":3000}' by the function '{{ value_json.linkquality }}'
2023-07-16 20:47:11.359 [DEBUG] [.internal.JinjaTransformationService] - transformation resulted in '87'
2023-07-16 20:47:11.359 [DEBUG] [.internal.JinjaTransformationService] - about to transform '{"battery":100,"illuminance":0,"illuminance_lux":0,"linkquality":87,"power_outage_count":null,"voltage":3000}' by the function '{{ value_json.illuminance }}'
2023-07-16 20:47:11.361 [DEBUG] [.internal.JinjaTransformationService] - transformation resulted in '0'
2023-07-16 20:47:11.361 [DEBUG] [.internal.JinjaTransformationService] - about to transform '{"battery":100,"illuminance":0,"illuminance_lux":0,"linkquality":87,"power_outage_count":null,"voltage":3000}' by the function '{{ value_json.illuminance_lux }}'
2023-07-16 20:47:11.363 [DEBUG] [.internal.JinjaTransformationService] - transformation resulted in '0'
2023-07-16 20:47:11.364 [DEBUG] [.internal.JinjaTransformationService] - about to transform '{"battery":100,"illuminance":0,"illuminance_lux":0,"linkquality":87,"power_outage_count":null,"voltage":3000}' by the function '{{ value_json.power_outage_count }}'
2023-07-16 20:47:11.365 [WARN ] [t.generic.ChannelStateTransformation] - Executing the JINJA-transformation failed: An error occurred while transformation. UnknownTokenException: Unknown token found: value_json.power_outage_count
2023-07-16 20:47:11.366 [DEBUG] [.internal.JinjaTransformationService] - about to transform '{"battery":100,"illuminance":0,"illuminance_lux":0,"linkquality":87,"power_outage_count":null,"voltage":3000}' by the function '{{ value_json.voltage }}'
2023-07-16 20:47:11.367 [DEBUG] [.internal.JinjaTransformationService] - transformation resulted in '3000'
2023-07-16 20:47:11.369 [DEBUG] [.internal.JinjaTransformationService] - about to transform '{"battery":100,"illuminance":0,"illuminance_lux":0,"linkquality":87,"power_outage_count":null,"voltage":3000}' by the function '{{ value_json.battery }}'
2023-07-16 20:47:11.370 [DEBUG] [.internal.JinjaTransformationService] - transformation resulted in '100'

Gargam3L0 · July 22, 2023, 7:52am

After some good steps for debugging (in github issue with @apella12 ) it seems there are 2 different issues in OpenHAB with the JINJA transformation:

Errors appear where there are ‘null’ values resulted from conversions.
‘power_on_behavior’ error seems to be due to missing parameters in translation because the Thing was discovered when this device parameter existed. Now that the device does not support it and ‘power_on_behavior’ was removed, we need to re-discover this device without ‘power_on_behavior’ channel.

apella12 · July 25, 2023, 7:41pm

I was reading the release notes for OH4 and noticed this [jsonpath] Fix exception when input string is empty by J-N-K · Pull Request #15283 · openhab/openhab-addons (github.com). Could be worth adding an issue to the jinja transformation as it is similar. I think a lot of the issue is the parameter is blank (“null”) initially until the something like a power outage occurs or you try to upgrade the firmware, etc

VasekCejka · December 13, 2023, 6:13pm

I have one sollution working for me - I have just filtered out these attributes from MQTT. I dont need them. You can do it in Zigbee2MQTT for the device in need in settings page, add “Filter attributes with regex from published payload.” in form like ^update(\_available)? (this will filter both update and update_available attributes).
If I would need an update, I can do it manually using Zigbee2MQTT, OTA page. However my experience was that the trouble with null occurs exactly with the devices, for which update page is not implemented in Zigbee2mqtt.
You can use this also for other problematic attributes, if you dont need them. I had some more like child_lock etc. But for them it was sufficient to set or get their value manually on Zigbee2mqtt Exposes page.