Is there a way to place reasonableness constraints on an item? Last night one of my temperature sensors was reporting temperatures over 5000 degrees (F) for a few hours. While this did not have any impact on me at the time since I was asleep, the data was persisted into the InfluxDB database and completely messed up the temperature graphs on the Grafana dashboard.
I’m hoping there is something I can do to prevent the invalid data from updating the items.
The temperature sensor is a zooZ ZSE44 (Zwave), v1.3
OpenHAB v3.4.2 running in Docker on a Synology NAS.
InfluxDB v2.6.1
There’s nothing you can do about the item, because as soon as openHAB has the data then InfluxDB has the data. Maybe there’s something you can do with InfluxDB (which I don’t use).
If you have a sensor that’s regularly troublesome, you could use a rule that updates a proxy item and rejects outlandish readings so that they don’t get persisted. I did something similar with a temperature sensor that would sometimes report 0 degrees at random.
You can use a profile written in jruby (which works on openhab 3.4) or wait for openhab 4. The state of this in openhab4 is still up in the air. Currently there’s SCRIPT profile that can let you do this (not sure if it’s in M1 but for sure it’s in SNAPSHOT), but if my PR is accepted, this will go away and be replaced with a language-specific profile, i.e. JS, RB, DSL, etc. in its place.
automation/ruby/profiles/my_profile_to_discard_bad_values.rb (you can name this file anything, of course - as long as it has a .rb extension, and reside somewhere inside automation/ruby/)
profile(:discard_anomalies) do |event, state:, configuration:|
next true if state && state.to_f < configuration["max"] # Assuming you're getting a raw numeric value in an implicit unit of °F
false # discard it
end
profile(:discard_anomalies) do |event, state:, configuration:|
next true if state && state < QuantityType.new(configuration["max"])
false # discard it
end
You don’t need jruby you could write a transformation to apply.
But there’s no existing profile to do this AFAIK, and I believe this would be a useful addition for many users so I suggest you raise this on the 4.0 wishlist thread., too.
I could be wrong, but AFAIK, the Javascript transformation profile in OpenHAB 3.4 cannot “veto” / stop the values, but it can alter it to blank, although I’m not sure what that will do to the persisted data. You’ll still be persisting something (a blank string, or zero or a null data point?).
This particular implementation of javascript transformation profile no longer exists in openhab 4.0 right now, but if my PR is merged, it will be once again supported although it won’t be a seamless upgrade; the profile config will need to be adjusted.
The only ways to do it are:
SCRIPT transformation profile in openhab 4.0 - the “Script transformation Profile” feature didn’t get merged into 3.x - so it only exists in 4.0 I’m not sure whether it made it to M1 or after that.
Using jruby profile is the only way to do it right now that can work on openhab 3.4 and will continue to work on openhab 4+.
If my PR is merged, then SCRIPT profile will be gone, and replaced with JS, RB, etc, but since this isn’t yet merged, this is not usable right now.
Of course you can write your own profile implementation if you understand the openHAB Java API, but this is far more complicated than any of the above options.
It only showed one log entry “Updated to 10” and the state remained 10.
So: add it to the list of options:
You can use JavaScript Transformation Profile in openhab 3.4 now and write a script that returns a blank string when the value is out of range, but in any case, you’ll have to change it when upgrading to openhab 4.
There is actually. You can exclude the Item from persistence and then use a rule that triggers when the Item changes and if the new value is reasonable use the persist Persistence action to save the value.
I believe there is a profile in the SmarthomeJ marketplace that can filter values that are above or below threshold. Navigate to Settings → JSON 3rd Party Add-On Service and put https://download.smarthomej.org/addons.json into the Add-On Service URLs field to make those add-ons available. Install the “Basic Profiles” add-on (under “Other Add-ons”) and see Basic Profiles for how to use it.
From what I can see on that page, there isn’t one that is suitable. The threshold profile works similarly to the range profile: it updates a switch item.
I considered adding to core, something similar to the range profile but acts as a filter. But I wonder how many people really need/would use it, and it can be done using script profile, especially inline script, just as easily.
Lots of people have asked for this sort of capability. I personally think it would be worth it to have it as it’s own profile. Resorting to a script transformation or coding your own profile in Ruby for this isn’t super intuitive and somewhat of a heavy lift for a lot of users.
Complain to your sensor vendor if possible. Sometimes these are known issues and can be fixed. There are some Zwave temperature sensors that have problems and can be improved with later firmware. Also note that you can filter out some values easily, but bad sensors can still give incorrect values within your “good” range.
Also, it’s not hard to go back and find outlier data in influxdb and remove them. Obviously it’s better to not have them there in the first place, but if they do get there, you can fix them. Here’s a sample that I did earlier today. 4872.2 degrees! (This is for a V1 influxdb.)
$ influx
Connected to http://localhost:8086 version 1.6.7~rc0
InfluxDB shell version: 1.8.10
> use openhab_db
Using database openhab_db
> select * from OutsideTemperature where value > 200
name: OutsideTemperature
time item value
---- ---- -----
1680339992494000000 OutsideTemperature 4872.2
> delete from OutsideTemperature where time = 1680339992494000000
Thanks to everyone for the help. I’ve managed to take a step forward and two steps back
At first I implemented the rule based approach that @rlkoshak suggested, mostly because I’m familar with rules files and it was easy for me to undersand. Once implemented I realized that the Item would still be updated with the invalid value so the everyHour strategy would still persist invalid values to InfluxDB. (My strategy is everyChange, everyHour, restoreOnStartup.) So I backed out those changes.
Next I tried the Ruby approach suggested by @JimT. That seemed to be working but I found that none of the temperature values were being sent to InfluxDB and in addition InfluxDB now had tags for profile, min and max (I added the min= parameter). I know nothing about Ruby so hopefully I have something messed up. My item definition is
profile(:discard_anomalies) do |event, state:, configuration:|
next true if state && state >= QuantityType.new(configuration["min"] && state <= QuantityType.new(configuration["max"])
false # discard it
end
One odd thing that I noticed is that after implementing this there were three new tags in InfluxDB: profile; min; & max. Perhaps there is something wrong with my Item configuration.
But the best part is that after restarting OpenHAB just on the off chance that it would fix it all of my data in InfluxDB disappeared from InfluxDB. I’m just starting down this persistence path so losing the data is no big deal, but I’m scratching my head trying to understand why restarting OpenHAB dropped the data from InfluxDB.
@jswim788, I suspect that the sample you provided is for Influx v1, and I’m using v2. I did manage to get the invalid data deleted using the following command but it deleted the data for all of the temperature sensors within the time range, not just the one with the error.
This one is easy. Remove the every hour rule in influxdb.persists for those items, and create them as a cron rule instead. Within this rule you can filter out the wrong value, but you’ll need to get the previous value if the current value is wrong (using persistence previousState(true), but I think there’s a bug in this) - so overall this is not really a simpler approach.
In other words, the [profile=.....] should be placed right after the incoming channel you wish to intercept. I also changed the min / max to a string that represents a proper quantity type, so they are parsed by QuantityType.new with the correct unit.
The script just had a syntax error, missing a closing parenthesis)
profile(:discard_anomalies) do |event, state:, configuration:|
next true if state && state >= QuantityType.new(configuration["min"]) && state <= QuantityType.new(configuration["max"])
false # discard it
end
Just to confuse you further, this is an alternative syntax:
profile(:discard_anomalies) do |event, state:, configuration:|
next false unless state
state >= QuantityType.new(configuration["min"]) && state <= QuantityType.new(configuration["max"])
end
this is because you added the [profile=xxxx, min=xx, max=xx] options to your influxdb metadata which was interpreted by influxdb as extra properties(?) of the “temperature” measurement.
Thanks. That all makes sense, but I do have one question. Would it be simpler to omit the " °F" from the item definition and use the script without the QuantityType?
You mean like in my post above? Or do you wish to completely not use QuantityType and just strip the unit into a float value and compare it as a simple numeric? That can be done too but why?
I’m just starting down this persistence path so losing the data is no big deal, but I’m scratching my head trying to understand why restarting OpenHAB dropped the data from InfluxDB.