Occasionally, an HTTP endpoint returns a temperature value of 2,147,483,647. Much hotter than expected, even for mid-July. The number is exactly 2^31, so I presume it’s an erroneous reading (Max Signed Int). Just in case, I’ll check if the basement for glowing plasma after I finish this post. It usually happens when the device is polled on startup.
Question simply:
What’s the best way to constrain a value outlier? Currently these data are being written to the persistent data, which messes up the scale of graphs.
About the setup: Using OpenHab 3 and an HTTP thing to periodically poll the sensor for temperature, humidity and other properties. The endpoint returns a delimited string of name-value pairs. I use the Regex transformation profile for each channel link to parse each item’s value.
What is the proper way to filter an outlier before an associated item is updated?
Transformation in the link seems like the right place, but doing number validation within a regex is painful/impossible/stupid (pick one), and It appears I can’t chain multiple transformations. Is there a way to do this that I’m not seeing?
Maybe use the JS transformation? You can use regexes in JS, and post process the result as needed there. The only ugly thing is that you’ll need one JS file per item.
For each linked item, I use a item-specific regex to pull out the value, and cast it into its decimal precision. For example OID4.1.13:
s/.OID4.1.13=(\d)(\d)(?:$|&OID.*)/$1.$2/g
Grabs the value 726 and sets it to 72.6 degrees.
Note some OIDs in the string returned blown-out values (-2147483647) causing the aforementioned problem
Are you suggesting a transformation on the whole string at a channel level, before parsing the string on a per-item basis? If so, I could see matching/discarding those special 2^31 strings which would likely solve my particular problem. However it maybe wouldn’t help somebody (or me in the future) if trying to bound a value between x and y, rather than discarding based on a known outlier.
Can you give a code/syntax example of chained transformations?
Transformations can be chained by separating them with the mathematical intersection character “∩”. Please note that the values will be discarded if one transformation fails (e.g. REGEX did not match).
This is all about transforming the incoming values at the Channel level, before it is passed onto the Item via its link.
Note that you can use the REGEX transformation as a filter. I’m rubbish at REGEX so am not sure if it’s possible, but perhaps it can look for values within a certain range - if it doesn’t find any it will fail, and that particular value will be discarded, and no update is sent to the Item. Or, as you say, somehow fail if it finds the specific massive number.
Post dates from two years ago, before the new HTTP binding was released - which now includes ability to chain transformations.
It is stated in the docs -
You could add some intelligence to your transformation with JS scripting - this could process the JSON, examine the extracted value, scale it if sensible or substitute something else if bonkers - UNDEF perhaps.
No double-transformation necessary in this case because (as you mentioned) the first transformation occurred at the channel level (sanitizing the WHOLE string for ALL linked items). I was confusing the two types when I first posted here.
My specific fix was to add this line to the channel definition to replace the blown out value with zero:
Curiously It seemed that I couldn’t use the regex to fully remove the string, It appears I had to substitute it with something (like a space, or a different value, e.g. 0). For example if I used // to remove the value entirely instead of substituting /0/ or / / e.g.
Then the channel would choke. There would be no error in the logs, but the things were not updated as if just not getting any data to parse. I don’t understand this. It appears to be valid syntax (e.g. in sed). Any ideas why this would fail?
This is a magical ability of the binding/channel. If the transformation returns empty/null, that is simply, ignored by design. No further processing towards a channel update occurs. This allows you to use the transformation to filter out unwanted updates.
Yeah, that’s a great feature, which this particular channel benefits from. However, I’m thinking I didn’t say it right. A practical example would be:
If the channel string is: ‘this sucks’
And the regex is:
‘s/(this)/that/g’
The channel transformation will properly return: ‘that sucks’
However if change the regex to ‘s/(this)//g’
I would expect the channel transformation to return just: ’ sucks’
(In the manner that sed would do).
Instead, the channel transformation seems to return nothing at all, and the channel is not updated.
Is this expected behavior? If the match string is to simply be removed, is there a special syntax, or token that should be used?
Thanks for chiming in. My example looks like the first basic example shown in the docs,. The only deviation is the length of the replacement string. You’re saying that’s unsupported? Maybe it just needs to not be defined as a capture group in OHs implementation, since I’m not actually augmenting the captured pattern. To your point–REGEX transformation doesn’t behave exactly like a regular REGEX. Will test.
So in my testing, using the example in the docs, it behaves as expected: My network does not work.withs/work/cast/gresults in"My netcast does not cast."
However, the example used in the docs fails in the manner described earlier if the length of the replacement is zero. e.g. My network does not work.withs/work//gresults in"" Nothing at all.
The expected result would be: "My net does not ."
It seems to me like this should be a supported regex. I suspect the parser is choking.
I know this is not an answer for your issue (yet). I had the same problem with some of my deployments and ended up writing a profile which works as a limiter. It does refuse values beside accepted level.
As I am still on OH 3.0.x I had to do it myself, but maybe there is a way in recent releases to do it out of the box.
I’ve never tried it but since an empty return is treated like a filter for some bindings, especially the MQTT binding, perhaps using a JS transformation that returns the empty string when the value is too high would cause the binding to throw out the message, similar to a REGEX transformation that doesn’t match anything in the message.
Obviously I’ve never tried it but it’s the first thing I would try.
