Applying averageSince to persisted quantities with units

jstro · May 19, 2021, 10:25pm

Hi folks,

I’m sorry but I’m struggling with this. I found some great tutorials, e.g., Working with Number:Temperature Items in rules, and a number of related posts, but I’m still lost.

I am attempting to perform a mathematical operation on two time-averaged persisted quantities, Temperature and Relative Humidity (a dewpoint calculation, to be specific). I did happen to find an example of dealing with persisted temperatures, so my average temperature is working. However, dealing with the RH with the percent unit has me stumped. It’s probably something glaringly obvious.

My temperature is persisted as, e.g., 75.254 °F, and my RH is persisted as 71.0 %.

Here is my DSL code. I’m using Windows OH 3.1.0.M4. Any advice greatly appreciated.

var T = new QuantityType(SmartWeatherAir_Temperature.averageSince(now.minusMinutes(5), "rrd4j"), ImperialUnits.FAHRENHEIT).toBigDecimal
// var RH = new QuantityType(SmartWeatherAir_RelativeHumidity.averageSince(now.minusMinutes(5), "rrd4j") .... ????

T = (T - 32) / 1.8  
val b = 18.68
val c = 257.14
val gamma = Math.log(RH / 100) + (b * T) / (c + T)
var DP = c * gamma / (b - gamma)
DP = DP * 1.8 + 32
OutsideDewpoint.postUpdate(DP)

rossko57 · May 19, 2021, 11:21pm

Give us a clue?

If your Items are Quantity types, then averageSince() should be returning a Quantity type too, and you don’t need to build a new one.

You don’t need to convert F to C, just ask the Quantity for the units you want (method .toUnit("°C"))

Obviously your RH retrieval/average does nothing because its commented out, but is this where your problem lies??

Background;
Number:Temperature types have a ‘system default’ unit, basically imperial or metric, C or F.
However
Some types do not have a system default - Number:Dimensionless (for %) is one such.

Interaction with persistence;
The persistence service only store numbers. That means it has to ‘reconstruct’ Quantities on retrieval by guessing the unit. (Note that means it’s hoping you haven’t changed units after data was persisted,but I doubt that’s your problem here)

So, to unit guessing uhhh reconstructing;
First port of call - look in the Items state presentation ‘pattern’ for a unit. This metadata is really only about presentation, but was long ago hijacked to also be referred to in some circumstances to seek a default unit for this Item.
Maybe you have no ‘pattern’ for Item default, next look for system default unit.

Let’s say you have no pattern for a Temperature type; that’s okay, it will store and retrieve in the system default (C or F) and that’s how your average should be presented.

Now let’s say you have no pattern for a Dimensionless type; here it gets weird. Uniquely among the Quantity types “no unit at all” is a valid unit. It’s taken to mean a ratio, a N-to-1 expression of a proportion.
I don’t even know if it looks for a system default in this case,but there is not one of those either so the effect is the same.

So no Item ‘pattern’, the state will be persisted and retrieved as a ratio,
If your Item state was 70%, 0.7 will be stored, and also retrieved apparently without unit (though really the unit is ‘1’).

The workaround here is to give your Items ‘pattern’ metadata with a real unit specified (no %unit% placeholder).

Your existing historic data is goosed of course, if this results in a unit change.

github.com/openhab/openhab-core

Persistence, Percentages and Default units.

opened 07:05PM - 23 Mar 21 UTC

closed 08:12PM - 17 Jul 23 UTC

Rossko57

persistence

I'm not truly sure if this issue lies with persistence services, or the oddball …handling of Dimensionless quantity types with a 'null' unit. The story; Item of type Number:Dimensionless, updated with valid quantity like "**26 %**" Notably, Item `"stateDescription": { ..."pattern": "%d %unit%",` , which is perfectly reasonable to display the units that the binding supplies. Such an Item will persist its state as **0.26**. This causes problems on retrieval e.g. `deltaSince()` will compare stored 0.26 with current 26% and not yield the expected result 0. There is no opportunity to work around that by manipulating units. My interpretation; persistence stores only the numeric part of a quantity, and on recovery uses some default unit to reconstruct the numeric into a quantity. An Item may have an individual default unit, derived from the pattern. This is not specified in this case (%unit%). I'm guessing persistence can then go on to look for a system default. Not all quantity types have one, I presume Dimensionless is one that does not. So in theory, there is no default here. However I think Dimensionless "special property" comes into play - a complete lack of unit is treated as a ratio or proportion. "99" is a valid quantity for Dimensionless, representing 99:1. And of course 26% converts to 0.26 on this basis. Again I'm guessing but I think "no default" for Dimensionless is treated as "use as a ratio" by persistence, and conversion takes place on writing. A practical and non-obvious workaround is to assign Item `"pattern": "%d %%" `to give a default % unit. Persistence now records 26. But this is a workaround, not a fix. Persistence is misbehaving by recording a "wrong" value when type happens to be Dimensionless and Item has no pattern, for example the deltaSince() then is useless. A quick fix might (I think!) be to give Dimensionless a system default unit of %. This would also help in other cases where people can get in a knot with "unitless" Dimensionless, like charting. I think that's a dirty fix though. It seems to me that persistence should not confuse "unitless" with "ratio units". I'm presuming in the true absence of a default unit, the numeric will be recorded without conversion. Detailed before-and-after example in forum post - https://community.openhab.org/t/persisting-percentages/119523

An alternate workaround is to leave it persisting as ratio, and convert to % after retrieval using toUnit()

jstro · May 20, 2021, 12:18am

Hi Rossko, that was a comprehensive reply, thanks for the effort. So yes that was dumb, I realized after I posted my question that perhaps changing the temperature unit was all I needed to do! And that does work. Thanks for pointing that out.

If I understand, then var T = SmartWeatherAir_Temperature.averageSince(now.minusMinutes(5), "rrd4j") should work, and it does. My default temperature unit is Fahrenheit, and this gives me a numerical average value of the fahrenheit quantity that is consistent with what I see recorded. So that checks out.

However, var RH = SmartWeatherAir_RelativeHumidity.averageSince(now.minusMinutes(5), "rrd4j") gives me some unexpected values. I pulled some values from the persistence via the API and, as you said, they are fractional values, e.g., 0.70 or occasionally a much higher precision fraction I suppose due to round off error (see below). However, the average value obtained is something strange:
2021-05-19 19:06:18.725 [INFO ] [org.openhab.core.model.script.DP ] - RH=2.944002483914810

So if currently the RH values being persisted are around 0.70, why is the average 2.944002483914810? Temperature is working fine, but not this.

This RH item is a Number:Dimensionless quantity:

Some values pulled from persistence:

    {
      "time": 1621467540000,
      "state": "0.7099999999999999"
    },
    {
      "time": 1621467600000,
      "state": "0.7099999999999999"
    },
    {
      "time": 1621467660000,
      "state": "0.7099999999999999"
    },
    {
      "time": 1621467720000,
      "state": "0.7099999999999999"
    },
    {
      "time": 1621467780000,
      "state": "0.7099999999999999"
    },
    {
      "time": 1621467840000,
      "state": "0.7199999999999999"
    },
    {
      "time": 1621467900000,
      "state": "0.7099999999999999"
    },
    {
      "time": 1621467960000,
      "state": "0.7099999999999999"
    }

EDIT TO ADD: I did try the pattern approach. It may be working, I’m waiting for the average to catch up with the new data…

I hope I’m providing enough info to diagnose this. Please let me know if there’s more I can provide. Much appreciated.

jstro · May 20, 2021, 12:30am

Rossko, the pattern idea does seem to be working. It’s providing the correct average of the persisted data. Funny how when the RH data is persisted as a fraction the average is wildly incorrect! That threw me for a loop.

Thanks again for the explanation. It seems it’s a complicated OH issue dealing with this dimensionless problem, but at least there is a work around. Again, much appreciated!

rossko57 · May 20, 2021, 1:09am

I’m not sure if the average takes into account the current value, which would be 70 odd.

jstro · May 20, 2021, 2:01am

Those aberrant averages were obtained before I changed anything with the pattern. The persisted values were all around 0.70, give or take. Strange.

rossko57 · May 20, 2021, 9:56am

Yes, so the average of several historic 0.7 and one (live) 70 can be 2.94.
The averaging is time weighted, not just simple sum+divide.

jstro · May 20, 2021, 11:04am

No no, what I’m saying is that when I first logged the average values for the RH and noticed the aberrant result, all of the persisted values for RH were fractional and all were around 0.7. This was before I changed anything with the pattern. I doubled checked this by pulling the persisted values from the API Explorer and confirmed all the RH values were fractional and were around 0.7. There were no 70’s or anything above 1.

rossko57 · May 20, 2021, 11:50am

Yes, what I’m saying is that averageSince may take into account the ‘live’ version of the Item state as well as historic records- which would be numeric 70. I really don’t know, but it fits.

opus · May 20, 2021, 12:30pm

Tried to check the averageSince on my persisted humidity value (on a tasmota device).
The API explorer shows all values being persisted as the percentage value (70 instead of .7). So the correct averageSince value on my system isn’t giving any clue. Sorry.