no proof as I don’t store a value with the only change in the tenth decimal in rrd (who would anyways?). But the way, rrd stores and rounds its data over time, there will be significant rounding problems with such values.
I don’t get, how it will be such a HUGE task. The actual behaviour (as of 3.3 - I’m not on 3.4 atm) is the one, which is a) simple and b) easy to understand for the user: the actual UoM of an item is the one, which gets persisted. end of story. => and that’s why it’s not expected behaviour for me, if an item changes its UoM just by sending a different unit with a value update.
I can imagine. But you don’t have to co-erce anything - just store the item’s value from its actual UoM at the time of persistence. I don’t expect persistence to calculate historic values, if I change the UoM later on. If I do I certainly can update the historic values myself in parallel as I change the UoM. (given, my persistence allows me to do that)
A binding provides a value including unit (e.g. Netatmo) and is linked to a dimension item. We don’t need a default unit here. If the measurement system is different (e.g. your locale provides m but the value is ft), then the value is converted to the unit of you locale. If the unit is in the same measurement system, then nothing is converted (e.g. m and km).
does that mean - regardless, what I configure at item state, the default Unit of the itemType is used for persistence?
That would be a widely breaking change for me and nearly all of my items will return the “wrong” historic units. I nearly never stored items in the default ItemType’s dimension. e.g. all my weather data is stored in “mm” - but now it would be “m”…
Basically, what I’m addressing there is that the binding provides a state description that says “this channel is in °C, and has a range of 20-40°C. Then my item has a unit of °F cause I’m a weirdo that lives in the US. The HomeKit addon automatically searches state descriptions for min and max values. But for temperatures, everything is in °C. So if a user configured “state description: °F, min 68, max 104”, HomeKit would happily present 20-40 by converting the state description values to C as well. This PR is the missing link so that the user doesn’t have to explicitly provide min/max if the binding provides them - in a different unit.
But in runs into issues - what happens if an item is linked to multiple channels that provide state descriptions? And this is where it gets relevant to the current discussion - what’s a number item’s unit if the user hasn’t specified one, but is linked to (multiple) channels that provides state descriptions with conflicting units?
Just as a real-life example of scaling range for discussion, we have users with car integration and weather stations. Speed quantities used range from roadspeed in tens of km/h to rainfall in mm/h.
I imagine ‘normalized’ Speed storage would be in m/s.
I think that’s going to produce some really tiny numbers for rainfall?
Don’t run too many circles, it’s not that complex.
It does not normalize, never. It only converts imperial ↔ metric. When you force doing so like I did by sending miles to a metric type item, it uses the default unit (meter) of the target system even if the source value (miles) wasn’t using the default for that source system (= feet).
I’ve seen this issue earlier and went with profile which can control quantity passed to/from item after it leaves binding. It still does not solve problem of persistence services which can’t determine what would be base unit for some cases.
Wow, go to a movie with the family and get a night’s sleep and this thread exploded!
You keep saying this but I’m not sure I completely agree. Most measurement units are relatively arbitrary. Somewhat recently a number of SI units were retconed to be based on physics, but they are still arbitrary. There is nothing in the universe that says a meter must be exactly the distance light travels in a vacuum in 1/299792458 seconds. Why 1/299792458 seconds? Because that’s pretty much exactly the length of a meter prior to it’s redefinition.
So no, the units are not defined by physics. They are defined by convention. Most of the world exclusively uses SI units. Some parts of the world mostly use imperial units (which have not ben retconed and are still based on historic conventions) and some parts of the world that use SI units for some things and imperial units for other things.
That’s what I thought too, and it’d still be a problem if the binding doesn’t support units. But in the case where the binding does support units it is converting the °C to *F as expected. Even the charts are correct. To be specific,
A plain number representing °C is published to an MQTT topic.
I set the MQTT Channel’s unit property to “°C”
I set the pattern in the State Description to “%.0f °F”
The value is converted from °C to °F and displayed with °F.
This is a bit of a concern of mine as well. But I haven’t sat down yet to think through whether or not this is a really big deal or not. My initial concern was repeated conversions between units on every restoreOnStartup which could result in the value drifting but I don’t think that will be the problem. At max only one conversion should occur on that initial save.
My understanding is:
there is a default unit for every Number:X type Item
that default unit is based on the locale (if you are in most of the world it means you’ll get SI units as the default, in the US and a few other places it means Imperial Units)
when a value is persisted, no matter what units the Item is carrying, it will be converted to the default units for that Number:X type
when restored, the Item’s state will carry that default unit, not what ever unit it had originally nor the unit defined by the state description
however, you can change the way it appears with the state description pattern per usual and you can use toUnit() on the QuantityType to convert it to a desired unit in a rule
That’s the way it’s always worked before.
I’m not so sure it always works that way now though. For instance, if it still worked that way all the time, what’s converting my temperature Item to °F? I have confirmed that persistence is saving the converted value.
The binding is delivering to the Item °C. Maybe temperature works different from length?
I wouldn’t put too much weight on the docs right now. I’m not sure we ever really understood UoM well enough to make the docs correct in the first place and they haven’t been updated since these latest changes.
Not in rrd4j.
This is a good argument for normalizing the values to the same unit before persisting.
But what’s the precision of the database?
I can’t remember, it’s either feet or yards. A mile is closer in scale to the sorts of things you would measure with km. I tried to suggest defaults for the imperial units that were close in scale to the SI unit defaults.
It probably doesn’t. But a lot of systems will punt and just represent decimal fractions instead of inches and fractions of an inch. So you’d get something like 25.5 ft instead of 25 ft 6 in. Though it’d be pretty awesome if it did it right.
That’s not what @rossko57 means. What he meant is that the precision and range of BigDecimal doesn’t matter because the range and precision of the database itself is going to be less.
It depends. BigDecimal gets its range and precision by using an alternative to IEEE-754. But pretty much all databased I’m aware of use IEEE-754 to represent floating point numbers. So myBigDecimal.toFloat() or toDouble() gets called to convert from the internal BigDecimal format to IEEE-754 which is limited in precision in strange and interesting ways.
If it were feasible at all I would push for that approach. But as long as we support rrd4j storing the units with the values is simply not feasible.
This! I hate inconsistency more than anything. The behavior of UoM seems to be inconsistent. There may be good reasons but I think we need to look through the code and set up experiments to exactly nail down the current behavior and then move on to what the behavior should be.
It would touch pretty much everything, core, add-ons, and require a pretty significant rework on charting in MainUI.
It may be heated but, at least for now, I think it’s been productive.
However, if this starts to spin out of control with no progress, I’ll close it down. We’ve had threads run out of control before and caused harm to the community. I don’t want to let that happen again.
Okeydoke. So when a weather binding posts mph windspeed to a Speed Item it will end up as m/s or something, but you will be able to get that converted back for GUI display, and (by some mechanism I don’t understand) for charts.
All the persisted data will be in m/s but again you can convert back in rules using .toUnit()
The same binding posting mm/h rainfall to a Speed will get mm/h.
But it’ll be persisted as m/s.
Huh? I don’t understand what you mean by that.
If the binding posts just the value xx, it’ll be interpreted as m/s (when that’s the default unit for speed). If it posts incl. the unit i.e. xx mm/h then OH API should spit an error because that’s a dimension mismatch, just like you cannot assign the value in OH console.
Persistence will persist the value only so that 2nd sentence does not make sense to me.
hmm. it did already, if I follow the thread here. And this is a problem for me - and I’d think many more. If 3.4 introduced the “default itemType defines persistence” functionality it is widely different from pretty much all installations I’d think.
I for example have a load of itemTypes in my installation - and pretty much none of them I’m afraid uses the default SI unit. There’s not one using Joule, not one using meter, etc…
That means, I have to somehow identify all my MariaDB-persisted items and recalculate them, before I move to 3.4? I can’t do this in all the rrd-files, so I can’t use those anymore for my OH-generated charts… That’s a whole lotta work for me - but I can manage. Someone with a standard pre-OH3.4 installation with only rrd4j-persistence will lose his historic item values completely?
No, so far it’s only impacted core. And in a relatively minor way.
It’s worth stating again. Right now, as I understand it the behavior for what unit is applied to the Item’s state is as follows:
Item Type
Update
SD
Units on Item
Number
no units
no units
Number
with units
no units (units get stripped)
Number:X
no units
default based on locale
Number:X
no units
defined
units defined on SD
Number:X
units different from locale
converted to locale
Number:X
units different from locale
defined
converted to locale
Note: “different from loacle” means °F is converted to °C in locales where °C is the system default, or feet is converted to meters in a local where meters is the default. Posting yards to a Number:Length Item in the US locale will remain yards and posting mm to a Number:Length Item in the rest of the world will remain mm.
The conversion only occurs when you are mixing SI and Imperial units.
There are a number of proposals on the table and under discussion here.
the PR @J-N-K linked to above is to add a defaultUnit that is user definable through Item metadata and independent from the State Description (SD)
how do we handle persistence?
what should be the role of SD in determining the actual units carried by the Item?
If 1 gets merged I believe the table becomes:
Item Type
Update
SD
defaultUnit
Units on Item
Number
no units
no units
Number
with units
no units (units get stripped)
Number:X
no units
default based on locale
Number:X
no units
defined
units defined on SD
Number:X
no units
defined
defaultUnit
Number:X
no units
defined
defined
defaultUnit
Number:X
units different from locale
converted to locale
Number:X
units different from locale
defined
converted to locale default
Number:X
units different from locale
defined
converted to locale default
Number:X
units different from locale
defined
defined
converted to locale default
Again, “converted to locale” means if your locale is using miles, km will be converted to miles. When the conversion occurs, it’s not clear if the SD and defaultUnit are taken into account of not (I show it above as not).
I can see arguments for the last two rows of the table being defaultUnit instead but my understanding of the PR is that wouldn’t be the case.
For 2, the discussion revolves around mainly how the values should be stored so that the proper units get applied on restoreOnStartup. Right now, the StateDescription pattern is used to determine the units on restoreOnStartup I think. But that won’t always be correct, especially if SD becomes strictly for display purposes, hence the discussion on normalizing the units prior to saving to the database.
I don’t think so actually. It would be relatively unusual for most users to be using units different from their locale, and if they do so, one can change their default measuring units under Settings → Regional → Measurement System. So if you are in Germany and want to use Imperial Units as the default, you can.
If this is true, change the setting described above prior to upgrading and no conversions will be made on your Item states.
Persistence still only saves the current value of the Item regardless of units so if there’s no conversion, your persisted values will remain the same as they always have.
Where people will run into problems is cases where they want to use a mix of SI and Imperial units. These users will exist but will be a relatively rare I would think. And in those cases, with the introduction of defaultUnit the user will be able to define what units the values get saved into persistence. So you’ll just need to add some metadata to keep saving the same units you’re used to.
I do think this threads the needle. It makes UoM more consistent and predictable while giving the end users the control they need to influence the units where they need to.
For completeness here’s the table for persistence.
Item Type
Units
SD
defaultUnit
Units Saved
Number
no units
value as is
Number:X
locale units
default based on locale (conversion to default scale?)
Number:X
locale units
defined
SD units (?)
Number:X
locale units
defined
defaultUnit
Number:X
locale units
defined
defined
defaultUnit
Number:X
different from locale
not possible
Number:X
different from locale
defined
not possible
Number:X
different from locale
defined
defaultUnit
Number:X
different from locale
defined
defined
defaultUnit
Note, by “conversion to default scale” above I mean would km be converted to m before saving?
No, not the default unit is used for persistence but the unit that is returned by .getUnit. This has always been the case and prior to 3.4 was
the unit derived from the state description
the locale’s default unit from the UnitProvider
null (in this case the plain number with unit stripped was persisted)
Since 3.4 all dimensions (i.e. Length already had, Acceleration did not) have default units, so point 3 is no longer valid.
That means: as long as you have a state description set which defines the unit, nothing changes. If you did not have a state description defining the unit, you can easily add a state description and everything is like it was before. No need to fiddle around with the database. If you add the state description (with the original unit, e.g. mm/h) BEFORE the upgrade to 3.4 you also prevent wrong entries in the database.
My proposed change decouples state description and item unit. The unit defined in the defaultUnit metadata will take precedence and getUnit returns
the unit defined in metadata defaultUnit
the unit derived from the state description
the locale’s default unit from the UnitProvider
It’s debatable if step 2 should be removed. This all independent from the internal representation in the item. There is simply no need to normalize the values. In fact IMO the conversion from °C to °F should also be removed. I assume it was only to provide sensible units without setting a state description and maybe a better modeling would be to provide a state description with the locale-default if there is no user-defined state description.
For programmatic access there is always .toUnit(...) to get the value in the unit you want.
Does that mean that already today with 3.4 if I have say an Number:Energy item with 1kW and no state description, it’ll “normalize” (kW → W) and persist the value 1000 (as 2. applies and the locale’s default unit is W now) ?
I’m all for removing it. With defaults now in, it’s no longer needed.
But for sure it’ll keep confusing generations to come wherever it applies should we leave it in.
We still need it at the moment, because it’s the only way to properly determine a unit for the item to use for bindings that provide a DecimalType only. The already mentioned rainfall is usually expressed in mm/h (at least in Europe). The locale default unit for Speed is m/s. If a binding now sends 1.2 the state description %.1f mm/h would provide an item unit of mm/h and the result is 1.2 mm/h. If we omit the state description from the logic the result would be 1.2 m/s (which is obviously wrong). With the new metadata you could set that to mm/h and the state description to m/d if you want and it would still be correct, because metadata takes precedence.
With a new per-Item default metadata for users, I would think the use of state description to also derive default units could/should be removed. Isolating presentation format from data storage format, a Good Thing.
There’s no reason to do that immediately - I’d suggest OH4 would be a good break point for removal.
but
this does also impact on @ccutrer point about binding-supplied state descriptions.
Binding author defines say options like “2 = Standby”, “max = 5”, pattern = %.1f kWh" - fair enough as the binding knows what this channel represents in the real world.
When an Item is linked, these binding-defined parameters are … what, copied to the Item? I don’t know, but it acts like they are - it’s a one-time action. You can hand-edit the Item afterwards to override those parameters. If you re-link, they appear again.
At the moment that stuff affects Item default units (via the pattern), but it’s almost by accident.
Let’s do away with that default units part (keeping the presentation effects), but allow new unit metadata to be given instead/as well.
That’s simply consistent with removing 2 above.
Seems straightforward, but an example of trhe knock-on effects to consider. I think a few hundred zwave device database entries would technically want the ‘default unit’ state description applied.
