UoM default units and consequences

Setup, Configuration and Use Items & Sitemaps
81

Huh? I don’t understand what you mean by that.
If the binding posts just the value xx, it’ll be interpreted as m/s (when that’s the default unit for speed). If it posts incl. the unit i.e. xx mm/h then OH API should spit an error because that’s a dimension mismatch, just like you cannot assign the value in OH console.
Persistence will persist the value only so that 2nd sentence does not make sense to me.

82

hmm. it did already, if I follow the thread here. And this is a problem for me - and I’d think many more. If 3.4 introduced the “default itemType defines persistence” functionality it is widely different from pretty much all installations I’d think.
I for example have a load of itemTypes in my installation - and pretty much none of them I’m afraid uses the default SI unit. There’s not one using Joule, not one using meter, etc…

That means, I have to somehow identify all my MariaDB-persisted items and recalculate them, before I move to 3.4? I can’t do this in all the rrd-files, so I can’t use those anymore for my OH-generated charts… That’s a whole lotta work for me - but I can manage. Someone with a standard pre-OH3.4 installation with only rrd4j-persistence will lose his historic item values completely?

83

No, so far it’s only impacted core. And in a relatively minor way.

It’s worth stating again. Right now, as I understand it the behavior for what unit is applied to the Item’s state is as follows:

Item Type Update SD Units on Item
Number no units no units
Number with units no units (units get stripped)
Number:X no units default based on locale
Number:X no units defined units defined on SD
Number:X units different from locale converted to locale
Number:X units different from locale defined converted to locale

Note: “different from loacle” means °F is converted to °C in locales where °C is the system default, or feet is converted to meters in a local where meters is the default. Posting yards to a Number:Length Item in the US locale will remain yards and posting mm to a Number:Length Item in the rest of the world will remain mm.

The conversion only occurs when you are mixing SI and Imperial units.

There are a number of proposals on the table and under discussion here.

  1. the PR @J-N-K linked to above is to add a defaultUnit that is user definable through Item metadata and independent from the State Description (SD)
  2. how do we handle persistence?
  3. what should be the role of SD in determining the actual units carried by the Item?

If 1 gets merged I believe the table becomes:

Item Type Update SD defaultUnit Units on Item
Number no units no units
Number with units no units (units get stripped)
Number:X no units default based on locale
Number:X no units defined units defined on SD
Number:X no units defined defaultUnit
Number:X no units defined defined defaultUnit
Number:X units different from locale converted to locale
Number:X units different from locale defined converted to locale default
Number:X units different from locale defined converted to locale default
Number:X units different from locale defined defined converted to locale default

Again, “converted to locale” means if your locale is using miles, km will be converted to miles. When the conversion occurs, it’s not clear if the SD and defaultUnit are taken into account of not (I show it above as not).

I can see arguments for the last two rows of the table being defaultUnit instead but my understanding of the PR is that wouldn’t be the case.

For 2, the discussion revolves around mainly how the values should be stored so that the proper units get applied on restoreOnStartup. Right now, the StateDescription pattern is used to determine the units on restoreOnStartup I think. But that won’t always be correct, especially if SD becomes strictly for display purposes, hence the discussion on normalizing the units prior to saving to the database.

I don’t think so actually. It would be relatively unusual for most users to be using units different from their locale, and if they do so, one can change their default measuring units under Settings → Regional → Measurement System. So if you are in Germany and want to use Imperial Units as the default, you can.

If this is true, change the setting described above prior to upgrading and no conversions will be made on your Item states.

Persistence still only saves the current value of the Item regardless of units so if there’s no conversion, your persisted values will remain the same as they always have.

Where people will run into problems is cases where they want to use a mix of SI and Imperial units. These users will exist but will be a relatively rare I would think. And in those cases, with the introduction of defaultUnit the user will be able to define what units the values get saved into persistence. So you’ll just need to add some metadata to keep saving the same units you’re used to.

I do think this threads the needle. It makes UoM more consistent and predictable while giving the end users the control they need to influence the units where they need to.

For completeness here’s the table for persistence.

Item Type Units SD defaultUnit Units Saved
Number no units value as is
Number:X locale units default based on locale (conversion to default scale?)
Number:X locale units defined SD units (?)
Number:X locale units defined defaultUnit
Number:X locale units defined defined defaultUnit
Number:X different from locale not possible
Number:X different from locale defined not possible
Number:X different from locale defined defaultUnit
Number:X different from locale defined defined defaultUnit

Note, by “conversion to default scale” above I mean would km be converted to m before saving?

1 Like
84

No, not the default unit is used for persistence but the unit that is returned by .getUnit. This has always been the case and prior to 3.4 was

  1. the unit derived from the state description
  2. the locale’s default unit from the UnitProvider
  3. null (in this case the plain number with unit stripped was persisted)

Since 3.4 all dimensions (i.e. Length already had, Acceleration did not) have default units, so point 3 is no longer valid.

That means: as long as you have a state description set which defines the unit, nothing changes. If you did not have a state description defining the unit, you can easily add a state description and everything is like it was before. No need to fiddle around with the database. If you add the state description (with the original unit, e.g. mm/h) BEFORE the upgrade to 3.4 you also prevent wrong entries in the database.

My proposed change decouples state description and item unit. The unit defined in the defaultUnit metadata will take precedence and getUnit returns

  1. the unit defined in metadata defaultUnit
  2. the unit derived from the state description
  3. the locale’s default unit from the UnitProvider

It’s debatable if step 2 should be removed. This all independent from the internal representation in the item. There is simply no need to normalize the values. In fact IMO the conversion from °C to °F should also be removed. I assume it was only to provide sensible units without setting a state description and maybe a better modeling would be to provide a state description with the locale-default if there is no user-defined state description.

For programmatic access there is always .toUnit(...) to get the value in the unit you want.

2 Likes
85

Does that mean that already today with 3.4 if I have say an Number:Energy item with 1kW and no state description, it’ll “normalize” (kW → W) and persist the value 1000 (as 2. applies and the locale’s default unit is W now) ?

I’m all for removing it. With defaults now in, it’s no longer needed.
But for sure it’ll keep confusing generations to come wherever it applies should we leave it in.

86

We still need it at the moment, because it’s the only way to properly determine a unit for the item to use for bindings that provide a DecimalType only. The already mentioned rainfall is usually expressed in mm/h (at least in Europe). The locale default unit for Speed is m/s. If a binding now sends 1.2 the state description %.1f mm/h would provide an item unit of mm/h and the result is 1.2 mm/h. If we omit the state description from the logic the result would be 1.2 m/s (which is obviously wrong). With the new metadata you could set that to mm/h and the state description to m/d if you want and it would still be correct, because metadata takes precedence.

87

Side issue only; that’s a valid unit for Speed and is used in real life to indicate rainfall, precipitation rate.

88

Right, I didn’t realize. So it’s a valid match and OH core should accept it.

89

With a new per-Item default metadata for users, I would think the use of state description to also derive default units could/should be removed. Isolating presentation format from data storage format, a Good Thing.

There’s no reason to do that immediately - I’d suggest OH4 would be a good break point for removal.

but
this does also impact on @ccutrer point about binding-supplied state descriptions.
Binding author defines say options like “2 = Standby”, “max = 5”, pattern = %.1f kWh" - fair enough as the binding knows what this channel represents in the real world.
When an Item is linked, these binding-defined parameters are … what, copied to the Item? I don’t know, but it acts like they are - it’s a one-time action. You can hand-edit the Item afterwards to override those parameters. If you re-link, they appear again.

At the moment that stuff affects Item default units (via the pattern), but it’s almost by accident.
Let’s do away with that default units part (keeping the presentation effects), but allow new unit metadata to be given instead/as well.
That’s simply consistent with removing 2 above.

Seems straightforward, but an example of trhe knock-on effects to consider. I think a few hundred zwave device database entries would technically want the ‘default unit’ state description applied.

90

I’m not sure it’s so bad. If the binding is supplying a SD pattern with units that means it knows what units the value is in and it should supply those units as part of the update. And I can’t think of a reason why it would supply different units in the update and the SD pattern it pushes.

If we take the SD pattern out of the chain the binding will still be telling us the units as part of the update. And since that takes precedence that becomes the default and we should remain in the status quo.

Except in the case where the supplied units are from a different measuring system from the system default. Based on @J-N-K’s post above, UoM default units and consequences - #84 by J-N-K, we may not need to worry about that either.

I agree, this auto-conversion is a little disconcerting and think it could be better handled other ways. Perhaps keep the unit from the binding or convert it if and only if defaultUnit is defined and different from what the binding supplied. That puts control of the conversion for persistence and Item state unit purposes in the hands of the user while minimizing the impact.

I do not like the idea of a binding being able to set the defaultUnit. It doesn’t need to.

91

The binding can’t and should not set the default unit.

  • If the bindings KNOWS the unit, it should send QuantityType updates, then there is no need for a default unit anyway.
  • If the binding doesn’t know the unit, it doesn’t make sense to provide a state description from the binding.

In fact we need to find a solution to Update existing Things for definition changes · Issue #1924 · openhab/openhab-core · GitHub so that we can change the channel types without breaking installations. Then we can send QuantityType wherever we want and if the link is to a plain Number item, the unit is stripped like before and otherwise the unit is correct - without any default unit.

The default unit is needed for bindings that do not know the unit.

1 Like
92

Reading quickly through this thread reminded me of:

and now made me check my persisted values for water consumption in the Miele@home binding. It turns out that all values from November 27th until December 11th were persisted as m³ rather than L, i.e. a factor 1000 off. This was introduced with 3.4M5 because of changed default unit and a bug.

The binding doesn’t provide a unit in the state description:

But provides a unit when updating the channel:

So I guess that made it change from L to m³ and thus influenced persistence. BUT… I believe this was fixed in M6, because all persisted values after that are correct. So if I understand this correctly, it’s okay for the binding (or even preferable?) not to provide a unit in state description, as long as it’s provided with the QuantityType - and this is the unit which will be persisted, regardless of any state description provided in the item definition?

I think there is one minor glitch, at least something I need to fully understand. I believe at some point I saw “0 m³” in a sitemap where I would have expected “0 L”, so I had to set state description on my item to “[%.1f L]” explicitly. This is probably because the channel had not been updated, because I hadn’t run my washing machine since I started openHAB. So in this case there was no unit available, and default unit was shown.

93

That’s the direction we seem to be headed.

If the binding knows the units, it should provide that in the state update.

That’s the part that is to be addressed and one of the major subjects of this thread. What about restoreOnStartup. Nothing about the units is saved to persistence. Therefore, without something to go on (e.g. the State Description Pattern right now) it has to choose something. So it chooses the system default.

This is the crux of the breaking change for 3.4 I think. With nothing else, restoreOnStartup will use the system default unit for that Item type when restoring. That’s why setting the state description pattern fixed it for you.

94

It’s perfectly fine to provide a state description (and very useful, e.g. to provide the number of significant digits). I only wanted to point out that technically the state description is not needed for UoM to work for everything that is directly related to the items state if the value is provided as QuantityType. That means you can use QuantityType.toUnit in rules or similar.

For persistence you need a valid return value from NumberItem.getUnit. That is either the locale default unit (in your case most likely m3) or the unit provided by the state description. My PR makes the state description independent from the item unit by introducing metadata.

@rlkoshak is right about the restoreOnStartup.

95

Sure, but when providing unit through QuantityType, perhaps it would be preferable in some cases to leave unit as %unit% and let locale default kick in? This way there would at least be cross-binding consistency instead of different bindings imposing different units for similar values.

It’s a bit of a pickle, because for a dishwasher, water consumption in liter makes more sense than m³ because of the small amount, so in that regard “L” would make sense over %unit%. However, this would mean that Americans would have to override the state description (in linked items) to US gallon or whatever they prefer, even when already having this as locale default unit.

Sorry for thinking loudly here, I’m just not sure what is the best approach from the individual binding perspective - so far I have preferred %unit% in combination with a proper/“API native” unit through QuantityType.

96

Perhaps I misunderstood (and still don’t get it?) something in the mix. I’ll test this in the holidays.
Thanks for your clarification, understand it will perhaos take a while (hihi)

97

… but only until your PR #3248 is in, right ?

Is that correct ?
It would mean we would not need the normalization before persisting that I proposed (after deriving unit from SD is removed).

98

[quote=“mstormi, post:97, topic:142360, full:true”]

That should be the case. I didn’t test it.

Regarding dropping the state description from the unit logic: The problem is that this will result in a change for those that currently use a state description. I’m not sure if we can find a way to make this change non-breaking.

99

You are right, ultimately it’s all just conventions.
But we have to follow at least some rules else we will be discussing each and every use case because there’s always someone who has used a non-default unit and does not want to adapt his data and/or existing post processing.
We should be treating the physically ‘right’ base as a prime candidate but we don’t have to use it when
it is obviously less commonly used. On Github there’s an example with J (= Ws) vs. kWh .

Scale prefixes however should be removed. Wh is obviously more appropriate as a base than kWh, but in some cases like Ws versus Wh we can choose what’s more commonly used (I’d opt for Wh rather than J here).
This will also minimize confusion at people to get hit by a change. A scale factor of 10^n is somewhat intuitive and acceptable, but a factor of 3600 is not, it’s completely messing up any charting attempts.

100

The whole idea of bindings interfering with Item properties makes me a bit queasy, and leads to “what happens if I link two channels?”, “what if I use a profile to link a different type?” etc. But those are ‘advanced’ user problems and are worked around by manual configuring.

Nevertheless this force-feed of state description is now accepted as standard and routine practice. I can’t see any special objection just for this default-unit purpose.

I suppose the key question is “Who knows best?” as in what units this particular measurement should be expressed in.
I’d argue that it is the binding (or binding configurer) because this is device-specific info.

If we accept “binding knows best”, then that IS covered by the binding supplying the appropriate unit in its actual updates via channel e.g. “1.4mm”

All we need to do now is stop the core framework ‘helpfully’ converting that update to state 0.004m if the user has done nothing else. (The whole point of these binding suggestions being to make it hands-off self configuring)

If we are going to prohibit setting the per-Item default unit from the binding, then we could/should still allow the binding to set the ‘pattern’ e.g. [%.1f mm]
Unfortunately we are plotting here to remove ‘pattern’ from consideration in determining default units.
So the Item state would be a converted 0.004m but the UI view would (sometimes) be 4mm.
Is that acceptable? I’m not sure.

If not, this looks like an argument for retaining ‘pattern’ in the hierarchy to determine default unit. (but only in the absence of per-Item default)

To complete the picture, if I have a C sensor I want to read in F, is that possible? Sure, manually set your new Item default-unit property to invoke auto-conversion. Binding supplies update in C, framework auto-converts to Item state F, display it any form you like.