Well, let’s revisit the “original” problem as scaling/normalization is that golden lamb that we all still seem to be dancing around.
Before 3.4 default units added, I had an Number:Energy item set to “1kW” and persisted.
Now on updating OH, restoreOnStartup is called to retrieve the value from persistence and the now running 3.4 applies a new default W.
If the internal representation is unconvertedly copied back i.e. same what was persisted, the state now will be 1W as only the unit (or scale part of the unit, actually) is replaced, or ?
As we want the state after restore to be 1kW or 1000W like before, the has to be multiplied by 1000 at some stage, correct ? Does that currently happen ? when exactly ?
Or has OH already been treating scale prefixes as part of the value, not the unit ?
If so, if I have a default unit W and send 1kW there, on assignment, it’ll have converted the 1 to 1000 and stored that as the internal value. It knew and still knows W is the implicit base unit to apply to the internally stored value. But in addition, it keeps the information that the value is to be displayed as kW by default (if no conversion is requested). So that info is merely a default display pattern than “the” unit.
BUT - and that’s what makes the difference when you consider the scale prefix to be a part of the value, not the unit - it does not mean the state is 1000kW.
Does anyone know ? Is anybody of you able to try it out and confirm or falsify ?
That’s why storage unit and display unit should be separated from each other. You might be storing values using kWh, but once reaching high enough value you might want to swap display to MWh with more decimal points. Doing it so with OH < 4.0 makes all persisted states a waste or manual migration.
I’m not splitting hairs. I’m trying to understand. Because in Imperial units, which usually don’t have nice scalar modifiers like SI units do (there’s no such thing as a kiloyard for example), the “physically correct” unit depends on what you are measuring. If you are measuring the area of a house, you’d use square feet. The size of a parcel of land is measured in acres, or if it’s a big farm/ranch hectares, or for really big areas like the sizes of cities in square miles.
If measuing the rate of rainfall you’d use inches per hour but the speed of a car would be miles per hour. You’d never measure the speed of a car in inches per second nor would you ever measure rainfall in miles per hour.
There is no base unit that is the always correct one. It’s always situational dependent. And conversions between them are not always even and straight forward. For example, a hectare is 2.47105381 acres or 107639.1 square feet.
This is where I’m coming from. A world where the correct default depends on the context of what you are measuring. For power and other types of units where the average person is less likely to encounter them maybe it’s a different story. But for the common ones like length, speed, volume, etc. there isn’t a correct default. It depends on what it is you are measuring.
Obviously we have to pick something but that’s Something has to be selected as a default when no other information is available, but the user needs to be able to override it both in how it’s displayed and how it’s persisted.
It doesn’t.
I think what we are dancing around is an edge case.
If the binding provides the unit in the update to the Item we are good.
If the binding doesn’t provide the unit in the update to the Item, here we are good too.
However, what if the binding provides the unit in the update but that unit is different from the system default for that Item (let’s make it easy and say it’s km where the default is m) and there is no defaultUnit configured nor a state description configured?
What is going to be saved to persistence? What’s going to be restored on startup?
We want to avoid, if we can, that we save km but restore it with m as the unit (unless it’s properly scaled).
I think all the other cases are well handled and to me at least it is clear where the units come from and where they go. But this one I think remains unclear what would happen and what should happen.
Only in the case where you don’t have a State Description defined. Otherwise, right now in 3.4, the unit in the State Description is used.
But your example is the same as what I am concerned about too.
Not shown here is the impact of defaultUnit, right? With defaultUnit set OH 4 would persist and restore assuming that unit, not the system default, right?
Correct. It works the same as the unit in the state description now. The difference is that currently persistence, representation and added unit (added when a decimal type is passed to the item) are coupled. With the defaultUnit the representation (state-description) is de-coupled from persistence/added unit (which both use defaultUnit).
Which is more or less what I meant. We have a set of defaults now to match what’s used in the majority of use cases in physics, but we can still selectively replace them if we identify unpopular choices (such as J → should be turned into Wh).
I think it does. It’s not as intuitive as in SI as the naming scheme doesn’t include the base unit’s name and scale factors are not 10^n, but it’s essentially the same. A yard is 3 feet so you could write “3ft” instead of “yd” with “3” being the scale prefix and ft being the base unit.
Thanks for that overview. Is that theory or did you really validate that in some way?
If I’m getting it right, we will not have a problem with persisted values in rows 1&2, only in 3
(btw someone already found that: After Update to OH3.4 one issue in ruleDSL - #14 by J-N-K).
4 is when sticking with <= 3.3 so we need not worry about and the user can fix it himself.
Good you highlighted the missing unit in 3.3 but 3.3 is nothing we need to take action on.
We should document that table in some central docs location and advise users to a) set state description today (preferrably before an upgrade to 3.4 so it should be part of breaking change docs) for any items they care about and b) (just proposal) to replace %unit% by the unit they want to use/see displayed.
For clarity, OH4 means after merging Jan’s outstanding PR#3248 to implement defaultUnit metadata, correct ?
Please note that this is only relevant for a small subset of items: Only Number items that are linked to channels that send a value WITH unit, WITHOUT a default unit in pre-3.4 openHAB and WITHOUT a state description that defines a unit.
A default unit was already present for
temperature (°C / °F)
pressure (Pa / inHg)
speed (km/h / mph)
length (m / in)
intensity (W/m2, in both system)
dimensionless (ONE, in both systems)
angle (°, in both systems)
I believe that greatly reduces the number of items you need to check. You can type Number: in the search field of the item’s page and see which need adjustment.
Edit: If you export all your items using curl -X 'GET' 'http://localhost:8080/rest/items?metadata=statedescription&recursive=false' -H 'accept: application/json' > items.json you can use
const fs = require('fs');
fs.readFile('/Users/jan/Documents/items.json', 'utf8', (err, data) => {
if (err) {
console.error(err);
return;
}
const items = JSON.parse(data);
let itemCount = 0;
items.forEach(item => {
if (item.type.startsWith('Number:')) {
if (item.stateDescription == undefined) {
console.log(item.type + ' ' + item.name + ' has no state description at all');
itemCount++;
} else if (item.stateDescription.pattern.includes('%unit%')) {
console.log(item.type + ' ' + item.name + ' has state description with unit placeholder');
itemCount++;
} else if (item.metadata == undefined || item.metadata.stateDescription == undefined) {
console.log(item.type + ' ' + item.name + ' has a state description not set in metadata');
itemCount++;
}
}
});
console.log('Total number of items that need your attention: ' + itemCount);
});
and it prints a list of all items that MIGHT need attention. It does not filter the ones that had a default unit before and also prints those that have a state description set by a binding (which is perfectly fine). You should pay attention to those missing a state description or have a state description with %unit% in it.
Regarding the “normalization”, I see the point of @Spaceman_Spiff. I have no strong opinion on that, as I already said: it doesn’t make a difference in what unit values are stored inside the item, representation is done by the state description. However, this will not solve the issue unless Add ItemStateUpdatedEvent and enable group channel-links by J-N-K · Pull Request #3141 · openhab/openhab-core · GitHub is merged, because currently the ItemStateEvent carries the unit that is sent by the binding/rule, not the one that is stored in the item.
Yes - and that is the correct unit which it should carry. This event should not be normalized.
The ItemStateUpdateEvent would then carry the state with the normalized unit and scale.
If you interact with external systems you have to normalize at some point.
So why not make it consistent and use the normalized value throughout openHAB.
That way persistence, API, Logs etc. is all in sync and it’s clear what is going on.
And as you said:
The internal storage of the value doesn’t matter because the datatype is precise enough
Graphical representation in the GUI is detached from the internal item state unit and scale
I’ve created the issue because I though it’s an elegant way to make UoM a really powerful and useful feature. It was more an “it would be great if it is that way in the long term” issue than an “this is an idea for a quick fix” issue.
You mean the graphical representation e.g. in the GUI (just to be sure - I think there was enough confusion in the last couple of days )
In short, is use the API explorer. I’ve you full out the door for one item, you’ll just need to change the Item name and the metadata to apply and hit submit again.
But like @J-N-K says, only a fraction of your 400 items will need to be adjusted.
This still makes removing state description from the unit determination logic breaking, but it allows the user to easily fix it by copying the state description pattern (if present) to the new metadata.
I just came across another issue if we internally normalize the item states: %unit% would always display the system unit (or default unit), not the unit from the command/update. If a binding selects the unit based on the value (e.g. distance of your car from home can be 600m or 435 km) this works automatically now, but not if we store a normalized state.
What does it show now if I post an update in °F to a temperature item in °C?
I still think this is okay and consistent.
If desired the graphical representation can be auto scaled to be 600 m instead of 0.6 km but of course this would be a new feature. Upside would be this would then work for every item state regardless of the binding.
[WARN ] [nternal.profiles.SystemOffsetProfile] - Received a QuantityType state '22.35 °C' with unit, but the offset is defined as a plain number without unit (0), please consider adding a unit to the profile offset.
)