Ever since the introduction of timeseries, there have been States that get refused when I try to add them via my binding code:
this.sendTimeSeries(channelUID, timeSeries);
For instance, if any of the items contain UnDefType.UNDEF then the send fails, rejecting the entire timeseries.
I believe it should accept the valid items and ignore the invalid ones, but from what I could gather in some of the code, it’s highly improbable this will ever change.
As such, I’m looking for a way to make sure that I don’t add states in my timeseries that are sure to be rejected.
Apart from the one above, do you think there are other obvious ones?
Or is there a way, from a ThingHandler to determine the channel preferred type and act accordingly?
I know little about time series, but it would be easy to modify the code you linked to silently drop any entries with UnDefType, and only error for types that aren’t either StringType or UnDefType, if that is indeed the desired behavior.
Somebody with insight in time series would have to decide what is desirable though - @laursen?
Yes, but it’s apparently not just String channels that are giving me grief, I’m observing it with Number or Quantity
And by looking at the code, there are quite a few places where timeseries can get rejected for invalid values.
Hence the question about a definitive list of refused values besides UnDefType but this might not be possible after all.
I’d say UNDEF is a valid item state but it is not a valid entry inside a TimeSeries. As isn’t NULL.
It just doesn’t make sense to have a series of values then suddenly no value. In that case the previous state would still be the valid value.
As @mstormi said, NULL and UNDEF should be the only two states that a timeseries cannot handle.
In JS the Item has items.MyItem.isUninitialized which will return true if the Item’s state is either NULL or UNDEF. You can filter them out of an array with something like:
Do you have a concrete use case where UnDefType shall occur in the TimeSeries because for me it would be more interesting what you want to express with this value than what’s technically possible.
Let me go deeper:
UnDefType.NULL
I never use this, even in updateState.
This ensures: If item state is NULL you never received ANY value.
UnDefType.UNDEF
I use this in 'updateState to express I received something but I cannot evaluate it
This ensures: If item state is UNDEF you receive updates but something is wrong
Example use case from my Mercedes binding:
Channel vehicle battery SoC is updated by binding. Battery capacity in kWh needs to be configured. If capacity isn’t configured I cannot calculate the channels for charged/uncharged kWh values => UNDEF
So I would be interested what you want to express with a NULL or UNDEF value within your TimeSeries.
UnDefType.UNDEF should mean, well, undefined surely? Persistence needs to see and handle it - rrd, jdbc and timescaledb at least support the concept of an unknown value. What’s the use case for unknowns in time series? Same as unknowns (NULLs) in SQL - they’re unknown. If you have data from an external source like metered data, billed data, discrete consumption data etc. unknowns matter. Making up values to replace them is not right.
It’s in the context of the Open Meteo binding where a user configures many channels for various forecast variables, along with a duration for which to receive predictions.
As it turns out, the API may return “null” to indicate that a forecast is not available for a variable at that given time. There might even be holes in the time series because some variables are only computed for noon after the initial 6 days full forecast for instance.
Inside the binding, I do this:
Instant timestamp = Instant.ofEpochSecond(time);
State state = getForecastState(channelId.toString(), values, valueIndex, forecast);
timeSeries.add(timestamp, state);
Each channel gets its own way to retrieve the value and so the returned State might be UNDEF.
This situation does not arise very often, but it appears that some of my users are asking for predictions too far in the future for locations where coverage is not optimal.
With the current code, this means the entire timeseries is rejected which is quite unfortunate.
Two solutions are possible here:
make openHAB accept timeseries with missing values
change my binding to not call timeSeries.add if the state is one of the invalid ones.
Looking at OH code, I came to the conclusion that lots of places would have to be changed, for no obvious immediate benefit, and no rapid delivery of the solution either.
So my conclusion here is that I’ll change my binding’s code and add a test for the returned State, now that you have confirmed only UNDEF and NULL would be applicable.
Just to be clear, values are not being made up to replace them. NULL and UNDEF states are simply not saved to persistence. They are omitted, not replaced. And no value is made up.
Strip out the null values before calling the timeseries so you can still add the data to the timeseries where it’s present.
It’s not quite that simple. Since you can’t have an UNDEF in a time series you cannot represent gaps or terminate it cleanly. Roll forward of a forecast time series can never UNDEF the state. Ok, it isn’t entirely made up, it’s the previous state - which is (probably) no longer right. If you use Channel state or persistence queries you’ll need to “know” what last update time and the timestamps should be and infer where any gaps are. Sometimes possible, yes, but…
I don’t know the time series implementation intimately, but I’m not sure I follow:
What exactly does this mean? I don’t think it matters what state the Item currently has. When you update time series (states) for a specific Item, why would you want to include an UNDEF or NULL state? What is your data source for creating the time series?
You may want to, but you don’t have to represent gaps, that’s purely academic.
With discrete value gathering in intervals for item updates you never know when a “undef” period ultimately starts, do you. You don’t know if the sensor failed to provide a value right after it just delivered the most recent one or if it just failed right before your current query.
It’s the nature of things. You must not expect OH or TimeSeries to provide a solution to that dilemma.You have to cope with that anyway and using the last value is as good as any other value. BTW you can also use Riemann interpolation.
Maybe theoretically; doesn’t the undef value show consistent granularity? Knowing a value is undef at some point in time provides information that you don’t have if the value is fully absent.
/me trying to explain db null values haha.
Anyway, I would really like to see a (simplified) practical use case that actually shows a day to day issue. Up til now it has mainly hypothetical
As explained in the above answer, it’s coming from me writing a weather binding where the forecast may be missing some values under circumstances out of my control or knowledge.
So this gave the following time series:
Time
00:00:00
01:00:00
02:00:00
03:00:00
04:00:00
05:00:00
06:00:00
07:00:00
08:00:00
09:00:00
10:00:00
11:00:00
12:00:00
Temperature
5.6°C
5.6°C
5.8°C
7.6°C
5.8°C
6.0°C
5.6°C
UNDEF
5.6°C
5.6°C
UNDEF
UNDEF
5.6°C
As you can see there are “holes” in the forecast and because I did not know about that, I did not do anything to prevent such a timeseries from being created.
As is being discussed here, this kind of timeseries is rejected by the items that ultimately receive it, so I saw two options:
change openHab code to “cope” with such a situation
change my binding to not create such a situation, skipping the timestamps where there is no value.
I totally understand that option 1 is complex and not really desirable, so I’m moving with option 2 that would generate series like this:
Time
00:00:00
01:00:00
02:00:00
03:00:00
04:00:00
05:00:00
06:00:00
08:00:00
09:00:00
12:00:00
Temperature
5.6°C
5.6°C
5.8°C
7.6°C
5.8°C
6.0°C
5.6°C
5.6°C
5.6°C
5.6°C
The timestamps for which there are no values are simply not present and it’s the responsibility of the person using the forecast to cater for such a situation.
You do have to represent gaps and it isn’t purely academic. Read the rational for known unknowns (i.e. NULLs) in SQL.
Here’s another use case:
Imagine I have a binding for utility providers that offers dynamic pricing (I do as it happens). It fetches future tariffs, so a forecast TimeSeries. Tariffs are valid for a period of time which may vary depending on supplier, contract or in theory even dynamically. Tariffs are stepped at the start of each period, they are not continuous. I use those tariffs to automate usage - high use on low (preferably negative) tariff, minimal use on high (sometimes painfully high) tariff.
Now imagine there’s a gap, a known unknown, or the next batch of tariffs is unavailable in time and we run off the end of the existing forecast TimeSeries.
Is the last tariff as good as any other value? Is it acceptable to interpolate what the tariff might be based on some set of historical tariffs?
