I try to rewrite my rules with the new Tibber binding and want to learn something on TimeSeries.
So my thinking is:
get a start and end date
fetch all states there with getAllStatesBetween
do my magic with forecasts
So I came up with the following short JS Scripting (ECMAScript 262 Edition 11):
var price = items.getItem("TIB_Current_Total_Price"); // that's the #spot-price channel with timeseries
var start = time.ZonedDateTime.now().minusHours(1); // I want the current hour plus 12 hours
var end = time.ZonedDateTime.now().plusHours(12); // so, here's the end
// now getAllStatesBetween
var allPrices = price.persistence.getAllStatesBetween(start, end);
// re-iterate the states and print them out
for (var i = 0; i < allPrices.length; i++) {
var historicItem = allPrices[i];
var state = historicItem.state;
var timestamp = historicItem.timestamp;
console.info("DEBUG: Time: " + timestamp + ", State: " + state);
}
So, why is there the “2025-08-29T17:00:00.133+02:00” entry in there? I guess, that’s the exact time, the binding changed the state of the item to the current hourly price?
Without just blindly deleting the second entry or deleting all values with microseconds, is there a way to avoid this in the first place?
What’s the persistence strategy on this Item. It might be sufficient to prevent it from saving updates and only saving changes. It certainly does look like the state comes from the Item updating as the future state becomes the current state. 133 msec seems a reasonable amount of time for that to have happened.
For the sake of learning and to maybe someday improve the LLMs, the above rule could be made a bit shorter:
var price = items.TIB_Current_Total_Price;
var start = time.toZDT('PT-1H'); // ISO8601 duration string
var end = time.toZDT('PT2H');
var allPrices = price.persistence.getAllStatesBetween(start, end);
allPrices.forEach( historicItem => {
const state = historicItem.state;
const timestamp = historicItem.timestamp;
console.info("DEBUG: Time: " + timestamp + ", State: " + state);
});
I like to have my items persisted on everyChange, so I can restoreOnStartup them. Plus the Forecast-items are in a simple Group with forecast to allow TimeSeries.
OK, so it’s not saving on updates, so that means this duplicate record is coming from a change. I think that might be a bug. If the Item is changed because a forecast record has become the current record, it shouldn’t treat that as a change to record in persistence, or it should replace the old forecast record instead of adding it as a new record.
Is it every past record that gets duplicated? I mean every record between now and one hour ago?
If it’s just the one duplicated record, there might be something else going on.
You’re right. The forecasted value changes exactly (or nearly exactly) at the time the forecast did it. So it should in theory not trigger the corresponding persistence event.
I think I can’t follow you… In that case it’s simple, every hour the price changes, so the forecast will be exactly what the item will change into. Is it that, what you mean? I can imagine, that other bindings will forecast some states, which then aren’t met?
As said in my TimeSeries setup post it’s still not transparent for me how TimeSeries are working. As mentioned in this post also for updating.
For my binding test machine I’m using inmemory persistence to check everything is working. For inmemory I cannot confirm this behavior, see below.
BUT
From binding point of view I send a a TimeSeries to price channels once a day.
The update magic to set the state at the right time is up to the persistence service. Please correct me if I’m wrong.
From rule
rule "Test Persistence"
when
System started
then
var states = Tibber_API_Spot_Price.getAllStatesBetween(ZonedDateTime.now.minusDays(1), ZonedDateTime.now.plusDays(1),"inmemory")
var i = 0
while(i<states.length) {
logInfo("Tibber Spot Price Series", states.get(i).getTimestamp.toString + " -> " + (states.get(i).getState as Number).doubleValue)
i = i + 1
}
end
_Perhaps also interesting for @weymann - because as we discuss in another thread, I got the TimeSeries working today at around 11:20, the item “changed” only since 12:00h and the TimeSeries seems to got up and running only after 13:00h (that’s btw the time I set in the binding, Bernd).
sorry, was a misinterpretation of mine, I had unknowingly a double console.info in my script.
no. sorry, was meant for Bernd. The last edit shows the actual openhab.log with only one instance writing logs. so yes, the problem exists.
But… I tend to think, that might be a bit tricky to get rid of? Because “normally” you aren’t so sure, that “forecasts” will be reached. Tibber prices are fixed, so that’s a given, but that’s not always the case with weather predictions or others… So, how can we be sure, that’s a “fixed forecast” and not a “random forecast”?
The past entries in the database should reflect the actual states the Item has been in. I would expect that the forecast entries to be removed at the time they are used to update the Item. If the Item was not changed, there shouldn’t be any entry saved (assuming an everyChanged strategy).
I can think of no case where it makes sense to preserve both the forecast entry and the entry caused when the forecast was used to change the Item. That makes no sense.
Future entries should reflect what we thing is going to happen but past entries should reflect only what actually happened.
Yes, forecasted states could be just removed from the persistence. But… normally there’s no actual counter measurement present, is there? I mean I could compare the forecasted weather states with let’s say my outside temperature or my actual PV production with the forecast.
So I think, “normally” TimeSeries represent predictions - and accidentially Tibber uses TimeSeries, which then come 100% true! All I can think of is weather forecasts, PV forecasts, … but do we use more than Tibber, which doesn’t use prediction, but future agreed upon values?
My point is it doesn’t matter whether the prediction comes true or not. Once we get to the past, any entries already in the database becuase they were predicted should be removed whether they came true or not. The only entries in the database should reflect the actual states that the Item assumed. If the entry is not the result of a update or change on the Item, it should not be there.
For the weather example, a new measurement will occur and that new measurement will be posted to the Item and the Item’s state is saved. The predicted entry in the database should be removed at that point.
In the tibber example, the predicted entry in the database should be removed prior to what ever mechanism updates the Item with that predicted state is posts that update. Then, because the Item was updated and/or changed that gets saved in the database.
But looking at the logs it’s not consistent.So perhaps there is a timing issue. I have no idea at this point as this sort of operation is what forecasts were created to support (the developers implementing it were doing it for energy prices, not weather) so there must be some sort of subtle bug or timing issue going on. Or there is something off with the Maris DB implementation.
Notice, in your logs posted above sometimes you get three entries for one change.
Thomas, you’re such a smart guy and I’m pretty sure you know how to deal with these’s values within context.
TimeSeries is just a technical concept to provide a state at a given timestamp that’s all! That’s really all!
For Tibber prices (stored as forecast) you’re sure at 13:00 the next day prices are fixed, right? 100 % fixed tight? With timestamp and state right? Yes, you are because it’s the stock ahead price"
For Weather or Solar forecast (stored as forecast) you’re sure that temperature, humididy, clouds,… will fit? No it depends and can be changed within refreshInterval
Ah. Ok. That’s new for me. But still I think we’re facing two different scenarios then.
the same Item is used for forecasts and present and historic states.
the Item solely predicts states and measured states are done by a different Item (and mostly also different binding)
ad 1)
In Tibber binding, there’s the #spotPrice channel used for predictions and curent/historic states, as they do not differ. AFAIR ENTSO-E binding also uses Timeseries. aWATTar uses 24 items to represent 24h of prices and no TimeSeries.
I can’t think of other bindings for that use case?
ad 2)
Weatherunderground and SolarForecast use Timeseries on separate items as weather stations or inverters.
Only in scenario 1 I’d like to only have the “forecasted” values not persisted (if the change a few hundred msecs after the hour is really the persisting of the “current” value).
In scenario 2 I like to have the forecasted data persisted, to later compare my measured itemss with the predicted ones. So I can check if I have to shift something in the forecast bindings. Even if the forecast changes over time, I still think it the “last forecast” can be persisted.
But the real reason I started this thread is energy price like Bernd mentioned:
I currently rely on the fact, the predicted prices are persisted without msecs. So I filter out and remove those prices from the array that have “longer” timestamps. Otherwise my planning ahead for home energy consumption gets scrambled if there’s multiple entries for the same time period (it will get messier in October when spot prices are calculated in 15mins periods for end users)
I can live with that. As long as that behaviour doesn’t change, I can manage. I was just trying to learn if there was something I miss.
But now bringing both Rich’s and Bernd’s thoughts together:
If there’s a way to avoid getting identical states persisted, I’m all in. If it’s not, I have to eliminate double (or even triple) entries of historical Timeseries data. I do understand, that a “everyUpdate” strategy will produce even more data points - and potentially identical states.
Can you create an issue for this in core? I think it should be possible to avoid this. It looks like when setting the current state from the forecast, it doesn’t consider the time and restores with current time.
In scenario 1 the Item gets updated by the actual measurement at some point by the binding. That value gets saved to persistence and the predicted value gets removed.
In scenario 2, the Item gets updated by what ever updates the Item from persistence when the time comes for that entry to become the current state of the Item. Again, that “new” value gets saved to persistence and the predicted value gets removed.
It’s the same action. The only difference is where the update comes from. But whatever mechanism manages this shouldn’t care where the update comes from. It probably couldn’t even know if it wanted to.
But the fact that you have three entries sometimes makes me thing something else is going on.
I was going to suggest the same. I think core is not handling this correctly, but I’m still not 100% certain what’s causing the behavior.
Really, it’s for what ever mechanism updates the Item from the timeseries. It needs to either prevent that update from being saved, or, as I suggested, remove the predicted value prior to updating the Item. Then the update will replace the predicted value through the usual mechanisms according to the persistence strategy.
It occurs to me you can work around this right now by changing the persistence strategy to not save on everyChange. Then the database will only contain those predicted values because the updates won’t get saved as duplicate records.
I was going to suggest the same. I think core is not handling this correctly, but I’m still not 100% certain what’s causing the behavior.