maximumSince isn't the maximum

I’m getting wind speed values every minute or so from a LaCrosse sensor and using a rule to update max values. The problem is the Weekly Max is less than the Daily Max - which is by definition not possible.

The sensor actually report km/h which I convert to mph, that part works fine.

The rule:
rule “Update_LaCrosse_Wind_Speed” when
Item LaCrosseWeather_Wind_Speed received update
then
var speed = (LaCrosseWeather_Wind_Speed.state as DecimalType).doubleValue
var speed_mph = speed / 1.609
postUpdate(LaCrosseWeather_Wind_Speed_mph, speed_mph)
LaCrosseWeather_Wind_Speed_24hMax.postUpdate(LaCrosseWeather_Wind_Speed_mph.maximumSince(now.minusDays(1)).state)
LaCrosseWeather_Wind_Speed_7dMax.postUpdate(LaCrosseWeather_Wind_Speed_mph.maximumSince(now.minusDays(7)).state)
end

The problem is for some reason LaCrosseWeather_Wind_Speed_7dMax is less than LaCrosseWeather_Wind_Speed_24hMax.

Since 24hMax is a subset of 7dMax it should be impossible for this to happen. The opposite is possible of course.

Any ideas on what’s going on here.

Please How to use code fences for code and logs.

What database are you persisting the data to? If it’s rrd4j it might be an artifact of how the data is compressed as it ages.

It takes some time for an update to actually get written out to the database so I would not expect that the call to maximumSince(now.minusDays(1)).state would include the value you just posted to the Item. However, enough time has passed after that call for the maximumSince(now.minusDays(7)) to include the most recent value. Would that explain the difference you see?

This does seems like an artifact of rrd4j but with slightly different symptoms. The minusDays(1) value seems mostly correct. The minusDays(7) value seems to be the current value or close to it.

Does this imply that maximumSince could have different values for different persistence databases? I can see more complicated statistics getting messed up based on compression but Max and Min are pretty straightforward concepts.

If so what is the “preferred” database?

Please have a look into the documentation for rrd4j. This database does consolidate older data, using the default setup only the stored values for each minute over the last 8! houres are kept. When requesting data covering more then 8 houres the values are stored as average of timesteps.
If you need the correct values for the maximum since a day and a week you could either setup rrd4j in a way to have those values ( for example consolidate the maximum value of each timestep) or move to a database that does not consolidate (compress!) the data.

1 Like

Hi @opus can you please explain a bit deeper what you mean by “consolidate the maximum value of each timestep” ??

EDIT: To be specific, I would like to archive the actual values of a temperature every minute over the last week (for a chart), and also the minimum and maximum values over the last year.

To start I suggest you have a close look into the rrd4j documentation.

As you can read in the documentation rrd4j does consolidate (or compress) to data in the archives that are reaching back longer then the period covered in the first archive. For the default setup the first archive covers 8 hours. In other words your desired data points for every minute are held only for this timeframe ( in the default setup).
The following archives which do cover longer periods consolidate the minute wise data points ( in case of the default setup the average of all data points for a timestep are taken).
The timeframe covered by each archive as well as the timesteps (how many data points are used to calculate a consolidated datapoint) and the used consolidation function can be set by a custom setup.
If you want to store a value for each minute over a week you need to adjust your setup in a way that archive does cover a week instead of 8 hours. The downside for this is that each graph calculated for a week or less would grab all those data points ( you would notice to longer time to calculate the graph).

For which timeframe should those max and min values be calculated?
Really for a year?, or min and max of each day?

Using rrd4j for this purpose has two downsides. First you need to persist the min and max separately and second the used time period (for example a day) would only be chance start a 00:00 ( rrd4j would start a new “day” calculation after each 24 hours after the initial calculation started).

In order to have all minute-wise datapoints for a week (10080 minutes) and the maximum of each 24 hour period (1440 minutes) for a year (365 days) the archive setup would be like:

MyCustomSetUp.archives=MAXIMUM,0.5,1,10080:MAXIMUM,0.5,1440,365

Requesting a chart from such a setting for a timeframe up to a week ago would give the actual values, longer periods would show the maximum values . At least I think so, NOT TESTED!

@opus many thanks for the very comprehensive answer; very helpful!

My actual use case is that I want to 1) display a chart of the outside temperature over a period of one week, 2) record the single absolute maximum outside temperature in the last 365 days, and 3) ditto record the single absolute minimum outside temperature in the last 365 days.

I am not completely sure, but I think an .archives like the one you suggest, might not work since it could not store both maximum and minimum values in one archive. So perhaps I need to link two Items to the same Channel and archive one using MAXIMUM, and the other using MINIMUM. Or something like that??

I am sure that you need two archives and two itmes for that one!

@opus I have read the rrd4j documentation, but I don’t understand the relationship between the rrd4j.cfg file and the rrd4j.persist file. Currently I have the former file empty, and a list of items in the latter. But I understand that to do what I want I need to add some definitions into rrd4j.cfg (??) but how do I link the entries in rrd4j.persist to the new definitions in rrd4j.cfg ??

.cfg tells rrd4j what to do. In this case, manage compaction etc.
.persist tells openHAB persistence what to do. Which Items to store when.

So if you want cfg to do certain things to a particular Item, you just need to use the same name everyone else is using for it. There’s no link as such.

Sorry for being so slow, but could you please be even more specific.

Currently I have an rrd4j.persist file with 30 Items in it. And currently I have no entries in the 'rrd4j.cfg` file, so the 30 Items are all being persisted according to the (invisible) “default” setting.

If I would add new settings in rrd4j.cfg for (say) min365 and max365, how can I tell the system which of my 30 Items shall continue to be persisted according to the existing “default” setting, and which according to the new min365 resp. max365 settings??

As Opus said, look at the rrd4j doc, which includes examples.

1 Like

Start reading Here

Create two packages of the three definitions (consisting of the three lines .def, .archives and .items) named min365 and max365. In their .items lines you specify for which items it should be used.
I actually can’t say if persisting a single item in two ways would work, you migth need a second item for that.

Dear @opus, Dear @rossko57,

Thanks for the link to the docs. Do you really think that I have not read them? Especially when I mentioned in the prior posts that I have read them. And that I said that notwithstanding the docs, they did not answer my question. Grrr.

The question relates to the “binding” between specific Items in the .persist file and specific settings in the .cfg file. @opus has kindly pointed out that in the case that an Item is mentioned in one (or possibly more) of the settings in .cfg AND also in .persist then it will be persisted according to that setting, and thus he implies an alternate case that if an Item is in .persist but NOT in a setting in .cfg then it will be persisted according to the (hidden) default .cfg entry.

(Really really guys, this is by no means clear from the docs).

All items configured in the .persist file will be persisted.
By default the settings of the default .cfg will be used by all of the above configured items.
If there is a custom .cfg it will be used for the items configured in the .items line of this .cfg. Concerning an item configured in more then one configuration I am not sure (as said above).
It migth be that in case of a single custom .cfg with no .items line, this setting would be used for all the above configured items.
I also THINK that all items which are not mentioned in .items line of the .cfg will be persisted with the default configuration.

Many thanks @opus. I will try to clarify your three think / might / not sure, topics, and then I will make a PR to add your answer above (with the respective clarifications) to the doc…

I’m sorry for this limited information, I am only a user of rrd4j. Until your questions I felt comfortable with my present knowledge, now I have to start digging into the code again.
Back when I initially digged into the code I came up with the explanation of the archives for the docs.
Let’s see who gets some answers first ( isn’t it the season to do something offline, I’d rather do coding stuff in winter time).
Updating the docs will be done afterwards.

Hi @opus, I didn’t mean this as a personal criticism; I just don’t want others to criticise me; there are evidently holes in the docs, which we can fix together.

1 Like

I did some testing:

Your setup can have several datasources (i.e. the different configuration done by the three lines .def, .archives and .items).
Each single item can ONLY be persisted by one of those setups (that should have been obvious to me from the start, since each item will be persisted into the file named after the item name! STUPID ME!)
If an item which is configured in the .persist file to be persisted by rrd4j is not mentioned in any datasource definition (i.e. in any .items line) the DEFAULT configuration will be used.
The question: If you configure a datasource with NO .items line, what datasource is used (default ?) is under investigation, I was not able to remove “remainders” of a custom setup in order to test that one. Although no rrd4j.cfg is present the items continue to be persisted as if the removed file would be present. SEARCHING…
[EDIT:]
After deleting the rrd4j.config (not .cfg!), the default configuration was used again.
Using a new rrd4j.cfg without a .items line resulted in all items are persisted in the default configuration (as expected).

1 Like

Hi @opus many thanks for your feedback. I gave myself a special project to rewrite the readme.md file for this service.

You can see the results here

I have not (yet) opened a PR for these changes, in case we discover anything else which needs to be improved. => Your input would be appreciated. :slight_smile:

1 Like