Charts via rrd4j - stopped filling persistence

[akashkumar]

@rossko57

t’s hard to see how the bsb binding could be involved, when you’ve some Items persisting and some not.
Still feels like an issue when persist starts and builds its list of items to persist (in this case “everything”) at startup time.

So it seems I was a bit hasty in assuming that the error was fixed. Apparently, after clearing the cache only the initial values were stored for all the values that I tested. Thereafter, the same problem reappears, i.e. only a few values are being stored, as visible in the graph below.

@rossko57

I still think selecting everything is lazy but do not think it would make any difference at all to this problem.

Perhaps that is the problem indeed. Initially I only started out with 50+ items, so I thought it would be fine. Maybe with 200+ it is reaching its limits. I do have an SSD attached (through an external hub) though, so normally it shouldn’t be a problem, but then who knows!

[opus (Jürgen Baginski)]

Which persistence services do you have installed?
Is rrd4j set as the default service?

[akashkumar]

@opus

The conclusion that no data was stored was drawn from a REST call for a group. Such would always give zero data!

My apologies. I wasn’t aware of that but even the other values aren’t stored. I had checked other values then too. Here is a log of another parameter. It seems only some values were stored after clearing the cache:
{"name":"Masterbed_Temperature","datapoints":"10","data":[{"time":1587630960000,"state":"21.0"},{"time":1587631200000,"state":"21.0"},{"time":1587631440000,"state":"21.0"},{"time":1587631680000,"state":"21.0"},{"time":1587631920000,"state":"21.0"},{"time":1587632160000,"state":"21.0"},{"time":1587632400000,"state":"20.934444444444445"},{"time":1587632640000,"state":"20.8"},{"time":1587632880000,"state":"20.8"},{"time":1587633120000,"state":"20.8"}]}

I can confirm that new values were collected by openhab (when I grep for this element in events.log), but not stored. As mentioned earlier, the timestamp still corresponds to the most recent element value read by Openhab, and NOT of the content in the rrd file.

@opus

? If you removed the old .rrd files, openHAB would have no way showing the old data! Either you removed other files or the graph is not using the files you think.

Well, the graph is still using those files. It just seems that when using the REST URL, there is a default time start used. When I chose an earlier date to start showing the values, they were all there. Of course when I remove the old files, that data is not shown.

@opus

In order to see the actually minute-wise stored data using the REST API on a rrd4j database you have to limit the requested timeframe to the last 8 houres (assuming your rrd4j setup is using the defaults).

Sure, but I can really confirm that for some elements all data is stored, while for the rest, no updates are made, just file time-stamp is updated.

[akashkumar]

@opus

IMHO if a group would be peraisted by rrd4j, a file like gTemperature.rrd would be created which could only have ONE value at a time.

So, I called the REST api call on the group element. I don’t have any group where all the elements are updated, but at least it does store some data as seen below. So, apparently, it is not true that Rest API calls to group elements always return no data.

{"name":"gTemperaturePlotOutside","datapoints":"43","data":[{"time":1587630780000,"state":"15.1"},{"time":1587630840000,"state":"15.1"},{"time":1587630900000,"state":"15.1"},{"time":1587630960000,"state":"15.1"},{"time":1587631020000,"state":"15.1"},
...
{"time":1587631080000,"state":"15.1"},{"time":1587631140000,"state":"15.1"},{"time":1587633240000,"state":"15.9"},{"time":1587633300000,"state":"15.9"}]}

Btw, I tried to update the rrd config file and a little after it reloaded, I got this error in openhab.log

2020-04-23 16:06:02.257 [WARN ] [ore.internal.scheduler.SchedulerImpl] - Scheduled job failed and stopped

java.lang.IllegalArgumentException: null

at java.nio.Buffer.position(Buffer.java:244) ~[?:1.8.0_222]

at org.rrd4j.core.RrdNioBackend.read(RrdNioBackend.java:172) ~[?:?]

Does anyone know why this error occurs, or how one could fix this?
Thanks a lot in advance!!

[rossko57]

@akashkumar

Here is a log of another parameter. It seems only some values were stored after clearing the cache:

That is a highly suspicious every-4-minutes. The kind of thing rrd4j would do when compressing data? (Which wouldn’t happen for “todays data” with usual settings)
Sure you are looking at the same database with same .persist file?

@akashkumar

it is not true that Rest API calls to group elements always return no data

No-one said that, quite.
Not all Groups have a state that could be persisted - it needs to be assigned a type and aggregation function.
Group persist generally means members, not Group itself.
You can specify a Group explicitly to get it persisted (if possible).
We’re not yet sure if * everything would include that mode.
Your data suggests that it does.

@akashkumar

Does anyone know why this error occurs,

As you think it is related to some mysterious change you made in your rrd config, maybe you wan to share that?

[akashkumar]

@rossko57

That is a highly suspicious every-4-minutes. The kind of thing rrd4j would do when compressing data? (Which wouldn’t happen for “todays data” with usual settings)
Sure you are looking at the same database with same .persist file?

Well, the data does also change every so often. Yes, I am very sure the data corresponds to the same .persist file…

@rossko57

As you think it is related to some mysterious change you made in your rrd config, maybe you wan to share that?

Sure, here it is. I had created the two commented lines to check whether limiting the number of variables that are persisted would help, but it didn’t. I must admit that I didn’t do a full clearing cache cycle though.

Strategies {
// for rrd charts, we need a cron strategy
everyMinute : "0 * * * * ?"
        default = everyChange
}
Items {
//        gTemperaturePlotInside : strategy = everyUpdate, everyMinute
//        gTemperaturePlotOutside : strategy = everyUpdate, everyMinute
        * : strategy = everyUpdate, everyMinute
}

Regarding groups, I currently have no aggregate function. The items are grouped for the sole purpose of plotting a graph easily.

[rossko57]

Okay, you edited rrdj4.persist not rrd4j.cfg which are different files with different purposes.

@akashkumar

gTemperaturePlotInside

Gotta read the docs carefully.

That will persist the state of the Group Item only (if it has one).
It will not persist the state of member Items.
If member Items are not persisted, there will be no data.
When you plot a Chart widget specifying a Group, it does not plot the data for the Group Item.
It fetches and plots the data for the individual member Items.
If there is no data, no chart.

Specifying
gTemperaturePlotInside*
in a persist file will persist members of that group.
But not the group’s own state. (who cares, its not used for charting)

[opus (Jürgen Baginski)]

@akashkumar

Here is a log of another parameter. It seems only some values were stored after clearing the cache:
{"name":"Masterbed_Temperature","datapoints":"10","data":[{"time":1587630960000,"state":"21.0"},{"time":1587631200000,"state":"21.0"},{"time":1587631440000,"state":"21.0"},{"time":1587631680000,"state":"21.0"},{"time":1587631920000,"state":"21.0"},{"time":1587632160000,"state":"21.0"},{"time":1587632400000,"state":"20.934444444444445"},{"time":1587632640000,"state":"20.8"},{"time":1587632880000,"state":"20.8"},{"time":1587633120000,"state":"20.8"}]}

This output of a REST call was using the (default) last 24 houres timeframe. Using this timeframe rrd4j would give you a value every 4 min, hence the timedifference of each timestamp is 240000!
@rossko57, that is the default setup of rrd4j, minute-wise data is kept only for 8 houres!

@akashkumar

I can confirm that new values were collected by openhab (when I grep for this element in events.log), but not stored. As mentioned earlier, the timestamp still corresponds to the most recent element value read by Openhab, and NOT of the content in the rrd file.

The event.log will show what openHAB “knows”, setting the rrd4j persistence service to log DEBUG messages would show what is persisted!

@akashkumar

It just seems that when using the REST URL, there is a default time start used.

Correct, without specifying the starttime, REST will use 24 houres ago and without using the endtime, REST will NOW!

[rossko57]

@opus

that is the default setup of rrd4j, minute-wise data is kept only for 8 houres!

Useful info, I thought it compacted daily by default.

[opus (Jürgen Baginski)]

I used to think the same way, after finding the correct setup I did put the description into the documentation. It’s on the end of that paragraph.

[akashkumar]

@rossko57

Okay, you edited rrdj4.persist not rrd4j.cfg which are different files with different purposes.

aah, my apologies. But since I hadn’t changed anything in that file, I thought it wasn’t relevant. I just had a look at that file as well. It is essentially blank containing only default commented lines.

@rossko57

Specifying
gTemperaturePlotInside*
in a persist file will persist members of that group.
But not the group’s own state. (who cares, its not used for charting)

Indeed, I really only want to plot the graph of individual items, and that purpose is achieved. In fact, the group*.rrd is really not necessary to be stored.

@rossko57

t will not persist the state of member Items.
If member Items are not persisted, there will be no data.
When you plot a Chart widget specifying a Group, it does not plot the data for the Group Item.

But then I do have data also of all the other items due to my *: strategy right. Or are you suggesting that I shouldn’t have been using rrd4j.persist in the first place? After all, I was getting data since the last year or so with the exact same configuration, unless some thing has changed in the newer version and made the format checking more strict.

Any suggestions on how to proceed short of restoring the backup from 3-4 days ago?

Btw, thanks a lot to all for trying to help so far!

[opus (Jürgen Baginski)]

@akashkumar

Any suggestions on how to proceed short of restoring the backup from 3-4 days ago?

I would only persist the items which I care for. Although rrd4j files never grow, persisting 200 items when caring only for a couple of them is putting load on the system ( a raspi!) that isn’t needed.

[akashkumar]

@opus

I would only persist the items which I care for.

Sure, will try it out and let you know how it goes!
I am also considering switching to InfluxDB+Grafana but that will probably be even more demanding for a Raspi…

[rossko57]

@akashkumar

But then I do have data also of all the other items due to my *: strategy right.

Should have. Something, somewhere is going wrong for you.

@akashkumar

I was getting data since the last year or so with the exact same configuration,

It didn’t have the comments lines. Might seem silly but persist files are picky about blank lines, for example.

My explanation about persist group/group* handling differences was meant to explain why your experiment with those commented lines could not work with your charting.

@akashkumar

unless some thing has changed in the newer version and made the format checking more strict.

Maybe. More likely nothing to do with your config, more to do with some system setting.
We haven’t seen solid evidence of missing data yet (although proving a negative is tricky).

@opus

persisting 200 items when caring only for a couple of them is putting load on the system

Adding to that, much depends on your overall system config. You’re generating at least 200 records per minute, plus any ad-hoc updates. Bindings like Modbus can generate 200 updates per second, which has bitten people. (Most don’t do anything like that, of course).

You’re logging each minute; do you need updates in-between? What for? everyChange would be more efficient … but again, what for?
These sort of considerations can become important on a resource strapped Pi.

[opus (Jürgen Baginski)]

@rossko57

You’re logging each minute; do you need updates in-between? What for? everyChange would be more efficient … but again, what for?

I’m sorry for for stepping in again, however rrd4j requires the every minute strategy.

[rossko57]

Yes. So, when logging every minute, do you also need to additionally log every update etc. etc.?
You might for some purposes wanting seconds accuracy, but then you wouldn’t use rrd4j anyway for that.

[opus (Jürgen Baginski)]

EveryUdate is not required. My guess is that most users do it without really reasoning. Those values would be aggregated after 8 houres anyway.

[akashkumar]

Thanks a lot. Adding only the required entries has helped. The result below shows that. Regarding

@rossko57

We haven’t seen solid evidence of missing data yet (although proving a negative is tricky).

Indeed. I guess one could use tools to extract information from the rrd files into text format that one can then really see whether the data is there. I will be happy to share my rrd file, if someone is interested.

[opus (Jürgen Baginski)]

If you really want to have a deep look into the stored data read Here.
It is possible to change data that way, however one has to note that the data corresponding to a special time has to be changed in each and every archive separately in order to keep a consistent database.

[rossko57]

@akashkumar

Adding only the required entries has helped.

It’s all a bit curious. That does suggest that maybe some of your 300ish-per-minute record writes were getting dropped, under stress say. The change in circumstances could originate anywhere. But that is not really an outlandish rate.