Charts via rrd4j - stopped filling persistence

binderth · March 30, 2018, 9:24am

since a few days, I’m experiencing no more updates for specific Charts.
as explained here:

I do have different file sizes for the exact same configuration of items and stuff
I experience different persistence behaviours despite having the same persistence configuration

what’s happening besides:

the values are coming in (and are stored in parallel without problems in a MySQL persistence)
I’m pretty sure, the red line (labelled “Küche” - “Kitchen”) was present before
there was an outage of the sensors in which there was no update coming in - but after that at least the three bigger files collected data - the smaller file didn’t…

[11:08:00] openhabian@openHAB2:~$ ls -la /srv/openhab2-userdata/persistence/rrd4j/
insgesamt 160
drwxrwxr-x+ 2 openhab openhabian  4096 Feb 13 15:25 .
drwxrwxr-x+ 5 openhab openhabian  4096 Dez 28 20:15 ..
-rw-rw-r--  1 openhab openhab    28280 Mär 30 11:01 RB_MiWeather_Humidity_Kitchen.rrd
-rw-rw-r--  1 openhab openhab    28280 Mär 30 11:01 RB_MiWeather_Pressure_Kitchen.rrd
-rw-rw-r--  1 openhab openhab    28280 Mär 30 10:02 RB_MiWeather_Temperature_Hall.rrd
-rw-rw-r--  1 openhab openhab     5664 Mär 30 11:01 RB_MiWeather_Temperature_Kitchen.rrd
-rw-rw-r--  1 openhab openhab    28280 Mär 30 11:16 RB_MiWeather_Temperature_Matresses.rrd
-rw-rw-r--  1 openhab openhab    28280 Mär 30 11:20 RB_MiWeather_Temperature_Outside.rrd
-rwxrwxr-x  1 openhab openhabian    32 Dez 18 13:44 Readme.txt

What’s wrong here?

weekly Chart Shows:

daily Chart Shows obviously:

monthly Chart Shows only:

legend: the biiig gap was an outage of the Pi.

items

Number RB_MiWeather_Temperature_Kitchen 	"Temperatursensor Küche [%.1f C]"			<temperature> 		(gRossbuehel, Huette, RBChart)	{ expire="1h", mqtt="<[mqttcottage:huette/sensoren/RB_MiWeather_Temperature_Kitchen:state:default]" }
Number RB_MiWeather_Temperature_Outside 	"Temperatursensor Aussen [%.1f C]"			<temperature> 		(gRossbuehel, Huette, RBChart)	{ expire="1h", mqtt="<[mqttcottage:huette/sensoren/RB_MiWeather_Temperature_Outside:state:default]" }
Number RB_MiWeather_Temperature_Hall 		"Temperatursensor Vorraum [%.1f C]"			<temperature> 		(gRossbuehel, Huette, RBChart)	{ expire="1h", mqtt="<[mqttcottage:huette/sensoren/RB_MiWeather_Temperature_Hall:state:default]" }
Number RB_MiWeather_Temperature_Matresses 	"Temperatursensor Matratzenlager [%.1f C]"	<temperature> 		(gRossbuehel, Huette, RBChart)	{ expire="1h", mqtt="<[mqttcottage:huette/sensoren/RB_MiWeather_Temperature_Matresses:state:default]" }

rrd4j.cfg

rossbuehel.def=GAUGE,900,U,U,60
rossbuehel.archives=AVERAGE,0.5,1,365:AVERAGE,0.5,7,300
rossbuehel.items=RB_MiWeather_Temperature_Kitchen, RB_MiWeather_Humidity_Kitchen, RB_MiWeather_Pressure_Kitchen, RB_MiWeather_Temperature_Outside, RB_MiWeather_Temperature_Hall, RB_MiWeather_Temperature_Matresses

rrd4j.persist

Strategies {
	everyMinute		: "0 * * * * ?"
	every5Minutes		: "0 */5 * * * ?"
	everyHour		: "0 0 * * * ?"
	every6Hours		: "0 0 */6 * * ?"
	everyDay		: "0 0 0 * * ?"
	default = everyChange
}


Items {
	RB_MiWeather_Temperature_Kitchen, RB_MiWeather_Humidity_Kitchen, RB_MiWeather_Pressure_Kitchen, RB_MiWeather_Temperature_Outside, RB_MiWeather_Temperature_Hall, RB_MiWeather_Temperature_Matresses : strategy = everyMinute, everyChange
}

opus · March 30, 2018, 9:56am

I still have NO CLUE why this one .rrd is smaller!
Regarding the stopp of persiting, I suspect that it stopped lasrt sunday morning at 03:00 (the change to DST). If you are on OH 2.2 (like me) and not a fairly actual snapshot of OH2.3 this was to be expected. A restart of OH did solve the problem for me and others.

As stated earlier I would stop OH, delete all .rrd files and restart OH (Yes that woulddelete the old data). I do expect all .rrd files being the same size afterwards (keeping fingers crosed).

binderth · March 30, 2018, 12:08pm

This wouldn’t explain that the one item (the smallest filesize…) still got updates. I suspect some Kind of weird misconfiguration or weird bug - or a combination…

I did have this solution in February, but nevertheless tried it again now.

unfortunately after stopping OH2, deleting the old *.rrds and restarting OH2:

[14:03:41] openhabian@openHAB2:~$ ls -la /srv/openhab2-userdata/persistence/rrd4j/
insgesamt 160
drwxrwxr-x+ 2 openhab openhabian  4096 Mär 30 14:04 .
drwxrwxr-x+ 5 openhab openhabian  4096 Dez 28 20:15 ..
-rw-rw-r--  1 openhab openhab    28280 Mär 30 14:04 RB_MiWeather_Humidity_Kitchen.rrd
-rw-rw-r--  1 openhab openhab    28280 Mär 30 14:04 RB_MiWeather_Pressure_Kitchen.rrd
-rw-rw-r--  1 openhab openhab    28280 Mär 30 14:03 RB_MiWeather_Temperature_Hall.rrd
-rw-rw-r--  1 openhab openhab     5664 Mär 30 14:03 RB_MiWeather_Temperature_Kitchen.rrd
-rw-rw-r--  1 openhab openhab    28280 Mär 30 14:03 RB_MiWeather_Temperature_Matresses.rrd
-rw-rw-r--  1 openhab openhab    28280 Mär 30 14:03 RB_MiWeather_Temperature_Outside.rrd
-rwxrwxr-x  1 openhab openhabian    32 Dez 18 13:44 Readme.txt

still the strange size drop for “Kitchen”… I really can’t see anything configured differently for Kitchen! - or I’m blind…

opus · March 30, 2018, 12:10pm

Could you post your rrd4j.cfg file?

binderth · March 30, 2018, 12:11pm

rossbuehel.def=GAUGE,900,U,U,60
rossbuehel.archives=AVERAGE,0.5,1,365:AVERAGE,0.5,7,300
rossbuehel.items=RB_MiWeather_Temperature_Kitchen, RB_MiWeather_Humidity_Kitchen, RB_MiWeather_Pressure_Kitchen, RB_MiWeather_Temperature_Outside, RB_MiWeather_Temperature_Hall, RB_MiWeather_Temperature_Matresses

see above… only Thing is the Umlaut in the description - but I don’t think that’s stored within the .rrd at all?

binderth · March 30, 2018, 12:13pm

at least, they’re getting populated now…

opus · March 30, 2018, 12:26pm

I see no obvious problem in the cfg. If you have searched the forum you probably saw comments about “_” in the item name not working. In my setup they are working with it.
I’d like to help, but I’m out of clues. I checked my files and I do have a .red with a smaller size as well. No problems with that observed so far.

binderth · March 30, 2018, 12:28pm

Thanks for your help! I didn’t stumple across the “_” issue until now. I’ll have a look and perhaps try with no Umlaut again…

opus · March 30, 2018, 1:09pm

I don’t think an Umlaut in a label causes the problem.

opus · March 31, 2018, 7:04am

I did some more research, the .rrd files of size 28280 are using the default_numeric archive setting. The files with a different size in my case have exactly that size as well. I’m still searching for the cause of that.
Presently I think rrd4j uses the default archive setting whenever the setup desired doesn’t fit or no setup is defined. I have used special values for the allowed min and max, which I forgot when I created a new logging item. Also the change of the rrd4j.persist before adding the new item to the rrd4j.cfg could cause the database touse the default archive. However those reasons wouldn’t fit your case, at least as I see it.

binderth · March 31, 2018, 12:35pm

Thanks for your help. Really appreciated!
yeah, I also think that’s not what I experience. funny thing is:

all configuration is the same
values for the sensor are all the same (at least for the temperatures, I have humidity and pressure also in the Strategy/item configuration)

I just added persistence on the other Pi running OH2 (the one, which originally has the sensors attached to). and surprise:

[14:32:53] openhabian@openHAB2-huette:~$ ls -la /srv/openhab2-userdata/persistence/rrd4j/
insgesamt 160
drwxrwxr-x+ 2 openhab openhabian  4096 Mär 31 14:33 .
drwxrwxr-x+ 5 openhab openhabian  4096 Dez 25 18:48 ..
-rw-rw-r--  1 openhab openhab     5664 Mär 31 14:33 RB_MiWeather_Humidity_Kitchen.rrd
-rw-rw-r--  1 openhab openhab    28280 Mär 31 14:33 RB_MiWeather_Pressure_Kitchen.rrd
-rw-rw-r--  1 openhab openhab    28280 Mär 31 14:33 RB_MiWeather_Temperature_Hall.rrd
-rw-rw-r--  1 openhab openhab    28280 Mär 31 14:33 RB_MiWeather_Temperature_Kitchen.rrd
-rw-rw-r--  1 openhab openhab    28280 Mär 31 14:34 RB_MiWeather_Temperature_Matresses.rrd
-rw-rw-r--  1 openhab openhab    28280 Mär 31 14:34 RB_MiWeather_Temperature_Outside.rrd
-rwxrwxr-x  1 openhab openhabian    32 Dez 18 13:44 Readme.txt

For this I even changed the order of the configuration - just to have “kitchen-temperature” not on first…

sigh

edit: oh yeah! the first item gets the wrong size! Just now I realise, it’s “humidity” - not “pressure”

opus · March 31, 2018, 12:42pm

Since you have probably had all the .cfg and .persist ready and no old .rrd files before starting OH, the second assumed cause can’t it in your case.
My head won’t stopp thinking about that! I even tried to have a look onto the code, but I got lost, So I started a question in the forum, even if nobody answers I can keep my findings (I had several cases where I referred to my answres already, I guess it comes with the age ).

Don’t forget to hide some easter-eggs

binderth · March 31, 2018, 12:47pm

yes - in the second OH2 items were already there. I did it that way

add rrd4j as “persistence” in addon.cfg
configured the rrd4j.cfg
added the “Chart”-Group in .items
added the persistence for the items in the .persist

did you read my edit, then? The very first item gets the smaller file size. I’m just figuring, if it’s the order in rrd4j.cfg or in the rrd4j.persist. My bets are in the first…

PS: My kids are too old for searching for easter-eggs… unfortunately. or luckily? you never know…

opus · March 31, 2018, 12:53pm

No, I didn’t see the edit.
So you have another item now that has the smaller size. Note that only the smaller size item is using your desired setup! Reading your procedure I think the sequence of events does this “trick” and not sequence in the .cfg nor .persist file.

P.S.: My kids as well, and the grandchild is only due to be.

binderth · March 31, 2018, 1:00pm

Just changed the config (in the rrd4j.cfg) again: it strongly seems the first item in that config. I do get randomly the first values. In case of my first test today it was “matresses temperature” - the second test just now was funny enough also “matresses temp”… for the last ones I’m sure it was differnet (as I tried like 10-12 times)

hmm. I just use the persistence to show the temperature over the last week or two. Would be nice to have a quick way of seeing months or a year also. So I don’t mind - the “real” data gets stored in mysql, though. The graphs should be sent via email, that’s why I needed a quick way for a JPG or PNG in that case.

opus · March 31, 2018, 1:09pm

If you are really want to show data for a year, your custom setup(the one with the smaller size) doesn’t give you that data. It stores data for 300*7 minutes only!!!
The default setup goes longer!
If need assistance for a setup according your needs, just call. I’m using a setup for charts covering 24 houres, a week and 4 weeks. All switchable from the UI.

binderth · March 31, 2018, 1:31pm

thanks for your help, I just used the examples provided in the docs…
So, for me, best would work:

accurate setting for the last 48h
more or less accurate for the last week
could be less accurate for everything else
Maximum could be a year or so, just to see it in comparison

I really don’t get the parameters and stuff. I suppose, the .archives= section does this? and you can define more than one configuration with the “:” as seperator? But I don’t get it 100%… So, if you can, please help me out!

opus · March 31, 2018, 2:08pm

Let’s take this example:

MyLogger.def=GAUGE,90,0,U,60
MyLogger.archives=MAX,.5,1,1440:MAX,.5,5,2016:MAX,.5,15,2668
MyLogger.items=MyItwm1, MyItem2, .....

First line:
“Gauge”; As stated in the Docs “represents the reading of e.g. a temperature sensor…”
“90”: The “heartbeat”, if no new value is stored after “heartbeat” seconds, the value is considered missing when charting.
“0” : The setting for the minimum value.
“U”: The setting for the maximum value (in this case Unlimited)
“60” The step size, sets the timeintervall(seconds) between consecutive readings.
Second line, the archives, each archive is defined by 4 values:
“MAX”: The “consolidation” function, which is used if more then one value is received during the steps size timeframe (see next point) this function is used to compress the data (in this case the maximum of all values is used, it could also be AVERAGE)
“.5”: “xff”, frankly, I have no clue!
“1”: Steps or Granularity. Steps (not the same as step in the first line!!!) are the number of consecutive readings that are used the create a single entry into the database for this timeintervall. The timeintervall covered is calculated by Step(seconds)*Steps.
“1440”: Rows, the number of Steps that are hold in this archive.
In the example we have 3 archives:
The first is using 1 Steps, which cover 60 seconds(or 1 minute) and holds 1440 values. So archive 1 holds data for 24 houres (24*60 minute Steps ).
The second is using 5 Steps, which cover 5*60 seconds(or 5 minutes) and holds 2016 values. So archive 2 holds data for 7 days (7*24*12 5-minute Steps).
The third is using 15 Steps, which cover 15*60 seconds(or 15 minutes) and holds 2668values. So archive 3 holds data for 4 weeks days (28*24*4 15-minute Steps).
I think the third line doesn’t need any explantion.

Remaining is the problem, why such configuration isn’t used all the time!

binderth · March 31, 2018, 3:44pm

Ok! Thanks a lot!

rossbuehel.def=GAUGE,600,U,U,60
rossbuehel.archives=AVERAGE,0.5,1,2880:AVERAGE,0.,5,2016:AVERAGE,0.5,15,2668:AVERAGE,0.5,240,8760
rossbuehel.items=RB_MiWeather_Pressure_Kitchen,RB_MiWeather_Temperature_Kitchen,RB_MiWeather_Humidity_Kitchen,RB_MiWeather_Temperature_Outside, RB_MiW

Now I understood, thanks a lot!
I added AVERAGE,0.5,240,8760, which should make a archive for 4h per day for a year, right?

PS: I have only new values every 6-12 minutes, so my heartbeat is 600
PPS: the 0.5 means, if more than half of the time for the heartbeat there’s no data, it’ll be undefined.

opus · March 31, 2018, 4:10pm

I think your calculation went the wrong direction, you are using correctly 240 to make 4 hour steps. However when saving 8760 of those, you have 8760 * 4 hours covered, which equals 4 years if I’m not mistaken.

I forgot to mention that each database can have only one consolidation function, so they have to be the same in all archives.

Thanks for the xff explanation.