Rrd4j does not store any data, only null in rrd files

opus · November 6, 2017, 8:17pm

In which way did you check wether rrd4j contains only NULL values? Through the REST API or by just editing the .rrd file?
Reading the log, it sounds as if only the database “ctr5min” got any items stored into, that is most probably because the two other are the default databases, which are not configured to store any item.
I’d suggest that you stop OpenHAB, delete all .rrd files, make a proper rrd4j.cfg file (at least the line

YourDatabaseName.items=HM_EG_HZ_Arbeit_Robert_vorne_an_Eingang_4_ActualTemperature

should be in there) and restart OpenHAB!

Betonklotz · November 7, 2017, 3:51pm

Hello Jürgen,

thanks for ths hint, but sadly it does not help.
What I did:

sudo systemctl status openhab2.service => openhab2 is running (what else…)
sudo systemctl stop openhab2.service
sudo systemctl status openhab2.service -> openhab2 is not active any longer
change to root folder
sudo find -iname *.rrd
./srv/openhab2-userdata/persistence/rrd4j/HM_EG_HZ_Arbeit_Robert_vorne_an_Eingang_4_ActualTemperature.rrd
./var/lib/openhab2/persistence/rrd4j/HM_EG_HZ_Arbeit_Robert_vorne_an_Eingang_4_ActualTemperature.rrd
delete both files
edit the /etc/openhab2/services/rrd4j.cfg and add

temperatur.def=GAUGE,120,10,40,60
temperatur.archives=AVERAGE,0.5,1,365:AVERAGE,0.5,7,300
temperatur.items=HM_EG_HZ_Arbeit_Robert_vorne_an_Eingang_4_ActualTemperature

sudo reboot
files
./srv/openhab2-userdata/persistence/rrd4j/HM_EG_HZ_Arbeit_Robert_vorne_an_Eingang_4_ActualTemperature.rrd
./var/lib/openhab2/persistence/rrd4j/HM_EG_HZ_Arbeit_Robert_vorne_an_Eingang_4_ActualTemperature.rrd
are created again automatically
output in log

2017-11-07 06:29:34.715 [DEBUG] [org.openhab.persistence.rrd4j       ] - BundleEvent STARTING - org.openhab.persistence.rrd4j
2017-11-07 06:29:34.720 [DEBUG] [stence.rrd4j.internal.RRD4jActivator] - RRD4j persistence bundle has been started.
2017-11-07 06:29:34.757 [DEBUG] [sistence.rrd4j.internal.RRD4jService] - Created ctr5min = GAUGE heartbeat = 60 min/max = 0.0/NaN step = 60 2 archives(s) = [ AVERAGE xff = 0.5 steps = 1 rows = 365 AVERAGE xff = 0.5 steps = 7 rows = 300] 1 items(s) = [HM_EG_HZ_Arbeit_Robert_vorne_an_Eingang_4_ActualTemperature ]
2017-11-07 06:29:34.759 [DEBUG] [sistence.rrd4j.internal.RRD4jService] - Created default_other = GAUGE heartbeat = 3600 min/max = NaN/NaN step = 1 6 archives(s) = [ MAX xff = 0.999 steps = 1 rows = 3600 MAX xff = 0.999 steps = 10 rows = 1440 MAX xff = 0.999 steps = 60 rows = 1440 MAX xff = 0.999 steps = 900 rows = 2880 MAX xff = 0.999 steps = 21600 rows = 1460 MAX xff = 0.999 steps = 86400 rows = 3650] 0 items(s) = []
2017-11-07 06:29:34.761 [DEBUG] [sistence.rrd4j.internal.RRD4jService] - Created default_numeric = GAUGE heartbeat = 60 min/max = NaN/NaN step = 60 6 archives(s) = [ AVERAGE xff = 0.5 steps = 1 rows = 480 AVERAGE xff = 0.5 steps = 4 rows = 360 AVERAGE xff = 0.5 steps = 14 rows = 644 AVERAGE xff = 0.5 steps = 60 rows = 720 AVERAGE xff = 0.5 steps = 720 rows = 730 AVERAGE xff = 0.5 steps = 10080 rows = 520] 0 items(s) = []
2017-11-07 06:29:34.764 [DEBUG] [sistence.rrd4j.internal.RRD4jService] - Created temperatur = GAUGE heartbeat = 900 min/max = 10.0/40.0 step = 60 2 archives(s) = [ AVERAGE xff = 0.5 steps = 1 rows = 365 AVERAGE xff = 0.5 steps = 7 rows = 300] 1 items(s) = [HM_EG_HZ_Arbeit_Robert_vorne_an_Eingang_4_ActualTemperature ]
2017-11-07 06:29:34.777 [DEBUG] [org.openhab.persistence.rrd4j       ] - ServiceEvent REGISTERED - {org.openhab.core.persistence.PersistenceService}={ctr5min.items=HM_EG_HZ_Arbeit_Robert_vorne_an_Eingang_4_ActualTemperature, component.name=org.openhab.persistence.rrd4j, temperatur.items=HM_EG_HZ_Arbeit_Robert_vorne_an_Eingang_4_ActualTemperature, ctr5min.def=GAUGE,60,0,U,60, service.pid=org.openhab.rrd4j, temperatur.archives=AVERAGE,0.5,1,365:AVERAGE,0.5,7,300, ctr5min.archives=AVERAGE,0.5,1,365:AVERAGE,0.5,7,300, component.id=164, temperatur.def=GAUGE,900,10,40,60, service.id=283, service.bundleid=194, service.scope=bundle} - org.openhab.persistence.rrd4j
2017-11-07 06:29:34.805 [DEBUG] [4j.internal.charts.RRD4jChartServlet] - Starting up rrd chart servlet at /rrdchart.png
2017-11-07 06:29:34.838 [DEBUG] [org.openhab.persistence.rrd4j       ] - ServiceEvent REGISTERED - {javax.servlet.ServletContext}={osgi.web.symbolicname=org.openhab.persistence.rrd4j, osgi.web.version=1.10.0, osgi.web.contextpath=/, service.id=286, service.bundleid=194, service.scope=singleton} - org.openhab.persistence.rrd4j
2017-11-07 06:29:34.847 [DEBUG] [org.openhab.persistence.rrd4j       ] - ServiceEvent REGISTERED - {org.openhab.ui.chart.ChartProvider}={component.name=org.openhab.persistence.rrd4j.chartservlet, component.id=165, service.id=285, service.bundleid=194, service.scope=bundle} - org.openhab.persistence.rrd4j
2017-11-07 06:29:34.849 [DEBUG] [org.openhab.persistence.rrd4j       ] - BundleEvent STARTED - org.openhab.persistence.rrd4j

How I checked the rrd is empty?

stopped openhab2
checked the file (e.g. via nano)
checked the timestamp of the file
made an apt-get for rrdtools, but they are not working (rrdtool dump *.rrd reported only ERROR: ‘HM_EG_HZ_Arbeit_Robert_vorne_an_Eingang_4_ActualTemperature.rrd’ is not an RRD file)

opus · November 7, 2017, 4:20pm

Neither opening the .rrd4j files using a editor nor the rrdtool will give you a valid answer. Use the REST API (you need to install REST Docs via PaperUi) in order to check if there is something useful in the databade.
Note that the nature of rrd4j databases is that the database file is created in the FINAL size at the beginning. So you won’t see any change from the “outside”.

Betonklotz · November 7, 2017, 5:07pm

Hmmm…

if access to the file is stopped and it’s opened via third party editor: shouldn’t be some content in it?
filesize should be constant, I’m aware of this fact. But timestamp of last access/save should be updated (depending on step while creating the rrd), or not?
a rrdtool dump normally dumps all contents of a rrd file. However: rrdtools are not working…
however, calls from the rest API showed following content:
/rest/items?recursive=false

[...]
8	
link	"http://xxx.xxx.xxx.xxx:8080/rest/items/HM_EG_HZ_Arbeit_Robert_vorne_an_Eingang_4_ActualTemperature"
state	"20.40"
stateDescription	
minimum	-10
maximum	56
step	0.1
pattern	"%.2f °C"
readOnly	true
options	[]
type	"Number"
name	"HM_EG_HZ_Arbeit_Robert_vorne_an_Eingang_4_ActualTemperature"
label	"Actual Temperature"
category	"Temperature"
tags	[]
groupNames	[]
[...]

/rest/services/org.eclipse.smarthome.persistence/config

default	"rrd4j"

/rest/persistence

0	
id	"rrd4j"
label	"rrd4j"
type	"Queryable"

/rest/persistence/items and also /rest/persistence/items?serviceId=rrd4j

[]

/rest/persistence/items/HM_EG_HZ_Arbeit_Robert_vorne_an_Eingang_4_ActualTemperature

name	"HM_EG_HZ_Arbeit_Robert_vorne_an_Eingang_4_ActualTemperature"
datapoints	"0"
data	[]

If I use any string which does not match an item I’ll get a http 500 as repsonse (JAVA NPE), if I use a valid item which is not stored in a rrd, response is same as above (0 datappoints and empty array)

Looks strange to me at all. I assumed at /rest/persistence/items some response and the final call just tells me I already thougt: no usefull data in the rrd at all

Betonklotz · November 7, 2017, 5:47pm

found openhab2 had some old data in the file /var/lib/openhab2/config/org/openhab/rrd4j.config
Now I changed it to

service.pid="org.openhab.rrd4j"
temperatur.archives="AVERAGE,0.5,1,365:AVERAGE,0.5,7,300"
temperatur.def="GAUGE,900,10,40,60"
temperatur.items="HM_EG_HZ_Arbeit_Robert_vorne_an_Eingang_4_ActualTemperature"

made a restart and durign bootup now the old crt5min entry is gone

2017-11-07 18:41:32.169 [DEBUG] [org.openhab.persistence.rrd4j       ] - BundleEvent STARTING - org.openhab.persistence.rrd4j
2017-11-07 18:41:32.172 [DEBUG] [stence.rrd4j.internal.RRD4jActivator] - RRD4j persistence bundle has been started.
2017-11-07 18:41:32.207 [DEBUG] [sistence.rrd4j.internal.RRD4jService] - Created default_other = GAUGE heartbeat = 3600 min/max = NaN/NaN step = 1 6 archives(s) = [ MAX xff = 0.999 steps = 1 rows = 3600 MAX xff = 0.999 steps = 10 rows = 1440 MAX xff = 0.999 steps = 60 rows = 1440 MAX xff = 0.999 steps = 900 rows = 2880 MAX xff = 0.999 steps = 21600 rows = 1460 MAX xff = 0.999 steps = 86400 rows = 3650] 0 items(s) = []
2017-11-07 18:41:32.208 [DEBUG] [sistence.rrd4j.internal.RRD4jService] - Created default_numeric = GAUGE heartbeat = 60 min/max = NaN/NaN step = 60 6 archives(s) = [ AVERAGE xff = 0.5 steps = 1 rows = 480 AVERAGE xff = 0.5 steps = 4 rows = 360 AVERAGE xff = 0.5 steps = 14 rows = 644 AVERAGE xff = 0.5 steps = 60 rows = 720 AVERAGE xff = 0.5 steps = 720 rows = 730 AVERAGE xff = 0.5 steps = 10080 rows = 520] 0 items(s) = []
2017-11-07 18:41:32.209 [DEBUG] [sistence.rrd4j.internal.RRD4jService] - Created temperatur = GAUGE heartbeat = 900 min/max = 10.0/40.0 step = 60 2 archives(s) = [ AVERAGE xff = 0.5 steps = 1 rows = 365 AVERAGE xff = 0.5 steps = 7 rows = 300] 1 items(s) = [HM_EG_HZ_Arbeit_Robert_vorne_an_Eingang_4_ActualTemperature ]
2017-11-07 18:41:32.223 [DEBUG] [org.openhab.persistence.rrd4j       ] - ServiceEvent REGISTERED - {org.openhab.core.persistence.PersistenceService}={service.pid=org.openhab.rrd4j, temperatur.def=GAUGE,900,10,40,60, component.name=org.openhab.persistence.rrd4j, component.id=166, temperatur.archives=AVERAGE,0.5,1,365:AVERAGE,0.5,7,300, temperatur.items=HM_EG_HZ_Arbeit_Robert_vorne_an_Eingang_4_ActualTemperature, service.id=284, service.bundleid=194, service.scope=bundle} - org.openhab.persistence.rrd4j
2017-11-07 18:41:32.249 [DEBUG] [4j.internal.charts.RRD4jChartServlet] - Starting up rrd chart servlet at /rrdchart.png
2017-11-07 18:41:32.282 [DEBUG] [org.openhab.persistence.rrd4j       ] - ServiceEvent REGISTERED - {javax.servlet.ServletContext}={osgi.web.symbolicname=org.openhab.persistence.rrd4j, osgi.web.version=1.10.0, osgi.web.contextpath=/, service.id=287, service.bundleid=194, service.scope=singleton} - org.openhab.persistence.rrd4j
2017-11-07 18:41:32.291 [DEBUG] [org.openhab.persistence.rrd4j       ] - ServiceEvent REGISTERED - {org.openhab.ui.chart.ChartProvider}={component.name=org.openhab.persistence.rrd4j.chartservlet, component.id=167, service.id=286, service.bundleid=194, service.scope=bundle} - org.openhab.persistence.rrd4j
2017-11-07 18:41:32.293 [DEBUG] [org.openhab.persistence.rrd4j       ] - BundleEvent STARTED - org.openhab.persistence.rrd4j

However, a call to /rest/persistence/items and also /rest/persistence/items?serviceId=rrd4j still gives an empty array, and finally also /rest/persistence/items/HM_EG_HZ_Arbeit_Robert_vorne_an_Eingang_4_ActualTemperature showed no entries

opus · November 7, 2017, 6:38pm

If REST Api is showing zero entries, then there are zero entries!
After you removed all old entries did you try the stop OH, remove all .rrd files procedure as well.
Other then that I’m out of clues. .

opus · November 7, 2017, 9:36pm

Looking a bit closer at the entries for your rrd4j.cfg:

What is that for? I have no such entry!

In the first archive you are saving 365 values, which are taken at every step (since you use a step of 60 seconds, that 365 minutes). In the second archive you saving the average from 7 steps(i.e. minutes) covering a timeframe of 2100 minutes (7*300). No comment on that!

You set the hearbeat to 900, that way stating that you would only consider a value missing if nothing arrived within 900 seconds(15 minutes!). The minimum and values you are using are remarkable, do you really want the neglect values below 10° and above 40°?
I wont say that is the cause of the problem, but since I’m out of ideas …

Betonklotz · November 8, 2017, 8:16am

Hello Jürgen,

thanks for your continous help, but sadly it still does not work.
I missed deleting the rrd files and restarting, did it but still the same: no content in the rrd (via REST API).

I did not enter the service.pid in the rrd4j.cfg file, that was done most likely by some automatic process. Also after I deleted this line and made a restart of the RPi, it was entered again by some unknown process.

The rrd config was just a c&p from examples, changed it now to

temperatur.archives="AVERAGE,0.5,5,8928:AVERAGE,0.5,30,8784:AVERAGE,0.5,60,17520:MAX,0.5,1440,1825:MIN,0.5,1440,1825"
temperatur.def="GAUGE,120,10,40,60"
temperatur.items="HM_EG_HZ_Arbeit_Robert_vorne_an_Eingang_4_ActualTemperature"

to have some real values. The restriction to accept only temperature between 10 and 40 °C and discard the rest is because the sensor is indoor (near my working desk). I’ll quit working if temp drops below 10°C and also at high temperature…
Output in the OH2 log is now

2017-11-08 08:24:35.287 [DEBUG] [org.openhab.persistence.rrd4j       ] - BundleEvent STARTING - org.openhab.persistence.rrd4j
2017-11-08 08:24:35.290 [DEBUG] [stence.rrd4j.internal.RRD4jActivator] - RRD4j persistence bundle has been started.
2017-11-08 08:24:35.319 [DEBUG] [org.openhab.persistence.rrd4j       ] - BundleEvent STARTED - org.openhab.persistence.rrd4j
2017-11-08 08:24:35.362 [DEBUG] [sistence.rrd4j.internal.RRD4jService] - Created default_other = GAUGE heartbeat = 3600 min/max = NaN/NaN step = 1 6 archives(s) = [ MAX xff = 0.999 steps = 1 rows = 3600 MAX xff = 0.999 steps = 10 rows = 1440 MAX xff = 0.999 steps = 60 rows = 1440 MAX xff = 0.999 steps = 900 rows = 2880 MAX xff = 0.999 steps = 21600 rows = 1460 MAX xff = 0.999 steps = 86400 rows = 3650] 0 items(s) = []
2017-11-08 08:24:35.363 [DEBUG] [sistence.rrd4j.internal.RRD4jService] - Created default_numeric = GAUGE heartbeat = 60 min/max = NaN/NaN step = 60 6 archives(s) = [ AVERAGE xff = 0.5 steps = 1 rows = 480 AVERAGE xff = 0.5 steps = 4 rows = 360 AVERAGE xff = 0.5 steps = 14 rows = 644 AVERAGE xff = 0.5 steps = 60 rows = 720 AVERAGE xff = 0.5 steps = 720 rows = 730 AVERAGE xff = 0.5 steps = 10080 rows = 520] 0 items(s) = []
2017-11-08 08:24:35.364 [DEBUG] [sistence.rrd4j.internal.RRD4jService] - Created temperatur = GAUGE heartbeat = 120 min/max = 10.0/40.0 step = 60 5 archives(s) = [ AVERAGE xff = 0.5 steps = 5 rows = 8928 AVERAGE xff = 0.5 steps = 30 rows = 8784 AVERAGE xff = 0.5 steps = 60 rows = 17520 MAX xff = 0.5 steps = 1440 rows = 1825 MIN xff = 0.5 steps = 1440 rows = 1825] 1 items(s) = [HM_EG_HZ_Arbeit_Robert_vorne_an_Eingang_4_ActualTemperature ]
2017-11-08 08:24:35.376 [DEBUG] [org.openhab.persistence.rrd4j       ] - ServiceEvent REGISTERED - {org.openhab.core.persistence.PersistenceService}={service.pid=org.openhab.rrd4j, temperatur.def=GAUGE,120,10,40,60, component.name=org.openhab.persistence.rrd4j, component.id=166, temperatur.archives=AVERAGE,0.5,5,8928:AVERAGE,0.5,30,8784:AVERAGE,0.5,60,17520:MAX,0.5,1440,1825:MIN,0.5,1440,1825, temperatur.items=HM_EG_HZ_Arbeit_Robert_vorne_an_Eingang_4_ActualTemperature, service.id=285, service.bundleid=194, service.scope=bundle} - org.openhab.persistence.rrd4j
2017-11-08 08:24:35.402 [DEBUG] [4j.internal.charts.RRD4jChartServlet] - Starting up rrd chart servlet at /rrdchart.png
2017-11-08 08:24:35.431 [DEBUG] [org.openhab.persistence.rrd4j       ] - ServiceEvent REGISTERED - {javax.servlet.ServletContext}={osgi.web.symbolicname=org.openhab.persistence.rrd4j, osgi.web.version=1.10.0, osgi.web.contextpath=/, service.id=288, service.bundleid=194, service.scope=singleton} - org.openhab.persistence.rrd4j
2017-11-08 08:24:35.440 [DEBUG] [org.openhab.persistence.rrd4j       ] - ServiceEvent REGISTERED - {org.openhab.ui.chart.ChartProvider}={component.name=org.openhab.persistence.rrd4j.chartservlet, component.id=167, service.id=287, service.bundleid=194, service.scope=bundle} - org.openhab.persistence.rrd4j

Some basic questions:

what is the correct place for the rrd4j configuration? First or second one?
/etc/openhab2/services/rrd4j.cfg (this one does not have the service.pid inside)
/var/lib/openhab2/config/org/openhab/rrd4j.config (has the service.pid inside after a sudo reboot of the RPi automatically)
I assume something is broken in the openhabian image and I’ll give it a second try from scratch.
Couldn’t I dig deeper inside, e.g. is some process running and fetching every minute the value and tries to store it or similar? I would have expected in the logs at least every minute (or whatever step is configured) some output where the actual temperature is gathered…

opus · November 8, 2017, 7:24pm

As far as I understand the first location 's the corect one. I guess the file gets automatically written in the second one.
I think anew install could solve, since I can’tell se a reason for this behaviour.
Using the DEBUG logging for rrd4j you should see a line for item stored.
IMHO there is no ned to digg deeper into, something is wrong with your setup and we will try to solve it.

Betonklotz · November 10, 2017, 5:07pm

Hello Jürgen,

sorry for taking so long with an answer, but in the meantime I treid some things…

just removed OH completely and made a fresh install via apt-get --> does not solve the issue
wiped the SD card, feteched (the same) openhabian image and started again -> does not solve the issue but gave me lot of pain due do used AES encryption I used for the Homematic devices

additionally I could report also the this issue Homematic GW failure

So the “stable” version of OH2 seems to be everything, but with Homematic not stable and usable.

At the moment I’m back on FHEM. OH2 is fancy and the GUI tries to allow an easy start. But the “magic” in the background needs much work to be used in some real environment for more than just few hours demonstration with Homematic

May I ask what version and HW you’re using? Is it worth to use the unstable?

opus · November 10, 2017, 6:45pm

I’m on OH 21. (Stable) on a RasPi2

I’m sorry to hear that it didn’t work for you.
I’m using openhabian, which I had not changed outside the OH requirements. My system is setup with just a CUL-stick and I’m using following bindings: Intertechno, Sonos, Tankerkoenig. Persistence: rrd4j, mapDB Other: VoiceRRS (correct name?)

Betonklotz · November 13, 2017, 5:12pm

Hello Jürgen,

it seems to be related to the Homegear Binding. Took another Pi and used Mosquitto to transfer some (dummy) data via MQTT and for this source also rrd4j worked like it should.
This makes me quite wondering, as the data from the Homematic devices seem to be present in OH2. Even more: I could control the actors and also via REST API I see the state of the senors. Even with PaperUI I controlled and read out all HomeMatic devices.
However: as Homegear seems to be new setup in the unstable version, I’ll give OH2 in few months another chance. Up to this, old FHEM needs to work a bit longer.
Thanks for your nice and quick support.

SachinD · March 25, 2018, 3:52pm

I am having the same issue, data from MQTT binding is persisted in RRD4J as well as JDBC, but any data from Xaomi Mi Home sensors doesn’t get persisted. The observation is similar to Robert’s, item changes are observed in the log, In case of RRD4J the files are created and are empty, same thing is observed in JDBC, tables get created but no data. Is it something to do with 2.x data sources ? Do bindings need to do something special to enable items to persist ?

opus · March 26, 2018, 5:10am

Could you post your Setup (related Items definitions and the .persisit file).
What makes you think the .rrd is empty (checking entries via rest API or editing a .rrd or …)?

SachinD · March 27, 2018, 1:10pm

Sorry Jurgen, problem was at my end. Forgot to define the group, items were tagged with the group. Might be a good idea for enhancement for the persistence subsystem to flag this as it creates the empty archives but does not log data.

finkm · March 28, 2018, 11:37am

Just an important FYI for everyone who reads this post: The first post in this thread contains an error in the filename:

/etc/openhab2/persistence/rr4dj.persist

should be

/etc/openhab2/persistence/rrd4j.persist

So if anybody is copy&pasting this like me, it will fail

jwiseman · May 2, 2019, 4:00pm

My RRD4j NULL errors doesn’t happen very often; when it does it’s tied to a specific zWave water/temp devices (have 3 of them).

Running OH 2.4 with OH 2.5 addon’s using a KAR.

2019-05-02 09:44:49.358 [ERROR] [sistence.rrd4j.internal.RRD4jService] - Could not create rrd4j database file ‘/volume1/@appstore/openHAB/userdata/persistence/rrd4j/SumpPumpSensor_Temp.rrd’: null

Any advise in having the RRD4j service ignore NULL entries?

Best, Jay

opus · May 2, 2019, 4:15pm

What strategy are you using, one or more?
The answer is IMHO disregard it in any case.

jwiseman · May 2, 2019, 4:18pm

strategy = everyChange, restoreOnStartup

Best, Jay

opus · May 2, 2019, 4:19pm

Cold be a race condition happening sometimes.