InfluxDB+Grafana persistence and graphing

Kim_Andersen · December 30, 2018, 4:53pm

Grafana is highly difficult, in my opinion, (I´m a Linux noob as well). I dont get the same errros you do, but I do seem to have problems rendering as well, if I render more than one chart at a time, the whole system lock up, (did the same with Grafana 5.1.4 after openhab has been upgraded from 2.3 to 2.4, thats why I thought updating Grafana to latest version would fix it… It didn´t).
So I can render one chart at at time… If I try to render more, my system locks up, crash and reboot. And Grafana log file is no good in this situation.

Something odd surely goes on, but I cant say for sure it´s because of Grafana. Since I updated openhab from 2.3 to 2.4, Grafana acting strange, and my system tend to crash and reboot a few minutes after I have rendered a few charts (one at a time).
I did make other changes as well, than just upgrade openhab. I also change from modbus1 to modbus 2 binding. But this was after openhab 2.3 has been running for a week or something like that, while Grafana was still acting strange.
I also get tons of binding errors and warnings when starting openhab, which never happened with openhab 2.3

Right now I´m close to give up on my system and start all over. I simply cant find the reason other than something seems very wrong on my system. I have a feeling its something with the openhab upgrade, in combination with grafana, maybe some permission problems somewhere.
But since I´m a total noob at Linux too, I have no knowledge of to how to correct or fix things like that.

PS. While writing this, I entered a sitemap on my system with two charts… System crashed and rebooted, again!
This is what Grafana log shows:

t=2018-12-30T17:51:53+0100 lvl=info msg=Rendering logger=rendering path="d-solo/pq1-vpgRz/casper-temperatur?orgId=1&panelId=2&from=now-12h&to=now&refresh=15s&width=1000&height=500"
t=2018-12-30T17:52:14+0100 lvl=info msg=Rendering logger=rendering path="d-solo/znwsvpRRz/lille-bad?refresh=30s&orgId=1&panelId=2&from=now-12h&to=now&width=1000&height=500"
t=2018-12-30T17:52:14+0100 lvl=info msg=Rendering logger=rendering path="d-solo/znwsvpRRz/lille-bad?refresh=30s&orgId=1&panelId=4&from=now-12h&to=now&width=1000&height=500"
t=2018-12-30T17:52:23+0100 lvl=eror msg="Phantomjs exited with non zero exit code" logger=rendering error="signal: killed"
t=2018-12-30T17:52:23+0100 lvl=eror msg="Phantomjs exited with non zero exit code" logger=rendering error="signal: killed"
t=2018-12-30T17:52:23+0100 lvl=eror msg="Rendering failed." logger=context userId=0 orgId=1 uname= error="signal: killed"
t=2018-12-30T17:52:23+0100 lvl=eror msg="Rendering failed." logger=context userId=0 orgId=1 uname= error="signal: killed"
t=2018-12-30T17:52:23+0100 lvl=eror msg="Request Completed" logger=context userId=0 orgId=1 uname= method=GET path=/render/d-solo/znwsvpRRz/lille-bad status=500 remote_addr=10.4.28.30 time_ms=9597 size=1722 referer="http://10.4.28.237:8080/basicui/app?sitemap=lillebad"
t=2018-12-30T17:52:23+0100 lvl=eror msg="Request Completed" logger=context userId=0 orgId=1 uname= method=GET path=/render/d-solo/znwsvpRRz/lille-bad status=500 remote_addr=10.4.28.30 time_ms=9656 size=1722 referer="http://10.4.28.237:8080/basicui/app?sitemap=lillebad"

First line is one sitemap with one chart.
Second line is a sitemap with two charts…
BAM! openhab restarted (notice my Rpi didn´t reboot, openhab just restart).

This is just great!

jimmyalias · December 30, 2018, 5:19pm

Maybe someone else on the forum can shine some light on our problems?

Kim_Andersen · December 30, 2018, 5:40pm

I wish.
But I still believe your problems is more concetrated to Grafana, where my problems are everywhere, and probably Linux related. Maybe some files are missing or perhaps damaged, even though I use SSD.

Tr8shCan · January 2, 2019, 4:24pm

This worked perfectly for me. Thx!

I didn’t have to create the /usr/share/grafana/tools/phantomjs folder, it already existed.

Joeh · January 3, 2019, 10:02am

Hello,

first of all, thanks a lot for this great tutorial!
Despite the good instructions I sadly got stuck and would be grateful for a hint.

I am running an Openhabian (OpenHAB 2.4) installation on a Raspberry Pi 2.
I followed the steps in post 1 respectively post 84 for the installation and setup of InfluxDB as well as the installation of Grafana (all on that pi). All without any errors.

However when trying to progress and setup Grafana I can’t reach http://monitoring-host:3000.

I double and triple checked all previous steps but couldn’t find anything. Any Ideas what could be wrong? Is there a way to find out whether Grafana was installed correctly? What would you recommend to do for debugging?

Kim_Andersen · January 3, 2019, 10:06am

try with the IP of your Rpi insted… Like this:
http://IP_RPI:3000

jimmyalias · January 3, 2019, 10:36am

Or go to the ip of your pi, there should be an option grafana. Http://:8080

Kim_Andersen · January 3, 2019, 10:58am

It will only be there if he installed influx and grafana through the openhabian-config.
If he follow the #1 post, it will not be there.

jimmyalias · January 3, 2019, 11:16am

Oops sorry, thats how i did it

Joeh · January 3, 2019, 11:41pm

Thanks a lot! Afterwards it is always so clear .
Good to know that installing through openhabian-config is also an option. I guess this could have been even a little easier for a beginner like me. Maybe this option could be included in the tutorial of post 1.
Out of curiosity, does the installation via openhabian-config also assist in configuration or are the steps for configuration the same?

Kim_Andersen · January 3, 2019, 11:54pm

Installing through openhabian-config does everything for you, except the persits file. That part you´ll have to do yourself, ofcouse…
Oh, and the binding ofcouse (through PaperUI). Dont forget the binding, like I just did

memphiz · January 6, 2019, 4:17pm

Same here:


t=2019-01-06T17:06:29+0100 lvl=eror msg="Phantomjs exited with non zero exit code" logger=rendering error="signal: killed"
t=2019-01-06T17:06:29+0100 lvl=eror msg="Phantomjs exited with non zero exit code" logger=rendering error="signal: killed"
t=2019-01-06T17:06:29+0100 lvl=eror msg="Rendering failed." logger=context userId=0 orgId=1 uname= error="signal: killed"
t=2019-01-06T17:06:29+0100 lvl=eror msg="Rendering failed." logger=context userId=0 orgId=1 uname= error="signal: killed"

… and openhab just restarts . But my installation is fresh from the scratch. So i’ve no clue how to fix it.

Best,
Olli

Kim_Andersen · January 6, 2019, 7:26pm

Neither have I

I have struggled with it for some time now on my main setup using an Rpi 3B and openhab 2.4.

At first I refused to accept is was due to the limited Rpi. So I installed a brand new openhab 2.4 on a Rpi3B+ with a very limited setup, (IHC binding, 5 channels and items, and the system binding to monitor whats happens), and then added Influxdb and Grafana from the openhabian-config…

On this test setup Grafana can render 5 charts at the same time. 6 charts will crash openhab and restart.

System monitor shows when Grafana is rendering the charts it takes up nearly 100% cpu and aprox 90-100% of the memory available.

This tells me, Rpi is no longer suitable to a system with many Grafana charts at the same time, at least not when using openhab 2.4 (and openhab 2.5, which I tested as well, though the difference is close to zero as the 2.5 is almost a copy of the released 2.4).

On my main system I can render 2 charts at the same time. 3 will make openhab crash and restart.
On the old openhab 2.3 (before I updated), I could render 4 charts without any problems. Openhab never crashed. I have tested Grafana 5.1.4 - 5.3.4 and 5.4.2 all doing the same.

Something changed in openhab 2.4. And since it simply crash I would say there is a fatal problem somewhere either in openhab 2.4 og a combination of openhab 2.4, Grafana and the limited Rpi 3B. No matter how limited a computer might be, it should never crash like that.

dr.dentz · January 17, 2019, 7:04pm

I get the a similar log error message.
However OpenHAB does not crash.
Did all the above steps, downloaded Phantomjs and CHMOD the directory.

I’m stuck now

t=2019-01-17T19:56:36+0100 lvl=eror msg=“Phantomjs exited with non zero exit code” logger=rendering error=“fork/exec /usr/share/grafana/tools/phantomjs/phantomjs: exec format error”
t=2019-01-17T19:56:36+0100 lvl=eror msg=“Rendering failed.” logger=context userId=1 orgId=1 uname=admin error=“fork/exec /usr/share/grafana/tools/phantomjs/phantomjs: exec format error”

Kim_Andersen · January 17, 2019, 7:52pm

The problem is far more serious that expected. It´s caused by Grafana (PhantomJS) which simply takes a huge amount of computer resources. PhantomJS throws an Java error, and Java process is beeing killed, which result in openhab restarts.
Since this is probably due to using PhantomJS, there isn´t much to do. PhantomJS isn´t supported anymore, and infact Grafana does really recommend it, (however I just noticed last night, that the windows version of Grafana comes with PhantomJS).

I changed to use webview insted. But if you plan to use webview in sitemaps and use remote/cloud connection, you´ll not be able to see the charts for some reason I can´t seem to find.

I gave up trying to use Grafana on my Rpi. Last night I installed the windows version on my Windows server. Then I can use rendering again.

It is very sad with these problems, and I believe it would be wise no longer to recommend rendering Grafana charts if using an Rpi. Webview is fine to recommend, but not without a notice about problems using webview for local URL´s, cause it simple doesn´t work.

rlkoshak · January 17, 2019, 8:37pm

The end result is the same but the explanation isn’t quite right. PhantomJS, being a JavaScript library, cannot throw an Java error. The problem is for some reason PhantomJS consumes so many resources that the Linux kernel freaks out and looks for something to kill to accommodate PhantomJS. For some reason it chooses openHAB, probably because openHAB is the only other process consuming a lot of resources which is running at the same or lower priority level as PhantomJS.

This is honestly the first time I’ve ever seen the kernel freak out and kill some other process like this. The logical approach would be to kill PhantomJS. But if for some reason openHAB is running at a lower priority level than PhantomJS I supposed that would explain it.

The cloud connector only proxies OH’s web server and OH’s REST API. When you use a generated static image chart that image gets served up to your sitemap using openHAB’s web server. When using a webview you are showing content from a different web server (Grafana’s server in this case). The cloud connector cannot proxy the connection to an external web server.

So if you want access to charts outside of your LAN you need to:

put Grafana with PhantomJS on some beefy machine, use the static image URL, and hope for the best
write your own external script to pull down the charts from Grafana and put the images in your html folder
expose your Grafana to the internet

I think the OP is a wiki.

Kim_Andersen · January 17, 2019, 9:00pm

As I recall, my log showed Java was killed, when PhantomsJS threw an error. When Java was killed, openhab stopped as well. But the end result is the same, as you say. Openhab restarts.

Yeah, I had a feeling that would be the case. Thats why I gave up and found another solution…

…which I did.
I installed Grafana on my old WindowsHomeServer2011 last night, (old HP SmartHomeMedia Server with an dual core Intel E6320 and 4GB of RAM). It works, but it sure takes alot of resources on that one as well. CPU goes 100% when rendering. But no crashes of openhab

rlkoshak · January 17, 2019, 10:34pm

The error and the killing came from the kernel in response to what PhantomJS was doing. It was definitely the kernel’s doing.

bccrew · January 18, 2019, 9:26am

Hi everyone,

I’m using the broadlink binding to monitor humidity and temperature in a room and I would like to build a nice graph with influxdb and grafana.

As you can see in the events.log the device put data every time the value has changed. So if the temperature stays the same for 20 minutes, no log entry is created. The binding has a parameter that can be set to x seconds for polling the device.

How can I force that a persistence file updates the influxdb every x time? For example, every 5 minutes (even though the temp or humidity is still the same)

==> /var/log/openhab2/events.log <==
2019-01-18 09:59:03.657 [vent.ItemStateChangedEvent] - BroadlinkA1192168112_Humidity changed from 42.1 to 42.0
2019-01-18 09:59:34.113 [vent.ItemStateChangedEvent] - BroadlinkA1192168112_Humidity changed from 42.0 to 42.1
2019-01-18 10:09:37.200 [vent.ItemStateChangedEvent] - BroadlinkA1192168112_Humidity changed from 42.1 to 42.0
2019-01-18 10:12:07.500 [vent.ItemStateChangedEvent] - BroadlinkA1192168112_Humidity changed from 42.0 to 42.1
2019-01-18 10:13:07.595 [vent.ItemStateChangedEvent] - BroadlinkA1192168112_Temperature changed from 19.299999237060547 to 19.200000762939453
2019-01-18 10:13:37.679 [vent.ItemStateChangedEvent] - BroadlinkA1192168112_Temperature changed from 19.200000762939453 to 19.299999237060547
2019-01-18 10:14:37.779 [vent.ItemStateChangedEvent] - BroadlinkA1192168112_Humidity changed from 42.1 to 42.0
2019-01-18 10:16:37.956 [vent.ItemStateChangedEvent] - BroadlinkA1192168112_Temperature changed from 19.299999237060547 to 19.200000762939453
2019-01-18 10:16:37.961 [vent.ItemStateChangedEvent] - BroadlinkA1192168112_Humidity changed from 42.0 to 42.1
2019-01-18 10:20:38.788 [vent.ItemStateChangedEvent] - BroadlinkA1192168112_Humidity changed from 42.1 to 42.2
2019-01-18 10:21:08.824 [vent.ItemStateChangedEvent] - BroadlinkA1192168112_Humidity changed from 42.2 to 42.1

rlkoshak · January 18, 2019, 3:23pm