Influx and grafana just stopped

NDR008 · June 26, 2019, 2:10am

Hi, my setup was running fine.

Influxdb, grafana, pre rendered grafana images periodically, etc.

Now grafana shows that it cannot reach the server.
And OH is full of this error (sorry cannot find the code formatting…)
FYI the problem started while nobody was at home, so it is a problem that developed alone. Memory failure?

==> /var/log/openhab2/openhab.log <==

2019-06-26 04:02:00.126 [ERROR] [org.influxdb.impl.BatchProcessor ] - Batch could not be sent. Data will be lost

retrofit.RetrofitError: Connection refused (Connection refused)

at retrofit.RetrofitError.networkError(RetrofitError.java:27) ~[216:org.openhab.persistence.influxdb:1.13.0]

at retrofit.RestAdapter$RestHandler.invokeRequest(RestAdapter.java:395) ~[216:org.openhab.persistence.influxdb:1.13.0]

at retrofit.RestAdapter$RestHandler.invoke(RestAdapter.java:240) ~[216:org.openhab.persistence.influxdb:1.13.0]

at org.influxdb.impl.$Proxy145.writePoints(Unknown Source) ~[216:org.openhab.persistence.influxdb:1.13.0]

at org.influxdb.impl.InfluxDBImpl.write(InfluxDBImpl.java:151) ~[216:org.openhab.persistence.influxdb:1.13.0]

at org.influxdb.impl.BatchProcessor.write(BatchProcessor.java:171) [216:org.openhab.persistence.influxdb:1.13.0]

at org.influxdb.impl.BatchProcessor$1.run(BatchProcessor.java:144) [216:org.openhab.persistence.influxdb:1.13.0]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:?]

at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:?]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:?]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:?]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:?]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:?]

at java.lang.Thread.run(Thread.java:748) [?:?]

Caused by: java.net.ConnectException: Connection refused (Connection refused)

at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:?]

at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:?]

at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:?]

at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:?]

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:?]

at java.net.Socket.connect(Socket.java:589) ~[?:?]

at com.squareup.okhttp.internal.Platform.connectSocket(Platform.java:100) ~[?:?]

at com.squareup.okhttp.internal.http.SocketConnector.connectRawSocket(SocketConnector.java:160) ~[?:?]

at com.squareup.okhttp.internal.http.SocketConnector.connectCleartext(SocketConnector.java:67) ~[?:?]

at com.squareup.okhttp.Connection.connect(Connection.java:152) ~[?:?]

at com.squareup.okhttp.Connection.connectAndSetOwner(Connection.java:185) ~[?:?]

at com.squareup.okhttp.OkHttpClient$1.connectAndSetOwner(OkHttpClient.java:128) ~[?:?]

at com.squareup.okhttp.internal.http.HttpEngine.nextConnection(HttpEngine.java:341) ~[?:?]

at com.squareup.okhttp.internal.http.HttpEngine.connect(HttpEngine.java:330) ~[?:?]

at com.squareup.okhttp.internal.http.HttpEngine.sendRequest(HttpEngine.java:248) ~[?:?]

at com.squareup.okhttp.Call.getResponse(Call.java:273) ~[?:?]

at com.squareup.okhttp.Call$ApplicationInterceptorChain.proceed(Call.java:230) ~[?:?]

at com.squareup.okhttp.Call.getResponseWithInterceptorChain(Call.java:201) ~[?:?]

at com.squareup.okhttp.Call.execute(Call.java:81) ~[?:?]

at retrofit.client.OkClient.execute(OkClient.java:53) ~[?:?]

at retrofit.RestAdapter$RestHandler.invokeRequest(RestAdapter.java:

NDR008 · June 26, 2019, 2:27am

Hey, when I try to check influxdb service status I get that there is no status… ???
How could that happen?
By the way, I installed influx and grafana via the openhabian script.

NDR008 · June 26, 2019, 2:43am

So I installed over that latest influxdb via apt-get.
Now service is running again, will see if the data is persisting later

BrutalBirdie · June 26, 2019, 6:28am

Since you stated you used openhabian I assume you are on a Pi, assuming even further using a SD-Card,
did you check it’s space?
Also how old is the SD-Card, it might be about to fail.
I switch all my Pis to SSD since it would eat SD-Cards like cereal.

NDR008 · June 26, 2019, 7:21am

Yeah, it has been a while. I am thinking of switching over to an ext HDD (disc type).
But the card is only 20% full.

After re-installing influx, I do not get the persistence error, but I now realise that even Grafana is not working as expected…

I hope I do not end up starting from scratch again.

BrutalBirdie · June 26, 2019, 8:35am

Well full is not the problem, SD-Card can get corrupted pretty fast with many read/write processes.
I would always advice to have a backup of your system.

Kim_Andersen · June 26, 2019, 9:14am

Whats the error you get from Grafana?
I´d struggle with this yesterday as well… A few weeks ago, I installed a fresh copy of openhab 2.5, used my old config files. Installed Influxdb from openhabian-config. But Grafana refused to connect to the service (I have Grafana running on my Windows server, where it has been running for several months without issues).
I´d realise that there is errors in the script when installing InfluxDb from openhabian-config. The user/pass doesn´t seem to get added. (the users is added, but not the passwords) as well as access granted doesn´t get passed.
After solving this matter, there were still issues with InfluxDB. I get a bad gateway (error 502) everytime, in Grafana. I had to change the host names for both Grafana and Influxdb… After that, Grafana connected again without any issues, and using the same user/pw. I cant explain why… The old config files should have worked just fine. But for some reason they didn´t.
If you get a Bad Gateway in Grafana… Make sure the hostname for Influx is set to the local IP, (mine was set to “localhost” in influxdb.cfg). I know, it doesn´t make sense, “localhost” should work just as well… But it doesnt!

rlkoshak · June 26, 2019, 2:05pm

In addition to Kim’s suggestions, there is also a recent change in Grafana. They now default [security] → allow_embedding to false. You need to set that to true in order to, for example, embed a panel from your dashboard on your sitemap or HABPanel.

NDR008 · June 26, 2019, 5:07pm

However as mentioned, I didn’t touch my setup or do any updates when things stopped working.

Kim_Andersen · June 26, 2019, 10:03pm

To be able to help, you have to provide some more info. What error do you get from Grafana?

NDR008 · June 27, 2019, 7:27am

How did you do that?
Do you still boot with an SD card and just have all directories on SSD?

Did you set things up with an SD first then migrate the partitions or installed from the first on SSD?

I want to do the same. Never realised that small SSDs cost almost the same as an microSD.

BrutalBirdie · June 27, 2019, 8:43am

Just google a bit, you will find the correct information.
I watched this video here

Kim_Andersen · June 27, 2019, 11:17am

If your Rpi3 is a B+ model, then it will boot from USB by default. If it´s not this version, then you´ll need to active boot from USB… Google is your friend on this matter…

Assuming your Rpi is a 3B+ model - You can grap the openhab image and write it directly to your the SSD, and then run from there, just like you did with your SD card.
I have not been using SD card for very long now. I use SSD´s. And I´ll never look back.

NDR008 · June 27, 2019, 12:45pm

I am thinking of trying with a conventional HDD, since Pi3’s USB speed is slow, and a conventional HDD should be more reliable anyway (and I have a few lying around).

My Pi is a 3B (not +), which to my understanding, can be made to boot from USB but permanently. So instead for now I’ll make a hybrid setup, that uses a microSD but loads the OS from ext HDD.