Openhab crashes like every 5 days or so

My openhab running on rpi stops running every 5 days or so, and I need to reset it(I mostly just cut the power, or rather my GF does it). The rpi is running just fine, its just OH2 that crashes for some reason and I have no idea how to figure out why. The last log says:


2018-07-31 11:59:42.649 [ItemStateChangedEvent     ] - Mobile_Wlan_Kim changed from ON to OFF

==> /var/log/openhab2/openhab.log <==

2018-07-31 11:59:55.114 [INFO ] [rvice.dhcp.ReceiveDHCPRequestPackets] - DHCP request for unknown address: 192.168.0.19

2018-07-31 12:00:00.032 [INFO ] [arthome.model.script.speedtest.rules] - --> speedtest executed...

==> /var/log/openhab2/events.log <==

2018-07-31 12:00:00.054 [ItemStateChangedEvent     ] - SpeedtestRunning changed from - to Doing meassurements...

2018-07-31 12:00:00.102 [ItemStateChangedEvent     ] - SpeedtestResultDate changed from 2018-07-31T11:00:00.041+0200 to 2018-07-31T12:00:00.050+0200

==> /var/log/openhab2/openhab.log <==

2018-07-31 12:00:00.245 [ERROR] [ntime.internal.engine.ExecuteRuleJob] - Error during the execution of rule Check Plant Status: cannot invoke method public abstract org.eclipse.smarthome.core.types.State org.eclipse.smarthome.core.items.Item.getState() on null

So I was wondering if i should make a cron rule to fire every night at 5am to restart OH, or if I should try to deactive bindings to see if that helps. Its just so hard to debug it, since it takes 5±3 days to happen. Maybe its a memory thing?

Any ideas?

In july it went down 5,8,10,12,16,24,26 …

What do you mean by “crash” - does the webpage/app become inaccessible (times out), or does Jetty server throw a 404? You mention RPI is still running fine - can you log in after a crash and check the status of the openhab service? Type in

sudo systemctl status openhab

…

Also, I’m wondering about those DHCP requests in the log - is your RPI set up for static IP? It’s possible that something is happening to the Pi’s network connection, based on the WLAN status and DHCP request events in your log.

Based on your answers, we should be able to help pin-point the issue. Let’s make sure the service is actually going down first, and it’s not a network/external issue.

Thanks, I do not think I have static IP:

cat /etc/network/interfaces
# interfaces(5) file used by ifup(8) and ifdown(8)
# Include files from /etc/network/interfaces.d:
source-directory /etc/network/interfaces.d
auto lo
iface lo inet loopback

auto eth0
allow-hotplug eth0
iface eth0 inet dhcp

Normally I can do : sudo systemctl restart openhab ,
Today when it crashed I could however not… this time the whole pi crashed.

I wouldn’t set the ip in the raspi configs (since you’ll have to do so every time you flash a new image), but you should be able to tie the ip to your pis MAC address in the router config (so it seta the same IP every time).

Are you running your Pi off the sad card or a usb stick? I’m thinking you might have a corrupted card…

Im running it of a sd card, and unfortunately I can not set the IP in the router since the router is from the cable provider and they do not allow to set static IP… They do not even show the device name!!

Do you have both a wired and wireless connection? I had same real instability until I went to just one or the other, not both. I was running out of memory and threads.

You can use this command in the Karaf console to monitor threads
shell:threads | grep HttpClient | wc -l

To connect to the console instructions are here:

also check your openhab log for out memory errors. They may not be at then end of the log. I often saw out of memory errors hours before things got totally hung up.

openhab> shell:threads | grep HttpClient | wc -l
        17

How do I search for out of memory in the logs?
In the frontail nothing shows up when searching for memory or error…

If you just opened frontail it only shows the most recent lines. You have to open the actual log file and search for it.

Also note the command is only showing (grep) HttpClient threads. there are others, my issue was related to too many HttpThreads, so that is what I used.

In the long run it was easiest to disable all bindings and then slowly turn them on to see which one was causing my issue. Knowing it was too many HttpClient threads made that process faster for me.

So is 17 a lot of threads? I also tried searching in the logs:

openhab> log | grep memory
openhab> log | grep error
openhab> log:display | grep error

However there were not much information here:


On how to search in the logs, and the karaf manual is long and complicated so I guess my syntax for searching in all the logs were wrong :frowning:

Don’t search in the karaf console. Just in the log at the operating system level. usually in the /var/log/openhab2 directory.

Yesterday it crashed again, however this time it will not restart. I can not even log on with putty. In my router it is however listed as connected devices. Does this means that the sd card is broken?

I figured it has to be the SD card, I am now reinstalling everything and hopefully this will also help with stability:)

Did you get a new sd card? If not unless a power outage corrupted the card you may encounter the same problem.

Yes a new sd card did the trick, i started out from scratch again with the latest version of openhabian(the version I had could not be upgraded(was the first release like 2 years ago). So it was time for it anyway. I got ola up running again flawless this time(thanks to great notes from last time), installed all the bindings again with paper UI, and copied over all my configuration files. The only thing that is not working is:

HSBType.toString() in my rules, so I expect there has been some changes to the methods… I will see now if the system will be more stable.

2 Likes

Its been way more stable after OH2.3 and new SD card.

1 Like

Here’s a link for repairing a broken SD card.

https://www.recantha.co.uk/blog/?p=1208

You may be able to get the old sd card working, I wouldn’t rely on it, but have just to test or play around. My brother had a RPI powered via the USB port on his TV (bad idea). I was surprised the sd card lasted for about a year, thats a good bit of on/off’s before it was trashed.

@H102 Thanks for the link, I have used test disk in the past. The only fear I would have is if the card had an actual hardware problem. I believe this article only will fix a card that got corrupted due to a power outage.

Here is my example, I recoverd my friends compact flash a month ago. The recovery worked and I was able to get the lost pictures off. However, the next time I tried to mount the card it gave an error. I ran test disk and fixed it again. This is because the card has something that is not functioning on a hardware level.

1 Like

Yea, it was multiple power outages that killed his card. That’s why I added that the RPI was connected/powered by the TV USB. Hopefully no one else will make that mistake.:face_with_raised_eyebrow:

Thanks for the extra info, I’ll stick that in my back pocket and hopefully never need it. :wink:

1 Like