SOLVED: OH stops after a day with OutOfMemory

It’s impossible to deduce a cause from this variety of symptoms, but if I was to guess you simply run out of memory and your system is busy paging.
You will have to mostly debug that on your own, this cannot be done from remote and is beyond the scope of this forum (and it’s time consuming so probably noone is willing to).

Some stuff to look after:

  • how much swap do you have ? use zram ?
    Should be fine if you really installed openHABian but see outputs of swapon -s and zramctl
    do you run stuff other than OH ? InfluxDB, Grafana ? Stop them, it’s just too much for a RPi3.
  • use htop and iostat to find out about memory consumption. G**gle for help.
  • setup monitoring to find out if memory usage is growing. It should settle after a while. If it does not you have a mem leak. Gradually disable components like HabPanel and bindings to find out when memory use starts to remain stable without these.
    Good luck.

It certainly used to be possible to build HABpanels that walloped the OH server. I believe there was work on this though.

I do recall that HABpanel was likely to flush out “weird” Items. You can try a test independent of HABpanel.
Just use REST API to get a list of all your Items, and see how long that takes.

I’ve got a critical issue during I had reduced the size of HABpanel by replace two customized icons with standard icons. I still think all my issues are related to memory problems with HABpanel. Before all my problems started I was editing the panel.

2020-11-03 09:43:04.685 [ERROR] [ore.thing.internal.ThingRegistryImpl] - Could not inform the ThingTracker 'org.eclipse.smarthome.core.thing.internal.ThingManagerImpl@17eaf76' about the 'THING_UPDATED' event!

java.lang.reflect.UndeclaredThrowableException: null

	at com.sun.proxy.$Proxy309.thingUpdated(Unknown Source) ~[?:?]

	at org.eclipse.smarthome.core.thing.internal.ThingManagerImpl.thingUpdated(ThingManagerImpl.java:498) ~[bundleFile:?]

	at org.eclipse.smarthome.core.thing.internal.ThingRegistryImpl.notifyTrackers(ThingRegistryImpl.java:220) [bundleFile:?]

	at org.eclipse.smarthome.core.thing.internal.ThingRegistryImpl.notifyListenersAboutUpdatedElement(ThingRegistryImpl.java:144) [bundleFile:?]

	at org.eclipse.smarthome.core.thing.internal.ThingRegistryImpl.notifyListenersAboutUpdatedElement(ThingRegistryImpl.java:1) [bundleFile:?]

	at org.eclipse.smarthome.core.common.registry.AbstractRegistry.updated(AbstractRegistry.java:291) [bundleFile:?]

	at org.eclipse.smarthome.core.common.registry.AbstractRegistry.updated(AbstractRegistry.java:1) [bundleFile:?]

	at org.eclipse.smarthome.core.common.registry.AbstractProvider.notifyListeners(AbstractProvider.java:62) [bundleFile:?]

	at org.eclipse.smarthome.core.common.registry.AbstractProvider.notifyListenersAboutUpdatedElement(AbstractProvider.java:87) [bundleFile:?]

	at org.eclipse.smarthome.core.common.registry.AbstractManagedProvider.update(AbstractManagedProvider.java:119) [bundleFile:?]

	at org.eclipse.smarthome.core.thing.internal.ThingManagerImpl$1$1.run(ThingManagerImpl.java:261) [bundleFile:?]

	at org.eclipse.smarthome.core.thing.internal.ThingManagerImpl$1$1.run(ThingManagerImpl.java:1) [bundleFile:?]

	at java.security.AccessController.doPrivileged(Native Method) [?:1.8.0_265]

	at org.eclipse.smarthome.core.thing.internal.ThingManagerImpl$1.thingUpdated(ThingManagerImpl.java:249) [bundleFile:?]

	at org.eclipse.smarthome.core.thing.binding.BaseThingHandler.updateProperty(BaseThingHandler.java:517) [bundleFile:?]

	at org.openhab.binding.milight.internal.handler.BridgeV6Handler.sessionStateChanged(BridgeV6Handler.java:104) [bundleFile:?]

	at org.openhab.binding.milight.internal.protocol.MilightV6SessionManager.sessionStateMachine(MilightV6SessionManager.java:500) [bundleFile:?]

	at org.openhab.binding.milight.internal.protocol.MilightV6SessionManager.run(MilightV6SessionManager.java:667) [bundleFile:?]

	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_265]

Caused by: java.lang.InterruptedException

	at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404) ~[?:1.8.0_265]

	at java.util.concurrent.FutureTask.get(FutureTask.java:204) ~[?:1.8.0_265]

	at org.eclipse.smarthome.core.internal.common.InvocationHandlerSync.invoke(InvocationHandlerSync.java:66) ~[?:?]

	... 19 more

Could it be physical faulty memory? (ie hardware failure).

To make a long story short: I solved ALL my different issues by follow one main strategy: Ensure, never run out of memory or let it come to a critical level of less free memory. I actually reached that aim by change couple of things.

1. Switched to the latest hardware

  • upgraded from Raspberry 3b to Raspberry 4b 4 Gbyte.
  • enlarged JAVA heap memory plus 1 Gbyte: EXTRA_JAVA_OPTS="-Xms1024m -Xmx1024m. So now 1.4 Gbyte of memory are stable allways in use and not growing anymore. My HABpanel needs a lot of memory. I run the UI on 3 different hardware platforms at the same time with two different HABpanel configurations. Also ZWAVE produced before on my RSP3 a lot of checksum errors. None checksum errors anymore on the new device! Most of issues were eliminated as I had switched to RSP4.

2. Fixed if possible open bugs and warnings first

I fixed if possible bugs in my source code, even so “harmless” bugs e.g. NULL based errors. They eating all the day a bit of the free memory! As well I improved my spaghetti coded rules:
I replaced most of my do-while loops and a lot of sleep threads by using the timer function.

3. Restarted with a clean up-to-date well configured software package

I started from scratch the system with the latest latest openhabian package. I decided to stay on OH2 for the moment, so I used the workaround of the downgrade option OH3 to OH2 from openhabians menu. The full automated installation script of openhabian - if I am right - installs fixed the latest OH3. But no problem, at the openhabian menu you can find all needed progs and tools to customize your system finally, if needed. E.g. support to transit the root to a HDD, the way I prefered. Last not least restored my backuped OH2 origin data.

So I am very happy to be able run finally a stable & super fast openhab solution. Thanks to all for your support!

1 Like