Tonight my entire OH crashed completely, loads of outofmemoryexceptions. So now I have restarted the entire server and started OH again, but after the restart all my Things are gone, I just get a list with “No Things yet”. Tried a second restart of OH, but the same thing happens. I don’t even really know where to start looking. Ideas?
This is OH 3.0 running on Armbian on a 64 bits arm server with 4 GB of RAM. I find nothing relevant in the logs.
Some of the Things that should normally be in the list is instead in the inbox, like for example all my Sonos speakers.
Would it be a good idea trying to recover from backup? I do a daily backup of entire OH…
Don’t panic! It sounds like your jsondb was corrupted.
If you have a daily backup, I would just use that - your jsondb also gets backed up regularly.
If you do a forum search for “ally things are gone” you will find a thread that deals exactly with your problem.
Oh. Seems something is blocked in the brain when panicking… Actually turned out my org.openhab.core.thing.Thing.json was 0 bytes. Restored it according to the instructions in the thread you linked to, restarted OH and they’re back!
I don’t really like this though. Second time in one month that OH completely crashed out with outofmemoryexceptions. I’ve used OH for many years before without any problems like this, seems to me there’s some memory leak introduced in 3.0…
What bindings are you running, and is there anything else on your server? Many people have been running OH3 for months without similar problems, so I’d think that it’s something specific to your system that’s causing the memory leak, and not openHAB core.
No, nothing has changed for ages, I have the same Bindings I’ve had for a very long time.
One exception is that I changed from jdbc-mysql persistence to jdbc-mariadb some months ago, but I have a hard time believing that would make any huge difference.
I did some log digging. The OH crash started out with this (followed by loads of other stuff, most of it containing “java.lang.OutOfMemoryError: Java heap space”). Don’t know if anyone can draw any conclusions from it?
2021-01-24 01:18:07.153 [WARN ] [clipse.jetty.server.HttpChannelState] - java.lang.OutOfMemoryError: Java heap space while invoking onComplete listener 1118093402
2021-01-24 01:18:07.153 [WARN ] [org.eclipse.jetty.io.ManagedSelector] - java.lang.OutOfMemoryError: Java heap space
2021-01-24 01:18:18.702 [WARN ] [com.zaxxer.hikari.pool.HikariPool ] - 45s325ms450μs481ns - Thread starvation or clock leap detected (housekeeper delta=yank-default).
2021-01-24 01:18:12.993 [WARN ] [mmon.WrappedScheduledExecutorService] - Scheduled runnable ended with an exception:
java.lang.OutOfMemoryError: Java heap space
2021-01-24 01:18:18.706 [ERROR] [org.apache.felix.fileinstall ] - In main loop, we have serious trouble
java.lang.OutOfMemoryError: Java heap space
2021-01-24 01:18:07.152 [WARN ] [e.jetty.util.thread.QueuedThreadPool] -
java.lang.OutOfMemoryError: Java heap space
2021-01-24 01:18:07.152 [WARN ] [e.jetty.util.thread.QueuedThreadPool] -
java.lang.OutOfMemoryError: Java heap space
2021-01-24 01:18:12.995 [WARN ] [mmon.WrappedScheduledExecutorService] - Scheduled runnable ended with an exception:
java.lang.OutOfMemoryError: Java heap space
Other relevant info:
My EXTRA_JAVA_OPTS is set to an empty string.
I’m running 64 bit Zulu 11.0.9.1.
My list of installed bindings: “exec,sonos,robonect,zwave,tellstick,mail,tibber,pushover”
Unfortunately exactly when and where the OutOfMemoryError occurs is not all that informative. It tells you where openHAB was when it ran out of memory but tells you nothing of the source of the memory leak.
I’ve seen sporadic reports of memory and CPU issues that seem mostly related to rules. I don’t know if there is a conclusion yet as to what might be the cause, whether it is actually associated with Rules DSL, and the such but there is at least one issue open I think on the problem.
As for the 0 bytes things JSONDB, if OH happens to crash while writing to the file the file will become corrupted. Hopefully they implement this correctly though and don’t start writing until the backup has completed so you will likely have lost nothing in your restore.
