OpenHAB crashing every few days

Setup, Configuration and Use Runtime
1

OpenHAB keeps crashing every few days (often overnight).
I get these errors in the docker logs (so not openhab.log).

I see stuff about memory and ipcamera, but I’m not sure what to do about it. Can anyone help?

Running OpenHAB Latest in Docker container on Pi4

Caused by: Attached Guest Language Frames (6)
java.lang.OutOfMemoryError: Java heap space
Caused by: Attached Guest Language Frames (6)
java.lang.OutOfMemoryError: Java heap space
Caused by: Attached Guest Language Frames (6)
java.lang.OutOfMemoryError: Java heap space
Caused by: Attached Guest Language Frames (6)
java.lang.OutOfMemoryError: Java heap space
Caused by: Attached Guest Language Frames (6)
java.lang.OutOfMemoryError: Java heap space
Caused by: Attached Guest Language Frames (6)
java.lang.OutOfMemoryError: Java heap space
Caused by: Attached Guest Language Frames (6)
java.lang.OutOfMemoryError: Java heap space
Caused by: Attached Guest Language Frames (6)
java.lang.OutOfMemoryError: Java heap space
Caused by: Attached Guest Language Frames (6)
java.lang.OutOfMemoryError: Java heap space
Caused by: Attached Guest Language Frames (6)
java.lang.OutOfMemoryError: Java heap space
Caused by: Attached Guest Language Frames (6)
java.lang.OutOfMemoryError: Java heap space
Caused by: Attached Guest Language Frames (6)
java.lang.OutOfMemoryError: Java heap space
Caused by: Attached Guest Language Frames (6)
java.lang.OutOfMemoryError: Java heap space
Caused by: Attached Guest Language Frames (6)
java.lang.OutOfMemoryError: Java heap space
Caused by: Attached Guest Language Frames (6)
java.lang.OutOfMemoryError: Java heap space
Caused by: Attached Guest Language Frames (6)
java.lang.OutOfMemoryError: Java heap space
Caused by: Attached Guest Language Frames (6)
io.reactivex.exceptions.UndeliverableException: The exception could not be delivered to the consumer because it has already canceled/disposed the flow or the exception has nowhere to go to begin with. Further reading: https://github.com/ReactiveX/RxJava/wiki/What's-different-in-2.0#error-handling | java.lang.OutOfMemoryError: Java heap space
at io.reactivex.plugins.RxJavaPlugins.onError(RxJavaPlugins.java:367)
at io.reactivex.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:69)
at io.reactivex.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:57)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.OutOfMemoryError: Java heap space
Caused by: Attached Guest Language Frames (6)
SLF4J: Failed toString() invocation on an object of type [io.netty.channel.socket.nio.NioSocketChannel]

Reported exception:
java.lang.NoClassDefFoundError: Could not initialize class io.netty.buffer.ByteBufUtil$HexUtil
Exception in thread "OH-scheduler-759" at io.netty.buffer.ByteBufUtil.hexDump(ByteBufUtil.java:161)
at io.netty.channel.DefaultChannelId.asShortText(DefaultChannelId.java:240)
at io.netty.channel.AbstractChannel.toString(AbstractChannel.java:412)
at org.slf4j.helpers.MessageFormatter.safeObjectAppend(MessageFormatter.java:291)
at org.slf4j.helpers.MessageFormatter.deeplyAppendParameter(MessageFormatter.java:263)
at org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:225)
at org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:160)
at org.slf4j.helpers.MessageFormatter.format(MessageFormatter.java:151)
at io.netty.util.internal.logging.LocationAwareSlf4JLogger.warn(LocationAwareSlf4JLogger.java:202)
at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:490)
at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:89)
at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:83)
at io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:86)
at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:339)
at io.netty.bootstrap.Bootstrap.doResolveAndConnect(Bootstrap.java:164)
at io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:148)
at org.openhab.binding.ipcamera.internal.onvif.OnvifConnection.sendOnvifRequest(OnvifConnection.java:580)
at org.openhab.binding.ipcamera.internal.onvif.OnvifConnection.connect(OnvifConnection.java:947)
at org.openhab.binding.ipcamera.internal.handler.IpCameraHandler.tryConnecting(IpCameraHandler.java:1770)
at org.openhab.binding.ipcamera.internal.handler.IpCameraHandler.resetAndRetryConnecting(IpCameraHandler.java:1800)
at org.openhab.binding.ipcamera.internal.handler.IpCameraHandler.cameraCommunicationError(IpCameraHandler.java:1455)
at org.openhab.binding.ipcamera.internal.handler.IpCameraHandler$2.operationComplete(IpCameraHandler.java:670)
at org.openhab.binding.ipcamera.internal.handler.IpCameraHandler$2.operationComplete(IpCameraHandler.java:1)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629)
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:262)
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:153)
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.OutOfMemoryError: Java heap space [in thread "nioEventLoopGroup-19-3"]
java.lang.OutOfMemoryError: Java heap space
Caused by: Attached Guest Language Frames (6)
Exception in thread "SocketListener(172-19-0-1.local.)" java.lang.IllegalStateException: Timer already cancelled.
at java.base/java.util.Timer.sched(Timer.java:409)
at java.base/java.util.Timer.schedule(Timer.java:205)
at javax.jmdns.impl.DNSTaskStarter$DNSTaskStarterImpl$StarterTimer.schedule(DNSTaskStarter.java:219)
at javax.jmdns.impl.tasks.Responder.start(Responder.java:98)
at javax.jmdns.impl.DNSTaskStarter$DNSTaskStarterImpl.startResponder(DNSTaskStarter.java:395)
at javax.jmdns.impl.JmDNSImpl.startResponder(JmDNSImpl.java:1784)
at javax.jmdns.impl.JmDNSImpl.handleQuery(JmDNSImpl.java:1574)
at javax.jmdns.impl.SocketListener.run(SocketListener.java:75)
2

I’m also getting this error in my openhab.log over and over again:
Seems related.

[WARN ] [io.netty.channel.DefaultChannelPipeline] - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
java.lang.NullPointerException: Cannot read field "dog_cat" because "aiResponse[0].value" is null
	at org.openhab.binding.ipcamera.internal.ReolinkHandler.channelRead(ReolinkHandler.java:123) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [bundleFile:4.1.99.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [bundleFile:4.1.99.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [bundleFile:4.1.99.Final]
	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) [bundleFile:4.1.99.Final]
	at org.openhab.binding.ipcamera.internal.handler.IpCameraHandler$CommonCameraHandler.channelRead(IpCameraHandler.java:295) [bundleFile:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [bundleFile:4.1.99.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [bundleFile:4.1.99.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [bundleFile:4.1.99.Final]
	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) [bundleFile:4.1.99.Final]
	at org.openhab.binding.ipcamera.internal.MyNettyAuthHandler.channelRead(MyNettyAuthHandler.java:182) [bundleFile:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [bundleFile:4.1.99.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [bundleFile:4.1.99.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [bundleFile:4.1.99.Final]
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) [bundleFile:4.1.99.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) [bundleFile:4.1.99.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) [bundleFile:4.1.99.Final]
	at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) [bundleFile:4.1.99.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [bundleFile:4.1.99.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [bundleFile:4.1.99.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [bundleFile:4.1.99.Final]
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [bundleFile:4.1.99.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [bundleFile:4.1.99.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [bundleFile:4.1.99.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [bundleFile:4.1.99.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [bundleFile:4.1.99.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [bundleFile:4.1.99.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [bundleFile:4.1.99.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [bundleFile:4.1.99.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [bundleFile:4.1.99.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [bundleFile:4.1.99.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) [bundleFile:4.1.99.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) [bundleFile:4.1.99.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [bundleFile:4.1.99.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [bundleFile:4.1.99.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [bundleFile:4.1.99.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [bundleFile:4.1.99.Final]
	at java.lang.Thread.run(Thread.java:840) [?:?]
3

Out of curiosity: How does the memory usage look like?

When I had severe freezing problems (OH on docket on RPi 3) it was because a lack of memory / extensive need for swapping.

4

Not sure what the memory is like during the crashing.

I’ve tracked it (or something) down to one camera thing.
It’s strange. It’s been working for a few weeks, now this. I have 3 poe doorbells, and 4 poe bullet cams all the same. It’s only channel 7 that starts throwing this error, if I change it, the error stops. Camera is working fine on the Reolink app.

Still not sure if this is what is causing the random crashing, or if it is just another problem at the same time.

5

Oh version?
Which scripting language used?
How much ram do you have?
Java Xmx setting?

6

OH 4.1.2 (I had been trying the milestone too, but it was happening then too)
Javascript
4GB Pi4

Don’t know what Java Xmx is

7

did you install openhab using openhabian, or apt, or manually?

What caused this is a memory leak / keeps growing, and your java maximum heap memory limit is probably set too low.

The memory leak might be caused by the camera thing (ipcamera binding?)

8

Installed as Docker container

I’ve disabled the one camera channel that was throwing errors and nothing has crashed so far.

9

Can you post config details about this problematic channel?

10 
Thing ipcamera:reolink:lane "Camera - Lane"
    [
    mjpegOptions= "-q:v 5 -r 2 -vf scale=640:-2 -update 1",
    ipAddress= "10.0.6.2",
    updateImageWhen= "0",
    gifPreroll= "0",
    onvifPort= "8000",
    ffmpegLocation= "/usr/bin/ffmpeg",
    ipWhitelist= "DISABLE",
    mp4OutOptions= "-c:v copy -c:a copy",
    pollTime= "1000",
    useToken= "false",
    password= "xxxxxx",
    port= "80",
    nvrChannel= "7",
    snapshotOptions= "-an -vsync vfr -q:v 2 -update 1",
    onvifMediaProfile= "1",
    gifOutOptions= "-r 2 -filter_complex
        scale=-2:360:flags=lanczos,setpts=0.5*PTS,split[o1][o2];[o1]palettegen[p];[o2]fifo[o3];[o3][p]paletteuse",
    hlsOutOptions= "-strict -2 -f lavfi -i aevalsrc=0 -acodec aac -vcodec copy
        -hls_flags delete_segments -hls_time 2 -hls_list_size 4",
    username= "admin"
    ]

It is nothing special. Pretty much all defaults (Reolink ipcamera). Just IP address and channel number are different between cameras.

Here is one that works, just a copy paste

Thing ipcamera:reolink:northcarpark "Camera - North Carpark"
    [
    mjpegOptions= "-q:v 5 -r 2 -vf scale=640:-2 -update 1",
    ipAddress= "10.0.6.2",
    updateImageWhen= "0",
    gifPreroll= "0",
    onvifPort= "8000",
    ffmpegLocation= "/usr/bin/ffmpeg",
    ipWhitelist= "DISABLE",
    mp4OutOptions= "-c:v copy -c:a copy",
    pollTime= "1000",
    useToken= "false",
    password= "xxxxxx",
    port= "80",
    nvrChannel= "6",
    snapshotOptions= "-an -vsync vfr -q:v 2 -update 1",
    onvifMediaProfile= "1",
    gifOutOptions= "-r 2 -filter_complex
        scale=-2:360:flags=lanczos,setpts=0.5*PTS,split[o1][o2];[o1]palettegen[p];[o2]fifo[o3];[o3][p]paletteuse",
    hlsOutOptions= "-strict -2 -f lavfi -i aevalsrc=0 -acodec aac -vcodec copy
        -hls_flags delete_segments -hls_time 2 -hls_list_size 4",
    username= "admin"
    ]
11

In your docker, add this environment variable (or if you already have it, add the argument to it)

    environment:
      EXTRA_JAVA_OPTS: "-Xmx2048m"
12

The dog_cat error is from a reolink camera, but it was fixed. Adding Reolink API to the IpCamera binding, beta testers needed - Add-ons / Bindings - openHAB Community Maybe you have something in the cache for that one camera. Maybe a comparison will help

13

Could it be from bouncing around between milestone, snapshot, and latest while trying to solve my other problem?

14

I will try this, but what does it do?

15 
   -Xmx<size>        set maximum Java heap size

If you hadn’t set it, it would probably be at 1GB

16

If this was for me, I’m sorry, but do not know. I just recognized the cat_dog error from when I was testing the Reolink IP camera binding back last May and thought it might help somehow. Maybe you can spot a difference on the Device UI code tab between the working and non-working cameras.

Edit: Another thought might be to delete the non-working camera and then Scan to pick it up again (hopefully without the error)

17

And this makes it 2GB? Why not make it 3GB? OpenHAB and InfluxDB will be the only apps ever run on this Pi

18

Thanks.

I wasn’t able to pick up my cameras from scanning. I had to create a thing and put the IP and NVR channel info in manually

19

you could try 3GB too and see which setting works.

20

I don’t know if I’d go to 3Gbs, 2 should work. I’d want to leave some for the host running docker. Also, if you have a memory leak, enlarging the max heap size will not cure the memory leak, it will just take longer before it crashes.