ONVIF camera occures Java heap space exceptions

Sofa1780 · April 7, 2021, 10:35am

Hi, all together,

I am fairly new to camera binding and trying to get the binding running faultlessly in my OpenHAB environment.
Regardings the documentation I turned the trace logs on as well as searching for a related topic here.
Unfortunately, I could not find any solution, yet.

Currently, I am running an OpenHAB Server on a Windows 10 WSL2 Docker image.
Everything is working well until I add at least one thing related to the IpCamear binding.

All bound cameras are configured the same way, as described bellow:

Thing ipcamera:onvif:BackDoor "Camera back door" @ "backyard" [
    ipAddress="192.168.178.106",
    username="username",
    password="password",
    onvifPort="8899",
    port="80",
    updateImageWhen="1",
    pollTime="60000",
    serverPort="5503",
    mjpegUrl="ffmpeg",
    mjpegOptions="-c:v libx265 -q:v 3 -vf scale=640:-2",
    ffmpegLocation="/usr/bin/ffmpeg",
    ffmpegOutput="/openhab/html/cameras/cam_backdoor/",
    ffmpegInput="rtsp://192.168.178.106:554/user=username_password=password_channel=1_stream=0.sdp?real_stream",
    snapshotUrl="http://192.168.178.106:80/webcapture.jpg?command=snap&amp;channel=1&amp;user=username&amp;password=password"
]

The items are mapped accordingly:

Switch  Cam_BackDoor_PollImage             "Camera back door - Poll Image" <camera>  {channel="ipcamera:onvif:BackDoor:pollImage"}

As well as the link within the sitemap:

Text label="Camera back door" icon="camera" {
            Video url="http://127.0.0.1:5503/snapshots.mjpeg" encoding="mjpeg"
            Switch item=Cam_BackDoor_PollImage label="Refrash image" mappings=[ON="Refresh"]
        }

With this configuration, I noticed, that the cameras are refreshed/polled several times during the day due to a communication error within the events.log:

2021-03-31 01:38:47.082 [INFO ] [ab.event.ThingStatusInfoChangedEvent] - Thing 'ipcamera:onvif:BackDoor' changed from ONLINE to OFFLINE (COMMUNICATION_ERROR): Connection Timeout: Check your IP and PORT are correct and the camera can be reached.
2021-03-31 01:39:54.128 [INFO ] [ab.event.ThingStatusInfoChangedEvent] - Thing 'ipcamera:onvif:BackDoor' changed from OFFLINE (COMMUNICATION_ERROR): Connection Timeout: Check your IP and PORT are correct and the camera can be reached. to ONLINE

2021-03-31 01:41:59.642 [INFO ] [ab.event.ThingStatusInfoChangedEvent] - Thing 'ipcamera:onvif:BackDoor' changed from ONLINE to OFFLINE (COMMUNICATION_ERROR): Connection Timeout: Check your IP and PORT are correct and the camera can be reached.
2021-03-31 01:42:03.161 [INFO ] [ab.event.ThingStatusInfoChangedEvent] - Thing 'ipcamera:onvif:BackDoor' changed from OFFLINE (COMMUNICATION_ERROR): Connection Timeout: Check your IP and PORT are correct and the camera can be reached. to ONLINE

2021-03-31 01:56:41.846 [INFO ] [ab.event.ThingStatusInfoChangedEvent] - Thing 'ipcamera:onvif:BackDoor' changed from ONLINE to OFFLINE (COMMUNICATION_ERROR): Connection Timeout: Check your IP and PORT are correct and the camera can be reached.
2021-03-31 01:56:50.760 [INFO ] [ab.event.ThingStatusInfoChangedEvent] - Thing 'ipcamera:onvif:BackDoor' changed from OFFLINE (COMMUNICATION_ERROR): Connection Timeout: Check your IP and PORT are correct and the camera can be reached. to ONLINE

2021-03-31 02:40:56.268 [INFO ] [ab.event.ThingStatusInfoChangedEvent] - Thing 'ipcamera:onvif:BackDoor' changed from ONLINE to OFFLINE (COMMUNICATION_ERROR): Connection Timeout: Check your IP and PORT are correct and the camera can be reached.
2021-03-31 02:41:03.833 [INFO ] [ab.event.ThingStatusInfoChangedEvent] - Thing 'ipcamera:onvif:BackDoor' changed from OFFLINE (COMMUNICATION_ERROR): Connection Timeout: Check your IP and PORT are correct and the camera can be reached. to ONLINE

Finally, the system breaks after about 20 to 24 hours with the following exception within the openhab.log:

2021-03-31 03:16:47.395 [WARN ] [io.netty.channel.AbstractChannel    ] - Force-closing a channel whose registration task was not accepted by an event loop: [id: 0x027dc1f5]
java.util.concurrent.RejectedExecutionException: event executor terminated
	at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:981) ~[bundleFile:4.1.42.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:388) ~[bundleFile:4.1.42.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:381) ~[bundleFile:4.1.42.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:880) ~[bundleFile:4.1.42.Final]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:472) [bundleFile:4.1.42.Final]
	at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:87) [bundleFile:4.1.42.Final]
	at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:81) [bundleFile:4.1.42.Final]
	at io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:86) [bundleFile:4.1.42.Final]
	at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:311) [bundleFile:4.1.42.Final]
	at io.netty.bootstrap.Bootstrap.doResolveAndConnect(Bootstrap.java:157) [bundleFile:4.1.42.Final]
	at io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:141) [bundleFile:4.1.42.Final]
	at org.openhab.binding.ipcamera.internal.onvif.OnvifConnection.sendOnvifRequest(OnvifConnection.java:518) [bundleFile:?]
	at org.openhab.binding.ipcamera.internal.onvif.OnvifConnection.connect(OnvifConnection.java:818) [bundleFile:?]
	at org.openhab.binding.ipcamera.internal.handler.IpCameraHandler.pollCameraRunnable(IpCameraHandler.java:1572) [bundleFile:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
2021-03-31 03:16:47.397 [ERROR] [ent.DefaultPromise.rejectedExecution] - Failed to submit a listener notification task. Event loop shut down?
java.util.concurrent.RejectedExecutionException: event executor terminated
	at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:981) ~[bundleFile:4.1.42.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:388) ~[bundleFile:4.1.42.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:381) ~[bundleFile:4.1.42.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:880) ~[bundleFile:4.1.42.Final]
	at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:841) [bundleFile:4.1.42.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:498) [bundleFile:4.1.42.Final]
	at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:183) [bundleFile:4.1.42.Final]
	at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:95) [bundleFile:4.1.42.Final]
	at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:30) [bundleFile:4.1.42.Final]
	at org.openhab.binding.ipcamera.internal.onvif.OnvifConnection.sendOnvifRequest(OnvifConnection.java:518) [bundleFile:?]
	at org.openhab.binding.ipcamera.internal.onvif.OnvifConnection.connect(OnvifConnection.java:818) [bundleFile:?]
	at org.openhab.binding.ipcamera.internal.handler.IpCameraHandler.pollCameraRunnable(IpCameraHandler.java:1572) [bundleFile:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]

This warnign and error log occures until the system is down with the las logs:

2021-03-31 03:28:12.188 [WARN ] [concurrent.SingleThreadEventExecutor] - An event executor terminated with non-empty task queue (9)
2021-03-31 03:28:12.188 [WARN ] [concurrent.SingleThreadEventExecutor] - An event executor terminated with non-empty task queue (10)
2021-03-31 03:28:12.190 [WARN ] [concurrent.SingleThreadEventExecutor] - An event executor terminated with non-empty task queue (11)
2021-03-31 03:28:12.193 [WARN ] [concurrent.SingleThreadEventExecutor] - Unexpected exception from an event executor: 
java.lang.OutOfMemoryError: Java heap space
2021-03-31 03:28:12.195 [WARN ] [til.concurrent.AbstractEventExecutor] - A task raised an exception. Task: ScheduledFutureTask@4b685a0d(uncancellable, task: Callable(task: io.netty.handler.timeout.IdleStateHandler$AllIdleTimeoutTask@75ba9067, result: null), deadline: 64216832269600, period: 0)
java.lang.OutOfMemoryError: Java heap space
2021-03-31 03:28:12.193 [WARN ] [concurrent.SingleThreadEventExecutor] - Unexpected exception from an event executor: 
java.lang.OutOfMemoryError: Java heap space
2021-03-31 03:28:12.196 [WARN ] [e.jetty.util.thread.QueuedThreadPool] - 
java.lang.OutOfMemoryError: Java heap space
2021-03-31 03:28:12.197 [WARN ] [concurrent.SingleThreadEventExecutor] - Unexpected exception from an event executor: 
java.lang.OutOfMemoryError: Java heap space
2021-03-31 03:28:12.204 [WARN ] [concurrent.SingleThreadEventExecutor] - An event executor terminated with non-empty task queue (40)
2021-03-31 03:28:12.205 [WARN ] [io.netty.channel.nio.NioEventLoop   ] - Unexpected exception in the selector loop.
java.lang.OutOfMemoryError: Java heap space
2021-03-31 03:28:12.206 [WARN ] [concurrent.SingleThreadEventExecutor] - An event executor terminated with non-empty task queue (1)
2021-03-31 03:28:12.206 [WARN ] [concurrent.SingleThreadEventExecutor] - An event executor terminated with non-empty task queue (7)
2021-03-31 03:28:12.320 [WARN ] [e.jetty.util.thread.QueuedThreadPool] - 
java.lang.OutOfMemoryError: Java heap space
2021-03-31 03:28:12.322 [WARN ] [e.jetty.util.thread.QueuedThreadPool] - 
java.lang.OutOfMemoryError: Java heap space
2021-03-31 03:28:12.325 [WARN ] [e.jetty.util.thread.QueuedThreadPool] - 
java.lang.OutOfMemoryError: Java heap space
2021-03-31 03:28:12.330 [WARN ] [concurrent.SingleThreadEventExecutor] - An event executor terminated with non-empty task queue (11)
2021-03-31 03:28:12.330 [WARN ] [e.jetty.util.thread.QueuedThreadPool] - 
java.lang.OutOfMemoryError: Java heap space

So with this knowledge, I tried to find similar issues and started reading the documentation a second and a third, without any progress.

Might you know why the errors occur or do you have any hints or suggestions on how I can solve the problem?

Best regards
Stephan

Integer · August 9, 2022, 6:58am

Hey @Sofa1780

I’m on OH 3.3 and I have exactly the same error every 8 sec. Did you solved this problem?
Man thanks for your help

matt1 · August 23, 2022, 12:37am

That will be because every 8 seconds the camera binding tries to connect to the cameras but fails as the system has no more space in the heap, ie OUT OF MEMORY ERROR.

This does not mean it is the ipcamera binding causing the memory/heap to be full, it may be something else that has a memory leak and the ipcamera binding is just a high user of memory and will complain when there is none left, the way to fix is to discover the binding or rule that is causing the memory to leak. You should be able to graph and view it happening using the system info binding and the ‘used heap percentage’ channel.