Memory Leak .. OH 3.1 M3

Hey I am currently on M3 Release and see that my Zwave Controller goes offline after roughly a week.
Zwave might be only a symptom I know and it might related to something else.
But its my only direction I have since other controllers etc. still work.
I had no specific logging turned on.
But at the point of time when the controller goes offline I see OOM exceptions:

2021-05-19 08:46:17.028 [WARN ] [mmon.WrappedScheduledExecutorService] - Scheduled runnable ended with an exception: 
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
	at java.lang.Thread.start0(Native Method) ~[?:?]
	at java.lang.Thread.start(Thread.java:803) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:937) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1583) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor.reExecutePeriodic(ScheduledThreadPoolExecutor.java:360) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:307) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
	at java.lang.Thread.start0(Native Method) ~[?:?]
	at java.lang.Thread.start(Thread.java:803) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:937) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1583) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor.reExecutePeriodic(ScheduledThreadPoolExecutor.java:360) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:307) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
	2021-05-19 08:50:06.678 [WARN ] [mmon.WrappedScheduledExecutorService] - Scheduled runnable ended with an exception: 
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
	at java.lang.Thread.start0(Native Method) ~[?:?]
	at java.lang.Thread.start(Thread.java:803) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:937) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1583) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:346) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:562) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor.execute(ScheduledThreadPoolExecutor.java:705) ~[?:?]
	at org.openhab.core.thing.internal.ThingManagerImpl.notifyThingsAboutBridgeStatusChange(ThingManagerImpl.java:954) ~[?:?]
	at org.openhab.core.thing.internal.ThingManagerImpl$1.handleBridgeStatusUpdate(ThingManagerImpl.java:241) ~[?:?]
	at org.openhab.core.thing.internal.ThingManagerImpl$1.statusUpdated(ThingManagerImpl.java:207) ~[?:?]
	at org.openhab.core.thing.binding.BaseThingHandler.updateStatus(BaseThingHandler.java:343) ~[?:?]
	at org.openhab.core.thing.binding.BaseThingHandler.updateStatus(BaseThingHandler.java:367) ~[?:?]
	at org.openhab.binding.somfytahoma.internal.handler.SomfyTahomaBridgeHandler.login(SomfyTahomaBridgeHandler.java:225) ~[?:?]
	at org.openhab.binding.somfytahoma.internal.handler.SomfyTahomaBridgeHandler.reLogin(SomfyTahomaBridgeHandler.java:726) ~[?:?]
	at org.openhab.binding.somfytahoma.internal.handler.SomfyTahomaBridgeHandler.getTahomaUpdates(SomfyTahomaBridgeHandler.java:389) ~[?:?]
	at org.openhab.binding.somfytahoma.internal.handler.SomfyTahomaBridgeHandler.lambda$2(SomfyTahomaBridgeHandler.java:170) ~[?:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]

That machine has 16GB Ram and about 5GB are used now. Not sure whats going on.

Any help appreciated.

Ooms are usually hard to find. Thread dump when you get it is a good start. If possible you could use a flight recorder to look at what objects are eating the memory.

I Recommend.looking at this thread for some inputs in how to debug.

Use jstack to thread dump.

Flight record (I have not tried this on openhab but on many other java applications)

@Seaside thank you! I digged more into the topic as I see OH3 running out of memory roughly once a week.I reviewed the mentioned forum posts already (not the other tools yet)

Maybe I can already find something with the more basic stuff I learnt on that forum posts:

- When I run threads --list:
~ I see thousands of:


couldn’t find what nioEventLoopGroup is actually for

~ Overall is there a info of how many threads etc. would be plausible on a healthy system?
Threads
Live threads 2942
Daemon threads 244
Peak 2961
Total started 87956

shell:info shows my Heap Size is ~4Gb
~ As I have 16GB Ram and a lot of that unused does it make sense to increase the Heap Size?
Memory (24 hours uptime at this point)
Current heap size 1,211,449 kbytes
Maximum heap size 4,069,376 kbytes
Committed heap size 1,396,736 kbytes
Pending objects 0
Garbage collector Name = ‘G1 Young Generation’, Collections = 1248, Time = 35.740 seconds
Garbage collector Name = ‘G1 Old Generation’, Collections = 0, Time = 0.000 seconds

thank you :slight_smile:

You would want to compare that statistics when you run oom. To me it looks ok, but hard to tell.

If you have a memory leak, using 16gb will make it take longer time before you run oom, but you will still run oom eventually. When java garbage collects, if you have a leak, will not be able to clear some object references each time gc is performed. Thus building up, with time, a lot of objects that can’t be gced. When java starts getting less and less memory it will continue to gc more frequently which will cause the cpu to spike, which is why you can expect to have the cpu at 100 percent towards the end, and openhab being slow in response.

mhh wanted to do that but I can’t even execute those commands :frowning:

Error in initialization script: /var/lib/openhab/etc/shell.init.script: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
Error in initialization script: /var/lib/openhab/etc/scripts/shell.colors.script: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
Error in initialization script: /var/lib/openhab/etc/scripts/shell.completion.script: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
openhab> shell:threads --list
Error executing command: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
openhab> shell:threads --list
Error executing command: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
openhab> shell:info
Error executing command: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
openhab>

I could still access the Main UI.
I deactivated 1 Thing; 1 of the IPCameras as I thought it is related to the IPCamera Binding and instantly the system became stable and accessible. But the ZWave Controller stays Offline.

After 1 Cam Thing deactivated:

Memory
  Current heap size           1,541,544 kbytes
  Maximum heap size           4,069,376 kbytes
  Committed heap size         1,683,456 kbytes
  Pending objects             0

Since last statics above … around 200k threads were startet.
Almost all of them that nioEventLoopGroup stuff

Threads
  Live threads                4828
  Daemon threads              260
  Peak                        4897
  Total started               204387

196230 │ nioEventLoopGroup-2196-1 │ RUNNABLE │ 847 │ 300

If you can’t access the console, better to do a thread dump. See my links.

The approach I would take here it’s to remove rules and bindings until you have a stable system. That’s of course easier said than done. If you suspect it has to do with groups remove the rules that are looping over groups or posting data to groups.

/s

with the Things from the IPCameraBinding disabled my stats look stable.
The current heap size even gets lower

Memory
  Current heap size           961,354 kbytes
  Maximum heap size           4,069,376 kbytes
  Committed heap size         1,381,376 kbytes
  Pending objects             0
  Garbage collector           Name = 'G1 Young Generation', Collections = 261, Time = 12.337 seconds
  Garbage collector           Name = 'G1 Old Generation', Collections = 0, Time = 0.000 seconds

Well yes, you might expect image handling to consume resource, and not doing it to free some up. What is important is to monitor over time and see if its still creeping up, albeit with more room now.

Without a IpCam Thing I am below 1gb heap now.
When I enable that Thing it consumes the available 4gb max in 7 days. Worst Case was 2 days.

What do you have the updateImageWhen: setting set to? Are any of your cameras not have this set to 0? Can you post the CODE tab output when you click on a camera thing? This gives a summery of what settings you have selected, just remember to remove your password before posting.

Thank you
the only parameter I think I actively changed is the Poll Time. all camers are same parameters (besides IP and serverport):

UID: ipcamera:dahua:192168184
label: dahua camera @192.168.1.110 name
thingTypeUID: ipcamera:dahua
configuration:
  mjpegOptions: -q:v 5 -r 2 -vf scale=640:-2 -update 1
  ipAddress: 192.168.1.110
  updateImageWhen: "0"
  onvifPort: 80
  gifPreroll: 0
  ffmpegLocation: /usr/bin/ffmpeg
  serverPort: 8085
  ffmpegOutput: /etc/openhab/html/camera1/
  ipWhitelist: DISABLE
  mp4OutOptions: -c:v copy -c:a copy
  pollTime: 30000
  password: password
  port: 80
  nvrChannel: 1
  snapshotOptions: -an -vsync vfr -q:v 2 -update 1
  ptzContinuous: false
  onvifMediaProfile: 0
  hlsOutOptions: -strict -2 -f lavfi -i aevalsrc=0 -acodec aac -vcodec copy
    -hls_flags delete_segments -hls_time 2 -hls_list_size 4
  gifOutOptions: -r 2 -filter_complex
    scale=-2:360:flags=lanczos,setpts=0.5*PTS,split[o1][o2];[o1]palettegen[p];[o2]fifo[o3];[o3][p]paletteuse
  username: admin

No, increasing heap size can be a negative thing to do. When JAVA is doing garbage collection it stops everything else until it is done, so by having a large heap size the amount of time your other processes get PAUSED for increases. There are plenty of articles on setting heap size on the web.

When you set memory aside for the heap, your system can not access it at all even if the heap is not close to being full. So your OS now only sees 12gb of ram as you removed 4 of it for the heap.

That is normal, if you remove/disable a thing, it willl free up space on the heap and cameras use a lot more then most other devices. The heap will grow and shrink all the time, what you need to look at is the max and min values that it hits after say a 24 hour period of time. Does these two values (min and max) keep growing each day or are they the same? If you keep running ‘shell:info’ 50 times over a 50 second period of time you soon get an idea of what it is doing at one point in time, but then you need to compare it. TIP: press the up arrow in linux to recall the last used command to save typing it in again.

You need to find out which binding or rule needs to be disabled to prevent the heap growing higher each day. By doing this you can find it quicker by not needing to wait for the heap to run out of space and throw OOME out of memory errors.

It would also help to post what bindings you use on your system, as there may be a known issue that someone can post about.

In regards to the ipCamera binding, the only known issue I am aware of is if your placing the cameras image into a IMAGE item as this floods the event bus with the image data. Since you have the setting on 0 this can not be it, unless your using the camera raw base64 data in rules that update an image item.

mhh ok
Since it started to happen after I used the IPCameraBinding my focus quickly turned towards it.
I also cross checked by deactivating all camera Things (binding still installed) → System is stable now for 6 days and heap looks fine.

I did this test before and then reactivated the camera things which caused the OOM errors after roughly 48 hours.

when I get the OOMs see my threads --list is flooded with 10thousands from this thread which I don’t know where it comes from:

I don’t see that type of thread at all when the camera things are disabled.

also I must say when I get the errors:
Error executing command: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
I am not certain that my heap is really full or another limit is reached. I am wondering about this because I checked heap on a regular basis and from 4gb heap (that was my default; didn’t change it actively) it was not near that max even if I checked few hours before that errors.
So my gut feeling is that actually I could be another limit? Possibly by 100k+ nioEventLoopGroup threads?

binding = zwave,sonos,samsungtv,astro,weatherunderground,chromecast,zigbee,http,fronius,gardena,miio,modbus,network,gpstracker,ntp,tr064,somfytahoma,ipcamera,mqtt,systeminfo,telegram,mail,homematic,openweathermap,shelly,wled

What happens if you pause all the cameras and enable 1 only at a time? Is it only a particular brand and model that it happens with? Are all the cameras the same?

I try that.

All the same IMOU C26E (discovered as DAHUA and I think also just a another brand of Dahua)

Can you enabled TRACE logging on the ipcamera binding with only 1 camera enabled and then restart the camera so the log shows it connecting and some of what the camera does when at idle?

thank you @matt1 for looking into it.
I will do the Trace logging and post once done.

Just in case it also helps please find below some errors I saw in the log from last night right at the point where openhab went instable.

2021-06-14 02:30:30.851 [WARN ] [netty.util.concurrent.DefaultPromise] - An exception was thrown by org.openhab.binding.ipcamera.internal.onvif.OnvifConnection$2.operationComplete()
java.lang.IllegalStateException: cannot await termination of the current thread
	at io.netty.util.concurrent.SingleThreadEventExecutor.awaitTermination(SingleThreadEventExecutor.java:807) ~[bundleFile:4.1.63.Final]
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.awaitTermination(MultithreadEventExecutorGroup.java:221) ~[bundleFile:4.1.63.Final]
	at org.openhab.binding.ipcamera.internal.onvif.OnvifConnection.disconnect(OnvifConnection.java:839) ~[?:?]
	at org.openhab.binding.ipcamera.internal.onvif.OnvifConnection$2.operationComplete(OnvifConnection.java:530) ~[?:?]
	at org.openhab.binding.ipcamera.internal.onvif.OnvifConnection$2.operationComplete(OnvifConnection.java:1) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) [bundleFile:4.1.63.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571) [bundleFile:4.1.63.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550) [bundleFile:4.1.63.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491) [bundleFile:4.1.63.Final]
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616) [bundleFile:4.1.63.Final]
	at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:609) [bundleFile:4.1.63.Final]
	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117) [bundleFile:4.1.63.Final]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:262) [bundleFile:4.1.63.Final]
	at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) [bundleFile:4.1.63.Final]
	at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) [bundleFile:4.1.63.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [bundleFile:4.1.63.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [bundleFile:4.1.63.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) [bundleFile:4.1.63.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [bundleFile:4.1.63.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [bundleFile:4.1.63.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [bundleFile:4.1.63.Final]
	at java.lang.Thread.run(Thread.java:834) [?:?]
2021-06-14 02:30:32.168 [INFO ] [era.internal.handler.IpCameraHandler] - The alarm stream was not running for camera 192.168.1.110, re-starting it now
2021-06-14 02:30:32.388 [INFO ] [era.internal.handler.IpCameraHandler] - The alarm stream was not running for camera 192.168.1.110, re-starting it now
2021-06-14 02:30:58.709 [INFO ] [era.internal.handler.IpCameraHandler] - The alarm stream was not running for camera 192.168.1.110, re-starting it now
2021-06-14 02:31:09.258 [INFO ] [era.internal.handler.IpCameraHandler] - The alarm stream was not running for camera 192.168.1.110, re-starting it now
2021-06-14 02:31:20.666 [INFO ] [era.internal.handler.IpCameraHandler] - The alarm stream was not running for camera 192.168.1.110, re-starting it now
2021-06-14 02:31:25.184 [INFO ] [era.internal.handler.IpCameraHandler] - The alarm stream was not running for camera 192.168.1.110, re-starting it now
2021-06-14 02:31:25.205 [WARN ] [netty.util.concurrent.DefaultPromise] - An exception was thrown by org.openhab.binding.ipcamera.internal.handler.IpCameraHandler$2.operationComplete()
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@6bd31f34[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@68073b96[Wrapped task = org.openhab.binding.ipcamera.internal.handler.IpCameraHandler$$Lambda$1677/0x00000007c1c53c40@2cfaf82b]] rejected from java.util.concurrent.ScheduledThreadPoolExecutor@4d3408e3[Shutting down, pool size = 4, active threads = 0, queued tasks = 0, completed tasks = 3]
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2055) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:340) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:680) ~[?:?]
	at org.openhab.binding.ipcamera.internal.handler.IpCameraHandler.bringCameraOnline(IpCameraHandler.java:1454) ~[?:?]
	at org.openhab.binding.ipcamera.internal.handler.IpCameraHandler$2.operationComplete(IpCameraHandler.java:614) ~[?:?]
	at org.openhab.binding.ipcamera.internal.handler.IpCameraHandler$2.operationComplete(IpCameraHandler.java:1) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) [bundleFile:4.1.63.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571) [bundleFile:4.1.63.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550) [bundleFile:4.1.63.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491) [bundleFile:4.1.63.Final]
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616) [bundleFile:4.1.63.Final]
	at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:605) [bundleFile:4.1.63.Final]
	at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104) [bundleFile:4.1.63.Final]
	at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) [bundleFile:4.1.63.Final]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:300) [bundleFile:4.1.63.Final]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:335) [bundleFile:4.1.63.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:707) [bundleFile:4.1.63.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) [bundleFile:4.1.63.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) [bundleFile:4.1.63.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [bundleFile:4.1.63.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [bundleFile:4.1.63.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [bundleFile:4.1.63.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [bundleFile:4.1.63.Final]
	at java.lang.Thread.run(Thread.java:834) [?:?]
2021-06-14 02:32:30.707 [INFO ] [era.internal.handler.IpCameraHandler] - The alarm stream was not running for camera 192.168.1.110, re-starting it now
2021-06-14 02:32:36.296 [INFO ] [era.internal.handler.IpCameraHandler] - The alarm stream was not running for camera 192.168.1.110, re-starting it now
2021-06-14 02:35:49.417 [ERROR] [ent.DefaultPromise.rejectedExecution] - Failed to submit a listener notification task. Event loop shut down?
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
	at java.lang.Thread.start0(Native Method) ~[?:?]
	at java.lang.Thread.start(Thread.java:803) ~[?:?]
	at io.netty.util.concurrent.GlobalEventExecutor.startThread(GlobalEventExecutor.java:232) ~[bundleFile:4.1.63.Final]
	at io.netty.util.concurrent.GlobalEventExecutor.execute(GlobalEventExecutor.java:208) ~[bundleFile:4.1.63.Final]
	at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:842) [bundleFile:4.1.63.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:499) [bundleFile:4.1.63.Final]
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616) [bundleFile:4.1.63.Final]
	at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:605) [bundleFile:4.1.63.Final]
	at io.netty.util.concurrent.DefaultPromise.setSuccess(DefaultPromise.java:96) [bundleFile:4.1.63.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:1051) [bundleFile:4.1.63.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [bundleFile:4.1.63.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [bundleFile:4.1.63.Final]
	at java.lang.Thread.run(Thread.java:834) [?:?]
2021-06-14 02:35:50.443 [INFO ] [penhab.core.model.script.Einspeisung] - Rule:PV_Umrechnungen triggered: Bezug
2021-06-14 02:36:10.670 [INFO ] [penhab.core.model.script.Einspeisung] - Rule:PV_Umrechnungen triggered: Bezug
2021-06-14 02:36:10.754 [INFO ] [era.internal.handler.IpCameraHandler] - The alarm stream was not running for camera 192.168.1.110, re-starting it now
2021-06-14 02:36:16.288 [WARN ] [io.netty.channel.AbstractChannel    ] - Force-closing a channel whose registration task was not accepted by an event loop: [id: 0xff59b179]
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
	at java.lang.Thread.start0(Native Method) ~[?:?]
	at java.lang.Thread.start(Thread.java:803) ~[?:?]
	at io.netty.util.concurrent.ThreadPerTaskExecutor.execute(ThreadPerTaskExecutor.java:32) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$1.execute(ThreadExecutorMap.java:57) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor.doStartThread(SingleThreadEventExecutor.java:978) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor.startThread(SingleThreadEventExecutor.java:947) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:830) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:818) ~[?:?]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:483) ~[?:?]
	at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:87) ~[?:?]
	at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:81) ~[?:?]
	at io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:86) ~[?:?]
	at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:323) ~[?:?]
	at io.netty.bootstrap.Bootstrap.doResolveAndConnect(Bootstrap.java:155) ~[?:?]
	at io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:139) ~[?:?]
	at org.openhab.binding.ipcamera.internal.onvif.OnvifConnection.sendOnvifRequest(OnvifConnection.java:517) ~[?:?]
	at org.openhab.binding.ipcamera.internal.onvif.OnvifConnection.connect(OnvifConnection.java:817) ~[?:?]
	at org.openhab.binding.ipcamera.internal.handler.IpCameraHandler.pollingCameraConnection(IpCameraHandler.java:1501) ~[?:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
2021-06-14 02:36:16.297 [ERROR] [ent.DefaultPromise.rejectedExecution] - Failed to submit a listener notification task. Event loop shut down?
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
	at java.lang.Thread.start0(Native Method) ~[?:?]
	at java.lang.Thread.start(Thread.java:803) ~[?:?]
	at io.netty.util.concurrent.ThreadPerTaskExecutor.execute(ThreadPerTaskExecutor.java:32) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$1.execute(ThreadExecutorMap.java:57) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor.doStartThread(SingleThreadEventExecutor.java:978) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor.startThread(SingleThreadEventExecutor.java:947) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:830) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:818) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:842) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:499) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:184) ~[?:?]
	at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:95) ~[?:?]
	at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:30) ~[?:?]
	at org.openhab.binding.ipcamera.internal.onvif.OnvifConnection.sendOnvifRequest(OnvifConnection.java:517) ~[?:?]
	at org.openhab.binding.ipcamera.internal.onvif.OnvifConnection.connect(OnvifConnection.java:817) ~[?:?]
	at org.openhab.binding.ipcamera.internal.handler.IpCameraHandler.pollingCameraConnection(IpCameraHandler.java:1501) ~[?:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]

This should only happen worse case every 8 seconds, but your log is showing it is 220milliseconds between them. Is there only 1 camera thing setup under the IP of 192.168.1.110 ? You have not setup the same camera multiple times by mistake? In a normal system you should only see that line get logged once when the camera first starts up and then you should not see it occur anymore from then on.

It really is not worth looking at logs AFTER the memory has run out, nothing will run correctly after there is no more ram to use. You need to send me the logs I requested on a freshly rebooted system that has not had the memory run out yet. I will be looking to see if the camera is triggering something weird to happen that is not normal. If that does not turn up any clues then I would suggest reloading the OS and openhab to see if it is corrupted data causing issues.