Hi!
Recently I replaced 2 cameras on my Dahua NVR (8 port XVR5108HS-I2) and something weird has started happening with generation of ipcamera.mjpeg (via ffmpeg) for those two cameras (the rest of them work fine and I didn’t change any settings for the binging - just the cameras were replaced). Basically the “stream” freezes very quickly. On an android tablet (two different ones) in Habpanel I pretty much see the stream for a second or two and it disappears. If I just open the ipcamera.mjpeg in a browser on my computer I will see the stream for a few minutes but it will very quickly freeze as well (strangely enough for one camera it very often seems to happen when a car passes by on a road).
The NVR that I mentioned above (Dahua XVR5108HS-I2) is an analog/BNC one. So the cameras are connected through BNC cables to the NVR. The two new cameras are POLICETECH Q4-B5200M-S2.
I have the main feed set to output 1920x1080 with CBR, 15fps and encoded in h265. The server running OH is an HP Proliant Microserver gen 8 with an upgraded CPU (Intel Xenon) and 12GB of RAM and is running on top of Ubuntu 20.04.2 LTS inside a virtual machine on VMWare ESXi. Looking at the CPU usage it is around 40% when all of the streams are being viewed in habpanel, there is enough free RAM left as well. So performance shouldn’t be an issue.
The browser console gives me nothing, and the binding trace log is not really any more useful either:
2021-08-22 03:24:53.154 [DEBUG] [pcamera.internal.StreamServerHandler] - Stream Server recieved request GET:/ipcamera.mjpeg
2021-08-22 03:24:53.156 [DEBUG] [hab.binding.ipcamera.internal.Ffmpeg] - Starting ffmpeg with this command now:******** ...
2021-08-22 03:24:55.756 [DEBUG] [hab.binding.ipcamera.internal.Ffmpeg] - [swscaler @ 0x555e4a0a7680] deprecated pixel format used, make sure you did set range correctly
2021-08-22 03:25:05.302 [DEBUG] [hab.binding.ipcamera.internal.Ffmpeg] - More than 1000 frames duplicated
2021-08-22 03:27:37.054 [DEBUG] [pcamera.internal.StreamServerHandler] - Stream server is going to close an idle channel.
2021-08-22 03:27:37.055 [DEBUG] [era.internal.handler.IpCameraHandler] - All ipcamera.mjpeg streams have stopped.
The parameters that I have set for the ffmpeg output are these:
-b:v 2000 -r 8 -vf scale=640:-2 -update 1
I have noticed that occasionally other cameras would disappear in the past as well. But this happened fairly infrequently - few times per week. I have also noticed that habpanel on those tablets keeps crashing like once per day. Without the camera streams there it seems stable. This crashing has become a lot more common with the replacement of those two cameras.
Now I’ve mentioned the camera stream freezing in the habpanel… Just to make it clear - this is happening everywhere - just that I primarily use these streams in habpanel.
I’ve been struggling with this for a week and I am completely lost. Any ideas on what could be wrong? I find it completely weird since only the cameras changed and nothing else.
My first thought is that your asking too much of the system and the stream (ffmpeg) stops, try reducing the resolution and the framerate as a test. Change to h264 to see what happens. Just because your not going above 40% cpu load does not mean much, as you need the data crunched within a certain time. 15fps means it has to process each video frame in under 66ms and if it can not keep up the data backs up. If you run the ffmpeg command manually you will see if the process stays above 1.0x speed, if it falls below you either need a faster cpu or lower the quality/fps or enable some kind of acceleration.
Since it happens when a car passes, this would be a more CPU intensive time to remove the h265 compression, so it seems to confirm my thoughts.
Did the older camera have different quality or compression?
Does not change my thoughts on the cause and running the command manually will allow you to see more information. Also it would be best to solve 1 problem at a time as they may be related or could be separate issues. The stream stopping is what the above is about, habpanel crashing is the other.
Hi!
What the older cameras had was a much lower resolution. Possibly bitrate was different as well (I kept the defaults that the NVR chose). The ffmpeg parameters were the same - I did not touch those.
So I did the test just now. I disabled all of the camera things except one of the new ones and I went to the NVR settings and switched compression to h264. Confirmed that with VLC. When reloaded habpanel I could see the stream and it appeared to be working… for about a minute. So instead of freezing after 2 seconds it freezed after a minute, but it was the same behavior. A few cars passed by on the road which went ok and then another one did and the thing just stuttered and froze like before.
Actually I just remembered something else. I was playing around with the NVR and openhab settings all day. The resolution for the new cameras is a bit weird. NVR shows it as 5M-N (1440x1620). But it’s not portrait. The NVR stretches so it looks perfectly normal in wide screen. However the binding/ffmpeg do not. And the output shown is actually taller than wider and looks very squished. I changed the resolution to 1920x1080 on the NVR but even then ffmpeg output is sometimes produced with this wrong aspect ratio. Reloading the stream sometimes helps.
EDIT: Well changing the camera resolution to the default (maximum) was apparently a bad idea. It literally crashed openhab after a few minutes. I could no longer access any UIs or anything. Rebooting the server, restarting openhab, … didn’t help. But as soon as I changed the resolution on the NVR back to 1080 the thing came online.
It looks like it ran out of memory.
2021-08-27 17:09:39.904 [ERROR] [ent.DefaultPromise.rejectedExecution] - Failed to submit a listener notification task. Event loop shut down?
java.util.concurrent.RejectedExecutionException: event executor terminated
at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:926) ~[bundleFile:4.1.63.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:353) ~[bundleFile:4.1.63.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:346) ~[bundleFile:4.1.63.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:828) ~[bundleFile:4.1.63.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:818) ~[bundleFile:4.1.63.Final]
at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:842) [bundleFile:4.1.63.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:499) [bundleFile:4.1.63.Final]
at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:184) [bundleFile:4.1.63.Final]
at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:95) [bundleFile:4.1.63.Final]
at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:30) [bundleFile:4.1.63.Final]
at io.netty.bootstrap.ServerBootstrap$ServerBootstrapAcceptor.channelRead(ServerBootstrap.java:215) [bundleFile:4.1.63.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [bundleFile:4.1.63.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [bundleFile:4.1.63.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [bundleFile:4.1.63.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [bundleFile:4.1.63.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [bundleFile:4.1.63.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [bundleFile:4.1.63.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [bundleFile:4.1.63.Final]
at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:97) [bundleFile:4.1.63.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [bundleFile:4.1.63.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) [bundleFile:4.1.63.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) [bundleFile:4.1.63.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [bundleFile:4.1.63.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [bundleFile:4.1.63.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [bundleFile:4.1.63.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [bundleFile:4.1.63.Final]
at java.lang.Thread.run(Thread.java:834) [?:?]
2021-08-27 17:09:41.305 [WARN ] [pcamera.internal.StreamServerHandler] - Exception caught from stream server:Java heap space
2021-08-27 17:09:41.305 [WARN ] [hannel.AbstractChannelHandlerContext] - An exception 'java.lang.OutOfMemoryError: Java heap space' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception:
java.lang.OutOfMemoryError: Java heap space
2021-08-27 17:09:41.306 [WARN ] [pcamera.internal.StreamServerHandler] - Exception caught from stream server:Java heap space
2021-08-27 17:09:41.306 [DEBUG] [pcamera.internal.StreamServerHandler] - Stream server is going to close an idle channel.
2021-08-27 17:09:42.550 [WARN ] [pcamera.internal.StreamServerHandler] - Exception caught from stream server:Java heap space
2021-08-27 17:09:44.442 [WARN ] [mmon.WrappedScheduledExecutorService] - Scheduled runnable ended with an exception:
java.lang.OutOfMemoryError: Java heap space
2021-08-27 17:09:44.444 [WARN ] [pcamera.internal.StreamServerHandler] - Exception caught from stream server:Java heap space
2021-08-27 17:09:48.799 [WARN ] [ab.core.internal.events.EventHandler] - Dispatching event to subscriber 'org.openhab.core.io.monitor.internal.EventLogger@4b269336' takes more than 5000ms.
Thanks. I think I will wait until it gets “officially” released before trying it again though. Yesterday evening I installed Shinobi in a Docker container in my other VM and configured it to produce MJPEG streams for me. And ever since switching Habpanel has been completely stable, cameras haven’t stopped working, … So it’s finally working as it should. The last time this was the case was when I was still using ffmpeg and ffserver directly to produce the streams. I still like the idea of having it integrated into OH so I will definitely give it a go at some point. But for now I think I found a solution that works for me.
I would recommend you keep doing it with that method if your wanting to do non stop streaming of 8 cameras above 1080 resolution and 15 fps. That is a lot of data all happening at once. The following is for you and anyone else that comes across the post…
I suspect you have the default java heap size settings and it was running out of ram. You can see what it is set to by going into the console and using the command shell:info.
Maximum heap size = 633,536 kbytes
That is what mine is set to but you may need more if your doing that much load.
Let’s say each frame is a 1mb picture, your using 8 streams and 15 frames each second, this comes to 120mb each second. The binding has buffers on the in and out so it may then need 3 to 4 times this amount of ram/heap, we don’t want the garbage collection to be running all the time. If your wanting to stream to multiple tablets at the same time, then it will require even more for each additional outgoing stream. I don’t know of any other binding which chews this amount of memory, so if your going to attempt such things the default heap size is probably not going to be enough.
java.lang.OutOfMemoryError: Java heap space
This shows you ran out of heap space, all other errors can be ignored as nothing works correctly when there is no more heap space.
I personally think its best to offload this to another server if your wanting non stop streams on multiple cameras as it will congest the network card, plus you probably will want other features that the packages offer. If like me you only want a few streams on demand that stops after a user closes a window then why not use the binding.
