IPCam: OutOfMemoryError and OH delays

Hi,

since a few weeks, my OH instance has problems with UI delays (I click on a widget and the reaction comes some seconds later) or OutOfMemoryErrors (OOME). The OOME where probably caused by IPCam binding (there was something in that way in the log file, but after shutdown -r, the log was cleaned).

Then I read in this topic something similar. Because the problems with UI delays comes now more often (=in shorter intervals), I want to follow the recommendations of that topic. But:

The topic recommends to set UPDATE_IMAGE=false in things file. But the current doc has no such config option; there are some other which sounds very familar:

  • updateImageWhen = Not set in my things file. Therefore default config is used, i.e. image channel never updates. Is this one equal to the recommended setting?
  • pollTime = Set to 5 seconds in my things file. Should it be disabled?

Here is an example from my things file:
Thing ipcamera:onvif:cam01 "IP-Kamera" @ "Carport" [ ipAddress="192.168.xx.yy", username="example", password="xyz", port=80, onvifPort=8000, serverPort=9997, pollTime=5000 ]

Thanks in advance for all hints!

Yes that is the newer version and if you have not set it, then the image channel will not be updating.

Have you seen this similar thread? How do you display the camera, with an image card or the custom widgets?

Memory Leak … OH 3.1 M3 - Setup, Configuration and Use / Installation - openHAB Community

I am running OH3 on a RPI4, 4GB (running latest nighly builds) . Since a few weeks I am experiencing OOMs as well. I get thousands of open threads, OH becomes extremely sluggish and then crashes. I have not been able to pinpoint why this started to happen.
I do not find anything obvious in the logs…

I am currently testing binding by binding. At the moment I have disabled the IP camera things, to check if they’re the culprits.

Screenshot showing available memory. Encircled in orange are system reboots when things are totally out of control.

In a couple of days it will be obvious if the camera binding has anything to do with this.

In fact, the other thread looks quite similar. Thanks for that hint!

OH version is 3.0.1. For displaying images I use

  • element of type “image” in sitemap; and
  • widget “image” in habpanel

It looks like, that the time span between these problems becomes shorter. At first time, I re-start OH after 4 weeks, then two weeks and now 10 days. Beginning of June I’ve installed a second IP cam.

If u know the console, check what version of netty is installed, I think the core updated to a new netty around the time people started to first report this. If so it is not hard to roll back the netty library that this binding uses, but yes the first step is confirm which binding is the cause.

shell:info command in the console allows you to look at thread count and more.

So far, the RAM usage looks much better than previously.
It increases a lot slower.


However, I will clone my SSD and setup a system in parallell, since I cannot really mess around to much with my system without upsetting the family. :wink:

I will do it the other way around on the cloned system, start with no bindings - and add them one by one.

Btw. my netty version is 4.1.63.Final.

The charts are really good - I’ve now installed the Systeminfo binding, too. Very interesting and helpful - thank you!

That may be the issue as they had a lot of regressions around that time.

Try the latest Netty or roll back to the known good 4.1.42.Final

Netty.news: Netty 4.1.66.Final released

PR that made the change to see if it lines up with when you saw issues appear.

Update Netty dependencies to 4.1.63.Final by wborn · Pull Request #10606 · openhab/openhab-addons (github.com)

But I want to stress that the first step is confirm which binding causes it otherwise you will waste your time looking at possible fixes which will never do anything for your problem.

@BananaBender
Also the second point is not to look at RAM as java does NOT USE the ram like normal programs, it uses a special slice of the ram called the HEAP. You can see the heap growing and shrinking with the console command shell:info But if people are saying the threads are going out of control that is something the system info binding can look at provided that they are growing BEFORE the system runs out of heap, as when java has no more memory to use you can not trust any readings as it is not working right anymore.

Lastly another angle to try is, stop using snapshots in a image card and switch to using a mjpeg stream as it uses different code in the binding. Snapshots.mjpeg should not be used in this test as it uses snapshots so wont be using different code. The binding has three different ways to get a picture, snapshot, mjpeg and also HLS, so change to another method as a test.

Just a follow up.
Since disabling the IPcamera things, I have had no memory issues.
This could of course all be coincidental. I will delete my camera things and set them up again, eliminating any configuration issues I might have had.

Does this binding use the serial connection? could it be part of the implementation of nrjavaserial as the serial interface with OH? which has a nasty bug described here

No, the binding only uses Netty for http communications and does not use a serial port. Strangely thou the binding does seem to interact with the Zwave and Zigbee bindings, how exactly I do not understand. See thread on USB serial dongles staying offline unless openHAB is started with the cameras paused.

At this point I feel it is most likely the image card that causes it as I have changed a number of my cameras to try and reproduce this using the custom widgets and different types of streams and have not seen any issues for 10 days non stop testing. I’ll try a image card next.

@Daniel_Linder and @BananaBender how are you displaying the camera/s? what card/widget and what type of steam and is it left running non stop on a tablet?

I’ve two IP cams. Both are equal configured like

Thing ipcamera:onvif:cam01 "IP-Kamera" @ "Carport" [ ipAddress="192.168.xx.yy", username="example", password="xyz", port=80, onvifPort=8000, serverPort=9997, pollTime=5000 ]

Differences in thing config are such params like label, IP, username/password and serverport.

I’m not using a streams in this context within OH. The IP cams are used as a “smart” motion sensor. For displaying images in UI are used:

  • Sitemap: Element of type “image”
  • Habpanel: widget “image”

Refresh-rate of the images is set to 5 seconds. But AFAIK, the UI is only refreshed, if it is opened (if you open an OH UI then the initial displayed image is from last visit, e.g. some hours old). Habpanel is used on a tablet and notebook; sitemap on smartphones. OH is not permantly opened on these devices - only on demand. Most aspects are automated by rules, therefore I use the devices for OH not very often (e.g. only in the morning to check the chart with the motion alarms, in case of any problems, etc.)

Besides the images, the cellMotionAlarm channel is used by following items:

Switch ipcam_outside_carport_cell_motion_alarm "IP-Kamera Carport" <motion> { channel="ipcamera:onvif:cam01:cellMotionAlarm" }
Switch ipcam_outside_carport_cell_motion_alarm_mapped "IP-Kamera Carport" [MAP(reolink.cellmotionalarm.map):%s]" <motion> { channel="ipcamera:onvif:cam01:cellMotionAlarm" }

Both items are used in both UIs (Habpanel and Sitemap). The mapped item for textual displaying of information. The other one for a Chart.

Two days ago, I’ve updated my OH from 3.1 to 3.2. Since that update it looks like, that the problem has been solved. The number of threads is not increasing anymore. I’m going to observe it in the next days, but until now it looks good!

Thank you (@all the OH developers and especially @matt1 ) for this x-mas present!!!