[SOLVED] IpCamera by Skinah: OutOfMemoryError Java heap space - Amcrest

Diogenes · August 8, 2019, 3:35pm

Sorry for the callout @matt1, but I think you’re the only one who can help with your binding. It’s been working well with a few hiccups (actually heap space errors, but I could run for days before I’d see them), but I updated to your snapshot a couple days ago to fix them, and now I can’t go for ten minutes without openHAB crashing due to Java heap space errors.

I followed your instructions about increasing the heap space, to no avail.

Also saw this thread; it doesn’t look like anything came of it besides your recommendation of increasing the heap space.

I set the log to TRACE, please find my attached log: openhab_IpCameraJavaHeapSpace_20190808part3.log (546.1 KB)

I’d really appreciate it if you could take a look at it.

Things:

// Amcrest IP2M-841E
Thing ipcamera:DAHUA:120 [IPADDRESS="xxx.xxx.xxx.120", PASSWORD="password", USERNAME="username", POLL_CAMERA_MS=3000]

// Amcrest IP2M-841W
Thing ipcamera:DAHUA:121 [IPADDRESS="xxx.xxx.xxx.121", PASSWORD="password", USERNAME="username", POLL_CAMERA_MS=3000]

// Amcrest IP8M-2493EW
Thing ipcamera:DAHUA:122 [IPADDRESS="xxx.xxx.xxx.122", PASSWORD="password", USERNAME="username", POLL_CAMERA_MS=3000]

// Amcrest IP8M-2493EW
Thing ipcamera:DAHUA:123 [IPADDRESS="xxx.xxx.xxx.123", PASSWORD="password", USERNAME="username", POLL_CAMERA_MS=3000]

Items:

// Camera 120 (xxx.xxx.xxx.120: Amcrest IP2M-841E)
Group   G_House_Amc120             "120 Camera"                                   <house>         (G_House, G_Cameras)
Image   I_House_Amc120_Jpg_im_c    "120 Snapshot JPG"                             <camera>        (G_House_Amc120, G_CamJpg_im)           { channel="ipcamera:DAHUA:120:image" }
Switch  I_House_Amc120_MotEn_sw_v  "120 Motion Alarm Enable [%s]"                                 (G_House_Amc120, G_CamMotEn_sw)
Switch  I_House_Amc120_MotAct_sw_c "120 Motion Alarm Active [%s]"                 <motion>        (G_House_Amc120, G_CamMotAct_sw)        { channel="ipcamera:DAHUA:120:motionAlarm" }
Switch  I_House_Amc120_MotTm_sw_v  "120 Motion Alarm Timer [%s]"                                  (G_House_Amc120, G_CamMotTm_sw)         { expire="3s,command=OFF" }               

// Camera 121 (xxx.xxx.xxx.121: Amcrest IP2M-841W)
Group   G_House_Amc121             "121 Camera"                                   <house>         (G_House, G_Cameras)
Image   I_House_Amc121_Jpg_im_c    "121 Snapshot JPG"                             <camera>        (G_House_Amc121, G_CamJpg_im)           { channel="ipcamera:DAHUA:121:image" }
Switch  I_House_Amc121_MotEn_sw_v  "121 Motion Alarm Enable [%s]"                                 (G_House_Amc121, G_CamMotEn_sw)
Switch  I_House_Amc121_MotAct_sw_c "121 Motion Alarm Active [%s]"                 <motion>        (G_House_Amc121, G_CamMotAct_sw)        { channel="ipcamera:DAHUA:121:motionAlarm" }
Switch  I_House_Amc121_MotTm_sw_v  "121 Motion Alarm Timer [%s]"                                  (G_House_Amc121, G_CamMotTm_sw)         { expire="3s,command=OFF" }               

// Camera 122 (xxx.xxx.xxx.122: Amcrest IP8M-2493EW)
Group   G_House_Amc122             "122 Camera"                                   <house>         (G_House, G_Cameras)
Image   I_House_Amc122_Jpg_im_c    "122 Snapshot JPG"                             <camera>        (G_House_Amc122, G_CamJpg_im)           { channel="ipcamera:DAHUA:122:image" }
Switch  I_House_Amc122_MotEn_sw_v  "122 Motion Alarm Enable [%s]"                                 (G_House_Amc122, G_CamMotEn_sw)
Switch  I_House_Amc122_MotAct_sw_c "122 Motion Alarm Active [%s]"                 <motion>        (G_House_Amc122, G_CamMotAct_sw)        { channel="ipcamera:DAHUA:122:motionAlarm" }
Switch  I_House_Amc122_MotTm_sw_v  "122 Motion Alarm Timer [%s]"                                  (G_House_Amc122, G_CamMotTm_sw)         { expire="3s,command=OFF" }               

// Camera 123 (xxx.xxx.xxx.123: Amcrest IP8M-2493EW)
Group   G_House_Amc123             "123 Camera"                                   <house>         (G_House, G_Cameras)
Image   I_House_Amc123_Jpg_im_c    "123 Snapshot JPG"                             <camera>        (G_House_Amc123, G_CamJpg_im)           { channel="ipcamera:DAHUA:123:image" }
Switch  I_House_Amc123_MotEn_sw_v  "123 Motion Alarm Enable [%s]"                                 (G_House_Amc123, G_CamMotEn_sw)
Switch  I_House_Amc123_MotAct_sw_c "123 Motion Alarm Active [%s]"                 <motion>        (G_House_Amc123, G_CamMotAct_sw)        { channel="ipcamera:DAHUA:123:motionAlarm" }
Switch  I_House_Amc123_MotTm_sw_v  "123 Motion Alarm Timer [%s]"                                  (G_House_Amc123, G_CamMotTm_sw)         { expire="3s,command=OFF" }

Memory when openHAB has crashed from IpCamera:

Memory
  Current heap size           334,784 kbytes
  Maximum heap size           506,816 kbytes
  Committed heap size         506,816 kbytes

 openhabian@openHABianPi:~$ free -h
              total        used        free      shared  buff/cache   available
Mem:           976M        764M         36M         11M        175M        151M
Swap:           99M        4.2M         95M

Thanks in advance for the help.

Hardware: Raspberry Pi3
OS: openhabian 2.4.0-1 (Release Build)
IpCamera: 2.5.0.201908030247 (SNAPSHOT downloaded today 2019.08.08)
Amcrest models: IP2M-841EW, IP2M-841B, IP8M-2493EW

matt1 · August 8, 2019, 8:36pm

I had a recent/same problem when I went from only one camera to four. One big change you can make is turning off the image channel from updating. My guess is the binding can send data faster then what your Pi and openhab can process when you have more then 3 cameras. Openhab seems to send the picture data down the event bus and this is the bottleneck which is not my coding. Also it appears to leak memory. Good news is I added a feature to turn off sending the data to openhabs frame work and these problems went away. No memory leaks, it halved the cpu usage and no signs of OOME heap space errors. The readme has details on turning off the UPDATE_IMAGE=false and how to get a picture using urls. I also did a howto post.

I would also plan on upgrading your Pi to something with at least 2gb of ram if not more. I use odroid c2 which has 2gb ram and had five cameras running for weeks with a heap size of 768 from memory which the Pi3 won’t handle.

Try turning off image first and then maybe one less camera till you upgrade.

Diogenes · August 8, 2019, 9:25pm

Thanks for the suggestions and prompt reply! Will give a go and post results.

matt1 · August 8, 2019, 11:48pm

Yes please do post what you find is the limit of a 1gb pi3. I would love to know why changing to the newer binding made it worse, did you add more cameras or change the poll rate at the same time or perhaps up the snapshot quality at the same time?

The how to i was referring to is here.

Diogenes · August 9, 2019, 12:46am

Thanks for the link, I was having trouble tracking it down!

Actually, I did just the reverse trying to solve the issue: increase poll rate in both the binding and the sitemap for all cameras from 1000 to 3000ms, and decreased the snapshot JPG quality in all camera settings. The only thing I can think of is I just installed two of the higher quality cameras outside (they were previously connected with the as-installed hardware, but just sitting under my office desk for bench testing).

I have motion alarms enabled on them both, so maybe with outside motion the cameras don’t respond to the requests? I doubt it, since I set the sensitivity pretty low and only in regions I care about, so the motion alarms were very infrequent. But perhaps I’ll disable the motion for a bit and see if things improve.

matt1 · August 9, 2019, 1:00am

It is purely the image channel stop it from updating nothing to do with motion or alarms, they use next to nothing. Every time it updates it will send a full dump of the picture to your log file and much more so it is a bottleneck and this binding stresses it out more than any other binding. Video is just a lot of data.

I have four cameras running all the time at 1sec poll rate and have tested with another two for six in total and no problems, but the moment I enable the image channel to update on more than 1-2 cameras the issues start. But your pi3 has less grunt then my c2 and less ram so I would expect you to have problems much sooner.

Disable the image channel and you may be able to run all four cameras at 1 sec poll rate, but at some point only 1gb of ram is simply not enough. Keen to hear what you find.

Diogenes · August 9, 2019, 2:25am

Thanks again for the link, I was able to set up my four cameras as you suggested, commenting out the image channel, adding

UPDATE_IMAGE=false

to the Things file, creating the HTML files, and using the Webview sitemap object. My results were mixed: BasicUI (FIrefox) would load the images about half the time the sitemap loaded, same if the URLs were entered directly in Firefox. No images would show on the Android openHAB app.

I understand why you made of note of keeping the HTML refresh interval higher than normal, because it takes so long to load. I’m not sure how you’re able to get 1 sec updates using Webview. I’ll have to keep playing with it tomorrow.

matt1 · August 9, 2019, 2:37am

That is not enough to stop it updating, you need to add this to the things file for each camera.

UPDATE_IMAGE=false

If you setup the first substream of the camera to be mjpeg in the cameras own setup then you can use the mjpeg stream to give live video feeds.

Diogenes · August 9, 2019, 10:32am

You’re right, I did add that line, but forgot to mention it (added it my post above). I’ll try the mjpeg, see if that’s better.

Diogenes · August 9, 2019, 12:20pm

Ok, couple things I tried:

Setup all cameras with substream 1, mjpeg @ 2fps. The two 2493EW’s displayed perfectly, using the following src:

<a href="http://xxx.xxx.xxx.xxx:8080/static/allcams.html"><img id=amc123 src=http://xxx.xxx.xxx.123/cgi-bin/mjpg/video.cgi?channel=1&subtype=1 width=99.4% border="0"></a><br>

The other two cameras (different models) refused to display with the same html file structure and settings. There must be something else in the camera settings I’m missing. Gonna have to double check those.

The openHAB Android app would not display the html images/streams. I suspect this is an app issue, and not your binding, since BasicUI (Firefox) worked fine. The html file seemed to load, because I could click on the broken-image icon in the app, and it would change from four icons to one icon, meaning the app was following the links in your four-camera format, but it just couldn’t load the images/streams. Something to look into…
As a test, I took the src from the html file and plugged it straight into the Webview sitemap object url. At first, the sitemap displayed an “unauthorized” error. I added login info and I was able to get all four cameras to display in the app:

Webview url="http://username:password@xxx.xxx.xxx.120/cgi-bin/mjpg/video.cgi?channel=1&subtype=1"

Curiously, only one of my two lower quality cameras would ever display at once in BasicUI (the other two higher quality ones had no issues). One of them would prompt for login info in BasicUI, while the other would load correctly. On refresh, that camera would display and the other one would prompt for login info. Strange.

Good things to know: *.html files aren’t always automatically updated/reloaded (at least on my system). Sometimes I had to restart the openhab2 service.

Gonna keep testing.

matt1 · August 9, 2019, 10:24pm

Have you read the read me file? See the snapshot and mjpeg sections as you are not using the bindings abilities and instead using the cameras direct urls. U will have to keep entering in logins that way.

Diogenes · August 12, 2019, 11:12am

Yep, I read them, and as I described, I was having trouble using the HTML approach and defined port numbers. So I described my testing procedure to make sure the cameras themselves were setup correctly and outputting the correct stream (which it appears they are). So I was trying to go back and figure out why only two cameras are able to stream using the HTML files and direct camera URLs, and I get nothing from the other two:

(x2) IP8M-2493EW: success
(x1) IP2M-841E: failed
(x1) IP2M-841W: failed

But I’m probably going about this the wrong way, and using the binding correctly will solve this issue. So, how do you define the port number in your example:

<a href="http://192.168.1.2:8080/static/backyardcam.html"><img id=cam1 src=http://192.168.1.2:40061/ipcamera.jpg width=49.4% border="0"></a>

A couple questions:

It looks like 192.168.1.2 is your openHAB IP, and you are not referencing the camera IP directly. What is unclear is where you define the port number 192.168.1.2:40061. Is this port 40061 the camera Thing SERVER_PORT? Does this also need to be defined/configured in the camera settings somewhere, perhaps for TCP or Multicast port?
Is the string after the port number, /ipcamera.jpg, a general placeholder that I need to fill in with the specific Amcrest model API call, or this a keyword your binding uses and should be left alone?

As always, thanks for the clarification and assistance.

matt1 · August 12, 2019, 10:12pm

Yes, no
Not a place holder, leave it the same and it is the port that changes for each camera.

Diogenes · August 12, 2019, 11:32pm

That did it! My issues were:

I did not notice the src used the IP of the openHAB server, and not the camera. So I thought the SERVER_PORT had to be configured on both sides (openHAB and the camera). Details, details…
Once the IP and port were corrected, I got an error: The request made from (IP) was not in the whitelist and will be ignored. Adding the line IP_WHITELIST="DISABLE" to each camera Thing fixed this.
Lastly, had to fix a typo in my HTML files, had src==. I did not receive any errors in the log (although I did not have TRACE set to any logger, so it might not have been displayed).

The last issues to solve would be some minor formatting of the Webview for BasicUI (but the streams in the openHAB Android app look perfect) and some lagging in loading multiple Webview in a single sitemap (but that’s probably due to my VPN speed and the fact that I’m halfway around the world from my system on business I also probably need to take the streams down to 1 fps.

All in all, @matt1 you’ve been a huge help, thank you. While your last few posts have fixed my streaming issues, I think I’ll mark your first reply as the solution, since it was there you suggested the URLs in the first place, as well as beefing up my hardware, which is what I’ll probably end up doing eventually (I really like the snapshot functions).

Thanks again for all your hard work on this binding!

zacofunny · September 28, 2019, 3:02pm

@mstormi, the only reason for me to install 64-bit Zulu java is that java 32-bit doesn’t allow me to increase Java Memory more than ~3GB. I have continuous problem with out of memory related to HIKVISION camera + ipcamera binding. I wanted to verify if increasing memory will solve the issue or not. 3GB should be more than enough for any yet the problem with memory still happen.

Now I summon @matt1. I’ve done everything I could to prevent leaks of memory, I’ve set UPDATE_IMAGE=false and started filtering habpanel items. But camera still drops the connection, very often with out of memory error like below. The interesting thing is that when I started filtering items to habpanel the Out of Heap Memory isse that was crashing the whole OpenHAB stopped happening, however the below drops still remain.

2019-09-28 16:33:34.303 [WARN ] [ing.ipcamera.handler.IpCameraHandler] - !!!! Camera has closed the channel     URL:/ISAPI/Streaming/channels/102/httppreview Cause reported is: {}
java.lang.OutOfMemoryError: null
        at sun.misc.Unsafe.allocateMemory(Native Method) ~[?:?]
        at io.netty.util.internal.PlatformDependent0.allocateDirectNoCleaner(PlatformDependent0.java:452) ~[210:io.netty.common:4.1.34.Final]
        at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:613) ~[210:io.netty.common:4.1.34.Final]
        at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:768) ~[207:io.netty.buffer:4.1.34.Final]
        at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:744) ~[207:io.netty.buffer:4.1.34.Final]
        at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:245) ~[207:io.netty.buffer:4.1.34.Final]
        at io.netty.buffer.PoolArena.allocate(PoolArena.java:215) ~[207:io.netty.buffer:4.1.34.Final]
        at io.netty.buffer.PoolArena.allocate(PoolArena.java:147) ~[207:io.netty.buffer:4.1.34.Final]
        at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:327) ~[207:io.netty.buffer:4.1.34.Final]
        at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187) ~[207:io.netty.buffer:4.1.34.Final]
        at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178) ~[207:io.netty.buffer:4.1.34.Final]
        at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:139) ~[207:io.netty.buffer:4.1.34.Final]
        at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114) ~[211:io.netty.transport:4.1.34.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:147) [211:io.netty.transport:4.1.34.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:677) [211:io.netty.transport:4.1.34.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:612) [211:io.netty.transport:4.1.34.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:529) [211:io.netty.transport:4.1.34.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:491) [211:io.netty.transport:4.1.34.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) [210:io.netty.common:4.1.34.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [210:io.netty.common:4.1.34.Final]
        at java.lang.Thread.run(Thread.java:748) [?:?]

matt1 · September 29, 2019, 12:57am

I am having no issues and dont need to restart with 4 cameras running updating every second but not updating the image channel. My uptime is 3-4 weeks before I do a reboot to fix another glitch. The moment I turn on the updateimagenow control or don’t use UPDATE_IMAGE=false I start to get memory leaking.

I use Odoird C2 with 2gb of ram and their standard ubuntu minimal image as a base. 32bit Zulu java, used multiple versions of it now but I am running an older one at the moment. Openhab 2.4 Stable.

All I can suggest is that you experiment to see what makes a difference on your system.
Try a different Java version.
Try 1 camera only.
Try not using Habpanel for a while.
Keep your UI’s closed when not actually needed.

Things like that to help narrow down what area is the issue in your setup. Unless I can reproduce it here, or have someone give a clue as to what is different between mine and your setup there that solves it for them there is not much I can do.

zacofunny · September 29, 2019, 5:34am

Thanks @matt1! I do respect your great work on the binding. I’m on OH 2.5M3. Do you use Hikvision cameras? I was sure you do. Would you mind to show your *.things file? Just want to compare.

matt1 · October 8, 2019, 10:00am

Yes I do use Hikvision, Amcrest and Instar cameras. The thing file examples are found in the readme file and were created from my setup.

Euphi · December 29, 2021, 9:50am

I know this is an old thread, but it seems I had a serious memory leak (and high CPU usage) caused by the IPCamera binding after upgrade to OH 3.2 .

The binding was installed but not used, however after the upgrade to 3.2, OH used much more CPU and froze after some time. Some “Out of heap space” error message were in logfile.

I found this thread, removed the IPCamera binding (and deleted remaining Items that linked to non-existing IPCamera things) and the problem was gone.

On the screenshot you can see the CPU load after Upgrade to 3.2 (13.12. evening), an even higher load before complete freeze, low CPU load after freeze, and new low CPU load after removing IPCamera binding (27.12.).

matt1 · December 29, 2021, 9:52pm

Open your own new thread, also install the system info binding and setup graphing of the free heap percentage. CPU load means nothing.