openHAB 3 runs out of memory / java heap space errors, CPU 100%+ after a few hours

You should see my taskbar pc a week after a forced windows update reboot, I almost never close any program. lol. But, OH3 doesn’t mind that, it just runs fine with Language server off.

Small tip from my side. Don’t save files to short after eachother, especially items files. It’s not new, OH2.x had the same ‘issues’. Not completely sure but when you save a large Item file, my guess is everything ‘attached’ to that file is compiled again (makes sense). I sometimes make this ‘mistake’ and a raspberry pi doesn’t like that.

I break all my items up into smaller files, because I’m obsessed with being organized.

Reproduces also for me:

2021-04-27 23:19:53.497 [WARN ] [ab.core.internal.events.EventHandler] - Dispatching event to subscriber 'org.openhab.core.io.monitor.internal.EventLogger@da0e20' takes more than 5000ms.

on 3.0.1 and when reproduces “VM Thread” consumes 100% of single core all the time.
Hard to tell what triggers it, as for me it’s sometimes happening every few days and sometimes after month.
I already have increased HEAP for Java:

EXTRA_JAVA_OPTS="-Xms400m -Xmx420m"

in /etc/default/openhab that was solving this issue on OH 2.x, but not helping with 3.x.

I had almost the same issue, everyday at round about 4 o‘clock my OH run out of memory.
Almost tried out everything even plain installation and than installed my backup.
Still a crash every night.
Then I went through my rules file and found out I had linked an item which does not exist and is triggered with many other items when my heating starts at 4 o‘clock. Since I repaired this it runs for more than two weeks without errors!

I am experiencing this myself with OH 3.1.0.M4 (docker). High CPU load, OH3 becoming more and more unresponsive resulting in “java heap space” log entries.

I am using DSL as well and had a situation where OH3 ran smoothly until I started VSC with language server enabled. CPU load went up immediately and never came down again to a normal level. Since then I disabled the language server feature in VSC and all is good now.

Did anyone file an issue already?

An issue is opened:

Hi Falk,
Can you point me to a description how I can disable the language server? Can’t find it, just found threads where they explained it can’t be disabled, just stopped via console.
Best regards, Stefan D.

Here it my memory usage:

Is there any way to log the memory usage for some longer time? I try to find some correlation with other events.

Sure, in Visual Studio Code got to “extensions (german: Erweiterungen)”, right klick openHAB and chose “extension setup (german: Erweiterungseinstellungen)”.
Scroll down to “Languageserver: Remote Enabled” and disable the function.

Edit: There may be another option which I have not tried in detail so far and should disable the language server entirely:

in openhab cli type

bundle:list | grep Language

This will give you the ID of this component. Now type:

bundle:stop ID

This should disable the language server on OH side. Not sure about potential side effects …

Thank you so much for that info! I set the same value in this file (running OH3.0.2 on RPI 4 8GB) and since then my system is running stable without any performance problems. System is up for 3 days now.
In the past I had to restart almost every day.

Hi all,
Since upgrade from 3.0 to 3.1 I have strong memory leak.
In 3.0 the system was very stable (running for weeks/months).
Now in 24h-48h I run into OOM (raspberry pi 4Go dedicated).
At 2go for OH3 java process, i have many error messages (rrdb, etc.) and Openhab hangs.
I tried EXTRA_JAVA_OPTS="-Xms1024m -Xmx2048m" but changed nothing.
I am trying to disable addons one by one and comment most of my rules but it is very long and fastidious.
Is there a method to see from which plugin(s) or rule(s) comes this memory leak ? Or Linux system commands ?
Currently I run “ps -p xxx -o rss=” and see the memory size increasing constantlty … until freeze or restart/reboot.
Many thanks for any tip/hint or info !

1 Like

I also had many java heap errors and system instability. About the same time I changed two things and since then Openhab works very stable. First I disabled LSP in VSCode and then I saw that there was a rule that used an item that did not exist anymore. After checking all the rules to make sure no items were used that no longer exist, I have no OOM or other stability issues and OpenHab remains very responsive. Anpro experienced the same phenomenon. I am currently using OpenHAB 3.1.0 - Release Build. I have made no changes to EXTRA_JAVA_OPTS.

1 Like

Thanks for your advice @Sebi
I will try to disable LSP (but I use a lot VSC, i am annoyed).
You mean you stop the bundle or uninstall ? if you stop only it restarts automatically at next startup no ?
I spent so many hours to find this memory leak … I am now at a point where I commented out almost 100% of my rules and uninstall every bundle except one (zwave) … and still memory increase constantly.
Wether there is a memory leak in core (or my install) or I dont have the right approach to debug.

Just for your information, I am in the exact same situation…

Trying to find out what causes my OOMs by disabling bindings…

Above graph is from the last 7 days. The “spike” was a shutdown where things were totally out of control. The last four days seems quite fine though, after I disabled a specific binding.

I won’t draw any conclusions yet though.

I only disabled LSP in openHAB extension in VSCode, I didn’t disable or uninstall anything in openHAB. To disable LSP in VSCode go to Extensions, openHAB, Manage and do a search for Languageserver option. Uncheck the option Languageserver: Remote Enabled.
afbeelding
Check User and Workspace settings.

Has anyone opened a bug on github to track the new memory leak you are seeing in 3.1? I’ve been on the 3.1 snapshots for a long time and I haven’t seen a memory leak so I’m wondering what the difference between our setups is.

Repeatable “Out of memory” found regarding large image items. I’ll document it here for others who might find it helpful.

OpenHab 3.1.0 Running on Pi4 4gb.
Bindings:

  • Amazon Echo
  • Astro
  • Epson Projector
  • Exec (used extensivly)
  • Harmony Hub
  • Ical
  • iRobot
  • Network
  • Onkyo
  • OpenGarage
  • OpenHab Zigbee
  • Open Weather
  • Roku (enabled but not in use yet)
  • System Info
  • Z-Wave
  • ZoneMinder

Extensive Use of HabPanel

I have 9 Lenovo M8 and M10 Tablets running habpanel Viewer to display screens, controls, cameras, etc. I have lots of large images and video streams that run over my wifi network. Some of these image items get stored as OpenHab image items.

Repeatable trigger events: Updating image items too quickly WILL trigger Java Heap OOM events. Example 9 tablets and 12 Zoneminder cameras pushing high res image items to OpenHab every few seconds will normally function correctly. HOWEVER if my wifi is congested or performing poorly, The tablets and camera images will try to update more frequently than they can finish. This leads to a pileup of image items spawning and murdering the java heap.

A hacky work around I have is an exec item that grabs the current max java heap percentage, If it exceeds 50% the rule that updates the images is skipped for an iteration. If the heap exceeds 90% a restart/reload command is sent to my tablets. This clears out any image queues that have pilled up.

If anyone with this problem has lots of .js files, they might like to look at what I just posted on this thread, in case it’s relevant:

Hi all,
Just for information, I figured out the main problem for my memory leaks : ipcamera addon.
The plugin was the cause of tons of threads, file descriptors and memory leaks.
Interresting command also in console : shell:info
I completly removed the bundle, waiting for a serious upgrade. Until this upgrade, I re-code all the stuff of the plugin in the rules …
Perhaps it can help someone else.

4 Likes

maybe @matt1, who is afaik mainting the ipcamera binding, can work together with you tracking down the issues you have noticed in the binding?