openHAB 3 runs out of memory / java heap space errors, CPU 100%+ after a few hours

Man this was my solution. VS code extension with language server enabled. Over last year or so I fought this, my OH install both 2.x and 3.x would run out of memory “randomly” after weeks of perfect operation. Usually after I made a very minor change or something. I would make the change and minimize vscode, within hours OH was dead, and I would piss around with it for a few days rebooting and lockingup and trying to figure out WHY things changed. I would get frustrated and give up closing VS code, and then boom it worked. Disable language server option, and no more problems.

This needs to be shouted louder, I dunno now many other nerds are playing with the VS Code extension and also chasing this frustrating mystery bug.

1 Like

Presumably just closing your VSC window when finished with it is just as effective?

In my case, just closing VS Code when finished is not a solution. Once I use VS Code with Languageserver enabled, it looks like OpenHAB will eventually run into problems. I used to code in the evening and then closed VS Code when going to sleep. Then I noticed that the “put house in sleep mode” rules did take a while to process. The morning after, the “house wakeup” rules sometimes failed. To prevent this, I restarted my RPi every time after I used VS Code. Then OpenHAB runs fine and rules are processed immediately when triggered.

Good info. So whatever gets messed up, stays messed up. On the face of it, a suspected memory leak in LSP. Given you’ve got more than just vague suspicions, would you be able to open a github issue on core for this?

For me it’s even more weird, dunno if it is related to this bug discussed here, but sine OH3 many of my rules file go crazy if I edited them (in VSCode). E.g. files with statically initialized variables like Maps lost their initialization and hence rules don’t work anymore until I restart OpenHAB. Very hard to experiment with rule changes with a behavior like this. Or log outputs are just not printed, and “System started” rules are not executed.
I have the feeling that complete rule DSL became very fragile with OH3.
And no, rossko57, I can’t be more specific as behavior is very erratic and I haven’t managed to find a safe way to reproduce, otherwise I would have filed a bug.
But maybe there are also others who experienced this, seems that I’m not alone with these issues.

Are you aware that each time you save a rule file all rules get reparsed/reinitialized?
It’s deterministic but easily looks to the user as erratic. FWIW that is not new in OH3.

Figure out at least what your problem is and when it starts happening please. Noone can act upon reports like these and they’re not helpful as long as they are as vague as this, lots of OH users might think it’s their problem while in fact it’s not but a different one. Not helpful at all in tracking things down so please be specific and precise in your reports.

There is also a uhh, unexpected behaviour (I’d say a bug) new to OH3 where editing an Items file causes linked rule triggers to stop working.

Your “crazy” symptoms might or might not be related to either of these conditions, no-one else but you can say because you’d need to be aware of actions leading up to it.

First I will look for a way to objectify and reproduce this and be more specific.

1 Like

It would be if I wasn’t forgetful, and/or drawn away in the middle of playing with OH for something my SO deems a better use of my time. :slight_smile:

You should see my taskbar pc a week after a forced windows update reboot, I almost never close any program. lol. But, OH3 doesn’t mind that, it just runs fine with Language server off.

Small tip from my side. Don’t save files to short after eachother, especially items files. It’s not new, OH2.x had the same ‘issues’. Not completely sure but when you save a large Item file, my guess is everything ‘attached’ to that file is compiled again (makes sense). I sometimes make this ‘mistake’ and a raspberry pi doesn’t like that.

I break all my items up into smaller files, because I’m obsessed with being organized.

Reproduces also for me:

2021-04-27 23:19:53.497 [WARN ] [ab.core.internal.events.EventHandler] - Dispatching event to subscriber 'org.openhab.core.io.monitor.internal.EventLogger@da0e20' takes more than 5000ms.

on 3.0.1 and when reproduces “VM Thread” consumes 100% of single core all the time.
Hard to tell what triggers it, as for me it’s sometimes happening every few days and sometimes after month.
I already have increased HEAP for Java:

EXTRA_JAVA_OPTS="-Xms400m -Xmx420m"

in /etc/default/openhab that was solving this issue on OH 2.x, but not helping with 3.x.

I had almost the same issue, everyday at round about 4 o‘clock my OH run out of memory.
Almost tried out everything even plain installation and than installed my backup.
Still a crash every night.
Then I went through my rules file and found out I had linked an item which does not exist and is triggered with many other items when my heating starts at 4 o‘clock. Since I repaired this it runs for more than two weeks without errors!

I am experiencing this myself with OH 3.1.0.M4 (docker). High CPU load, OH3 becoming more and more unresponsive resulting in “java heap space” log entries.

I am using DSL as well and had a situation where OH3 ran smoothly until I started VSC with language server enabled. CPU load went up immediately and never came down again to a normal level. Since then I disabled the language server feature in VSC and all is good now.

Did anyone file an issue already?

An issue is opened:

Hi Falk,
Can you point me to a description how I can disable the language server? Can’t find it, just found threads where they explained it can’t be disabled, just stopped via console.
Best regards, Stefan D.

Here it my memory usage:

Is there any way to log the memory usage for some longer time? I try to find some correlation with other events.

Sure, in Visual Studio Code got to “extensions (german: Erweiterungen)”, right klick openHAB and chose “extension setup (german: Erweiterungseinstellungen)”.
Scroll down to “Languageserver: Remote Enabled” and disable the function.

Edit: There may be another option which I have not tried in detail so far and should disable the language server entirely:

in openhab cli type

bundle:list | grep Language

This will give you the ID of this component. Now type:

bundle:stop ID

This should disable the language server on OH side. Not sure about potential side effects …

Thank you so much for that info! I set the same value in this file (running OH3.0.2 on RPI 4 8GB) and since then my system is running stable without any performance problems. System is up for 3 days now.
In the past I had to restart almost every day.

Hi all,
Since upgrade from 3.0 to 3.1 I have strong memory leak.
In 3.0 the system was very stable (running for weeks/months).
Now in 24h-48h I run into OOM (raspberry pi 4Go dedicated).
At 2go for OH3 java process, i have many error messages (rrdb, etc.) and Openhab hangs.
I tried EXTRA_JAVA_OPTS="-Xms1024m -Xmx2048m" but changed nothing.
I am trying to disable addons one by one and comment most of my rules but it is very long and fastidious.
Is there a method to see from which plugin(s) or rule(s) comes this memory leak ? Or Linux system commands ?
Currently I run “ps -p xxx -o rss=” and see the memory size increasing constantlty … until freeze or restart/reboot.
Many thanks for any tip/hint or info !

1 Like