Openhab filling up Memory and Swap

Ok, so yesterday it got me too.
I’ve created a dump with dev:dump-create and a ZIP got created. Then I downloaded Eclipse Memory Analyser Tool (MAT) for Mac but then it requested me to install the old Java 6 (WTF?) so I gave up.

Can anybody do something with my dump to get this memory leak fixed or should I keep it for security reasons?

Thanks,
Sascha

I have this issue too. I have never had an OH setup that was more stable that staying up for a few days - I thought this was normal!
I have a rule setup that warns me when available memory gets below a set point - I then reboot the pi (before it dies).
I find it difficult to understand how an issue like this has gone un-fixed for years (while at the same time claiming that OH is stable)!

I’m sorry that I don’t have the skills to help resolve this, but in my opinion, getting OH to be truly stable should be a priority.

A complex matter, here’s some simple answers: 1) there’s only very few people to have this problem 2) no user has been providing the input needed to debug this (noone has tracked it down to use of a particular binding or whatever component which is required that a developer could start looking into) … and (well, sorry) 3) many people like you only ever complain but don’t think of helping instead.

This is also often misunderstood. Sure there can be real memory leaks at times, but most of the time there’s default heap size limits in use which if reached make Java terminate (“crash”).
But regular use with a fair amount of bindings can already require more memory than this limit will allow for.
That’s not a OH problem but a user problem then - use of inadequate system settings.
BTW: the default maximum heap on ARM/Zulu is 256M.
Anyone (and in particular anyone to have a ‘crash’ problem) should raise the limit rather than shout at OH.

Ok, I hear what you are saying. Given that I don’t have the skills to debug the issue, what can I do to help someone who does have the skills, find the solution?
I am very willing to help in any way I can. If it is a case of providing dumps, or other things to help track it down - I just need to know.
I’m not a linux person, so I simply downloaded openHABian image and went to it from there. If there are settings that need to be changed to help make the system stable, please let me know and I will do it. Or at least point me in the direction.
I don’t honestly think we are doing OpenHAB any favours by saying “it’s not an OpenHAB problem, it’s a user problem”

1 Like

Gavin:
Not all of us have the skills to debug this type of very complex issue, no worries! Christoph suggested some ways a user could help but my eyes glazed over about half way thru reading his suggestions, they are beyond someone who isn’t a Linux ‘power user’ or developer. But even just reporting an issue is useful to the developers and a small contribution. As a community this is our strength.

*shrug* well it is like it is.
The major part of the problem is to properly describe when it happens and to dissect it.

There’s the openHAB core, and there’s the specific bindings, actions and eventually specific config a user runs (an infinite number of combinations), and there’s the system (OS) this all runs on.
Potentially there’s memory leaks in bindings and the core/API. It’s the user’s task to nail them down on his machine, with his config. No developer can do that for you because every OH installation is unique so the leak usually only shows up in your combination of bindings and config and does not show if someone else tries to reproduce it.

You say you already use openHABian so check and eventually raise the limit (the EXTRA_JAVA_OPTS line in /etc/default/openhab2).
That should at least increase the time it takes until the limit is reached and Java crashes.

To find the leaking entity, you need to run OH with a reduced set of bindings and functionality and gradually add more of them (no need to restart, just change config via PaperUI or files).
DO ALL OF THIS WHILE WATCHING memory consumption (with tools like htop or by generating memory dumps every now and then) until you notice a substantial increase in memory use over time (without actions of yours that might explain this). Then your last change probably includes the component to have a memory leak, at least these are the candidates you should have a closer look at next. Try running with and without them, try identifying actions (device discovery, API calls you made, …? ) that result in increased memory footprint.
Once you identify a component, config or action to show this problem, you can open a Github issue and we (ideally) can assign this to the component’s developer. But take care to well describe the exact conditions that need to be present to make the problem show up.

2 Likes

Thanks for your advice. I have been, and continue to record memory use regularly. I am also working my way through bindings and add-ons in an effort to figure out what, or what combinations are causing the issue.
It has also been suggested that I increase the swap (currently set at 100M), do you think that is a good plan?

yes, having more swap ready never hurts. With just 100M you eventually already hit the virtual process size limit. Make it 2xRAM size.

So far I have not seen anything posted that describes this is the same as the OP. It is probably better you start your own thread unless it is caused by the android app running non stop and the issue does not occur if you use basicui, see post

I believe playing withswap is a waste of time, not going to hurt but unlikely to help. See Markus’ previous post about java option. Increase the heap / stack size and see if the problem takes longer to occur or is fixed.

Hey,

I’m no expert and I had the same issue. What I’ve done is overclock my raspberry pi added cooling with a fan and heat sinks. My system is now up for more then 6h and it stay’s stable my swap mem has not gone up and my mem has maxed ad 56% even when i programming in studio or connecting with ssh.I hope it stays this way! Keep you posted :slight_smile:
I have a RPI 3b OH2.4.
my overclock specs

arm_freq=1350
core_freq=500
sdram_freq=550
sdram_schmoo=0x02000020
over_voltage_sdram_c=4
over_voltage_sdram_p=6
over_voltage_sdram_i=4
over_voltage=4

I don’t recommend anyone overclocks. If you OC the sd card and it can not cope, it will silently corrupt your data and then you will get crashes… Overclocking will not change anything to do with running out of memory and can only raise your odds of crashing. If you need more CPU, ram or faster uSD then go and buy a Pi4 or a faster x86 based system.

2 Likes

I second. A Home Automation server has to be 1) stable, 2) stable and 3) stable and 4) you shouldn’t run anything else on there.
If 4) a RPi is way fast enough to run openHAB. So remove overclocking. It doesn’t add significant value and is potentially harmful to stability. And move any other programs to a separate HW.

1 Like

Perhaps somebody needs to drill that in to the devs over at Home Assistant :smiley:

Hi,
My pi runs openhabian and openhabian only with an Mqtt server mosquito, so that isn’t the problem. And like I said in the beginning i’m Not an expert I only try to solve my problem. I had to reboot my pi after 3 or 4 days because of the memory. Also when I deleted every rule except for one to turn on and off a light or after a clean install or 2.
That I didn’t had to overclock Ok, but the is my test and if it breaks so be it… I bought a pi 3b+ and a 4b so I going to keep on testing and hope that it will go on for years without rebooting :slightly_smiling_face:.

If you have a memory leak see my post. If you don’t there’s no need to reboot. Eventually adjust Xms/Xmx settings.

I recommend to move as much out of the jvm as possible. Use the mosquitto c++ mqtt broker instead of the java embedded one. Use c++ deconz instead of java zigbee. Use v8/javascript nodered instead of the java rule engine. Use c++/rust/go based mqtt bridges for other protocols.

Cheers, David

This begs the question, then what is left? MQTT replaces the event bus. There are no bindings. No Rules. So OH is just a UI and maybe persistence? With an approach like this, what value does OH bring?

1 Like

The value of openHAB are the addons, not the core IMO. I once thought different, but tbh, my openHAB crashes often because of memory leaks. I have seen almost all of core code. A lot of leaks and high memory consumption is caused by dependencies, not openHAB code itself. (Think of xtend, jupnp, the old rest interface library we use and so on.)

But the openHAB code is also spaghetti sometimes and not well understood. As seen by the recent huge memory leak that was unintentionally caused by Markus patch to fix a Thing-always-offline bug.
A binding can nowadays register actions for the NRE. I scratched my head when I saw the corresponding code in core.

It would be so much easier if openHAB would be a composition of different processes and we could exactly pinpoint where memory goes and cpu time is spend.

Cheers, David

Easier for developers? Yes. Easier for testers? Yes. Easier for average users? Unless all of this composition of separate processes is accomplished completely transparently, no.

1 Like

But Rich: It is. Docker compose for example. Everything is started up, isolated, secure with a single command.