Openhab filling up Memory and Swap

No but I can report the inverse: I don’t use Chrome and despite running latest OH code and quite a number of bindings, I do NOT have any memory leakage to report since months.
Maybe a direction worth inspecting into.

Have to correct myself, mem usage did grow significantly in the last 2 days.
Still don’t use Chrome however, so it’s rather unlikely your issue (and mine) is caused by that.

This has not happened for a long time and started showing after I moved from 2.5M1 to 2.5M2, nothing else I’m aware of to have changed significantly.

So that’s all the re-integrated ESH stuff to likely contain a memory leak.

So I used to have major problem with RPI going down once a week due to Java Heap memory. I did increase the swap memory in the java options and after this its been up for 63470minutes. But almost at the same time my wall tablet battery died and I had no Idea that a client running the OH app could make the server memory full. I was really close to switching over to home assistant but since the problem now is gone I will stick with OH until it happens again. For me having a stable system when you rent out the flat is very very very important…

Now that I’ve shutdown my “permanent” clients, OH process size remained the same but resident size dropped significantly while usage on swap increased on about the same order of magnitude.
So there were mem pages the OS identified as not being accessed in a long time and paged them out.
As I wrote this is since I moved from 2.5M1 to M2 and didn’t show up in M1 (my instance ran from almost day1 until M2 was released) so it was introduced by the changes between these two milestones.
Which is somewhat understandable because to re-integrate ESH was a big effort all by itself. Now let’s go bug hunting.

Does anyone know of a way I can persist the Java heap size so I can graph it in Grafana? I need to do something like capture the value of “Current heap size” from “shell:info” on the OH console.

Configure certificate based login to the Karaf console. https://karaf.apache.org/manual/latest/security (create the ssh key pair, create an entry in /var/lib/openhab2/etc/keys.properties

`openhab=<id_rsa.pub key>,_g_:admingroup`

NOTE: The corresponding private key needs to be stored in the .ssh folder of the user that will be executing the command(s). If you will run this from openHAB it needs to be stored in ~openhab/.ssh.

# Current heap size
ssh -p 8101 openhab@localhost shell:info | grep 'Current heap' | cut -d ' ' -f 16

# Maximum heap size
ssh -p 8101 openhab@localhost shell:info | grep 'Maximum heap' | cut -d -f 16

# Committed heap size
ssh -p 8101 openhab@localhost shell:info | grep 'Committed heap' | cut -d -f 14

Run these commands from a Rule that updates openHAB Items that get persisted, or update OH Items using curl from a cron triggered script, or save the values directly into a database using your database's API.

fyi since i changed to my “workaround” (avoiding android app as permanent client) some months ago i didn’t had any memory problems with openhab…
even though i extended the system with new rules/things/items… but i didn’t do any tests with the openhab
android app since then… as far as i can remember there were some updates on the android app some time ago…
Best Regards,
Hans

Hi Gavin - I’m having the same java memory issue and I’d like to try your solution for rebooting - could you please post a quick outline of the rules and how you are monitoring memory usage on the Pi? Could be useful for others!

thanks

Julian

Rule to notify on low memory - uses the standard sysinfo binding. Memory_Available_alert is a virtual numeric value used in the site map to set the point of alert.

    if(Memory_Available.state < Memory_Available_alert.state  as DecimalType && memory_alert === false){
		var msg = "Memory Available (< " + Memory_Available_alert.state + "mb) " + Memory_Available.state + "mb remaining"
		logWarn(ruleName,msg)
		sendPushoverMessage(pushoverBuilder(msg).withUser(pi_pushgavin).withTitle("Openhab").withPriority(0).withSound("cosmic"))												
		memory_alert = true
	}

Rule to reboot the Pi using a virtual switch

rule "Reboot Pi"

when
	Item Reboot_PI changed from OFF to ON
then
	val String ruleName = "Reboot Pi"

	val String msg = "Openhab is Rebooting the PI"
	logInfo(ruleName,msg)
	sendPushoverMessage(pushoverBuilder(msg).withUser(command_pushgavin).withTitle("Openhab").withPriority(0).withSound("cosmic"))									
	Reboot_PI.postUpdate(OFF)
	Thread::sleep(5000) // sleep for 5 seconds

	executeCommandLine('"sudo" "reboot"', 100)

end

BTW - since changing to the Pi4 memory has not been an issue.
You could also combine these two rules to automatically reboot the pi - but I did not like that idea.
I should also mention that there is a flag “memory_alert” set to note that a message has been sent - this is declared at the top of the rule file and reset by a rule that runs at midnight each night. This stops the issue of getting lots of messages.

Out of good old fashioned paranoia, I have my openhab system reboot itself daily. Root has a cron job that just has systemctl reboot.

I did it this way so I could pick an idle tine in the dead of night, rather than having OH reboot when someone might be trying to use it.

Guys, you shouldn’t be rebooting your systems because a proper openHAB doesn’t grow in size once it reached its working set size. If yours do, find the error. If you cannot find it, downgrade to the latest version that does not have this problem.
BTW, “reboot” does not make use of shutdown scripts, so you’re seriously putting the integrity of your system at risk. This is known to e.g. break zram.

1 Like

Thanks Gavin that’s perfect. I’ll probably just use the first rule to start with to keep tabs on the memory usage. I’ve moved my in wall tablets off the android app to habpanel. Apparently the app has a memory leak which may be my issue.

Assuming you refer to openHAB java process ever-growing, that wouldn’t be the app (to run on your phone) but the REST API on server side.

Yes sorry Markus should have been more precise! The problem is on the server - I’ve run memory analyzer and I get a suspect leak

3,973 instances of “org.eclipse.jetty.io.SocketChannelEndPoint” , loaded by org.eclipse.jetty.io occupy 181,587,432 (73.71%) bytes.

Please file that issue on GitHub to openhab-core and provide the details there, too, that developers need to fix it such as bindings you use and the full output of your memory analysis.

It would be best to first confirm that this is an issue in 2.5M2.

What version of openHAB are you running. And can you check which version of jetty-io is active?

I didn’t add the reboot to prevent memory leaks. I added it out if general paranoia and the desire to have a stable system without fuss. It is something I do with most long-running processes.

Also, I stated I used systemctl to do the reboot, and it clearly, visibly, runs the shutdown scripts. I am surprised that a plain “reboot” does not, as it should be triggering systemctl as well. That difference seems unexpected and keeps me further away from zram, as there is clearly an unexpected risk there.

Do as you like. For me (as a Cloud architect/server engineer) and our customers that would be inacceptable, but you’re the customer here.

Well so was I and in fact learned meanwhile that I was at least partially wrong.
In latest openHABian, i.e. buster based, reboot/halt actually do trigger shutdown scripts, so that’s safe to use.

Well, as I said you’re the customer. But given that NOT to run zram is a well-known risk, too, I’d reconsider my decision if I was in your place.

Hi here are the versions:

openHAB Distribution Version Information
----------------------------------------
build-no        : Release Build
online-repo     : https://dl.bintray.com/openhab/mvn/online-repo/2.4

Repository        Version
----------------------------------------
openhab-distro  : 2.4.0
smarthome       : 0.10.0.oh240
openhab-core    : 2.4.0
openhab1-addons : 1.13.0
openhab2-addons : 2.4.0
karaf           : 4.2.1

org.eclipse.jetty.server:9.4.11.v20180605]

##        Ip = 192.168.0.14
##   Release = Raspbian GNU/Linux 9 (stretch)
##    Kernel = Linux 4.14.98-v7+
##  Platform = Raspberry Pi 3 Model B Plus Rev 1.3
##    Uptime = 1 day(s). 0:48:6
## CPU Usage = 1.79% avg over     4 cpu(s) (4 core(s) x 1 socket(s))
##  CPU Load = 1m: 0.64, 5m: 0.56, 15m: 0.43
##    Memory = Free: 0.09GB (10%), Used: nullGB (90%), Total: 0.95GB
##      Swap = Free: 0.09GB (100%), Used: 0.00GB (0%), Total: 0.09GB
##      Root = Free: 11.60GB (83%), Used: 2.30GB (17%), Total: 14.53GB
##   Updates = 31 apt updates available.
##  Sessions = 1 session(s)
## Processes = 114 running processes of 32768 maximum processes