Does anyone know of a way I can persist the Java heap size so I can graph it in Grafana? I need to do something like capture the value of “Current heap size” from “shell:info” on the OH console.
Configure certificate based login to the Karaf console. https://karaf.apache.org/manual/latest/security (create the ssh key pair, create an entry in /var/lib/openhab2/etc/keys.properties
`openhab=<id_rsa.pub key>,_g_:admingroup`
NOTE: The corresponding private key needs to be stored in the .ssh folder of the user that will be executing the command(s). If you will run this from openHAB it needs to be stored in ~openhab/.ssh.
# Current heap size
ssh -p 8101 openhab@localhost shell:info | grep 'Current heap' | cut -d ' ' -f 16
# Maximum heap size
ssh -p 8101 openhab@localhost shell:info | grep 'Maximum heap' | cut -d -f 16
# Committed heap size
ssh -p 8101 openhab@localhost shell:info | grep 'Committed heap' | cut -d -f 14
Run these commands from a Rule that updates openHAB Items that get persisted, or update OH Items using curl from a cron triggered script, or save the values directly into a database using your database's API.fyi since i changed to my “workaround” (avoiding android app as permanent client) some months ago i didn’t had any memory problems with openhab…
even though i extended the system with new rules/things/items… but i didn’t do any tests with the openhab
android app since then… as far as i can remember there were some updates on the android app some time ago…
Best Regards,
Hans
Hi Gavin - I’m having the same java memory issue and I’d like to try your solution for rebooting - could you please post a quick outline of the rules and how you are monitoring memory usage on the Pi? Could be useful for others!
thanks
Julian
Rule to notify on low memory - uses the standard sysinfo binding. Memory_Available_alert is a virtual numeric value used in the site map to set the point of alert.
if(Memory_Available.state < Memory_Available_alert.state as DecimalType && memory_alert === false){
var msg = "Memory Available (< " + Memory_Available_alert.state + "mb) " + Memory_Available.state + "mb remaining"
logWarn(ruleName,msg)
sendPushoverMessage(pushoverBuilder(msg).withUser(pi_pushgavin).withTitle("Openhab").withPriority(0).withSound("cosmic"))
memory_alert = true
}
Rule to reboot the Pi using a virtual switch
rule "Reboot Pi"
when
Item Reboot_PI changed from OFF to ON
then
val String ruleName = "Reboot Pi"
val String msg = "Openhab is Rebooting the PI"
logInfo(ruleName,msg)
sendPushoverMessage(pushoverBuilder(msg).withUser(command_pushgavin).withTitle("Openhab").withPriority(0).withSound("cosmic"))
Reboot_PI.postUpdate(OFF)
Thread::sleep(5000) // sleep for 5 seconds
executeCommandLine('"sudo" "reboot"', 100)
end
BTW - since changing to the Pi4 memory has not been an issue.
You could also combine these two rules to automatically reboot the pi - but I did not like that idea.
I should also mention that there is a flag “memory_alert” set to note that a message has been sent - this is declared at the top of the rule file and reset by a rule that runs at midnight each night. This stops the issue of getting lots of messages.
Out of good old fashioned paranoia, I have my openhab system reboot itself daily. Root has a cron job that just has systemctl reboot.
I did it this way so I could pick an idle tine in the dead of night, rather than having OH reboot when someone might be trying to use it.
Guys, you shouldn’t be rebooting your systems because a proper openHAB doesn’t grow in size once it reached its working set size. If yours do, find the error. If you cannot find it, downgrade to the latest version that does not have this problem.
BTW, “reboot” does not make use of shutdown scripts, so you’re seriously putting the integrity of your system at risk. This is known to e.g. break zram.
1 Like
Thanks Gavin that’s perfect. I’ll probably just use the first rule to start with to keep tabs on the memory usage. I’ve moved my in wall tablets off the android app to habpanel. Apparently the app has a memory leak which may be my issue.
Assuming you refer to openHAB java process ever-growing, that wouldn’t be the app (to run on your phone) but the REST API on server side.
Yes sorry Markus should have been more precise! The problem is on the server - I’ve run memory analyzer and I get a suspect leak
3,973 instances of “org.eclipse.jetty.io.SocketChannelEndPoint” , loaded by “org.eclipse.jetty.io” occupy 181,587,432 (73.71%) bytes.
Please file that issue on GitHub to openhab-core and provide the details there, too, that developers need to fix it such as bindings you use and the full output of your memory analysis.
It would be best to first confirm that this is an issue in 2.5M2.
What version of openHAB are you running. And can you check which version of jetty-io is active?
I didn’t add the reboot to prevent memory leaks. I added it out if general paranoia and the desire to have a stable system without fuss. It is something I do with most long-running processes.
Also, I stated I used systemctl to do the reboot, and it clearly, visibly, runs the shutdown scripts. I am surprised that a plain “reboot” does not, as it should be triggering systemctl as well. That difference seems unexpected and keeps me further away from zram, as there is clearly an unexpected risk there.
Do as you like. For me (as a Cloud architect/server engineer) and our customers that would be inacceptable, but you’re the customer here.
Well so was I and in fact learned meanwhile that I was at least partially wrong.
In latest openHABian, i.e. buster based, reboot/halt actually do trigger shutdown scripts, so that’s safe to use.
Well, as I said you’re the customer. But given that NOT to run zram is a well-known risk, too, I’d reconsider my decision if I was in your place.
Hi here are the versions:
openHAB Distribution Version Information
----------------------------------------
build-no : Release Build
online-repo : https://dl.bintray.com/openhab/mvn/online-repo/2.4
Repository Version
----------------------------------------
openhab-distro : 2.4.0
smarthome : 0.10.0.oh240
openhab-core : 2.4.0
openhab1-addons : 1.13.0
openhab2-addons : 2.4.0
karaf : 4.2.1
org.eclipse.jetty.server:9.4.11.v20180605]
## Ip = 192.168.0.14
## Release = Raspbian GNU/Linux 9 (stretch)
## Kernel = Linux 4.14.98-v7+
## Platform = Raspberry Pi 3 Model B Plus Rev 1.3
## Uptime = 1 day(s). 0:48:6
## CPU Usage = 1.79% avg over 4 cpu(s) (4 core(s) x 1 socket(s))
## CPU Load = 1m: 0.64, 5m: 0.56, 15m: 0.43
## Memory = Free: 0.09GB (10%), Used: nullGB (90%), Total: 0.95GB
## Swap = Free: 0.09GB (100%), Used: 0.00GB (0%), Total: 0.09GB
## Root = Free: 11.60GB (83%), Used: 2.30GB (17%), Total: 14.53GB
## Updates = 31 apt updates available.
## Sessions = 1 session(s)
## Processes = 114 running processes of 32768 maximum processesHi no I’m on still on 2.4 - should I upgrade?
I suggest to at least test to see if the issue still occurs in 2.5M2 before reporting in GH. There have been a lot of changes and fixes since 2.4… and IIRC the issue you reported has been addressed.
Last night, with the very patient help of @5iver, I managed to graph the current heap size. I’d previously had the java settings set to 3GB and 6GB:
EXTRA_JAVA_OPTS="-Xms3072m -Xmx6144m"
I reduced them to 1GB and 2GB:
EXTRA_JAVA_OPTS="-Xms1024m -Xmx2048m"
for testing.
This afternoon I noticed that the automatic lighting in the house had stopped working so I checked the heap size graph and saw that the java heap had grown hugely at 13:30:
I restarted the OH service at about 14:10. The strange thing is is that I can’t think of anything significant that occurred at 13:30. I’ve had a nose around openhab.log and events.log and didn’t spot anything obvious.
Maybe I could work on a rule that checks if the heap size has grown 50% above the average of the last 10 minutes and if so creates a heap dump? Does anyone have suggestions?
1 Like
Take a look at… https://github.com/openhab/openhab-core/issues/829#issuecomment-499032957. You might also get some use out of jconsole.