Something is causing openhab to run out of memory and I cannot figure out what it is. It takes a few days, then the OS (Ubuntu 12.04 on x86_64) kills the java process to get its memory back, to the tune of 4GB resident size.
I used to run openhab for months without ever having this issue, now I’m at 4.5GB of address space after less than a day, so what has changed?
I’m now running a very recent 1.8 build with the InsteonPLM binding (well, that’s my baby so that cannot possibly be buggy) and as of lately the JSR 223 engine.
I ran jmap -dump, and ran jhat on the dump file, but nothing there, maybe 150 megs accounted for.
Then I ran pmap on the java process, and indeed all is anonymous memory, most likely heap.
Is it possible that the JSR engine is causing this? Anybody else seeing something like it? Any suggestions as to how to debug this?
I’m also seeing some strange behaviors in my installation since I installed the jsr223 engine. Many times I see “java.lang.OutOfMemoryError: PermGen space” messages in my openhab.log file and the whole system becomes unresponsive until I restart openhab.
I’m going to disable the jsr223 binding and have a look at the logs, let’s see if anything changes.
I use the Insteon PLM binding with JSR223 and I haven’t seen that issue. I know you have many more Insteon devices than me so that could be one difference. All my rules are JSR223 and I’m running on a Raspberry Pi Model 2 B.
My server has currently been up for 5 days since the last time I restarted it. I do see memory increasing slightly over the week but nothing like you are describing. I think the recent 20 MB memory jump may have been because I added the MQTT binding and set up event bus publishing to experiment with external rule engines.
This is what the resident size (in kilobytes) looks like as a function of time (in seconds), without any python rules running, but with the jsr binding in the addons folder. Ugh.
How can I find out where the memory goes? jvisualvm didn’t give me any clue what’s going on. Are there any other debugging methods? I am running with -Xmx256m -Xms256m. If the memory leak was a regular leak in the openhab java code, shouldn’t that terminate the process well before it reaches a GB of resident size?
Can I ask you what tools are you using to store openhab’s memory consumption? The Systeminfo binding?
Many thanks for your help
Yes, I’m using the Systeminfo binding and saving the data to rrd4j.
Maybe jmap could give some clues?
jmap the first thing I tried, see my first post. It accounted for a small fraction of total memory. All looked benign.
I ran openhab again tonight, this time without the jsr engine in the addons folder. Something is still eating away at the memory.
Will try disabling all rules, xtext and jsr, see if that helps.
Sorry, I overlooked that you had tried jmap already.However, it almost sounds like there’s a leak in native code somewhere given the JVM doesn’t appear to be managing it.
I have had massive problems with java heap space out of memory in Openhab. I wrote about this in this thread:
Switching from 64-bit Ubuntu to 32-bit Ubuntu (and at the same time switch from 64- to 32-bit Java) solved the problem for me.
I do not know the underlying cause for this, but perhaps it could be worth a try for you as well.
I switched off and removed the jsr engine, and disabled all xtext based rules. It’s still leaking a bit, but a fraction of what it was before. That’s first evidence that the leak is somewhere in the xtext rules.
All the while the insteonplm binding was running, so that one does not seem to leak outright.
Next thing will be to renable jsr engine and rules, leaving the xtext rules off.
This now with the JSR engine back in place, insteonplm binding running, but all xtext based rules switched off.
Now it’s leaking about 200 bytes/sec, much less than with the xtext rules switched on.
Looks like neither jsr nor insteonplm are to be blamed. The saga continues…
It’s anecdotal, at best, but this is consistent with the fact that I’m not seeing the memory issues and I use JSR223 and Insteon wtih no Xtext rules.
Now I have everything running except for one complex xtext based rule. Memory consumption stayed flat for most of the day, then jumped this morning at 6:38am. Nothing suspicious in any of the openhab logs around that time. Nothing in syslog either.
I ended up tossing out my gargantuan xtest rule and replaced it with a jython/jsr223 rule. Oh so much easier to write, and that much easier to read. And look, it also took care of the memory leak, sort of:
I just love it when “debugging” means wholesale ditching of one approach and going with another, without having any idea what was wrong with the former, and just praying that the latter will hold up better.
I really wish openhab core and bindings were split up into separate address spaces, interacting via network messages. Would make it easier to localize leaks like that. I’m also pretty disappointed that I was not able to track down memory leaks using jmap, jhat, or other freely available tools like visualvm.