Something is causing openhab to run out of memory and I cannot figure out what it is. It takes a few days, then the OS (Ubuntu 12.04 on x86_64) kills the java process to get its memory back, to the tune of 4GB resident size.
I used to run openhab for months without ever having this issue, now I’m at 4.5GB of address space after less than a day, so what has changed?
I’m now running a very recent 1.8 build with the InsteonPLM binding (well, that’s my baby so that cannot possibly be buggy) and as of lately the JSR 223 engine.
I ran jmap -dump, and ran jhat on the dump file, but nothing there, maybe 150 megs accounted for.
Then I ran pmap on the java process, and indeed all is anonymous memory, most likely heap.
Is it possible that the JSR engine is causing this? Anybody else seeing something like it? Any suggestions as to how to debug this?
I’m also seeing some strange behaviors in my installation since I installed the jsr223 engine. Many times I see “java.lang.OutOfMemoryError: PermGen space” messages in my openhab.log file and the whole system becomes unresponsive until I restart openhab.
I’m going to disable the jsr223 binding and have a look at the logs, let’s see if anything changes.
I use the Insteon PLM binding with JSR223 and I haven’t seen that issue. I know you have many more Insteon devices than me so that could be one difference. All my rules are JSR223 and I’m running on a Raspberry Pi Model 2 B.
My server has currently been up for 5 days since the last time I restarted it. I do see memory increasing slightly over the week but nothing like you are describing. I think the recent 20 MB memory jump may have been because I added the MQTT binding and set up event bus publishing to experiment with external rule engines.
This is what the resident size (in kilobytes) looks like as a function of time (in seconds), without any python rules running, but with the jsr binding in the addons folder. Ugh.
How can I find out where the memory goes? jvisualvm didn’t give me any clue what’s going on. Are there any other debugging methods? I am running with -Xmx256m -Xms256m. If the memory leak was a regular leak in the openhab java code, shouldn’t that terminate the process well before it reaches a GB of resident size?
Sorry, I overlooked that you had tried jmap already.However, it almost sounds like there’s a leak in native code somewhere given the JVM doesn’t appear to be managing it.
I switched off and removed the jsr engine, and disabled all xtext based rules. It’s still leaking a bit, but a fraction of what it was before. That’s first evidence that the leak is somewhere in the xtext rules.
All the while the insteonplm binding was running, so that one does not seem to leak outright.
Next thing will be to renable jsr engine and rules, leaving the xtext rules off.
Now I have everything running except for one complex xtext based rule. Memory consumption stayed flat for most of the day, then jumped this morning at 6:38am. Nothing suspicious in any of the openhab logs around that time. Nothing in syslog either.
I ended up tossing out my gargantuan xtest rule and replaced it with a jython/jsr223 rule. Oh so much easier to write, and that much easier to read. And look, it also took care of the memory leak, sort of:
I just love it when “debugging” means wholesale ditching of one approach and going with another, without having any idea what was wrong with the former, and just praying that the latter will hold up better.
I really wish openhab core and bindings were split up into separate address spaces, interacting via network messages. Would make it easier to localize leaks like that. I’m also pretty disappointed that I was not able to track down memory leaks using jmap, jhat, or other freely available tools like visualvm.
