I suspect REST-Api as a memory leak

[Sascha_]

Hi all,

I’m seeing regular OOM with oom-kills of the java process on my raspberry pi 3 with zram.
I’ve already disabled all bindings and rules exept for my arduino-bridge using the REST-API very often.

I’ve collected a heap dump after some time before an OOM and took a look using eclipse memory analyser:

For God’s sake, I can’t see the culprit - just WeakCacheProxy which to me as a Sys Admin and sparetime programmer is off the hook.

Next thing I’m about to try is to set up a new Openhab instance in a VM and hammer it the same way my arduino bridge does. Is that sensible?

I’ll report back in this topic.

Best regards,
Sascha

[J-N-K (Jan N. Klug)]

You better check if the same is true with OH3, there has been quite some work in that area and core-issues will not be fixed for OH2.5.

[Sascha_]

Will do, thanks for the hint!

[splatch (Łukasz Dywicki)]

If you encounter out of memory error removing all the load from the system will not help you finding the issue. Memory profile of Java Virtual Machine is completely different during the load.

First example you have - the WeakCache is unlikely to be a trouble. The weak caches in Java means their entries are removed once they are not necessary. There are also soft caches and some variations with key/values. I do not think it is the issue.

The EMF on other hand might be issue, however if you disabled all the load from bindings and other systems then even parsed rules will get significant amount of memory.

Most often finding OOM is limited to single problem “finding what holds objects in the memory”. It is not necessary the amount of memory (see that String, int[] or long[] take a lot of it), but amount of memory plus references which doesn’t let given object to be evicted from memory. Most often it is some collection which grows over time and does not release own entries.

My suggestion to you is to bring back the load and do heap dump on out of memory error. You can do that with two additional parameters XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=<directory>. Once you will collect two heap dumps you will be able to see which elements retain most of the memory.
I can also recommend you Yourkit which prints a little bit more information than Eclipse Memory Analyzer. One of greatest functions is “paths from GC roots” which allows you to find answer for above question - who holds the object reference. Once you know it finding root cause is much easier.

Cheers,
Łukasz

[Sascha_]

Thank you @J-N-K for the hint with OH3 and @splatch for the detailed hints regarding to debug

I’ve searched for OH3 packages but sadly didn’t find any, so I’ll go Łukasz’s way in the meantime.

Currently, my OH is killed my the OS so java does not automatically write a dump. My Extra-Args were just up until now

EXTRA_JAVA_OPTS="-Xmx512m -Djna.boot.library.path=/usr/lib/jni -XX:+HeapDumpOnOutOfMemoryError"

I’ve just chaned it so that I hopefully now get dumps:

EXTRA_JAVA_OPTS="-Djna.boot.library.path=/usr/lib/jni -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/openhab2"

Using YourKit (thanks for the hint) I’ve seen this in my old dump:

So three questions remain:

1. Where would I find OH3 packages to test for the same behaviour in OH3?
2. Are my EXTRA_JAVA_OPTS now suitable for java to OOM and write a dump?
3. What does the snapshot show exactly?

Thanks and Cheers,
Sascha

[splatch (Łukasz Dywicki)]

The extra args looks fine to me.

With regard to screenshot - the first two rows seems to hold the same data. Take a look on paths from gc roots (the tab you’ve cut) to see where these collections are created.
Yourkit has also additional column (if you scroll left) which shows you “reclaimed memory” or something like that. Try sorting objects by that column. It will show you different elements at the top. Check also paths from GC roots to see if these are managed by OH in some ways.

Best,
Łukasz

[Sascha_]

Thanks for all your tipps!

After going with openhabians /etc/default/openhab2 settings:

EXTRA_JAVA_OPTS="-Xms250m -Xmx350m -Dfile.encoding=UTF8 -Djna.boot.library.path=/usr/lib/jni -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/openhab2"

Everything is fine and RAM usage on a steady base