Java runs amok weekly

I ran openhab 1.x and 2.x a number of years on a Mac mini (indeed one of less obvious platforms).
In general openhab ran quite stable.
Certain upgrades/patch situations did produce sometimes errors that were fixed with reinstall, clean cache, etc.
However I don’t think I ever faced your fan / 200% cpu issue.
But indeed if you see a storm of warnings and/or errors it might end up with high memory usage, triggering java garbage collection.
As you are on a Mac you can easily increase such java memory settings normally.
I do hope your Mac mini itself is not totally frozen because only Java cannot do / provoke this as it will only use one cpu I think ( assuming you have quad core - 8gb). The only way I got a Mac hanging was when it nearly had no disk space ( eg a time machine or an camera storage eating your total space and I did not set disk quota ).
Meanwhile I migrated since 1 year to a RPI 3b+.

Hi Peter, thanks for your input. Actually one can’t easily get at any Java settings anymore on Mac…the control panel, which would have allowed the adjustments you mentioned re stack size etc, has been removed since before OS Mojave. Also, The Java Plugin and Java WebStart technologies that were deprecated in JDK 9 and marked as candidates for removal in JDK 10, have now been removed. So more fiddling around… :roll_eyes:
No the Mac never froze under the Java run-away situation so that’s good. Anyway you’ve gone the way of what seems the majority of users and migrated to a RPi. I didn’t seem to fit in and feel the glowing invitations upfront of Openhab being Multiplatform evaporated when it was intimated that I was an oddity with scant hope of solutions and should really get with the plan and use a RPi… raised eyebrows and tut-tutting’ ?.. To be honest, I feel that robust software shouldn’t need it’s own dedicated space/device like this and should be able to run/live harmoniously on most powerful machines (Mac, PC, etc) along with the other applications…if one has to ‘Molly-coddle’ a program like this, then I may as well revert back to a dedicated hub like I had before… Fibaro and Vera. I honestly tried this migration to openhab to escape the ‘dedicated hub’ paradigm! Anyway, I may now try the ‘Home Assistant’ opensource application software to see how that performs…
Thanks again for you interest,
Denis

Hi Denis,

Openhab is build on Java, so it’s effectively multi-platform/device.
Openhab is also open source so yes, you can encounter issues that are related to a specific version but also to your specific configuration.
I really think there is no reason for you to leave your MacOS platform. It ran quite fine for me.
Of course as this OS has probably less openhab users, less people can help you.

What is my underlying story?
My son build for school also his own KNX automation project using openhab where he used a RPI. So we had to learn the PI.
He used ultrasonic devices to measure the water level, and that was reasonably easy using the I/O of the RPI and a small python script. Such I/O pins are not available on a Mac mini (ok unless you put some I/O via USB…). And last but not least the cost was important, my son had a limited budget.
So my son convinced me to switch… but I was really satisfied with the stability on the Mac.
We also had some difficult times on the PI at the start. Sometimes related to a specific bug, sometimes related to things on the PI itself, sometimes things in the configuration (UI versus text files) that got somehow messed up, etc. That is the glory & joy of open source :grinning:
One tip, if you consider moving your config from a Mac to a PI: doing a backup via the zip file and restoring it on the PI does not work well (some paths are not translated correctly - should have created a bug for that :roll_eyes:).

Coming back to your question. You can modify the Java settings (although I only tried this a few times). In the bin directly where the Karaf is, there’s a setenv file, containing the java options.
You could extend this with specific memory size params if needed:

set java options

export JAVA_OPTS=“${JAVA_OPTS}
-Dopenhab.home=${OPENHAB_HOME}
-Dopenhab.conf=${OPENHAB_CONF}
-Dopenhab.runtime=${OPENHAB_RUNTIME}
-Dopenhab.userdata=${OPENHAB_USERDATA}
-Dopenhab.logdir=${OPENHAB_LOGDIR}
-Dfelix.cm.dir=${OPENHAB_USERDATA}/config
-Djava.library.path=${OPENHAB_USERDATA}/tmp/lib
-Djetty.host=${HTTP_ADDRESS}
-Djetty.http.compliance=RFC2616
-Dorg.ops4j.pax.web.listening.addresses=${HTTP_ADDRESS}
-Dorg.osgi.service.http.port=${HTTP_PORT}
-Dorg.osgi.service.http.port.secure=${HTTPS_PORT}”

set JVM options

ARCH=uname -m
EXTRA_JAVA_OPTS_COMMON=“-Djava.awt.headless=true”
EXTRA_JAVA_OPTS_ARCH=“”
case “$ARCH” in
arm) ;;
aarch) ;;
*) EXTRA_JAVA_OPTS_ARCH=“-XX:+UseG1GC” ;;
esac
export EXTRA_JAVA_OPTS=“${EXTRA_JAVA_OPTS_COMMON} ${EXTRA_JAVA_OPTS_ARCH} ${EXTRA_JAVA_OPTS}”

I would be however surprised that this is really the issue if you have a small configuration.
Do you use specific add-ons, bindings, use complex rules?
The log file in debug mode can reveal often an issue, lead to the root cause.
Also often a clean-cache and restart can help.

For info, on the pi, I am still running a stable release openHAB 2.5.4-1.

Peter

I have to laugh at this because ‘Home Assistant’ recently announced dropping support for all devices except pi, n2 and Nuc outside a VM. Basically they are trying to push for limited hardware choice that runs their own OS. The people using the N2 report that is is buggy in the last 3 major recent releases, and now I see that device has had the ‘recommended’ tag removed. So that basically now means it only runs on a PI or in a VM if you follow the main recommendation.

Whilst openHAB works on multiple platforms, and many different devices it will always be true of any opensource project that the most trouble free way is where the majority of users are. Most testing done, most bug reports and most people that can provide help. I’m sure you will fix this issue, but you will need to do some testing by disabling bindings and rules to narrow the cause down. Once you can say it is X feature causing the problem it will be easier to help you out.

Hi Skinah,
I didn’t know that… thanks for the heads up… you’ve probably saved me a lot of disappointment so I probably shouldn’t jump ship yet… :thinking: Apparently I can install Python and ‘Virtualenv’ (Virtual Environment) to handle version differences on my MacMini (using the Terminal) and see. I can run any Virtual OS on the Mac using ‘Parallels VM’ if necessary. Anyway this is off topic, sorry…there’s definitely a general love affair with the Rpi :grin: given its low cost, power and small form factor etc but I still like the elegance of recycling an older (still powerful) Mac. Indeed you are correct that Opensource is a bumpy road and unfortunately very few know how to change a wheel when the Mac gets a flat…
I’ll follow your suggestions and update the forum if/when I discover something.
Thanks for your detailed and helpful reply… people are very generous with their time.
Denis

I don’t think your issue will be Mac related, if you moved your same setup to a pi and Linux I would not be surprised if it did the same thing. Until the cause is narrowed down it is silly to point fingers and talk about the issue which at this point is not known.

I understand your frustrated and hopefully my advice Using shell:info or the logs turn up a way you can see the fault happening so you don’t need to wait a full week for the issue to occur.

I indeed run openHAB on a MacMini from 2011 and it works perfectly on it.
There are rare occasions where I see a huge CPU load as well though and I believe to have tracked that down to JVM updates becoming available - whenever my Mac pops up a window asking for updating the (Java8) JVM to the latest patch level, I sometimes/often saw such issues. As Java8 is out of maintenance by now, this problem should no longer occur, though…

Hi Ross,
Java went wild between 9 pm last night (my time) and 8 am this morning.
NOTHING shows in logs which are in ‘debug mode’.
Karaf screenshots of Normal and Run-amok mode attached. Thousands of extra threads running and Garbage collection seems very busy. Are these clues…?
Logs:




regards,

Hi Skinah,
I have posted details of current run-amok mode. You mentioned ‘Garbage Collection’ originally. What do you think?
Thanks and regards,

What’s that HABpanel start and stopping every minute about? Is that normally happening for the rest of the day?

Hi Ross, no that was just me working on habpanel, making changes and resyncing to my iPad to see how the UI was shaping up. I finished working on it around 9 and left the Mac alone then. I’ve only been trying out Habpanel recently and the previous Java episodes happened without any Habpanel running so I’m not convinced it’s the fault of Habpanel.
Also, it was impossible to ‘logout’ of Openhab on the Karaf this morning…it just ignored me. I had to ‘force quit’ Java process from the Unix Terminal and also trashed the ‘tmp’ and the ‘cache’ folders. Openhab still wouldn’t relaunch because 2 instances of Java then started up on the next launch! Eventually, I had to restart the Mac and now everything is fine again…until the next Java episode… :roll_eyes:
Thanks for your continued interest…

It all looks fine to me except your Java version is 252 and the latest one for macOS is:

8u265b11
Zulu: 8.48.0.53

See Kai’s reply above, I think you should update your Java.

The running threads are not out of control and the heap size is normal. The garbage collection has only used 29 seconds of the 5 days to clean up so it is also not out of control.

Hi Skinah,
Thanks for that assessment. Basic question I know…but I’m not sure about the Java update procedure here… presume I logout/quit openhab… is the update on the Openhab resources page and can it overwrite/update the previous Java without uninstall… don’t want to make things worse…
UPDATE: Figured it out. All updated to latest versions as you recommended. Thanks for your help. We’ll see now what happens this week :slightly_smiling_face:

Hello again Skinah…and everyone else on this topic, Java gone wild again between late last night and this morning. Running at 128% CPU with fans blasting. Updated to latest Java stuff yesterday. Logs show nothing at all unusual. I’m starting to despair… :frowning:
Shell and log screen shots attached. Anybody any new insights…? Thanks to all so far.

May we see your events.log for he period?

Is that HABpanel activity just you working again?

Hi Ross,
Yes that was me messing about with Habpanel. Events log is rather long. Copied into a text file with ‘txt’ suffix…hope that’s readable to all?
Events_log.txt (263.3 KB)
regards,

Ross, details of Java wild process using Activity Monitor on the Mac (saves me from tormenting Unix code):
Activity monitor for Java process Activity Stats for Java Sample of java.txt (264.8 KB)
Maybe this will shed some more light on what’s going on… otherwise I’m baffled.
The ‘java.txt’ file was produced by clicking the ‘sample’ button.
Thanks again.

2020-10-13 07:56:59.285 [vent.ChannelTriggeredEvent] - astro:sun:local:rise#event triggered START
2020-10-13 07:56:59.285 [vent.ChannelTriggeredEvent] - astro:sun:local:civilDawn#event triggered END
2020-10-13 07:56:59.286 [vent.ChannelTriggeredEvent] - astro:sun:local:civilDawn#event triggered END
2020-10-13 07:56:59.286 [vent.ChannelTriggeredEvent] - astro:sun:local:rise#event triggered START

You’re getting doubled Astro events, which is interesting but unlikely to be a problem. Just make a note for now, and see if it persists after anything else gets sorted out.

2020-10-13 02:54:24.198 [me.event.ThingUpdatedEvent] - Thing 'zwave:device:c0ec4738:node21' has been updated.
2020-10-13 02:54:24.203 [me.event.ThingUpdatedEvent] - Thing 'zwave:device:c0ec4738:node9' has been updated.
2020-10-13 02:56:22.227 [me.event.ThingUpdatedEvent] - Thing 'zwave:device:c0ec4738:node11' has been updated.
2020-10-13 02:56:24.694 [me.event.ThingUpdatedEvent] - Thing 'zwave:device:c0ec4738:node9' has been updated.
2020-10-13 02:56:26.478 [me.event.ThingUpdatedEvent] - Thing 'zwave:device:c0ec4738:node12' has been updated.
2020-10-13 02:56:28.432 [me.event.ThingUpdatedEvent] - Thing 'zwave:device:c0ec4738:node11' has been updated.
2020-10-13 02:57:07.789 [me.event.ThingUpdatedEvent] - Thing 'zwave:device:c0ec4738:node13' has been updated.
2020-10-13 02:57:09.506 [me.event.ThingUpdatedEvent] - Thing 'zwave:device:c0ec4738:node12' has been updated.
2020-10-13 02:57:16.933 [me.event.ThingUpdatedEvent] - Thing 'zwave:device:c0ec4738:node13' has been updated.
2020-10-13 02:57:17.287 [me.event.ThingUpdatedEvent] - Thing 'zwave:device:c0ec4738:node18' has been updated.
2020-10-13 02:57:19.416 [me.event.ThingUpdatedEvent] - Thing 'zwave:device:c0ec4738:node14' has been updated.
2020-10-13 02:57:20.261 [me.event.ThingUpdatedEvent] - Thing 'zwave:device:c0ec4738:node22' has been updated.
2020-10-13 02:57:30.684 [me.event.ThingUpdatedEvent] - Thing 'zwave:device:c0ec4738:node18' has been updated.
2020-10-13 02:57:34.128 [me.event.ThingUpdatedEvent] - Thing 'zwave:device:c0ec4738:node22' has been updated.
2020-10-13 02:57:36.306 [me.event.ThingUpdatedEvent] - Thing 'zwave:device:c0ec4738:node23' has been updated.

This is not usual.
“Thing updated” is referring to the configuration of the Thing, not status like online/offline.
It suggests your configuration does not quite match reality.
What happens at this time of morning is a zwave “network heal”, which can be a fairly disruptive process on zwave (but who cares at openHAB end).
However, knock-on problems can arise from Thing updates, unexpected reinitialization of various stuff like the binding itself. I don’t know if twenty such Thing updates lead to a queue of twenty reinitialzes …

2020-10-13 07:16:18.920 [me.event.ThingUpdatedEvent] - Thing 'zwave:device:c0ec4738:node15' has been updated.
...
2020-10-13 10:00:32.928 [me.event.ThingUpdatedEvent] - Thing 'zwave:device:c0ec4738:node20' has been updated.

Other Thing updates trickle in later on as devices wakeup, we’re talking long-running events here.

For purposes of elimination, you can I believe disable the zwave nightly heal.

For a fix, you may need to delete and rediscover each Thing so that they match the discovery. This is often needed if pre-existing Things are carried forward to updated binding that changed some aspect. If you re-use the same Thing uID, pre-existing Items etc. should reattach.

Hi Ross, thank you for all your time on this. I appreciate it.
I have disabled the z-wave nightly heal now from within the HABmin panel and saved the change.
What are the consequences…short or long term…of not having a wave heal?
Will I wait and see before deleting the ‘Things’?
I am using an Aeotec z-wave stick as my zwave radio. It can be brought about the house to add things as required whose details are then stored within it. Do I need to actually wipe its memory /reset or will I just delete the ‘things’ from Openhab and let Openhab rediscover them? If for some reason they get a different ‘Thing uID’ will I just overwrite it in the Paper UI with the original ID? I don’t want to lose all the setup/connections between the present items and the present Things.
regards,

It’s to do with pathway mapping. As I understand it, everything will work, you only need to heal when you reconfigure the mesh by moving/adding/removing nodes. And then I think it’s more “desirable” than “needed”.

No, leave it alone! Only look at OH Things.

As I understand it should not, because the unchanged stick will supply the same id