Java runs amok weekly

AFAIK mac will allow you to start the Karaf console, see here:

In the console type in

shell:info

This should work on mac, pc and linux and allow you to see if there are run away threads and details on the heap.

The other thing which I dont believe anyone has posted yet, is to remove a binding 1 by 1 until you find the trouble maker. It can also be a rule, so disable all rules to see if it fixes it. Once you work out the cause by process of elimination, people can help you out a little easier.

I understand your frustration. I run OH in an Ubuntu VM, previously on ESXi and more recently on Proxmox, and I too used to encounter the same problem you have described. Java memory usage would shoot up, CPU usage would reach 100% and the only fix was to restart the OH service from the CLI.

I didn’t take notes but I believe the issue started with v2.3 and it is only since about two or three months ago that I noticed that the problem no longer occurs. I used to be able to trigger the issue by opening the PaperUI in Chrome, within a few hours or sometimes minutes the CPU and memory usage would shoot up.

Whilst I was trying to diagnose the problem I ended up writing a rule that persisted the java heap size so that I could graph it with Grafana in order to determine when exactly the problem started. I can share the rule with you if you would find that useful.

I wish I could tell you what fixed it, I can only assume it was a bug in one of the bindings I use that was fixed in one of the monthly add-on releases but I can’t say for sure.

What bindings do you have installed? Is it possible to uninstall one at a time and see if that has any effect? I appreciate that might be difficult since the bindings are running some automations in your home.

Hi again Markus,
Thank you very much for your detailed reply and advise. I really do appreciate your generosity with your time and indeed all the community out there. Sorry I misunderstood you and got a bit narky about OS and systems etc… I guess it’s down to the investment and hopes for Openhab and my attachment to Macs :smile: …yes an emotive thing but sure we’re all human…
regards, Denis

2 Likes

No worries, you’re welcome. It’s hard to keep the emotions out.

Hi Skinah,
I didn’t know that command (still an amateur with Unix) thanks for that! I ran the command in the Terminal (Mac equivalent of your console) and took a screenshot. It’s for the system running NORMAL and Java behaving properly and only using <1% of CPU. Can I attach a screenshot in theses forums or will I need to copy n paste it?
regards,

Hello Steve,
At last… I’m not alone…another user who understands my exact situation…! :grin: … I had read that someone else has linked this behaviour to having opened the Paper or the Basic Web UI’s…
So yes I would be delighted if you would kindly send me (with any helpful/explanatory notes) your rule and how to use Grafana… gawd there’s so much to learn and absorb one gets bewildered… :crazy_face:
Once again it’s interesting that both Hilbrand Bouwkamp and yourself reckon its a Binding/rule problem rather than a core Java issue.
thanks and regards,
Denis

There are many millions of java users.
There are thousands of openHAB users.
There are a handful of users with your particular combination of bindings and versions.
There is one user with your rules and devices environment.

The output is text so it should copy paste fine, but you can also attach screenshots unless the forum stops a new comer from doing it…
Main thing is to record what it is when things are going fine, then try to get the details when it faults as it can greatly speed up diagnosing the issue if you can see something in there climbing up over time. No need to wait till it faults if you can see it happening.

Ross I’m not sure what you’re trying to say here…?? Is it a criticism of me asking the user community for help on what seems to be a rarely encountered issue or a vote that’s its a hopeless cause to solve…??
Denis

I think he is just saying that it is really difficult to give general advice or solution, as every single installation is unique and somtimes the combination of used bindings and rules lead to an error.

Hans, yes indeed installations are unique but there are always generalities that apply and it’s good and notable that many users have tried to point me in the right direction with tips and suggestions…
Denis

It’s diagnostic guidance. You report an issue. Is it an unusual issue? Yes. What else is unusual about your situation? That’s probably a good place to start looking. Java for example is not unusual.

You are going to have to do stuff here, it won’t get resolved by posting. Have you looked in your openhab.log and events.log for unusual messages? If you don’t know what is unusual, we can help. But we can’t look for you.

Hi Ross, thanks for your update. I’m not being lazy and I don’t mind doing the leg work… but for an relative novice, I would need to know what I’m looking for in those logs…that’s the kind of help I was seeking… I’m still climbing the steep learning curve. So I’ve no idea what would constitute ‘unusual’ messages and would gladly look for them if ye can suggest any. Skinah has been very helpful with suggestion to look at the shell Karaf. It’s all voluntary and people’s time so I do appreciate that…
regards,

Just go and look. We have to assume some basic common sense here, messages including the word “error” are likely to be interesting, while those saying “door updated from open to close” are probably not - unless you know the door was not supposed to open. We can’t tell you that.

There will not be messages saying “This service is about to go wrong”, so we are looking here for clues not answers. There may be none. There may be gaps or pauses or complete storms in your logs.

Hi Ross, found the following list of ‘jetty request fails’ which are showing prior to my last two logouts/shutdowns of openhab when I noticed Java in overload.
“2020-10-03 11:33:30.489 [WARN ] [io.openhabcloud.internal.CloudClient] - Jetty request 13058721 failed: null
2020-10-03 11:33:32.946 [WARN ] [io.openhabcloud.internal.CloudClient] - Jetty request 13058762 failed: null
2020-10-03 11:33:34.147 [WARN ] [io.openhabcloud.internal.CloudClient] - Jetty request 13058809 failed: null
2020-10-03 11:33:44.832 [WARN ] [io.openhabcloud.internal.CloudClient] - Jetty request 13058817 failed: null
2020-10-03 11:33:46.288 [WARN ] [io.openhabcloud.internal.CloudClient] - Jetty request 13058920 failed: null
2020-10-03 11:33:54.736 [WARN ] [io.openhabcloud.internal.CloudClient] - Jetty request 13058970 failed: null
2020-10-03 11:33:58.514 [WARN ] [io.openhabcloud.internal.CloudClient] - Jetty request 13058987 failed: null
2020-10-03 11:34:00.195 [WARN ] [io.openhabcloud.internal.CloudClient] - Jetty request 13058968 failed: null
2020-10-03 11:34:00.198 [WARN ] [io.openhabcloud.internal.CloudClient] - Jetty request 13059022 failed: null
2020-10-03 12:27:03.241 [WARN ] [io.openhabcloud.internal.CloudClient] - Jetty request 13083286 failed: null
2020-10-03 12:27:12.922 [WARN ] [io.openhabcloud.internal.CloudClient] - Jetty request 13083295 failed: null
2020-10-03 12:27:15.122 [WARN ] [io.openhabcloud.internal.CloudClient] - Jetty request 13083391 failed: null
2020-10-03 13:28:28.136 [WARN ] [io.openhabcloud.internal.CloudClient] - Jetty request 13109953 failed: null
2020-10-03 13:28:33.795 [WARN ] [io.openhabcloud.internal.CloudClient] - Jetty request 13109968 failed: null
2020-10-03 13:28:35.529 [WARN ] [io.openhabcloud.internal.CloudClient] - Jetty request 13110027 failed: null
2020-10-03 13:29:04.715 [WARN ] [io.openhabcloud.internal.CloudClient] - Jetty request 13110260 failed: null
2020-10-03 13:29:12.336 [WARN ] [io.openhabcloud.internal.CloudClient] - Jetty request 13110269 failed: null
2020-10-03 13:29:33.762 [WARN ] [io.openhabcloud.internal.CloudClient] - Jetty request 13110329 failed: null
2020-10-03 13:29:46.211 [WARN ] [io.openhabcloud.internal.CloudClient] - Jetty request 13110486 failed: null
2020-10-03 13:29:48.326 [WARN ] [io.openhabcloud.internal.CloudClient] - Jetty request 13110549 failed: null
2020-10-03 13:29:53.566 [WARN ] [io.openhabcloud.internal.CloudClient] - Jetty request 13110562 failed: null
2020-10-03 13:30:04.727 [WARN ] [io.openhabcloud.internal.CloudClient] - Jetty request 13110626 failed: null
2020-10-03 13:30:08.848 [WARN ] [io.openhabcloud.internal.CloudClient] - Jetty request 13110772 failed: null
2020-10-03 13:30:09.907 [WARN ] [io.openhabcloud.internal.CloudClient] - Jetty request 13110804 failed: null
2020-10-03 13:30:12.155 [WARN ] [io.openhabcloud.internal.CloudClient] - Jetty request 13110807 failed: null
2020-10-04 23:41:47.445 [WARN ] [io.openhabcloud.internal.CloudClient] - Jetty request 78683 failed: null
2020-10-05 14:53:52.575 [WARN ] [io.openhabcloud.internal.CloudClient] - Jetty request 261553 failed: null
2020-10-05 14:53:57.226 [WARN ] [io.openhabcloud.internal.CloudClient] - Jetty request 261559 failed: null
2020-10-05 14:53:59.865 [WARN ] [io.openhabcloud.internal.CloudClient] - Jetty request 261585 failed: null
2020-10-05 14:54:05.645 [WARN ] [io.openhabcloud.internal.CloudClient] - Jetty request 261593 failed: null
2020-10-05 14:54:08.325 [WARN ] [io.openhabcloud.internal.CloudClient] - Jetty request 261630 failed: null
2020-10-05 15:01:33.180 [WARN ] [io.openhabcloud.internal.CloudClient] - Jetty request 264197 failed: null”
I don’t know if this has caused Java to overload or whether it’s a resulting symptom of Java gone crazy for a different reason. What do you make of this?
regards,

Well, cause and symptoms are hard to separate.

Jetty errors like that are usually connected with a UI, it’s about openHAB to UI communication.
So maybe a symptom, claggy OH is not responding to UI.
Maybe linked to cause - I’ve seen old posts where long term UI connections gobble resources.

Can also be about other remote services though.

At this point, I guess we need to know what sort of UI you have in general use, what remote services you are using.

Maybe it’s caused by openhabcloud service. You could uninstall it and see if the “Java overload” appears again!

Then do further investigations!

You could also do a debug on openhab cloud service:

Open openHAB karaf console:

log:set DEBUG org.openhab.io.openhabcloud
1 Like

Hi Ross, I only occasionally use the Local Web Basic UI on the Mac itself and only when I’m working with ‘Visual Studio Code’ (yes it has a built in UI viewer) to cross check what the UI is showing as I build stuff.
99% of time I use the iPhone app which connnects both locally on the LAN and also connects very well remotely (through the openhab cloud naturally).
Services & Bindings installed: Astro, Automower, HarmonyHub, OpenWeatherMap, Tradfri and z-wave and of course openHABcloud.

Hi Alex, thanks for your reply. If I turn on debug using the Karaf as you suggest, what will I be looking for and also what’s the syntax to turn it off afterwards? …sorry if that’s a stupid question…
regards,

log:set DEFAULT org.openhab.io.openhabcloud

There will be the following in your log: [DEBUG] [io.openhabcloud.internal.CloudClient].

Look at these entries…