High CPU usage, possibly due to serial?

TRS-80 · December 30, 2019, 3:40am

Platform information:
- Hardware: Cubietruck Allwinner A20 armhf 2GB RAM
- OS: Debian (Armbian) Jessie
- Java Runtime Environment: Zulu 8
- openHAB version: 2.4.0

Last couple times I ssh in to my openHAB server running on Cubietruck, I notice high system load:

System load:   4.17 4.87 4.15

Doing htop shows openhab hogging a lot of resources:


  1  [|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||100.0%]     Tasks: 47, 251 thr; 12 running
  2  [|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||100.0%]     Load average: 11.55 12.27 14.03 
  Mem[|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||842/2010MB]     Uptime: 26 days, 11:29:25
  Swp[||                                                                                                          16/1005MB]

  PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
25699 openhab    20   0  491M  367M 13840 S 157. 18.3 71h13:01 /usr/bin/java -Dopenhab.home=/usr/share/openhab2 -Dopenhab.conf=/etc/openhab2 -Dopenhab.runtime=/usr/share/openhab2/runtime -Dopenhab.userdata=/var/lib/openhab2 -Dopenhab.logdir=/var/log/open
26980 openhab    20   0  491M  367M 13840 R 30.0 18.3  1h22:54 /usr/bin/java -Dopenhab.home=/usr/share/openhab2 -Dopenhab.conf=/etc/openhab2 -Dopenhab.runtime=/usr/share/openhab2/runtime -Dopenhab.userdata=/var/lib/openhab2 -Dopenhab.logdir=/var/log/open
22779 openhab    20   0  491M  367M 13840 R 28.8 18.3  1h25:23 /usr/bin/java -Dopenhab.home=/usr/share/openhab2 -Dopenhab.conf=/etc/openhab2 -Dopenhab.runtime=/usr/share/openhab2/runtime -Dopenhab.userdata=/var/lib/openhab2 -Dopenhab.logdir=/var/log/open
30438 openhab    20   0  491M  367M 13840 R 27.6 18.3  1h21:15 /usr/bin/java -Dopenhab.home=/usr/share/openhab2 -Dopenhab.conf=/etc/openhab2 -Dopenhab.runtime=/usr/share/openhab2/runtime -Dopenhab.userdata=/var/lib/openhab2 -Dopenhab.logdir=/var/log/open
19738 openhab    20   0  491M  367M 13840 R 26.5 18.3  1h27:42 /usr/bin/java -Dopenhab.home=/usr/share/openhab2 -Dopenhab.conf=/etc/openhab2 -Dopenhab.runtime=/usr/share/openhab2/runtime -Dopenhab.userdata=/var/lib/openhab2 -Dopenhab.logdir=/var/log/open
26999 openhab    20   0  491M  367M 13840 R 22.9 18.3  1h22:58 /usr/bin/java -Dopenhab.home=/usr/share/openhab2 -Dopenhab.conf=/etc/openhab2 -Dopenhab.runtime=/usr/share/openhab2/runtime -Dopenhab.userdata=/var/lib/openhab2 -Dopenhab.logdir=/var/log/open
26145 openhab    20   0  491M  367M 13840 R 13.5 18.3  7h29:17 /usr/bin/java -Dopenhab.home=/usr/share/openhab2 -Dopenhab.conf=/etc/openhab2 -Dopenhab.runtime=/usr/share/openhab2/runtime -Dopenhab.userdata=/var/lib/openhab2 -Dopenhab.logdir=/var/log/open

So I start to investigate. I search forums and internet and find similar threads about high CPU usage. In those threads I found some commands to issue in openHAB console like threads --list which I executed and found several threads with high CPU time:

211    │ ESH-usb-serial-discovery-linux-sysfs-1                                                       │ TIMED_WAITING │ 26523307 │ 17697940
2188   │ Thread-105                                                                                   │ RUNNABLE      │ 31352788 │ 9442970
2585   │ Thread-112                                                                                   │ RUNNABLE      │ 30686536 │ 9191600
2595   │ Thread-113                                                                                   │ RUNNABLE      │ 30693875 │ 9184630
14123  │ Thread-247                                                                                   │ RUNNABLE      │ 30432034 │ 9089630
14130  │ Thread-248                                                                                   │ RUNNABLE      │ 30437704 │ 9083700
14455  │ Thread-250                                                                                   │ TIMED_WAITING │ 1        │ 0
31916  │ Thread-432                                                                                   │ RUNNABLE      │ 5877711  │ 1642970
31926  │ Thread-433                                                                                   │ RUNNABLE      │ 5881276  │ 1651180
32575  │ Thread-440                                                                                   │ RUNNABLE      │ 5514174  │ 1505700
37395  │ Thread-488                                                                                   │ RUNNABLE      │ 5187324  │ 1384310
37791  │ Thread-493                                                                                   │ RUNNABLE      │ 4983929  │ 1309180
37808  │ Thread-494                                                                                   │ RUNNABLE      │ 4983894  │ 1309570
73507  │ Thread-864                                                                                   │ TIMED_WAITING │ 0        │ 0
73521  │ Thread-865                                                                                   │ RUNNABLE      │ 4379802  │ 1182390
73805  │ Thread-869                                                                                   │ TIMED_WAITING │ 0        │ 0
80504  │ ESH-discovery-9000                                                                           │ TIMED_WAITING │ 3102     │ 2540
81670  │ Thread-960                                                                                   │ RUNNABLE      │ 3581328  │ 1158900
83671  │ Thread-985                                                                                   │ RUNNABLE      │ 3441377  │ 1100270
86184  │ Thread-1015                                                                                  │ RUNNABLE      │ 3295117  │ 1042530
86199  │ Thread-1016                                                                                  │ RUNNABLE      │ 3294444  │ 1042790
88440  │ Thread-1044                                                                                  │ RUNNABLE      │ 3191726  │ 1002800

Thread-xxx does not tell much so I issue command threads xxxx on several (not all) and they all said the same thing:

openhab> threads 2188
Thread 2188 Thread-105 RUNNABLE
Stacktrace:
gnu.io.RXTXPort.eventLoop line: -2
gnu.io.RXTXPort$MonitorThread.run line: 1611
openhab> threads 2585
Thread 2585 Thread-112 RUNNABLE
Stacktrace:
gnu.io.RXTXPort.eventLoop line: -2
gnu.io.RXTXPort$MonitorThread.run line: 1611
openhab> threads 2595
Thread 2595 Thread-113 RUNNABLE
Stacktrace:
gnu.io.RXTXPort.eventLoop line: -2
gnu.io.RXTXPort$MonitorThread.run line: 1611
openhab> threads 14123
Thread 14123 Thread-247 RUNNABLE
Stacktrace:
gnu.io.RXTXPort.eventLoop line: -2
gnu.io.RXTXPort$MonitorThread.run line: 1611
openhab> threads 14130
Thread 14130 Thread-248 RUNNABLE
Stacktrace:
gnu.io.RXTXPort.eventLoop line: -2
gnu.io.RXTXPort$MonitorThread.run line: 1611

So, it appears something is going on with serial port, but not sure how to diagnose from here?

Only change I can recall making recently was to enable Sanity Check in MySensors binding, because the (USB attached serial) gateway seemed to me to keep dropping out. Since I made that change it seems more reliable, but maybe that caused this? Prior to that I never noticed anyway problems like this.

I suppose I just wanted to touch base here before raising an issue in that binding GitHub to confirm I am on the right track?

Bruce_Osborne · December 30, 2019, 12:36pm

Why are you not running the current 2.5 stable version? 2.4 is over a year old and not actively supported.
If you have restdocs installed, uninstall before upgrading and reinstall after. The location changed and not uninstalling the old one causes issues after upgrading.

mstormi · December 30, 2019, 12:41pm

To my knowledge none of these have an “unknown” cause.
Most of the time it’s rules to reload or bindings to constantly restart which is causing this.
Looking after threads won’t help as it doesn’t get you a direction to look at.
Increase debugging on org.openhab.* and org.eclipse.* and browse this to get a hint what is going on.

TRS-80 · December 30, 2019, 7:43pm

Well if you take issue with that, you’re going to love this… It’s because the server I’m running openHAB on is still on Debian Jessie as I haven’t had time to do the upgrade (and I don’t think 2.5 is available in the repository for Jessie). I have a lot of services running on there that we rely on every day. Anyway, I am off work for the holidays and that is one of the main things I would like to get done. I will actually install everything again on new (upgraded) hardware in parallel, and make sure services are working one by one before I shut them down on the old machine.

Thanks for that advice. Any other specific tips? I imagine there is some guide somewhere on upgrading from 2.4 to 2.5 (haven’t searched for it yet, but I will)? Although, as I said it will be somewhat like a new installation, although I do plan on copying my config folder over (it is under Git version control, so should be pretty easy to do that).

I thought what I posted above seemed to give a pretty good indication of the proximate cause, no?

Will do, thanks for advice.

Bruce_Osborne · December 30, 2019, 7:48pm

OH is in its own repo, independent of OS version. I like using openHABian following their Linux install instructions. That makes it easier to get the proper Java version, etc.

The 2.5 Blog post with a link to the Release Notes is here.

Here are the instructions I like to follow. I think the normal Linux installation docs are currently a mess.

TRS-80 · December 30, 2019, 8:04pm

Now that I think about it, of course it makes sense that the proper Java version is much more important than OS version, as openHAB is running on Java, and therefore doesn’t really need any other packages from the underlying system (coffee is just now starting to kick in ).

I don’t recall finding them to be so terrible. But I will take a look at those. Thanks again!

Bruce_Osborne · December 30, 2019, 8:05pm

Now there is no good link to Zulu Java, for instance. It is no too clear but you do not need or WANT the openhab2-addons package. It actually breaks your system currently.

TRS-80 · December 30, 2019, 8:28pm

At the top of that page it says “A home automation enthusiast doesn’t have to be a Linux enthusiast!” but I am a Linux enthusiast!

Hence I prefer to do all commands one by one on the command line, so I know exactly what is going on at each step (I even keep detailed notes of same!). Curses interfaces have their place (controlling mpd instances for instance ) but when doing sysadmin stuff I much prefer the most granular, manual way possible. For the same reason I don’t like “wizards” or Docker images, etc. as IMHO they only obfuscate things from you.

To each their own however. I am sure that openhab-config tool has helped many people who are otherwise unfamiliar with Linux. But even then, I am of the opinion that they would be better off in the end learning more Linux, in order to be better prepared to administer the system they are putting together. But that’s just like, my opinion, man.

I just checked back on the Installation Overview as well as Linux docs, and both pretty clearly link to Zulu. And I don’t recall having trouble finding that info back when I installed like a year or more ago, either. I would even say the docs have improved since then, as they more clearly now explain about licensing issues surrounding Oracle version of Java. Nice! Because people need to be told about these kinds of things!

EDIT: I just looked, and I also see on the Linux page:

When you choose to install an add-on, openHAB will download it from the internet on request. If you plan on disconnecting your machine from the internet, then you will want to also install the add-ons package.
sudo apt-get install openhab2-addons

Also, a similar instruction is given for Yum or Dnf Based Systems.

I am not trying to be argumentative, and I appreciate your help. I know you are (at least) a very experienced user. But perhaps it has been a while since you read those pages?

Bruce_Osborne · December 30, 2019, 8:33pm

Zulu reorganized their downloads and website. Our installation docs point to how to install Zulu but not where to download it. Just to add complexity, there are multiple download variants for the Raspberry Pi ARM architecture.

When I when through this recently I ended up using the download URLs from openHABian to be sure I got the correct variant on my test Pi.

TRS-80 · December 30, 2019, 8:39pm

I do remember at the time I installed (over 1 year ago) their website being somewhat convoluted. I was able to find all the information I needed though, eventually. You are right about finding the correct version to download, it was less clear then that you need the embedded version, and even right now just reading it, that is still not completely clear (it seems to be presented as an alternative). Maybe I should take a stab at improving the wording of that…

mstormi · December 30, 2019, 10:45pm

It doesn’t say it is not for enthusiasts as well, does it ?

Give openHABian a chance. You’ll see it’s just scripts.
So insert a set -x if you like to see all the commands
It’s a good idea to start with that because it is pretty comprehensive.
And if you want help later on it’s much more convenient for the both of us to simply state “I run openHABian” than to start explaining each and every detail of your OS setup that may or may not be relevant to an issue.

Mainly because people demand one recommendation, and Zulu is proven to run well on ARM.
But it doesn’t mean others don’t work, you’re free to choose any other Java. But you should have a good reason to deviate from the mainstream.

gitMiguel · December 30, 2019, 11:23pm

Sounds like: https://github.com/NeuronRobotics/nrjavaserial/issues/36
The event listener loop is consuming cpu cycles.

If I were you I would update openHAB and Java to the latest and see if it helps.

milty456 · November 10, 2020, 8:34pm

Hi, I just installed 2.5(openhab) AND am having similar issues; haven’t configured anything; my CPU spikes to 200%…I have 2 cores installed for the VM; ubuntu 20.04. If i remove a core, it seems fine; if i put a core back in, the java process goes nuts and I can’t load openhab webpage; any thoughts on how to debug this? Thanks

namraccr · November 11, 2020, 12:20am

So you’re doing nothing with serial, yet chose to revive a year-old thread about serial?
Start a new thread, after you read the docs to learn how to ask a good question.

milty456 · November 11, 2020, 12:23am

Thanks for the help; love the passive aggressiveness on this one.

I simply thought maybe the person who had the issue had some more tips to help me troubleshoot.