Karaf console is very slow

Setup, Configuration and Use Runtime
, ,
1

Hello!
I’m running Openhab on my Raspberry Pi 4B and the Karaf console responds very slowly. It takes quite some time to connect, text input is delayed considerably and text output such as viewing the log takes forever. CPU usage goes up noticeably while doing any of these things, but it almost never gets over 60 %.
So far, I only found threads about the Karaf console being so slow that users get timeouts. This is not the case for me as the console is still way too fast for that to happen; however, it is slow enough to make using it somewhat annoying.
I open the console by first ssh-ing onto my Pi regularly and then connecting to the Karaf console using ssh -p 8101 openhab@localhost. The regular bash command line outside of the Karaf console responds just fine. I haven’t touched any Karaf-related configs yet since Openhab’s installation.

Does anyone else have this issue or any solution to this?

Platform information:

  • Hardware: Raspberry Pi 4B (4 GB RAM)
  • OS: Arch Linux ARM (Up-to-date)
  • Java Runtime Environment: Zulu (Java 8, 32-bit) (Up-to-date)
  • openHAB version: 2.5.4 (Up-to-date)
2

32-bit or 64-bit Java?
Java 8 or 11?
The questions are detailed for a reason.

3

It’s 32-bit Java 8.

4

I would suggest to check integrations which you run in order to exclude a configuration or third party error.

Based on this awesome article how to find a busy thread, you can check following script:

#!/bin/bash
PID=$(top -n1 | grep -m1 java | perl -pe 's/\e\[?.*?[\@-~] ?//g' | cut -f1 -d' ')
NID=$(printf "%x" $(top -n1 -H | grep -m1 java | perl -pe 's/\e\[?.*?[\@-~] ?//g' | cut -f1 -d' '))
jstack $PID | grep -A500 $NID | grep -m1 "^$" -B 500

Best,
Łukasz

1 Like
5

Thanks for the hint. Unfortunately, this snipped doesn’t work. The top -n1 -H | grep -m1 java portion fails because top somehow directly prints the thread names instead of java in each line, but on the other hand finding the problematic thread manually just got easier.

Looks like the Karaf console thread itself is the one eating up my CPU.

PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     ZEIT+ BEFEHL
970 openhab   20   0  773124 275860  14076 R  99,3   6,9   0:42.24 Karaf ssh conso

I didn’t accidentally not copy le there. That seems to be the actual thread name. The thread’s CPU usage only gets that high when I actively try to do stuff with the console. When I leave the console idling around, it hardly uses the CPU at all.

6

Karaf truncates long names.

7

I’ve checked registered issues around the topic. There are two places - first is jline, an underlying library which serves shell functionality:
https://github.com/jline/jline3/search?q=cpu&type=Issues

Second is Karaf itself. I’ve found KARAF-6512 - Dead lock in LogServiceImpl which is related to ssh and jline and following comment:

If you turn sshd logger to debug, and you do log:tail via ssh, you create an invite loop.

When you turn sshd logger to debug, each network packet are logged. So if you display the log via ssh, you have a circle.

There’s no possible fix for that, the only workaround is to avoid DEBUG for sshd logger.

Not sure if that’s the case for you. Anyhow getting 100% of CPU is clear indication of an infinite loop.

8

My OpenSSH daemon has its log level set to INFO, but Karaf uses its own SSH daemon, doesn’t it? Where can I find the config for that or check the current log level?

I don’t think some infinite loop is my propblem though. As I mentioned, CPU usage only rises while I’m actively using the console. When I lift my hands from my keyboard but remain connected to the console, CPU usage returns to normal. I’d expect it to stay high if it’s caused by an infinite loop.

9

Check file named org.ops4j.pax.logging.cfg or simply from shell log:get [category]. Second command will tell you what is desired logging level for given area ie. log:get org.apache.sshd.

10

Thanks. The log level is WARN. Shouldn’t be a problem then.