[SOLVED] OpenHAB2 setup on CentOS 7 - Dashboard is looping "StartedDashboard" "Stopped Dashboard" every 26 secs

hilorio · July 22, 2019, 11:46pm

Dear openHAB Community,

On my CentOs 7 openHAB Dashboard is starting and then stopping in a cycle with a period around 26 secs (process IDs changing; watchdog daemon?)
For the few moments Dashboard is started I can pull it from port 8080 to remote browser successfully
“Stopped Dashboard” after 10 seconds (pretty reliably 10 secs; with or without access from the browser)

I disabled SELinux to see if this has any influence to no avail.

Please accept my apologies, I am out of ideas by now. I read through quite a log of the community posts but did not find anything like the above.

Any help appreciated. Thank you very much!

Robert

Platform information:
- Hardware: vmware host with i7-3820 CPU @ 3.60GHz, VM has dual-core, 2GB RAM, 16 GB storage
- OS: CentOS7, CentOS Linux release 7.6.1810 (Core), 3.10.0-957.el7.x86_64
- Java Runtime Environment: Zulu
  openjdk version “1.8.0_222”
  OpenJDK Runtime Environment (Zulu 8.40.0.25-CA-linux64) (build 1.8.0_222-b10)
  OpenJDK 64-Bit Server VM (Zulu 8.40.0.25-CA-linux64) (build 25.222-b10, mixed mode)
- openHAB version:
  openhab2-2.4.0-1.noarch,
  yum install on 22-JUL-2019,
  repo pointing to https://dl.bintray.com/openhab/rpm-repo2/stable


[root@openhab rob]# uname -a
Linux openhab.intern 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

[root@openhab rob]# cat /etc/centos-release
CentOS Linux release 7.6.1810 (Core)

[root@openhab rob]# sestatus
SELinux status:                 disabled

[root@openhab rob]# firewall-cmd --zone=public --list-ports
8080/tcp 8443/tcp

[root@openhab rob]# java -version
openjdk version "1.8.0_222"
OpenJDK Runtime Environment (Zulu 8.40.0.25-CA-linux64) (build 1.8.0_222-b10)
OpenJDK 64-Bit Server VM (Zulu 8.40.0.25-CA-linux64) (build 25.222-b10, mixed mode)

[root@openhab rob]# cat /etc/yum.repos.d/openhab.repo
[openHAB-Stable]
name=openHAB 2.x.x Stable
baseurl=https://dl.bintray.com/openhab/rpm-repo2/stable
gpgcheck=1
gpgkey=https://bintray.com/user/downloadSubjectPublicKey?username=openhab
enabled=1

[root@openhab rob]# rpm -qa | grep openhab2
openhab2-2.4.0-1.noarch
openhab2-addons-2.4.0-1.noarch

[root@openhab rob]# netstat -an | grep 8080
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN

[root@openhab rob]# ps -ef | grep openhab
avahi     5585     1  0 01:10 ?        00:00:00 avahi-daemon: running [openhab.local]
openhab  17942     1  0 01:20 ?        00:00:00 /bin/sh /usr/share/openhab2/runtime/bin/karaf debug
openhab  18083 17942 99 01:20 ?        00:00:11 /usr/bin/java -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005 -Dopenhab.home=/usr/share/openhab2 -Dopenhab.conf=/etc/openhab2 -Dopenhab.runtime=/usr/share/openhab2/runtime -Dopenhab.userdata=/var/lib/openhab2 -Dopenhab.logdir=/var/log/openhab2 -Dfelix.cm.dir=/var/lib/openhab2/config -Djava.library.path=/var/lib/openhab2/tmp/lib -Djetty.host=0.0.0.0 -Djetty.http.compliance=RFC2616 -Dorg.ops4j.pax.web.listening.addresses=0.0.0.0 -Dorg.osgi.service.http.port=8080 -Dorg.osgi.service.http.port.secure=8443 -Djava.awt.headless=true -XX:+UseG1GC -Djava.net.preferIPv4Stack=true -Djava.endorsed.dirs=/usr/lib/jvm/zulu-8/jre/jre/lib/endorsed:/usr/lib/jvm/zulu-8/jre/lib/endorsed:/usr/share/openhab2/runtime/lib/endorsed -Djava.ext.dirs=/usr/lib/jvm/zulu-8/jre/jre/lib/ext:/usr/lib/jvm/zulu-8/jre/lib/ext:/usr/share/openhab2/runtime/lib/ext -Dkaraf.instances=/var/lib/openhab2/tmp/instances -Dkaraf.home=/usr/share/openhab2/runtime -Dkaraf.base=/var/lib/openhab2 -Dkaraf.data=/var/lib/openhab2 -Dkaraf.etc=/var/lib/openhab2/etc -Dkaraf.logs=/var/log/openhab2 -Dkaraf.restart.jvm.supported=true -Djava.io.tmpdir=/var/lib/openhab2/tmp -Djava.util.logging.config.file=/var/lib/openhab2/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -classpath /usr/share/openhab2/runtime/lib/boot/activation-1.1.1.jar:/usr/share/openhab2/runtime/lib/boot/jaxb-api-2.3.0.jar:/usr/share/openhab2/runtime/lib/boot/jaxb-core-2.2.11.jar:/usr/share/openhab2/runtime/lib/boot/jaxb-impl-2.2.11.jar:/usr/share/openhab2/runtime/lib/boot/org.apache.karaf.diagnostic.boot-4.2.1.jar:/usr/share/openhab2/runtime/lib/boot/org.apache.karaf.jaas.boot-4.2.1.jar:/usr/share/openhab2/runtime/lib/boot/org.apache.karaf.main-4.2.1.jar:/usr/share/openhab2/runtime/lib/boot/org.osgi.core-6.0.0.jar org.apache.karaf.main.Main
root     18182  7291  0 01:20 pts/0    00:00:00 grep --color=auto openhab

[root@openhab rob]# sudo systemctl status openhab2
openhab2.service - openHAB 2 - empowering the smart home
   Loaded: loaded (/usr/lib/systemd/system/openhab2.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2019-07-23 01:20:17 CEST; 13s ago
     Docs: https://www.openhab.org/docs/
           https://community.openhab.org
  Process: 17782 ExecStop=/usr/share/openhab2/runtime/bin/karaf stop (code=exited, status=1/FAILURE)
 Main PID: 17942 (karaf)
    Tasks: 99
   CGroup: /system.slice/openhab2.service
           ├─17942 /bin/sh /usr/share/openhab2/runtime/bin/karaf debug
           └─18083 /usr/bin/java -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005 -Dopenhab.home=/usr/share/openhab2 -Dopenhab.conf=/etc/openhab2 -Dopenhab.runtime=/usr/share/open...

Jul 23 01:20:20 openhab.sol-xo.intern karaf[17942]:   ____  ____  ___  ____  / / / /   |  / __ )
Jul 23 01:20:20 openhab.sol-xo.intern karaf[17942]:  / __ \/ __ \/ _ \/ __ \/ /_/ / /| | / __  |
Jul 23 01:20:20 openhab.sol-xo.intern karaf[17942]: / /_/ / /_/ /  __/ / / / __  / ___ |/ /_/ /
Jul 23 01:20:20 openhab.sol-xo.intern karaf[17942]: \____/ .___/\___/_/ /_/_/ /_/_/  |_/_____/
Jul 23 01:20:20 openhab.sol-xo.intern karaf[17942]:     /_/                        2.4.0
Jul 23 01:20:20 openhab.sol-xo.intern karaf[17942]:                                Release Build
Jul 23 01:20:20 openhab.sol-xo.intern karaf[17942]: Hit '<tab>' for a list of available commands
Jul 23 01:20:20 openhab.sol-xo.intern karaf[17942]: and '[cmd] --help' for help on a specific command.
Jul 23 01:20:20 openhab.sol-xo.intern karaf[17942]: Hit '<ctrl-d>' or type 'system:shutdown' or 'logout' to shutdown openHAB.
Jul 23 01:20:21 openhab.sol-xo.intern karaf[17942]: openhab>

[root@openhab rob]# sudo journalctl | tail -13
Jul 23 01:20:20 openhab.sol-xo.intern karaf[17942]: / /_/ / /_/ /  __/ / / / __  / ___ |/ /_/ /
Jul 23 01:20:20 openhab.sol-xo.intern karaf[17942]: \____/ .___/\___/_/ /_/_/ /_/_/  |_/_____/
Jul 23 01:20:20 openhab.sol-xo.intern karaf[17942]:     /_/                        2.4.0
Jul 23 01:20:20 openhab.sol-xo.intern karaf[17942]:                                Release Build
Jul 23 01:20:20 openhab.sol-xo.intern karaf[17942]: Hit '<tab>' for a list of available commands
Jul 23 01:20:20 openhab.sol-xo.intern karaf[17942]: and '[cmd] --help' for help on a specific command.
Jul 23 01:20:20 openhab.sol-xo.intern karaf[17942]: Hit '<ctrl-d>' or type 'system:shutdown' or 'logout' to shutdown openHAB.
Jul 23 01:20:21 openhab.sol-xo.intern karaf[17942]: openhab>
Jul 23 01:20:31 openhab.sol-xo.intern sudo[18221]: rob : TTY=pts/0 ; PWD=/home/rob ; USER=root ; COMMAND=/bin/systemctl status openhab2
Jul 23 01:20:31 openhab.sol-xo.intern sudo[18221]: pam_unix(sudo:session): session opened for user root by rob(uid=0)
Jul 23 01:20:31 openhab.sol-xo.intern sudo[18221]: pam_unix(sudo:session): session closed for user root
Jul 23 01:20:38 openhab.sol-xo.intern sudo[18227]: rob : TTY=pts/0 ; PWD=/home/rob ; USER=root ; COMMAND=/bin/journalctl
Jul 23 01:20:38 openhab.sol-xo.intern sudo[18227]: pam_unix(sudo:session): session opened for user root by rob(uid=0)

[root@openhab rob]# ls -l /var/log/openhab2/
total 196
-rw-r--r--. 1 openhab openhab      0 Jul 22 23:24 audit.log
-rw-r--r--. 1 openhab openhab      0 Jul 22 23:24 events.log
-rw-r--r--. 1 openhab openhab 140006 Jul 23 01:20 openhab.log
-rwxr-xr-x. 1 openhab openhab      0 Dec 17  2018 Readme.txt
[root@openhab rob]# openhab-cli showlogs

==> /var/log/openhab2/audit.log <==

==> /var/log/openhab2/events.log <==

==> /var/log/openhab2/openhab.log <==
2019-07-23 01:20:09.904 [INFO ] [.dashboard.internal.DashboardService] - Stopped Dashboard
2019-07-23 01:20:19.706 [WARN ] [org.jline                           ] - Unable to create a system terminal, creating a dumb terminal (enable debug logging for more information)
2019-07-23 01:20:26.189 [INFO ] [thome.model.lsp.internal.ModelServer] - Started Language Server Protocol (LSP) service on port 5007
2019-07-23 01:20:26.708 [INFO ] [.dashboard.internal.DashboardService] - Started Dashboard at http://10.1.1.6:8080
2019-07-23 01:20:26.709 [INFO ] [.dashboard.internal.DashboardService] - Started Dashboard at https://10.1.1.6:8443
2019-07-23 01:20:36.120 [INFO ] [.dashboard.internal.DashboardService] - Stopped Dashboard
2019-07-23 01:20:45.710 [WARN ] [org.jline                           ] - Unable to create a system terminal, creating a dumb terminal (enable debug logging for more information)
2019-07-23 01:20:52.932 [INFO ] [thome.model.lsp.internal.ModelServer] - Started Language Server Protocol (LSP) service on port 5007
2019-07-23 01:20:53.696 [INFO ] [.dashboard.internal.DashboardService] - Started Dashboard at http://10.1.1.6:8080
2019-07-23 01:20:53.696 [INFO ] [.dashboard.internal.DashboardService] - Started Dashboard at https://10.1.1.6:8443
2019-07-23 01:21:03.149 [INFO ] [.dashboard.internal.DashboardService] - Stopped Dashboard


And by the way:

[rob@openhab ~]$ cat /etc/default/openhab2 | egrep "($EXTRA_JAVA_OPTS|=debug)"
EXTRA_JAVA_OPTS="-Djava.net.preferIPv4Stack=true "
OPENHAB_STARTMODE=debug

Thedannymullen · July 23, 2019, 2:30am

A few ideas.

A potential work around but I found it to figure things out. Manually install openhabian and have it install openhab. Not sure if it will work on non Debian based but maybe.
Make sure the correct java installed. I had issues installing openhab on Linux due to java. Especially 64bit.

hilorio · July 23, 2019, 12:18pm

Thank you Danny for adding your experience! I read about the ARM issues with serial ports on 64 bit JVMs so I thought this a non-issue on x86-64. However, following your experience, I installed zulu-8 in 32 bit and nailed the JAVA_HOME and JAVA env vars for Karaf to the 32 bit version.

[root@openhab bin]# /usr/lib/jvm/zulu8.40.0.25-ca-jdk8.0.222-linux_i686/jre/bin/java -version
openjdk version "1.8.0_222"
OpenJDK Runtime Environment (Zulu 8.40.0.25-CA-linux32) (build 1.8.0_222-b10)
OpenJDK Server VM (Zulu 8.40.0.25-CA-linux32) (build 25.222-b10, mixed mode)

I positively confirmed that the 32 bit version is used in the output of ps aux. Still, the start/stop loops continue to persist with the 32 bit version.

Can you please give me a hint on what linux versions you successfully installed openHAB and whether you used the package manager or openHABian to make it work?

CentOS is the standard but I might consider Debian for openHAB if it works better.

Thank you very much!

hilorio · July 23, 2019, 2:21pm

Actually it looks like openHAB is running just fine by itself.

When starting with

[rob@openhab ~]$ sudo openhab-cli start
[sudo] password for rob:

A systemd service configuration exists...
Use 'sudo /bin/systemctl start openhab2.service' to start an openHAB service
Launching an instance in this terminal..
Launching the openHAB runtime...
JAVA_HOME=/usr/lib/jvm/zulu-8/jre,   JAVA=/bin/java
JAVA=/bin/java

                          __  _____    ____
  ____  ____  ___  ____  / / / /   |  / __ )
 / __ \/ __ \/ _ \/ __ \/ /_/ / /| | / __  |
/ /_/ / /_/ /  __/ / / / __  / ___ |/ /_/ /
\____/ .___/\___/_/ /_/_/ /_/_/  |_/_____/
    /_/                        2.4.0
                               Release Build

Hit '<tab>' for a list of available commands
and '[cmd] --help' for help on a specific command.
Hit '<ctrl-d>' or type 'system:shutdown' or 'logout' to shutdown openHAB.

openhab>

everything apparently is just fine and the Karaf container does not exit.

So to me it looks like the systemd service management is killing the process probably for some sort of exit code. Maybe the warning about the jline system terminal has some influence on this.

hilorio · July 23, 2019, 10:07pm

My conclusion is that Apache Karafe or its Karafe-delivered systemd Unit definition do not fit the systemd model.
After starting Karafe the server process is spawned and the start script returns. The latter makes systemd believe that the service did not start properly and it then runs the stop process (maybe for cleanup?). Since the openHAB2 systemd Unit definition for Karafe defines a restart on failed, the start/stop scripts are called in a loop.

So here …

Solved item with post by Gad_Ofir

I found some commands to make systemd use the SysV rc.d scripts and this did the trick for me.
* systemctl start openhab2 --> works
* systemctl stop openhab2 --> works
* systemctl restart openhab2 --> works
* openhab2 is started after boot/reboot --> works

I am not 100% sure this covers all circumstances in the boot process, but for me it feels good enough to move forward.