Docker container does not start on Raspberry lite 64: openhab/runtime/lib/endorsed is not supported

I use an RPI3b+ on the latest Raspberry lite 64.
I used the Pi imager and set the OS up from scratch. Boots flawlessly.
The I try to use a plain (as provided) docker container - and fail.

Thats what I did:

install docker

curl -fsSL https://get.Docker.com -o get-Docker.sh;
sudo sh get-Docker.sh;
sudo usermod -aG docker $USER
newgrp docker

prepare openhab folders

sudo useradd -r -s /sbin/nologin openhab
sudo usermod -a -G openhab pi
sudo mkdir -p /opt/openhab/{conf,userdata,addons}
sudo chown -R openhab:openhab /opt/openhab

docker test

test with docker helloWorld executes just fine.

docker run

docker run
–name openhab
–net=host
-v /etc/localtime:/etc/localtime:ro
-v /etc/timezone:/etc/timezone:ro
-v /opt/openhab/conf:/openhab/conf
-v /opt/openhab/userdata:/openhab/userdata
-v /opt/openhab/addons:/openhab/addons
-d
-e USER_ID=999
-e GROUP_ID=994
-e OPENHAB_HTTP_PORT=8080
-e OPENHAB_HTTPS_PORT=8444
-e CRYPTO_POLICY=unlimited
–restart=always
openhab/openhab:3.4.2

docker logs openhab reads:


++ cmp /openhab/userdata/etc/version.properties /openhab/dist/userdata/etc/version.properties
+ '[' '!' -z ']'
+ chown -R openhab:openhab /openhab
+ sync
+ '[' -d /etc/cont-init.d ']'
+ sync
+ '[' false == false ']'
++ IFS=' '
++ echo gosu openhab tini -s ./start.sh
+ '[' 'gosu openhab tini -s ./start.sh' == 'gosu openhab tini -s ./start.sh' ']'
+ command=($@ server)
+ exec gosu openhab tini -s ./start.sh server
Launching the openHAB runtime...
Aborted (core dumped)
Aborted (core dumped)
./runtime/bin/karaf: 247: [: Illegal number:
./runtime/bin/karaf: 97: [: Illegal number:
./runtime/bin/karaf: 300: [: Illegal number:
-Djava.endorsed.dirs=/usr/lib/jvm/temurin-11-jdk-arm64/jre/lib/endorsed:/usr/lib/jvm/temurin-11-jdk-arm64/lib/endorsed:/openhab/runtime/lib/endorsed is not supported. Endorsed standards and standalone APIs
in modular form will be supported via the concept of upgradeable modules.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

console says:

pi@raspberrypi:~ $ docker attach pi-openhab-1
Aborted (core dumped)
./runtime/bin/karaf: 247: [: Illegal number:
Aborted (core dumped)
Aborted (core dumped)
./runtime/bin/karaf: 247: [: Illegal number:
./runtime/bin/karaf: 97: [: Illegal number:
./runtime/bin/karaf: 300: [: Illegal number:
-Djava.endorsed.dirs=/usr/lib/jvm/temurin-11-jdk-arm64/jre/lib/endorsed:/usr/lib/jvm/temurin-11-jdk-arm64/lib/endorsed:/openhab/runtime/lib/endorsed is not supported. Endorsed standards and standalone APIs
in modular form will be supported via the concept of upgradeable modules.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

Edit:
In th meantime I tried several other versions, last one working is 3.1.0.
3.2.0, 3.3.0 and 3.4.x do not start.

For me the arm64 image works without issues on a RPi 4 running Ubuntu 22.04.2.

I think the main difference between those OH versions is that Debian 11 is used as base image since OH 3.2 (instead of Debian 10).

Do other Debian 11 based images work properly for you?

The openjdk:11 image also uses Debian 11 and Java 11 so you could test if that works by just letting it print the Java version:

docker run --rm openjdk:11 java --version

If you run an old Raspberry Pi OS version as host OS it does not properly support time64. Java would also complain about this by printing “No monotonic clock was available”. But I would not expect this to be an issue if you run a recent bullseye based Raspberry Pi OS version.

See: Docker openhab:3.2.0-snapshot stuck at unhealthy with OpenJDK Client VM warning: No monotonic clock was available

Perhaps you can also post the output of:

docker info

The openjdk reported

Status: Downloaded newer image for openjdk:11
openjdk 11.0.16 2022-07-19
OpenJDK Runtime Environment 18.9 (build 11.0.16+8)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.16+8, mixed mode, sharing)

docker infor reports:

pi@rpi3i:~ $ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.16.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 2
  Running: 2
  Paused: 0
  Stopped: 0
 Images: 3
 Server Version: 23.0.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2456e983eb9e37e47538f59ea18f2043c9a73640
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.84-v8+
 Operating System: Debian GNU/Linux 11 (bullseye)
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 909.6MiB
 Name: rpi3i
 ID: 8a9f070e-1ecf-45a6-a7ea-37fbe899ea8c
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No memory limit support
WARNING: No swap limit support
pi@rpi3i:~ $

Just tried Ubuntu 22.04.2 LTS 64 bit. Same error.

 Aborted (core dumped)
openhab      | ./runtime/bin/karaf: 247: [: Illegal number:
openhab      | Aborted (core dumped)
openhab      | Aborted (core dumped)
openhab      | ./runtime/bin/karaf: 247: [: Illegal number:
openhab      | ./runtime/bin/karaf: 97: [: Illegal number:
openhab      | ./runtime/bin/karaf: 300: [: Illegal number:
openhab      | Error: Could not create the Java Virtual Machine.
openhab      | Error: A fatal exception has occurred. Program will exit.
openhab      | -Djava.endorsed.dirs='' is not supported. Endorsed standards and standalone APIs
openhab      | in modular form will be supported via the concept of upgradeable modules.
openhab exited with code 1

It looks OK to me! Maybe there are some (kernel/sys/docker) log files that explain why the crashes occur?

Maybe if I find some time I can also try it on my own RPi3+ that is still lying around somewhere for issues like this. :wink:

I’ll keep trying on my end. I suspect it may have to do with me not providing my config folder to the docker. I started it straight from the dockerhub.

If it is volume related you could also test if it does work without mapping the volumes with these options.

The volume dirs should be completely empty when first starting the image otherwise they won’t be properly initialized.

If I start it with empty directories, does start up. (tested with 3.4.1)
It does not start if

  • I do not mount the directories at all.
  • if the directories are populated with my config from openhabian or
  • if any other docker container runs additionally. Then the RPI runs out or memory (1GB)

I was able to reproduce this on my own RPi 3. :slightly_frowning_face: It looks like the JVM crashes and it also creates hs_err_pid log files in the userdata dir for this. Do you also have these? I’ll have to further investigate what’s causing these crashes.

As a workaround for now you could also try the Alpine image which does seem to work.

1 Like

I am also experiencing this error on my Raspberry 3B+ running 64bit RaspiOS Debian Bullseye:

Launching the openHAB runtime...
Aborted (core dumped)
./runtime/bin/karaf: 247: [: Illegal number: 
./runtime/bin/karaf: 97: [: Illegal number: 
./runtime/bin/karaf: 300: [: Illegal number: 
-Djava.endorsed.dirs=/usr/lib/jvm/temurin-11-jdk-arm64/jre/lib/endorsed:/usr/lib/jvm/temurin-11-jdk-arm64/lib/endorsed:/openhab/runtime/lib/endorsed is not supported. Endorsed standards and standalone APIs
in modular form will be supported via the concept of upgradeable modules.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

openJDK runs fine in Docker:

$ docker run --rm openjdk:11 java --version
Status: Downloaded newer image for openjdk:11
openjdk 11.0.16 2022-07-19
OpenJDK Runtime Environment 18.9 (build 11.0.16+8)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.16+8, mixed mode, sharing)

Edit: I uploaded the log files from my /opt/openhab/userdata directory.

If I switch the JVM from Temurin to the standard Debian OpenJDK it no longer seems to crash on my Raspberry Pi 3. You can test that change using the wborn/openhab:3.4.3 images I just uploaded.

Was a definitive solution ever found for this problem?

I have experienced this exact error for the first time trying to upgrade two different containers from 4.0M3 image → 4.0M4 image.

We can switch JVMs if someone can provide positive feedback on if the images using the standard Debian OpenJDK solves their issues. :slight_smile:

I build one for 4.0.0.M4 too now so maybe you can test it: wborn/openhab:4.0.0.M4

What hardware architecture and OS (32/64-bit) do you use ?

1 Like

I’m running on a linux box with Ubuntu 22.04 x64. I’ve been running some version of the same container for my production system since OH3.0 and never seen this issue until now.

Wonderful. I’ll give it a test after work today and report back.

I just tested this image with a duplicate of production my container and get the same error still. I have a second smaller container that I sometimes run but I’m seeing the same error in that one with any M4 images.

I do much of my docker admin with Portainer, and I’m aware of the ENV concern. I have never had to delete the old ENV vars to get image updates working, but just in case, I also tried this image on a duplicate of my container with just my custom ports and user and group IDs and I still get the same errors.

It could also be due to the Debian 12 base image update. I ran into some similar issues on another project while using Java in Debian 12 containers. Instead of the JVM crashing it exits due to insufficient memory being available. Still have to figure out why that is. :confused:

What do you see when you use the image to print the Java version?

docker run --rm openhab/openhab:4.0.0.M4 java -version

Yep, there does seem to be a memory issue:

++ echo java -version
+ '[' 'java -version' == 'gosu openhab tini -s ./start.sh' ']'
+ exec java -version
[0.025s][warning][os,thread] Failed to start thread "GC Thread#0" - pthread_create failed (EPERM) for attributes: stacksize: 1024k, guardsize: 4k, detached.
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create worker GC thread. Out of system resources.
# An error report file with more information is saved as:
# /openhab/hs_err_pid1.log

It might be better to use Debian 11 again until it is clear why the memory usage increases with Debian 12.

I pushed new 3.4.4 and 4.0.0.M4 images based on openhab-docker#423 to my Docker Hub repo:

  • wborn/openhab:3.4.4
  • wborn/openhab:4.0.0.M4

Hopefully that fixes the crashes and memory issues and doesn’t introduce new issues.

Maybe you can give it a test?

2 Likes

Yes! The debian 11 image works just fine.

Thank you for all the effort.