[Solved] Regular JRE crashes on Fedora 27 on Pine64

cmuellner · February 15, 2018, 9:10am

I’ve installed OpenHAB2 on my Pine64, running Fedora 27 (aarch64)
and observing very frequent crashes of the Java Runtime Environment
with journalctl -f.

A typical message looks like this:

Feb 15 09:39:19 openhab karaf[17184]: #
Feb 15 09:39:19 openhab karaf[17184]: # A fatal error has been detected by the Java Runtime Environment:
Feb 15 09:39:19 openhab karaf[17184]: #
Feb 15 09:39:19 openhab karaf[17184]: #  SIGSEGV (0xb) at pc=0x0000ffff93ea1a84, pid=17184, tid=0x0000ffff802cc1f0
Feb 15 09:39:19 openhab karaf[17184]: #
Feb 15 09:39:19 openhab karaf[17184]: # JRE version: OpenJDK Runtime Environment (8.0_161-b14) (build 1.8.0_161-b14)
Feb 15 09:39:19 openhab karaf[17184]: # Java VM: OpenJDK 64-Bit Server VM (25.161-b14 mixed mode linux-aarch64 compressed oops)
Feb 15 09:39:19 openhab karaf[17184]: # Problematic frame:
Feb 15 09:39:19 openhab karaf[17184]: # V  [libjvm.so+0x394a84]
Feb 15 09:39:19 openhab karaf[17184]: #
Feb 15 09:39:19 openhab karaf[17184]: # Core dump written. Default location: /var/lib/openhab2/core or core.17184
Feb 15 09:39:19 openhab karaf[17184]: #
Feb 15 09:39:19 openhab karaf[17184]: # An error report file with more information is saved as:
Feb 15 09:39:19 openhab karaf[17184]: # /var/lib/openhab2/hs_err_pid17184.log
Feb 15 09:39:19 openhab karaf[17184]: [error occurred during error reporting , id 0xb]
Feb 15 09:39:19 openhab karaf[17184]: #
Feb 15 09:39:19 openhab karaf[17184]: # If you would like to submit a bug report, please visit:
Feb 15 09:39:19 openhab karaf[17184]: #   http://bugreport.java.com/bugreport/crash.jsp
Feb 15 09:39:19 openhab karaf[17184]: #

That message is followed by a stacktrace. When looking at several stacktraces
I’ve not seen any similarities so far (different number of threads, and different
locations in code).

A typical stacktrace looks like this:

Feb 15 09:41:17 openhab systemd-coredump[17619]: Process 17184 (java) of user 995 dumped core.
                                                 
                                                 Stack trace of thread 17372:
                                                 #0  0x0000ffff948dce48 raise (libc.so.6)
                                                 #1  0x0000ffff948dec68 abort (libc.so.6)
                                                 #2  0x0000ffff943515f4 _ZN2os5abortEb (libjvm.so)
                                                 #3  0x0000ffff9456b464 _ZN7VMError14report_and_dieEv (libjvm.so)
                                                 #4  0x0000ffff9456b954 _ZL13crash_handleriP9siginfo_tPv (libjvm.so)
                                                 #5  0x0000ffff94b3466c n/a (linux-vdso.so.1)
                                                 #6  0x0000ffff94b3466c n/a (linux-vdso.so.1)
                                                 #7  0x0000ffff93e986c8 _ZN12ciMethodData16dump_replay_dataEP12outputStream (libjvm.so)
                                                 #8  0x0000ffff9456b10c _ZN7VMError14report_and_dieEv (libjvm.so)
                                                 #9  0x0000ffff9435c5d0 JVM_handle_linux_signal (libjvm.so)
                                                 #10 0x0000ffff9434ed44 _Z13signalHandleriP9siginfo_tPv (libjvm.so)
                                                 #11 0x0000ffff94b3466c n/a (linux-vdso.so.1)
                                                 #12 0x0000ffff94b3466c n/a (linux-vdso.so.1)
                                                 #13 0x0000ffff93ea242c _ZN15ciObjectFactory12get_metadataEP8Metadata (libjvm.so)
                                                 #14 0x0000ffff93e9ba6c _ZN17ciVirtualCallData14translate_fromEPK11ProfileData (libjvm.so)
                                                 #15 0x0000ffff93e9a668 _ZN12ciMethodData9load_dataEv (libjvm.so)
                                                 #16 0x0000ffff93e8e5b8 _ZN8ciMethod18ensure_method_dataE12methodHandle (libjvm.so)
                                                 #17 0x0000ffff93e8ea78 _ZN8ciMethod18ensure_method_dataEv (libjvm.so)
                                                 #18 0x0000ffff93dd94cc _ZN12GraphBuilder15try_inline_fullEP8ciMethodbN9Bytecodes4CodeEP11Instruction (libjvm.so)
                                                 #19 0x0000ffff93dd9ee4 _ZN12GraphBuilder10try_inlineEP8ciMethodbN9Bytecodes4CodeEP11Instruction (libjvm.so)
                                                 #20 0x0000ffff93dda9a8 _ZN12GraphBuilder6invokeEN9Bytecodes4CodeE (libjvm.so)
                                                 #21 0x0000ffff93dd6bb4 _ZN12GraphBuilder27iterate_bytecodes_for_blockEi (libjvm.so)
                                                 #22 0x0000ffff93dd8610 _ZN12GraphBuilder18iterate_all_blocksEb (libjvm.so)
                                                 #23 0x0000ffff93dd8a5c _ZN12GraphBuilderC2EP11CompilationP7IRScope (libjvm.so)
                                                 #24 0x0000ffff93dde208 _ZN7IRScopeC1EP11CompilationPS_iP8ciMethodib (libjvm.so)
                                                 #25 0x0000ffff93dded28 _ZN2IRC2EP11CompilationP8ciMethodi (libjvm.so)
                                                 #26 0x0000ffff93dc057c _ZN11Compilation9build_hirEv.part.93 (libjvm.so)
                                                 #27 0x0000ffff93dc2458 _ZN11Compilation19compile_java_methodEv (libjvm.so)
                                                 #28 0x0000ffff93dc2638 _ZN11Compilation14compile_methodEv (libjvm.so)
                                                 #29 0x0000ffff93dc2a20 _ZN11CompilationC1EP16AbstractCompilerP5ciEnvP8ciMethodiP10BufferBlob (libjvm.so)
                                                 #30 0x0000ffff93dc34f8 _ZN8Compiler14compile_methodEP5ciEnvP8ciMethodi (libjvm.so)
                                                 #31 0x0000ffff93f15f00 _ZN13CompileBroker25invoke_compiler_on_methodEP11CompileTask (libjvm.so)
                                                 #32 0x0000ffff93f17e88 _ZN13CompileBroker20compiler_thread_loopEv (libjvm.so)
                                                 #33 0x0000ffff9450dc40 _ZN10JavaThread17thread_main_innerEv (libjvm.so)
                                                 #34 0x0000ffff9450df80 _ZN10JavaThread3runEv (libjvm.so)
                                                 #35 0x0000ffff94350558 _ZL10java_startP6Thread (libjvm.so)
                                                 #36 0x0000ffff94ac8204 start_thread (libpthread.so.0)
                                                 #37 0x0000ffff94996f1c thread_start (libc.so.6)
                                                 
                                                 Stack trace of thread 17363:
                                                 #0  0x0000ffff94acec94 pthread_cond_wait@@GLIBC_2.17 (libpthread.so.0)
                                                 #1  0x0000ffff94359588 _ZN2os13PlatformEvent4parkEv (libjvm.so)
                                                 #2  0x0000ffff9430f8f8 _ZN7Monitor5IWaitEP6Threadl (libjvm.so)
                                                 #3  0x0000ffff943111d8 _ZN7Monitor4waitEblb (libjvm.so)
                                                 #4  0x0000ffff9402e070 _ZN13GCTaskManager8get_taskEj (libjvm.so)
                                                 #5  0x0000ffff9402fddc _ZN12GCTaskThread3runEv (libjvm.so)
                                                 #6  0x0000ffff94350558 _ZL10java_startP6Thread (libjvm.so)
                                                 #7  0x0000ffff94ac8204 start_thread (libpthread.so.0)
                                                 #8  0x0000ffff94996f1c thread_start (libc.so.6)
                                                 
                                                 Stack trace of thread 17184:
                                                 #0  0x0000ffff94ac96e8 pthread_join (libpthread.so.0)
                                                 #1  0x0000ffff94a76fb4 ContinueInNewThread0 (libjli.so)
                                                 #2  0x0000ffff94a73924 ContinueInNewThread (libjli.so)
                                                 #3  0x0000ffff94a73f80 JLI_Launch (libjli.so)
                                                 #4  0x0000000000400638 main (java)
                                                 #5  0x0000ffff948c7bec __libc_start_main (libc.so.6)
                                                 #6  0x0000000000400670 _start (java)
                                                 #7  0x0000000000400670 _start (java)

Any thoughts?
Any similar experience reports with OpenHABian on Pine64?

Dim · February 15, 2018, 9:24am

try either Oracle or Zulu versions of Java to see if this happens
try also 32bit JVM

cmuellner · February 15, 2018, 12:07pm

Thanks for the suggestion!
I’ve installed Zulu VM and could not reproduce the crashes anymore so far.

In case someone stumbles over the same issue, here are the steps I’ve taken to address the issue:

Download Zulu VM. Zulu offers two versions of their arm64 JDK:
a) The normal arm64 one (157 MiB):
http://cdn.azul.com/zulu-embedded/bin/ezdk-1.8.0_152-8.25.0.79-eval-linux_aarch64.tar.gz
b) The Qualcomm specific one (71 MiB):
https://cdn.azul.com/zulu-embedded/qc/ezdk-1.8.0_152-8.25.0.79-eval-linux_aarch64.tar.gz
As you can see, they did not manage to name them different, so be careful what you download.
I choose the normal one.
Extract with tar xf ezdk-1.8.0_152-8.25.0.79-eval-linux_aarch64.tar.gz and copy the extracted
folder to /usr/lib/jvm with sudo cp -a ezdk-1.8.0_152-8.25.0.79-eval-linux_aarch64 /usr/lib/jvm.
Install as java alternative by issuing the following command:
alternatives --install /usr/bin/java java /usr/lib/jvm/ezdk-1.8.0_152-8.25.0.79-eval-linux_aarch64/jre/bin/java 1
Set Zulu VM as default java binary with the following command:
alternatives --config java #Select the ezdk
Test if java if installation succeeded, by running java -version:

openjdk version “1.8.0_152”
OpenJDK Runtime Environment (Zulu Embedded 8.25.0.79-linux-aarch64) (build 1.8.0_152-b79)
OpenJDK 64-Bit Server VM (Zulu Embedded 8.25.0.79-linux-aarch64) (build 25.152-b79, mixed mode, Evaluation)

Restart OpenHAB2 with sudo systemctl restart openhab2

Dim · February 16, 2018, 3:59am

very nice description @cmuellner !
@Kai : another case where the “standard” OpenJDK is having “hiccups” while Zulu (which is an “optimized” version of OpenJDK works fine)