JVM Crash

touitoui · October 23, 2021, 7:11pm

Hi,

recently, when adding points from things to semantic model, my JVM crashes. This is an example log of such a crash: hs_err_pid22071.log (104.7 KB)

As this is most probably not an openhab but an OpenJDK issue, I should probably change JVM vendor. Can you recommend alternatives?

mstormi · October 23, 2021, 9:11pm

Mind the posting rules please. Minimum would have been to mention what you are using.

How to ask a good question / Help Us Help You - Tutorials & Examples - openHAB Community

Wolfgang_S · October 24, 2021, 8:21am

They are described in the prerequisites: Installation Overview - prerequisites | openHAB

touitoui · October 24, 2021, 8:51am

Platform information:

Hardware: Raspberry Pi 2 Model B Rev 1.1
OS: Raspbian GNU/Linux 10 (buster)
openHABian: openHAB3 Branch
Java Runtime Environment:
openjdk version “11.0.11” 2021-04-20
OpenJDK Runtime Environment AdoptOpenJDK-11.0.11+9 (build 11.0.11+9)
OpenJDK Server VM AdoptOpenJDK-11.0.11+9 (build 11.0.11+9, mixed mod
openHAB version: openHAB 3.1.0

Just tried with AdoptOpenJDK, same symptoms: hs_err_pid15221.log (105.5 KB)

This was just one bugfix version older than my initial jvm, so no wonder. Will try another implementation.

touitoui · October 24, 2021, 12:54pm

Next try, OpenJDK Runtime Environment Zulu11.50+19-CA (11.0.12+7), crashes also, but with a different frame message: hs_err_pid21957.log (109.2 KB)

I’m slowly running out of JVM implementations… The last bigger change to my system was adding the ipcamera binding. Will delete it and try again. Although I don’t have much hope that this is the root cause.

touitoui · October 24, 2021, 12:59pm

org.openhab.persistence.rrd4j.internal.RRD4jPersistenceService.getDB is in the stack trace before the crash occurs.

touitoui · October 24, 2021, 1:06pm

Deleting the rrd4j persistence altogether seems to fix it. Fortunately I don’t use rrd4j anyway, so this should work for me.

Sascha_Billian · October 24, 2021, 3:06pm

There has been a problem with RRD4J persistence
<wrong> in combination with ZRAM</wrong> (EDIT: According to the GitHub issue, this is not related with ZRAM). In very rare occasions one of the rrd files got corrupted resulting in java crashes:

github.com/openhab/openhab-core

java crashing in persistence handling

opened 12:56PM - 30 Dec 20 UTC

mstormi

external bug

My OH Java (3.0 release as well as latest snapshot) keeps crashing every few min…utes, Java hs_err_pidxxxxx.log below It's always in rrd4j processing. At times I saw an error like ``` 2020-12-30 13:19:12.834 [ERROR] [d4j.internal.RRD4jPersistenceService] - Could not create rrd4j database file '/var/lib/openhab/persistence/rrd4j/K_Waschkeller_Thermo_Batterie.rrd': Invalid file header. File [/var/lib/openhab/persistence/rrd4j/K_Waschkeller_Thermo_Batterie.rrd] is not a RRD4J RRD file ``` But it warns of various different item.rrd files it cannot persist into. And I have already deleted all of my persisted .rrd files in /var/lib/openhab/persistence/rrd4j so Java complains about a file it created itself in those few minutes it ran. [ EDIT: eventually it did not really "remove" the file because the persistence dir is on ZRAM and what's on disk before ZRAM start then will be used ... complicated to explain but if there's an error in *that* file deleting it doesn't help. ] Either way, Java must not crash even if the persistence dir is gone or files in there have bad permissions. ``` # # A fatal error has been detected by the Java Runtime Environment: # # SIGBUS (0x7) at pc=0x76ba3d84, pid=16426, tid=17020 # # JRE version: OpenJDK Runtime Environment Zulu11.43+88-CA (11.0.9+11) (build 11.0.9+11-LTS) # Java VM: OpenJDK Client VM Zulu11.43+88-CA (11.0.9+11-LTS, mixed mode, serial gc, linux-arm) # Problematic frame: # V [libjvm.so+0x57cd84] _Copy_conjoint_jints_atomic+0x24 # # No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again # # If you would like to submit a bug report, please visit: # http://www.azulsystems.com/support/ # --------------- S U M M A R Y ------------ Command Line: -XX:-UsePerfData -Dopenhab.home=/usr/share/openhab -Dopenhab.conf=/etc/openhab -Dopenhab.runtime=/usr/share/openhab/run time -Dopenhab.userdata=/var/lib/openhab -Dopenhab.logdir=/var/log/openhab -Dfelix.cm.dir=/var/lib/openhab/config -Djava.library.path =/var/lib/openhab/tmp/lib -Djetty.host=0.0.0.0 -Djetty.http.compliance=RFC2616 -Dnashorn.args=--no-deprecation-warning -Dorg.ops4j.pa x.web.listening.addresses=0.0.0.0 -Dorg.osgi.service.http.port=8080 -Dorg.osgi.service.http.port.secure=8443 -Djava.awt.headless=true -Xms192m -Xmx320m --add-reads=java.xml=java.logging --add-exports=java.base/org.apache.karaf.specs.locator=java.xml,ALL-UNNAMED --pa tch-module=java.base=/usr/share/openhab/runtime/lib/endorsed/org.apache.karaf.specs.locator-4.2.7.jar --patch-module=java.xml=/usr/sh are/openhab/runtime/lib/endorsed/org.apache.karaf.specs.java.xml-4.2.7.jar --add-opens=java.base/java.security=ALL-UNNAMED --add-open s=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens= java.naming/javax.naming.spi=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport.tcp=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNN AMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.text=ALL-UNNAMED --add-opens=java.desktop/java.aw t.font=ALL-UNNAMED --add-exports=java.base/sun.net.www.protocol.http=ALL-UNNAMED --add-exports=java.base/sun.net.www.protocol.https=A LL-UNNAMED --add-exports=java.base/sun.net.www.protocol.jar=ALL-UNNAMED --add-exports=jdk.xml.dom/org.w3c.dom.html=ALL-UNNAMED --add- exports=jdk.naming.rmi/com.sun.jndi.url.rmi=ALL-UNNAMED -Dkaraf.instances=/var/lib/openhab/tmp/instances -Dkaraf.home=/usr/share/open hab/runtime -Dkaraf.base=/var/lib/openhab -Dkaraf.data=/var/lib/openhab -Dkaraf.etc=/var/lib/openhab/etc -Dkaraf.log=/var/log/openhab -Dkaraf.restart.jvm.supported=true -Djava.io.tmpdir=/var/lib/openhab/tmp -Djava.util.logging.config.file=/var/lib/openhab/etc/java.u til.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true org.apache.karaf.main.Main Host: rev 4 (v7l), 4 cores, 974M, Raspbian GNU/Linux 10 (buster) Time: Wed Dec 30 13:19:12 2020 CET elapsed time: 123.833624 seconds (0d 0h 2m 3s) --------------- T H R E A D --------------- Current thread (0x5bc73c00): JavaThread "OH-items-4" [_thread_in_vm, id=17020, stack(0x55d60000,0x55db0000)] Stack: [0x55d60000,0x55db0000], sp=0x55dae494, free space=313k Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code) V [libjvm.so+0x57cd84] _Copy_conjoint_jints_atomic+0x24 C 0x55dae4bc J 1637 jdk.internal.misc.Unsafe.copyMemory0(Ljava/lang/Object;JLjava/lang/Object;JJ)V java.base@11.0.9 (0 bytes) @ 0x7458c330 [0x745 8c280+0x000000b0] J 7917 c1 jdk.internal.misc.Unsafe.copyMemory(Ljava/lang/Object;JLjava/lang/Object;JJ)V java.base@11.0.9 (33 bytes) @ 0x74a08a64 [0x7 4a08860+0x00000204] j java.nio.DirectByteBuffer.put([BII)Ljava/nio/ByteBuffer;+110 java.base@11.0.9 j java.nio.ByteBuffer.put([B)Ljava/nio/ByteBuffer;+5 java.base@11.0.9 j org.rrd4j.core.RrdNioBackend.write(J[B)V+22 j org.rrd4j.core.RrdBackend.writeString(JLjava/lang/String;I)V+80 j org.rrd4j.core.RrdBackend.writeString(JLjava/lang/String;)V+188 j org.rrd4j.core.RrdPrimitive.writeString(Ljava/lang/String;)V+9 j org.rrd4j.core.RrdString.set(Ljava/lang/String;)V+40 j org.rrd4j.core.Header.<init>(Lorg/rrd4j/core/RrdDb;Lorg/rrd4j/core/RrdDef;)V+135 j org.rrd4j.core.RrdDb.<init>(Lorg/rrd4j/core/RrdDef;Lorg/rrd4j/core/RrdBackendFactory;)V+103 j org.rrd4j.core.RrdDb.<init>(Lorg/rrd4j/core/RrdDef;)V+3 j org.openhab.persistence.rrd4j.internal.RRD4jPersistenceService.getDB(Ljava/lang/String;)Lorg/rrd4j/core/RrdDb;+109 j org.openhab.persistence.rrd4j.internal.RRD4jPersistenceService.store(Lorg/openhab/core/items/Item;Ljava/lang/String;)V+49 j org.openhab.core.persistence.internal.PersistenceManagerImpl.handleStateEvent(Lorg/openhab/core/items/Item;Z)V+162 j org.openhab.core.persistence.internal.PersistenceManagerImpl.stateChanged(Lorg/openhab/core/items/Item;Lorg/openhab/core/types/Sta te;Lorg/openhab/core/types/State;)V+3 j org.openhab.core.items.GenericItem.lambda$1(Lorg/openhab/core/items/StateChangeListener;Lorg/openhab/core/types/State;ZLorg/openha b/core/types/State;)V+17 j org.openhab.core.items.GenericItem$$Lambda$931.run()V+20 J 3982 c1 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V java.base@11.0.9 (187 bytes) @ 0x74413260 [0x74413020+0x00000240] j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5 java.base@11.0.9 j java.lang.Thread.run()V+11 java.base@11.0.9 v ~StubRoutines::call_stub V [libjvm.so+0x371aec] JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, Thread*)+0x2b4 V [libjvm.so+0x3702e8] JavaCalls::call_virtual(JavaValue*, Handle, Klass*, Symbol*, Symbol*, Thread*)+0x140 V [libjvm.so+0x412470] thread_entry(JavaThread*, Thread*)+0xa0 V [libjvm.so+0x68d7ac] JavaThread::thread_main_inner()+0xf0 V [libjvm.so+0x68a970] Thread::call_run()+0x158 V [libjvm.so+0x579a4c] thread_native_entry(Thread*)+0x108 C [libpthread.so.0+0x6494] start_thread+0xcc Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) J 1637 jdk.internal.misc.Unsafe.copyMemory0(Ljava/lang/Object;JLjava/lang/Object;JJ)V java.base@11.0.9 (0 bytes) @ 0x7458c300 [0x745 8c280+0x00000080] J 7917 c1 jdk.internal.misc.Unsafe.copyMemory(Ljava/lang/Object;JLjava/lang/Object;JJ)V java.base@11.0.9 (33 bytes) @ 0x74a08a64 [0x7 4a08860+0x00000204] j java.nio.DirectByteBuffer.put([BII)Ljava/nio/ByteBuffer;+110 java.base@11.0.9 j java.nio.ByteBuffer.put([B)Ljava/nio/ByteBuffer;+5 java.base@11.0.9 j org.rrd4j.core.RrdNioBackend.write(J[B)V+22 j org.rrd4j.core.RrdBackend.writeString(JLjava/lang/String;I)V+80 j org.rrd4j.core.RrdBackend.writeString(JLjava/lang/String;)V+188 j org.rrd4j.core.RrdPrimitive.writeString(Ljava/lang/String;)V+9 j org.rrd4j.core.RrdString.set(Ljava/lang/String;)V+40 j org.rrd4j.core.Header.<init>(Lorg/rrd4j/core/RrdDb;Lorg/rrd4j/core/RrdDef;)V+135 j org.rrd4j.core.RrdDb.<init>(Lorg/rrd4j/core/RrdDef;Lorg/rrd4j/core/RrdBackendFactory;)V+103 j org.rrd4j.core.RrdDb.<init>(Lorg/rrd4j/core/RrdDef;)V+3 j org.openhab.persistence.rrd4j.internal.RRD4jPersistenceService.getDB(Ljava/lang/String;)Lorg/rrd4j/core/RrdDb;+109 j org.openhab.persistence.rrd4j.internal.RRD4jPersistenceService.store(Lorg/openhab/core/items/Item;Ljava/lang/String;)V+49 j org.openhab.core.persistence.internal.PersistenceManagerImpl.handleStateEvent(Lorg/openhab/core/items/Item;Z)V+162 j org.openhab.core.persistence.internal.PersistenceManagerImpl.stateChanged(Lorg/openhab/core/items/Item;Lorg/openhab/core/types/Sta te;Lorg/openhab/core/types/State;)V+3 j org.openhab.core.items.GenericItem.lambda$1(Lorg/openhab/core/items/StateChangeListener;Lorg/openhab/core/types/State;ZLorg/openha b/core/types/State;)V+17 j org.openhab.core.items.GenericItem$$Lambda$931.run()V+20 J 3982 c1 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V java.base@11.0.9 (187 bytes) @ 0x74413260 [0x74413020+0x00000240] j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5 java.base@11.0.9 j java.lang.Thread.run()V+11 java.base@11.0.9 v ~StubRoutines::call_stub siginfo: si_signo: 7 (SIGBUS), si_code: 2 (BUS_ADRERR), si_addr: 0x53c7b000 ```

But that’s just a wild guess. But you can compare your logs with the ones from the issues.
However, even if there’s still a bug open, it didn’t happen to me for quite some time and i’m using rrd4j with ZRAM enabled.

touitoui · October 24, 2021, 3:29pm

Does not sound wild at all to me, thanks for pointing that out. Pretty sure it’s the same issue. For me it happened when creating enough items at the same time. Creating just one item was fine.