openHAB 3.1: problems with ThreadPoolExecutor and subsequenlty OutOfMemory errors

  • Platform information:
    • Hardware: Raspberry Pi 4+ (4GB)
    • OS: openHABian [openHAB3]{2021-05-05T15:53:37-06:00}(3e35b6d) => Raspbian GNU/Linux 10 (buster: 5.10.17-v7l+ #1421)
    • Java Runtime Environment:
openjdk version "11.0.11" 2021-04-20 LTS
OpenJDK Runtime Environment Zulu11.48+21-CA (build 11.0.11+9-LTS)
OpenJDK Client VM Zulu11.48+21-CA (build 11.0.11+9-LTS, mixed mode)
  • openHAB version: 3.1.0 - Release Build

Issue
After upgrading to OH3.1 I encounter some OOM-related errors, which lead to unexpected behaviour, most notably strangely enough, that I can read the KNX-statuses, but not write telegrams.

current status before reboot

● openhab.service - openHAB - empowering the smart home
   Loaded: loaded (/lib/systemd/system/openhab.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2021-07-10 16:53:44 CEST; 14h ago
     Docs: https://www.openhab.org/docs/
           https://community.openhab.org
 Main PID: 671 (java)
    Tasks: 4912 (limit: 4915)
   CGroup: /system.slice/openhab.service
           └─671 /usr/bin/java -XX:-UsePerfData -Dopenhab.home=/usr/share/openhab -Dopenhab.conf=/etc/openhab -Dopenhab.runtime=/usr/share/openhab/runtime -Dopenhab.userdata=/var/lib/openhab -Dopenhab.logdir=/var/log/openhab -Dfelix.cm.dir=/var/lib/openhab/config -Djava.library.path=/var/lib/openhab/tmp/lib -Djetty.host=0.0.0.0 -Djetty.http.compliance=RFC2616 -Dnashorn.args=--no-deprecation-warning -Dorg.apache.cxf.osgi.http.transport.disable=true -Dorg.ops4j.pax.web.listening.addresses=0.0.0.0 -Dorg.osgi.service.http.port=8080 -Dorg.osgi.service.http.port.secure=8443 -Djava.awt.headless=true -Xms16m -Xmx256m --add-reads=java.xml=java.logging --add-exports=java.base/org.apache.karaf.specs.locator=java.xml,ALL-UNNAMED --patch-module java.base=/usr/share/openhab/runtime/lib/endorsed/org.apache.karaf.specs.locator-4.3.2.jar --patch-module java.xml=/usr/share/openhab/runtime/lib/endorsed/org.apache.karaf.specs.java.xml-4.3.2.jar --add-opens java.base/java.security=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.naming/javax.naming.spi=ALL-UNNAMED --add-opens java.rmi/sun.rmi.transport.tcp=ALL-UNNAMED --add-opens java.base/java.io=ALL-UNNAMED --add-opens java.base/java.lang.reflect=ALL-UNNAMED --add-opens java.base/java.text=ALL-UNNAMED --add-opens java.base/java.time=ALL-UNNAMED --add-opens java.desktop/java.awt.font=ALL-UNNAMED --add-exports=java.base/sun.net.www.protocol.file=ALL-UNNAMED --add-exports=java.base/sun.net.www.protocol.ftp=ALL-UNNAMED --add-exports=java.base/sun.net.www.protocol.http=ALL-UNNAMED --add-exports=java.base/sun.net.www.protocol.https=ALL-UNNAMED --add-exports=java.base/sun.net.www.protocol.jar=ALL-UNNAMED --add-exports=java.base/sun.net.www.content.text=ALL-UNNAMED --add-exports=jdk.xml.dom/org.w3c.dom.html=ALL-UNNAMED --add-exports=jdk.naming.rmi/com.sun.jndi.url.rmi=ALL-UNNAMED --add-exports java.security.sasl/com.sun.security.sasl=ALL-UNNAMED -Dkaraf.instances=/var/lib/openhab/tmp/instances -Dkaraf.home=/usr/share/openhab/runtime -Dkaraf.base=/var/lib/openhab -Dkaraf.data=/var/lib/openhab -Dkaraf.etc=/var/lib/openhab/etc -Dkaraf.log=/var/log/openhab -Dkaraf.restart.jvm.supported=true -Djava.io.tmpdir=/var/lib/openhab/tmp -Djava.util.logging.config.file=/var/lib/openhab/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -classpath /usr/share/openhab/runtime/lib/boot/org.apache.karaf.diagnostic.boot-4.3.2.jar:/usr/share/openhab/runtime/lib/boot/org.apache.karaf.jaas.boot-4.3.2.jar:/usr/share/openhab/runtime/lib/boot/org.apache.karaf.main-4.3.2.jar:/usr/share/openhab/runtime/lib/boot/org.apache.karaf.specs.activator-4.3.2.jar:/usr/share/openhab/runtime/lib/boot/osgi.core-7.0.0.jar:/usr/share/openhab/runtime/lib/jdk9plus/istack-commons-runtime-3.0.10.jar:/usr/share/openhab/runtime/lib/jdk9plus/jakarta.xml.bind-api-2.3.3.jar:/usr/share/openhab/runtime/lib/jdk9plus/javax.activation-1.2.0.jar:/usr/share/openhab/runtime/lib/jdk9plus/javax.annotation-api-1.3.1.jar:/usr/share/openhab/runtime/lib/jdk9plus/jaxb-runtime-2.3.3.jar:/usr/share/openhab/runtime/lib/jdk9plus/txw2-2.3.3.jar org.apache.karaf.main.Main

Jul 11 07:20:53 openhabMain karaf[671]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Jul 11 07:20:53 openhabMain karaf[671]:         at java.base/java.lang.Thread.run(Thread.java:829)
Jul 11 07:21:03 openhabMain karaf[671]: [52034.639s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:21:03 openhabMain karaf[671]: [52034.642s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:21:03 openhabMain karaf[671]: [52034.643s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:21:03 openhabMain karaf[671]: [52035.151s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:21:03 openhabMain karaf[671]: [52035.163s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:21:04 openhabMain karaf[671]: [52035.670s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:21:04 openhabMain karaf[671]: [52035.736s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:26:02 openhabMain karaf[671]: [52334.256s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.

yesterday night I also got a OutOfMemory somewhere in between that status, but just sudo reboot to get to sleep again.
Right now, it’s in the syslog:

Jul 11 07:04:10 openhabMain systemd[1]: Stopping Network Time Synchronization...
Jul 11 07:04:10 openhabMain systemd[1]: systemd-timesyncd.service: Succeeded.
Jul 11 07:04:10 openhabMain systemd[1]: Stopped Network Time Synchronization.
Jul 11 07:04:10 openhabMain systemd[1]: Starting Network Time Synchronization...
Jul 11 07:04:10 openhabMain systemd[1]: Started Network Time Synchronization.
Jul 11 07:04:11 openhabMain systemd-timesyncd[18342]: Synchronized to time server for the first time 185.11.138.90:123 (0.debian.pool.ntp.org).
Jul 11 07:05:01 openhabMain CRON[18362]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Jul 11 07:07:04 openhabMain karaf[671]: [51196.253s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:07:04 openhabMain karaf[671]: [51196.354s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:00:00 openhabMain karaf[671]: [50772.199s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:00:00 openhabMain karaf[671]: [50772.205s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:00:37 openhabMain karaf[671]: [50809.054s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:00:37 openhabMain karaf[671]: [50809.055s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:00:37 openhabMain karaf[671]: Exception in thread "OH-items-3713" java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
Jul 11 07:00:37 openhabMain karaf[671]: #011at java.base/java.lang.Thread.start0(Native Method)
Jul 11 07:00:37 openhabMain karaf[671]: #011at java.base/java.lang.Thread.start(Thread.java:798)
Jul 11 07:00:37 openhabMain karaf[671]: #011at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:937)
Jul 11 07:00:37 openhabMain karaf[671]: #011at java.base/java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1005)
Jul 11 07:00:37 openhabMain karaf[671]: #011at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
Jul 11 07:00:37 openhabMain karaf[671]: #011at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Jul 11 07:00:37 openhabMain karaf[671]: #011at java.base/java.lang.Thread.run(Thread.java:829)

It repeats unregularly (?) until I did the reboot

long syslog
Jul 11 07:10:35 openhabMain karaf[671]: [51407.014s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:10:35 openhabMain karaf[671]: [51407.015s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:10:35 openhabMain karaf[671]: Exception in thread "OH-items-3880" java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
Jul 11 07:10:35 openhabMain karaf[671]: [51407.016s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:10:35 openhabMain karaf[671]: #011at java.base/java.lang.Thread.start0(Native Method)
Jul 11 07:10:35 openhabMain karaf[671]: #011at java.base/java.lang.Thread.start(Thread.java:798)
Jul 11 07:10:35 openhabMain karaf[671]: #011at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:937)
Jul 11 07:10:35 openhabMain karaf[671]: #011at java.base/java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1005)
Jul 11 07:10:35 openhabMain karaf[671]: #011at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
Jul 11 07:10:35 openhabMain karaf[671]: #011at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Jul 11 07:10:35 openhabMain karaf[671]: #011at java.base/java.lang.Thread.run(Thread.java:829)
Jul 11 07:10:35 openhabMain karaf[671]: Exception in thread "items-queue" java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
Jul 11 07:10:35 openhabMain karaf[671]: #011at java.base/java.lang.Thread.start0(Native Method)
Jul 11 07:10:35 openhabMain karaf[671]: #011at java.base/java.lang.Thread.start(Thread.java:798)
Jul 11 07:10:35 openhabMain karaf[671]: #011at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:937)
Jul 11 07:10:35 openhabMain karaf[671]: #011at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1354)
Jul 11 07:10:35 openhabMain karaf[671]: #011at org.openhab.core.common.QueueingThreadPoolExecutor.access$0(QueueingThreadPoolExecutor.java:1)
Jul 11 07:10:35 openhabMain karaf[671]: #011at org.openhab.core.common.QueueingThreadPoolExecutor$1.run(QueueingThreadPoolExecutor.java:196)
Jul 11 07:10:35 openhabMain karaf[671]: #011at java.base/java.lang.Thread.run(Thread.java:829)
Jul 11 07:10:39 openhabMain karaf[671]: [51411.069s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:10:39 openhabMain karaf[671]: Exception in thread "OH-items-4455" java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached[51411.070s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:10:39 openhabMain karaf[671]: Exception in thread "items-queue" #011at java.base/java.lang.Thread.start0(Native Method)
Jul 11 07:10:39 openhabMain karaf[671]: #011at java.base/java.lang.Thread.start(Thread.java:798)
Jul 11 07:10:39 openhabMain karaf[671]: #011at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:937)
Jul 11 07:10:39 openhabMain karaf[671]: #011at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1354)
Jul 11 07:10:39 openhabMain karaf[671]: #011at org.openhab.core.common.QueueingThreadPoolExecutor.execute(QueueingThreadPoolExecutor.java:167)
Jul 11 07:10:39 openhabMain karaf[671]: #011at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
Jul 11 07:10:39 openhabMain karaf[671]: #011at org.openhab.persistence.mapdb.internal.MapDbPersistenceService.commit(MapDbPersistenceService.java:221)
Jul 11 07:10:39 openhabMain karaf[671]: #011at org.openhab.persistence.mapdb.internal.MapDbPersistenceService.store(MapDbPersistenceService.java:187)
Jul 11 07:10:39 openhabMain karaf[671]: #011at org.openhab.core.persistence.internal.PersistenceManagerImpl.handleStateEvent(PersistenceManagerImpl.java:152)
Jul 11 07:10:39 openhabMain karaf[671]: #011at org.openhab.core.persistence.internal.PersistenceManagerImpl.stateChanged(PersistenceManagerImpl.java:473)
Jul 11 07:10:39 openhabMain karaf[671]: #011at org.openhab.core.items.GenericItem.lambda$1(GenericItem.java:259)
Jul 11 07:10:39 openhabMain karaf[671]: #011at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
Jul 11 07:10:39 openhabMain karaf[671]: #011at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Jul 11 07:10:39 openhabMain karaf[671]: #011at java.base/java.lang.Thread.run(Thread.java:829)
Jul 11 07:10:39 openhabMain karaf[671]: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
Jul 11 07:10:39 openhabMain karaf[671]: #011at java.base/java.lang.Thread.start0(Native Method)
Jul 11 07:10:39 openhabMain karaf[671]: #011at java.base/java.lang.Thread.start(Thread.java:798)
Jul 11 07:10:39 openhabMain karaf[671]: #011at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:937)
Jul 11 07:10:39 openhabMain karaf[671]: #011at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1354)
Jul 11 07:10:39 openhabMain karaf[671]: #011at org.openhab.core.common.QueueingThreadPoolExecutor.access$0(QueueingThreadPoolExecutor.java:1)
Jul 11 07:10:39 openhabMain karaf[671]: #011at org.openhab.core.common.QueueingThreadPoolExecutor$1.run(QueueingThreadPoolExecutor.java:196)
Jul 11 07:10:39 openhabMain karaf[671]: #011at java.base/java.lang.Thread.run(Thread.java:829)
Jul 11 07:13:24 openhabMain systemd[1]: Stopping Network Time Synchronization...
Jul 11 07:13:24 openhabMain systemd[1]: systemd-timesyncd.service: Succeeded.
Jul 11 07:13:24 openhabMain systemd[1]: Stopped Network Time Synchronization.
Jul 11 07:13:24 openhabMain systemd[1]: Starting Network Time Synchronization...
Jul 11 07:13:24 openhabMain systemd[1]: Started Network Time Synchronization.
Jul 11 07:13:24 openhabMain systemd-timesyncd[18565]: Synchronized to time server for the first time 194.36.144.87:123 (0.debian.pool.ntp.org).
Jul 11 07:15:01 openhabMain CRON[18599]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Jul 11 07:16:05 openhabMain karaf[671]: [51736.522s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:17:01 openhabMain CRON[18649]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul 11 07:20:29 openhabMain karaf[671]: [52000.452s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:29 openhabMain karaf[671]: [52000.654s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:32 openhabMain karaf[671]: [52004.304s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:32 openhabMain karaf[671]: [52004.309s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:32 openhabMain karaf[671]: [52004.313s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:33 openhabMain karaf[671]: [52004.647s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:33 openhabMain karaf[671]: [52004.664s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:33 openhabMain karaf[671]: [52004.666s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:33 openhabMain karaf[671]: [52004.909s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:33 openhabMain karaf[671]: [52004.923s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.688s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.692s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.696s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.698s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.700s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.702s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.704s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.706s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.708s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.710s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.711s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.714s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.715s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.717s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.719s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.721s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.723s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.725s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.727s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.729s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.731s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.733s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.735s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.738s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.740s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.741s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.744s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.746s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.748s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.750s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.752s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.753s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.756s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.757s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.759s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.761s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.763s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.764s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.767s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.771s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.775s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.776s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.778s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.780s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.782s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.784s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.785s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.787s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.788s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.790s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.792s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.793s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.795s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.796s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.799s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.800s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.804s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.805s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.809s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.810s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.814s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:34 openhabMain karaf[671]: [52005.816s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:35 openhabMain karaf[671]: [52006.694s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:35 openhabMain karaf[671]: [52006.698s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:35 openhabMain karaf[671]: [52006.700s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:35 openhabMain karaf[671]: [52006.702s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:35 openhabMain karaf[671]: [52006.704s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:35 openhabMain karaf[671]: [52006.706s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:35 openhabMain karaf[671]: [52006.708s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:35 openhabMain karaf[671]: [52006.709s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:35 openhabMain karaf[671]: [52006.711s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:35 openhabMain karaf[671]: [52006.716s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:35 openhabMain karaf[671]: [52006.719s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:35 openhabMain karaf[671]: [52006.722s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:35 openhabMain karaf[671]: [52006.724s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:35 openhabMain karaf[671]: [52006.725s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:35 openhabMain karaf[671]: [52006.729s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:35 openhabMain karaf[671]: [52006.730s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:35 openhabMain karaf[671]: [52006.734s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:35 openhabMain karaf[671]: [52006.737s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:35 openhabMain karaf[671]: [52006.741s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:35 openhabMain karaf[671]: [52006.743s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:35 openhabMain karaf[671]: [52006.744s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:38 openhabMain karaf[671]: [52009.949s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:38 openhabMain karaf[671]: [52009.961s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:40 openhabMain karaf[671]: [52011.778s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:40 openhabMain karaf[671]: [52011.779s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:43 openhabMain karaf[671]: [52014.988s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:43 openhabMain karaf[671]: [52015.001s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:48 openhabMain karaf[671]: [52020.023s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:48 openhabMain karaf[671]: [52020.036s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:53 openhabMain karaf[671]: [52025.077s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:53 openhabMain karaf[671]: [52025.083s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:20:53 openhabMain karaf[671]: Exception in thread "OH-common-2607" java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
Jul 11 07:20:53 openhabMain karaf[671]: #011at java.base/java.lang.Thread.start0(Native Method)
Jul 11 07:20:53 openhabMain karaf[671]: #011at java.base/java.lang.Thread.start(Thread.java:798)
Jul 11 07:20:53 openhabMain karaf[671]: #011at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:937)
Jul 11 07:20:53 openhabMain karaf[671]: #011at java.base/java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1005)
Jul 11 07:20:53 openhabMain karaf[671]: #011at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
Jul 11 07:20:53 openhabMain karaf[671]: #011at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Jul 11 07:20:53 openhabMain karaf[671]: #011at java.base/java.lang.Thread.run(Thread.java:829)
Jul 11 07:21:03 openhabMain karaf[671]: [52034.639s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:21:03 openhabMain karaf[671]: [52034.642s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:21:03 openhabMain karaf[671]: [52034.643s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:21:03 openhabMain karaf[671]: [52035.151s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:21:03 openhabMain karaf[671]: [52035.163s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:21:04 openhabMain karaf[671]: [52035.670s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:21:04 openhabMain karaf[671]: [52035.736s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
Jul 11 07:23:12 openhabMain systemd[1]: Stopping Network Time Synchronization...
Jul 11 07:23:12 openhabMain systemd[1]: systemd-timesyncd.service: Succeeded.
Jul 11 07:23:12 openhabMain systemd[1]: Stopped Network Time Synchronization.
Jul 11 07:23:12 openhabMain systemd[1]: Starting Network Time Synchronization...
Jul 11 07:23:12 openhabMain systemd[1]: Started Network Time Synchronization.
Jul 11 07:23:12 openhabMain systemd-timesyncd[18902]: Synchronized to time server for the first time 162.159.200.123:123 (0.debian.pool.ntp.org).
Jul 11 07:25:01 openhabMain CRON[18954]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Jul 11 07:26:02 openhabMain karaf[671]: [52334.256s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.

Don’t know if this is related to it, but I get in openhab.log:

2021-07-09 04:07:29.628 [WARN ] [org.eclipse.jetty.io.ManagedSelector] - java.nio.channels.ClosedSelectorException

Can I check something on this, and by what it’s caused?

It could be, I found the culprit.
Using Pi-Hole to block certain domains/URLs, I obviously somehow blocked “dp-gw-na.amazon.de” lately, which lead to repeated “connection refused” error messages in the log:

2021-07-10 04:32:03.174 [INFO ] [control.internal.WebSocketConnection] - Web Socket error
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777) ~[?:?]
	at org.eclipse.jetty.io.SelectorManager.doFinishConnect(SelectorManager.java:355) ~[?:?]
	at org.eclipse.jetty.io.ManagedSelector.processConnect(ManagedSelector.java:347) ~[?:?]
	at org.eclipse.jetty.io.ManagedSelector.access$1700(ManagedSelector.java:65) ~[?:?]
	at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.processSelected(ManagedSelector.java:676) ~[?:?]
	at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:535) ~[?:?]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360) ~[bundleFile:9.4.40.v20210413]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184) ~[bundleFile:9.4.40.v20210413]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) ~[bundleFile:9.4.40.v20210413]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135) ~[bundleFile:9.4.40.v20210413]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882) [bundleFile:9.4.40.v20210413]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036) [bundleFile:9.4.40.v20210413]
	at java.lang.Thread.run(Thread.java:829) [?:?]

after whitelisting said domain the memory-leak seems to be gone:

Would this be considered a bug in the alexa smarthome binding?

If that’s your likely cause, yes, a binding should gracefully handle connection failures. Seems you should be able to reproduce it easily enough.

I’ll have it under observation until tomorrow, if the used memory stays reasonable I remove the whitelisting and try again - it should pop up again after 18-24h

as expected, I ran into the problem quickly after de-whitelisting the amazon-domain in my pi-hole.
I opened an bug-ticket on github:

2 Likes

And this issue is new since 3.1? I can’t remember something changed in that code since 3.0.

i don’t know, if it’s 3.1 only. My bet is, with some gravity-updates for my pi-hole a few days ago, the amazon-Server got into the blocked domain list…
the blocked domain isn’t necessary for my echo devices and I guess correctly so on the “Ad block” list. Everything works as intended, no flaws with Alexa at all
…and I’m accidentially on 3.1 since minute 21 the final version was available! :wink:

1 Like