Hello,
i have the following installation. Raspi4 4GB, SD32GB
openHAB 3.4.0.M3
Recently, every 1-2 weeks openhab does not react anymore, so no more MQTT communications, almost dead. When entering SSH i can see that the root folder is almost FULL and of course the logs are taking gigabytes…
i’m not a java/openhab expert to totally understand these logs, for me it looks like influxdb connection is getting problems, but maybe thats not the root cause. i would be pleased if experts could have a look and tell me what makes my system crash after a while…did not happen before latest updates, but not sure if it was the last or the one before Milestone…
I now checked the logs:
daemon log from LINUX:
Oct 29 08:25:15 SmartPi karaf[679]: #011at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Oct 29 08:25:15 SmartPi karaf[679]: #011at java.base/java.lang.Thread.run(Thread.java:829)
Oct 29 08:25:15 SmartPi karaf[679]: java.lang.NullPointerException
Oct 29 08:25:15 SmartPi karaf[679]: #011at java.base/java.util.concurrent.LinkedBlockingQueue.dequeue(LinkedBlockingQueue.java:214)
Oct 29 08:25:15 SmartPi karaf[679]: #011at java.base/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
Oct 29 08:25:15 SmartPi karaf[679]: #011at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
Oct 29 08:25:15 SmartPi karaf[679]: #011at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
Oct 29 08:25:15 SmartPi karaf[679]: #011at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Oct 29 08:25:15 SmartPi karaf[679]: #011at java.base/java.lang.Thread.run(Thread.java:829)
Oct 29 08:25:15 SmartPi karaf[679]: Exception in thread "EventAdminAsyncThread #6636937" java.lang.NullPointerException
Oct 29 08:25:15 SmartPi karaf[679]: Exception in thread "EventAdminAsyncThread #6636941" Exception in thread "EventAdminAsyncThread #6636940" #011at java.base/java.util.concurrent.LinkedBlockingQueue.dequeue(LinkedBlockingQueue.java:214)
Oct 29 08:25:15 SmartPi karaf[679]: #011at java.base/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
Oct 29 08:25:15 SmartPi karaf[679]: Exception in thread "EventAdminAsyncThread #6636942" #011at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
Oct 29 08:25:15 SmartPi karaf[679]: Exception in thread "EventAdminAsyncThread #6636944" Exception in thread "EventAdminAsyncThread #6636943" #011at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:111>
Oct 29 08:25:15 SmartPi karaf[679]: #011at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Oct 29 08:25:15 SmartPi karaf[679]: Exception in thread "EventAdminAsyncThread #6636945" Exception in thread "EventAdminAsyncThread #6636946" #011at java.base/java.lang.Thread.run(Thread.java:829)
Oct 29 08:25:15 SmartPi karaf[679]: java.lang.NullPointerException
Oct 29 08:25:15 SmartPi karaf[679]: #011at java.base/java.util.concurrent.LinkedBlockingQueue.dequeue(LinkedBlockingQueue.java:214)
Oct 29 08:25:15 SmartPi karaf[679]: #011at java.base/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
Oct 29 08:25:15 SmartPi karaf[679]: #011at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
Oct 29 08:25:15 SmartPi karaf[679]: #011at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
Oct 29 08:25:15 SmartPi karaf[679]: #011at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Oct 29 08:25:15 SmartPi karaf[679]: #011at java.base/java.lang.Thread.run(Thread.java:829)
Oct 29 08:25:15 SmartPi karaf[679]: java.lang.NullPointerException
syslog from linux:
Oct 29 08:25:13 SmartPi karaf[679]: Exception in thread "EventAdminAsyncThread #6636820" java.lang.NullPointerException
Oct 29 08:25:13 SmartPi karaf[679]: #011at java.base/java.util.concurrent.LinkedBlockingQueue.dequeue(LinkedBlockingQueue.java:214)
Oct 29 08:25:13 SmartPi karaf[679]: #011at java.base/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
Oct 29 08:25:13 SmartPi karaf[679]: #011at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
Oct 29 08:25:13 SmartPi karaf[679]: #011at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
Oct 29 08:25:13 SmartPi karaf[679]: #011at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Oct 29 08:25:13 SmartPi karaf[679]: #011at java.base/java.lang.Thread.run(Thread.java:829)
Oct 29 08:25:13 SmartPi karaf[679]: Exception in thread "EventAdminAsyncThread #6636822" java.lang.NullPointerException
Oct 29 08:25:13 SmartPi karaf[679]: #011at java.base/java.util.concurrent.LinkedBlockingQueue.dequeue(LinkedBlockingQueue.java:214)
Oct 29 08:25:13 SmartPi karaf[679]: #011at java.base/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
Oct 29 08:25:13 SmartPi karaf[679]: Exception in thread "EventAdminAsyncThread #6636824" #011at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
Oct 29 08:25:13 SmartPi karaf[679]: #011at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
Oct 29 08:25:13 SmartPi karaf[679]: Exception in thread "EventAdminAsyncThread #6636823" #011at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Oct 29 08:25:13 SmartPi karaf[679]: #011at java.base/java.lang.Thread.run(Thread.java:829)
Oct 29 08:25:13 SmartPi karaf[679]: java.lang.NullPointerException
Oct 29 08:25:13 SmartPi karaf[679]: #011at java.base/java.util.concurrent.LinkedBlockingQueue.dequeue(LinkedBlockingQueue.java:214)
Oct 29 08:25:13 SmartPi karaf[679]: #011at java.base/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
Oct 29 08:25:13 SmartPi karaf[679]: #011at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
Oct 29 08:25:13 SmartPi karaf[679]: Exception in thread "EventAdminAsyncThread #6636826" #011at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
Oct 29 08:25:13 SmartPi karaf[679]: #011at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Oct 29 08:25:13 SmartPi karaf[679]: #011at java.base/java.lang.Thread.run(Thread.java:829)
Oct 29 08:25:13 SmartPi karaf[679]: java.lang.NullPointerException
Oct 29 08:25:13 SmartPi karaf[679]: #011at java.base/java.util.concurrent.LinkedBlockingQueue.dequeue(LinkedBlockingQueue.java:214)
Oct 29 08:25:13 SmartPi karaf[679]: #011at java.base/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
Oct 29 08:25:13 SmartPi karaf[679]: #011at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
Oct 29 08:25:13 SmartPi karaf[679]: #011at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
Oct 29 08:25:13 SmartPi karaf[679]: #011at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Oct 29 08:25:13 SmartPi karaf[679]: #011at java.base/java.lang.Thread.run(Thread.java:829)
Oct 29 08:25:13 SmartPi karaf[679]: java.lang.NullPointerException
Oct 29 08:25:13 SmartPi karaf[679]: #011at java.base/java.util.concurrent.LinkedBlockingQueue.dequeue(LinkedBlockingQueue.java:214)
Oct 29 08:25:13 SmartPi karaf[679]: #011at java.base/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
Oct 29 08:25:13 SmartPi karaf[679]: #011at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
Oct 29 08:25:13 SmartPi karaf[679]: #011at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
Oct 29 08:25:13 SmartPi karaf[679]: #011at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Oct 29 08:25:13 SmartPi karaf[679]: #011at java.base/java.lang.Thread.run(Thread.java:829)
Oct 29 08:25:13 SmartPi karaf[679]: Exception in thread "EventAdminAsyncThread #6636827" java.lang.NullPointerException
Oct 29 08:25:13 SmartPi karaf[679]: Exception in thread "EventAdminAsyncThread #6636825" #011at java.base/java.util.concurrent.LinkedBlockingQueue.dequeue(LinkedBlockingQueue.java:214)
and of course the openhab log section when this crash/extreme logging starts to happen:
2022-10-29 00:11:07.528 [WARN ] [e.io.http.servlet.BaseOpenHABServlet] - Received unknown command '---' for item 'RestO2state'
2022-10-29 00:11:10.238 [ERROR] [org.influxdb.impl.BatchProcessor ] - Batch could not be sent. Data will be lost
org.influxdb.InfluxDBIOException: java.net.SocketTimeoutException: timeout
at org.influxdb.impl.InfluxDBImpl.execute(InfluxDBImpl.java:831) ~[bundleFile:?]
at org.influxdb.impl.InfluxDBImpl.write(InfluxDBImpl.java:460) ~[bundleFile:?]
at org.influxdb.impl.OneShotBatchWriter.write(OneShotBatchWriter.java:22) ~[bundleFile:?]
at org.influxdb.impl.BatchProcessor.write(BatchProcessor.java:340) [bundleFile:?]
at org.influxdb.impl.BatchProcessor$1.run(BatchProcessor.java:287) [bundleFile:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.net.SocketTimeoutException: timeout
at okio.Okio$4.newTimeoutException(Okio.java:232) ~[bundleFile:?]
at okio.AsyncTimeout.exit(AsyncTimeout.java:286) ~[bundleFile:?]
at okio.AsyncTimeout$2.read(AsyncTimeout.java:241) ~[bundleFile:?]
at okio.RealBufferedSource.indexOf(RealBufferedSource.java:358) ~[bundleFile:?]
at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:230) ~[bundleFile:?]
at okhttp3.internal.http1.Http1ExchangeCodec.readHeaderLine(Http1ExchangeCodec.java:242) ~[bundleFile:?]
at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.java:213) ~[bundleFile:?]
at okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.java:115) ~[bundleFile:?]
at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:94) ~[bundleFile:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[bundleFile:?]
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:43) ~[bundleFile:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[bundleFile:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[bundleFile:?]
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94) ~[bundleFile:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[bundleFile:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[bundleFile:?]
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[bundleFile:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[bundleFile:?]
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88) ~[bundleFile:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[bundleFile:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[bundleFile:?]
at org.influxdb.impl.BasicAuthInterceptor.intercept(BasicAuthInterceptor.java:22) ~[bundleFile:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[bundleFile:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[bundleFile:?]
at org.influxdb.impl.GzipRequestInterceptor.intercept(GzipRequestInterceptor.java:42) ~[bundleFile:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[bundleFile:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[bundleFile:?]
at okhttp3.logging.HttpLoggingInterceptor.intercept(HttpLoggingInterceptor.java:152) ~[bundleFile:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[bundleFile:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[bundleFile:?]
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:229) ~[bundleFile:?]
at okhttp3.RealCall.execute(RealCall.java:81) ~[bundleFile:?]
at retrofit2.OkHttpCall.execute(OkHttpCall.java:188) ~[bundleFile:?]
at org.influxdb.impl.InfluxDBImpl.execute(InfluxDBImpl.java:819) ~[bundleFile:?]
at retrofit2.OkHttpCall.execute(OkHttpCall.java:188) ~[bundleFile:?]
at org.influxdb.impl.InfluxDBImpl.execute(InfluxDBImpl.java:819) ~[bundleFile:?]
... 10 more
Caused by: java.net.SocketException: Socket closed
at java.net.SocketInputStream.read(SocketInputStream.java:183) ~[?:?]
at java.net.SocketInputStream.read(SocketInputStream.java:140) ~[?:?]
at okio.Okio$2.read(Okio.java:140) ~[bundleFile:?]
at okio.AsyncTimeout$2.read(AsyncTimeout.java:237) ~[bundleFile:?]
at okio.RealBufferedSource.indexOf(RealBufferedSource.java:358) ~[bundleFile:?]
at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:230) ~[bundleFile:?]
at okhttp3.internal.http1.Http1ExchangeCodec.readHeaderLine(Http1ExchangeCodec.java:242) ~[bundleFile:?]
at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.java:213) ~[bundleFile:?]
at okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.java:115) ~[bundleFile:?]
at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:94) ~[bundleFile:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[bundleFile:?]
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:43) ~[bundleFile:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[bundleFile:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[bundleFile:?]
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94) ~[bundleFile:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[bundleFile:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[bundleFile:?]
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[bundleFile:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[bundleFile:?]
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88) ~[bundleFile:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[bundleFile:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[bundleFile:?]
at org.influxdb.impl.BasicAuthInterceptor.intercept(BasicAuthInterceptor.java:22) ~[bundleFile:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[bundleFile:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[bundleFile:?]
at org.influxdb.impl.GzipRequestInterceptor.intercept(GzipRequestInterceptor.java:42) ~[bundleFile:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[bundleFile:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[bundleFile:?]
at okhttp3.logging.HttpLoggingInterceptor.intercept(HttpLoggingInterceptor.java:152) ~[bundleFile:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[bundleFile:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[bundleFile:?]
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:229) ~[bundleFile:?]
at okhttp3.RealCall.execute(RealCall.java:81) ~[bundleFile:?]
at retrofit2.OkHttpCall.execute(OkHttpCall.java:188) ~[bundleFile:?]
at org.influxdb.impl.InfluxDBImpl.execute(InfluxDBImpl.java:819) ~[bundleFile:?]
... 10 more
2022-10-29 00:11:12.203 [WARN ] [org.apache.karaf.services.eventadmin] - EventAdmin: Exception: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
at java.lang.Thread.start0(Native Method) ~[?:?]
at java.lang.Thread.start(Thread.java:798) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:937) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1343) ~[?:?]
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118) ~[?:?]
at org.apache.felix.eventadmin.impl.tasks.DefaultThreadPool.executeTask(DefaultThreadPool.java:134) ~[?:?]
at org.apache.felix.eventadmin.impl.tasks.AsyncDeliverTasks.execute(AsyncDeliverTasks.java:102) ~[?:?]
at org.apache.felix.eventadmin.impl.handler.EventAdminImpl.postEvent(EventAdminImpl.java:180) ~[?:?]
at org.apache.felix.eventadmin.impl.security.EventAdminSecurityDecorator.postEvent(EventAdminSecurityDecorator.java:79) ~[?:?]
at jdk.internal.reflect.GeneratedMethodAccessor238.invoke(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at org.eclipse.osgi.internal.log.EventAdminLogListener.lambda$0(EventAdminLogListener.java:74) ~[org.eclipse.osgi-3.17.200.jar:?]
at java.securit