Several items from different bindings stop updating after a while

The strange thing is that only some of the Items stop updating. I will try to uninstall the tado° binding and see if Items will stop updating as well.

After uninstalling the tado° binding it just took longer until items stopped updating. So I think it can’t be the tado° binding itself.

As I found some java.lang.OutOfMemoryError, I think I will try to monitor the memory.

Any other ideas where to investigate further?

Are all non updating items related to the same binding? Then probably the installed version of the binding might have a problem. You should check in the OH-Log for Java exceptions related to that binding.

I had such problems with the Somfy Tahoma binding (Version 20180611). It worked a couple of days and suddenly it stopped without obvious reason. I could find Java exceptions in the log with timestamp close to the stopp of item updates. I also experienced the same with the dev version of the Z-Wave binding a couple of days ago. In each case, the binding developers were able to provide an new version and now they run stable.

I am not sure if your problem is similar but it might be. OH keeps on running, some bindings still update while others stopp working? Then this might be the cause.

I experience a similar problem where item of several bindings are not reacting or are not updated suddenly after several weeks of normal operation. This seems not to be a problem of one particular binding but OH itself.

I’ll make a guess that something hogs the limited execution threads available for quartz to run timed jobs, which some bindings rely on to e.g. periodic poll or recalculate.

I don’t know how you’d go about tracking that down. Start with a critical look at any rules timers you may have.

To track down if threads are blocked, possible causing the bindings not to update use this:

2 Likes

Thanks, will try next time the issue appears. However, there seems to be a lot of information given with “threads --monitors --locks” … curios to see how to identify the problem …

It does give a lot of information. I can’t remember how a problem actual looks like (I haven’t any problems either at the moment). However I think you can look for the names that start with org.openhab.binding. These are threads in bindings. It can be a valid thread, like serial bindings waiting for input. But it can also be blocked thread in a binding. It’s very likely blocking threads occur before the whole system blocks. Because it will block only when all threads are exhausted. If you have the problem again and you can’t find the exact threads feel free to post the output here.

Problem is back earlier than expected. Nuki lock does not react on commands while status is been read correctly. This is the extract from “threads --monitors --locks”:

"ESH-httpClient-common-104" Id=104 in TIMED_WAITING on lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@18caf4f
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
    at java.lang.Thread.run(Unknown Source)

"ESH-httpClient-common-105" Id=105 in TIMED_WAITING on lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@18caf4f
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
    at java.lang.Thread.run(Unknown Source)

"ESH-httpClient-common-106" Id=106 in TIMED_WAITING on lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@18caf4f
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
    at java.lang.Thread.run(Unknown Source)

"ESH-httpClient-common-107" Id=107 in TIMED_WAITING on lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@18caf4f
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
    at java.lang.Thread.run(Unknown Source)

"ESH-OSGiEventManager" Id=111 in TIMED_WAITING on lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@da9705
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)
    at org.eclipse.smarthome.core.internal.events.ThreadedEventHandler.lambda$0(ThreadedEventHandler.java:61)
    at org.eclipse.smarthome.core.internal.events.ThreadedEventHandler$$Lambda$384/6095163.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)

and also

"ESH-httpClient-common-104" Id=104 in TIMED_WAITING on lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@18caf4f
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
    at java.lang.Thread.run(Unknown Source)

"ESH-httpClient-common-105" Id=105 in TIMED_WAITING on lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@18caf4f
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
    at java.lang.Thread.run(Unknown Source)

"ESH-httpClient-common-106" Id=106 in TIMED_WAITING on lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@18caf4f
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
    at java.lang.Thread.run(Unknown Source)

"ESH-httpClient-common-107" Id=107 in TIMED_WAITING on lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@18caf4f
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
    at java.lang.Thread.run(Unknown Source)

"ESH-OSGiEventManager" Id=111 in TIMED_WAITING on lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@da9705
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)
    at org.eclipse.smarthome.core.internal.events.ThreadedEventHandler.lambda$0(ThreadedEventHandler.java:61)
    at org.eclipse.smarthome.core.internal.events.ThreadedEventHandler$$Lambda$384/6095163.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)

and later …

"ESH-thingHandler-5" Id=202 in BLOCKED on lock=org.openhab.binding.nuki.internal.dataexchange.NukiHttpClient@1ad8ead
     owned by ESH-thingHandler-1 Id=194
    at org.openhab.binding.nuki.internal.dataexchange.NukiHttpClient.getBridgeLockAction(NukiHttpClient.java:165)
    at org.openhab.binding.nuki.handler.NukiSmartLockHandler.handleCommandDecimalType(NukiSmartLockHandler.java:208)
    at org.openhab.binding.nuki.handler.NukiSmartLockHandler.lambda$3(NukiSmartLockHandler.java:75)
    at org.openhab.binding.nuki.handler.NukiSmartLockHandler$$Lambda$1016/14610683.run(Unknown Source)
    at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)

    Locked synchronizers: count = 1
      - java.util.concurrent.ThreadPoolExecutor$Worker@1732856

/tmp/oh_block.txt lines 587-622/1655 37%
      - java.util.concurrent.ThreadPoolExecutor$Worker@37b3b7

"ESH-thingHandler-3" Id=196 in BLOCKED on lock=org.openhab.binding.nuki.internal.dataexchange.NukiHttpClient@1ad8ead
     owned by ESH-thingHandler-1 Id=194
    at org.openhab.binding.nuki.internal.dataexchange.NukiHttpClient.getBridgeLockAction(NukiHttpClient.java:165)
    at org.openhab.binding.nuki.handler.NukiSmartLockHandler.handleCommandDecimalType(NukiSmartLockHandler.java:208)
    at org.openhab.binding.nuki.handler.NukiSmartLockHandler.lambda$3(NukiSmartLockHandler.java:75)
    at org.openhab.binding.nuki.handler.NukiSmartLockHandler$$Lambda$1016/14610683.run(Unknown Source)
    at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)

    Locked synchronizers: count = 1
      - java.util.concurrent.ThreadPoolExecutor$Worker@12a8178

"ESH-thingHandler-5" Id=202 in BLOCKED on lock=org.openhab.binding.nuki.internal.dataexchange.NukiHttpClient@1ad8ead
     owned by ESH-thingHandler-1 Id=194
    at org.openhab.binding.nuki.internal.dataexchange.NukiHttpClient.getBridgeLockAction(NukiHttpClient.java:165)
    at org.openhab.binding.nuki.handler.NukiSmartLockHandler.handleCommandDecimalType(NukiSmartLockHandler.java:208)
    at org.openhab.binding.nuki.handler.NukiSmartLockHandler.lambda$3(NukiSmartLockHandler.java:75)
    at org.openhab.binding.nuki.handler.NukiSmartLockHandler$$Lambda$1016/14610683.run(Unknown Source)
    at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)

    Locked synchronizers: count = 1
      - java.util.concurrent.ThreadPoolExecutor$Worker@1732856

some time later there is the following error message in the log:

00:35:09.266 [ERROR] [jersey.server.ServerRuntime$Responder] - An I/O error has occurred while writing a response message entity to the container output stream.
org.glassfish.jersey.server.internal.process.MappableException: org.eclipse.jetty.io.EofException
        at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:92) ~[?:?]
        at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[171:org.glassfish.jersey.core.jersey-common:2.22.2]
        at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1130) ~[171:org.glassfish.jersey.core.jersey-common:2.22.2]
        at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:711) [172:org.glassfish.jersey.core.jersey-server:2.22.2]
        at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:444) [172:org.glassfish.jersey.core.jersey-server:2.22.2]
        at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:434) [172:org.glassfish.jersey.core.jersey-server:2.22.2]
        at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:329) [172:org.glassfish.jersey.core.jersey-server:2.22.2]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [171:org.glassfish.jersey.core.jersey-common:2.22.2]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [171:org.glassfish.jersey.core.jersey-common:2.22.2]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [171:org.glassfish.jersey.core.jersey-common:2.22.2]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [171:org.glassfish.jersey.core.jersey-common:2.22.2]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [171:org.glassfish.jersey.core.jersey-common:2.22.2]
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [171:org.glassfish.jersey.core.jersey-common:2.22.2]
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [172:org.glassfish.jersey.core.jersey-server:2.22.2]
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [172:org.glassfish.jersey.core.jersey-server:2.22.2]
        at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) [169:org.glassfish.jersey.containers.jersey-container-servlet-core:2.22.2]
        at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) [169:org.glassfish.jersey.containers.jersey-container-servlet-core:2.22.2]
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) [169:org.glassfish.jersey.containers.jersey-container-servlet-core:2.22.2]
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) [169:org.glassfish.jersey.containers.jersey-container-servlet-core:2.22.2]
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) [169:org.glassfish.jersey.containers.jersey-container-servlet-core:2.22.2]
        at com.eclipsesource.jaxrs.publisher.internal.ServletContainerBridge.service(ServletContainerBridge.java:76) [20:com.eclipsesource.jaxrs.publisher:5.3.1.201602281253]
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865) [85:org.eclipse.jetty.servlet:9.4.12.v20180830]
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:535) [85:org.eclipse.jetty.servlet:9.4.12.v20180830]
        at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:71) [186:org.ops4j.pax.web.pax-web-jetty:7.2.5]
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) [84:org.eclipse.jetty.server:9.4.12.v20180830]
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) [82:org.eclipse.jetty.security:9.4.12.v20180830]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) [84:org.eclipse.jetty.server:9.4.12.v20180830]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) [84:org.eclipse.jetty.server:9.4.12.v20180830]
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) [84:org.eclipse.jetty.server:9.4.12.v20180830]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) [84:org.eclipse.jetty.server:9.4.12.v20180830]
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1340) [84:org.eclipse.jetty.server:9.4.12.v20180830]
        at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:293) [186:org.ops4j.pax.web.pax-web-jetty:7.2.5]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) [84:org.eclipse.jetty.server:9.4.12.v20180830]
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) [85:org.eclipse.jetty.servlet:9.4.12.v20180830]
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) [84:org.eclipse.jetty.server:9.4.12.v20180830]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) [84:org.eclipse.jetty.server:9.4.12.v20180830]
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1242) [84:org.eclipse.jetty.server:9.4.12.v20180830]
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) [84:org.eclipse.jetty.server:9.4.12.v20180830]
        at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:80) [186:org.ops4j.pax.web.pax-web-jetty:7.2.5]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) [84:org.eclipse.jetty.server:9.4.12.v20180830]
        at org.eclipse.jetty.server.Server.handle(Server.java:503) [84:org.eclipse.jetty.server:9.4.12.v20180830]
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364) [84:org.eclipse.jetty.server:9.4.12.v20180830]
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) [84:org.eclipse.jetty.server:9.4.12.v20180830]
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) [75:org.eclipse.jetty.io:9.4.12.v20180830]
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) [75:org.eclipse.jetty.io:9.4.12.v20180830]
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) [75:org.eclipse.jetty.io:9.4.12.v20180830]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) [87:org.eclipse.jetty.util:9.4.12.v20180830]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) [87:org.eclipse.jetty.util:9.4.12.v20180830]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) [87:org.eclipse.jetty.util:9.4.12.v20180830]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) [87:org.eclipse.jetty.util:9.4.12.v20180830]
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) [87:org.eclipse.jetty.util:9.4.12.v20180830]
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) [87:org.eclipse.jetty.util:9.4.12.v20180830]
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) [87:org.eclipse.jetty.util:9.4.12.v20180830]
        at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: org.eclipse.jetty.io.EofException
        at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:286) ~[75:org.eclipse.jetty.io:9.4.12.v20180830]
        at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:393) ~[75:org.eclipse.jetty.io:9.4.12.v20180830]
        at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277) ~[75:org.eclipse.jetty.io:9.4.12.v20180830]
        at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:380) ~[75:org.eclipse.jetty.io:9.4.12.v20180830]
        at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:802) ~[84:org.eclipse.jetty.server:9.4.12.v20180830]
        at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241) ~[87:org.eclipse.jetty.util:9.4.12.v20180830]
        at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224) ~[87:org.eclipse.jetty.util:9.4.12.v20180830]
        at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:538) ~[84:org.eclipse.jetty.server:9.4.12.v20180830]
        at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:831) ~[84:org.eclipse.jetty.server:9.4.12.v20180830]
        at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:887) ~[84:org.eclipse.jetty.server:9.4.12.v20180830]
        at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:239) ~[84:org.eclipse.jetty.server:9.4.12.v20180830]
        at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:215) ~[84:org.eclipse.jetty.server:9.4.12.v20180830]
        at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:540) ~[84:org.eclipse.jetty.server:9.4.12.v20180830]
        at java.io.OutputStream.write(Unknown Source) ~[?:?]
        at org.glassfish.jersey.servlet.internal.ResponseWriter$NonCloseableOutputStreamWrapper.write(ResponseWriter.java:320) ~[?:?]
        at org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:218) ~[?:?]
        at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.write(WriterInterceptorExecutor.java:294) ~[?:?]
        at org.eclipse.smarthome.io.rest.core.internal.GsonProvider.writeTo(GsonProvider.java:71) ~[?:?]
        at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:265) ~[?:?]
        at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:250) ~[?:?]
        at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[?:?]
        at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:106) ~[?:?]
        at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[?:?]
        at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:86) ~[?:?]
        ... 53 more
Caused by: java.io.IOException: Datenübergabe unterbrochen (broken pipe)
        at sun.nio.ch.FileDispatcherImpl.writev0(Native Method) ~[?:?]
        at sun.nio.ch.SocketDispatcher.writev(Unknown Source) ~[?:?]
        at sun.nio.ch.IOUtil.write(Unknown Source) ~[?:?]
        at sun.nio.ch.SocketChannelImpl.write(Unknown Source) ~[?:?]
        at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:266) ~[75:org.eclipse.jetty.io:9.4.12.v20180830]
        at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:393) ~[75:org.eclipse.jetty.io:9.4.12.v20180830]
        at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277) ~[75:org.eclipse.jetty.io:9.4.12.v20180830]
        at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:380) ~[75:org.eclipse.jetty.io:9.4.12.v20180830]
        at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:802) ~[84:org.eclipse.jetty.server:9.4.12.v20180830]
        at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241) ~[87:org.eclipse.jetty.util:9.4.12.v20180830]
        at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224) ~[87:org.eclipse.jetty.util:9.4.12.v20180830]
        at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:538) ~[84:org.eclipse.jetty.server:9.4.12.v20180830]
        at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:831) ~[84:org.eclipse.jetty.server:9.4.12.v20180830]
        at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:887) ~[84:org.eclipse.jetty.server:9.4.12.v20180830]
        at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:239) ~[84:org.eclipse.jetty.server:9.4.12.v20180830]
        at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:215) ~[84:org.eclipse.jetty.server:9.4.12.v20180830]
        at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:540) ~[84:org.eclipse.jetty.server:9.4.12.v20180830]
        at java.io.OutputStream.write(Unknown Source) ~[?:?]
        at org.glassfish.jersey.servlet.internal.ResponseWriter$NonCloseableOutputStreamWrapper.write(ResponseWriter.java:320) ~[?:?]
        at org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:218) ~[?:?]
        at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.write(WriterInterceptorExecutor.java:294) ~[?:?]
        at org.eclipse.smarthome.io.rest.core.internal.GsonProvider.writeTo(GsonProvider.java:71) ~[?:?]
        at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:265) ~[?:?]
        at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:250) ~[?:?]
        at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[?:?]
        at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:106) ~[?:?]
        at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[?:?]
        at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:86) ~[?:?]
        ... 53 more

I can’t identify a root cause though … does this help?

Thanks

These blocked threads might be the problem. It could be no issue, but If these threads are still there after several minutes that’s very likely an indication of a problem. Based on the source code of the binding, I could not directly pinpoint if a problem, but the code could use some attention. The handle of commands is performed in scheduled threads and part of that is synchronized. In general the combination of threads and synchronization is a likely cause for deadlocks and I would expect something like that is happening here.

So I’ll ping @mkatter here, the original developer of this binding, to see if he has time to look into this (and also to make some general improvements that should improve overall stability. I can provide help with that.)

Thanks @hilbrand, this is very much appreciated!

Yes, the blocked nuki threads are still there (after hours).
In addition, I experience problems with hue and vitotronic as well. Items are not reacting or not being updated. I don’t see the blocked hue binding threads though. Sometimes, I do see the following instead:

"ESH-thingHandler-5" Id=202 in BLOCKED on lock=org.openhab.binding.nuki.internal.dataexchange.NukiHttpClient@1ad8ead
    at org.openhab.binding.nuki.internal.dataexchange.NukiHttpClient.getBridgeLockAction(NukiHttpClient.java:165)
    at org.openhab.binding.nuki.handler.NukiSmartLockHandler.handleCommandDecimalType(NukiSmartLockHandler.java:208)
    at org.openhab.binding.nuki.handler.NukiSmartLockHandler.lambda$3(NukiSmartLockHandler.java:75)
    at org.openhab.binding.nuki.handler.NukiSmartLockHandler$$Lambda$1016/14610683.run(Unknown Source)
    at org.openhab.binding.vitotronic.handler.VitotronicBridgeHandler$2.run(VitotronicBridgeHandler.java:276)
    at org.openhab.binding.garadget.internal.Connection.sendCommand(Connection.java:276)
    at org.openhab.binding.garadget.internal.Connection.sendCommand(Connection.java:182)
    at org.openhab.binding.garadget.internal.GaradgetBinding.execute(GaradgetBinding.java:247)
    at org.openhab.core.binding.AbstractActiveBinding$BindingActiveService.execute(AbstractActiveBinding.java:148)

whereas the garadget entries disappear after a while. However, I do pull data via the garadget binding every second. This may be a reason why I do see this tasks that often.

The vitotronic entry on the other side remains as well and is stuck as well. I don’t see updates on the data. Bottom line: there are more bindings impacted than just nuki.

The scheduled tasks of the bindings use a limited shared thread pool. If a binding goes rough and deadlocks on those threads it will deplete the available threads and the other bindings will stall too. Some bindings use own threads for certain tasks, and those tasks won’t be effected. So you might still see some updates on certain bindings. In this case the nuki binding seems too be causing this problem.

Update: just stopped the nuki binding via “bundle:stop” and that seemed to resolve all hangups of the other bindings as far as I can see. Restarted nuki binding again.
The vitotronic entry remains but the binding seems to work and provides updates on items again.

ok, thanks @hilbrand.
Focusing on the nuki binding seems to be the right thing to do. Anything I can help with?

… one more thing. Since system appeared to be not fully right finally, i tried to stop OH by using “system:shutdown”. This throws following messages obviously preventing the system to go down:

17:05:50.984 [WARN ] [se.jetty.util.thread.QueuedThreadPool] - QueuedThreadPool[ESH-httpClient-common]@18474c8{STOPPING,10<=10<=40,i=1,q=4}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$313/28600488@81af1] Couldn't stop Thread[ESH-httpClient-common-99,5,main]
17:05:50.985 [WARN ] [se.jetty.util.thread.QueuedThreadPool] - QueuedThreadPool[ESH-httpClient-common]@18474c8{STOPPING,10<=10<=40,i=1,q=4}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$313/28600488@81af1] Couldn't stop Thread[ESH-httpClient-common-101,5,main]
17:06:08.030 [WARN ] [se.jetty.util.thread.QueuedThreadPool] - QueuedThreadPool[WebSocketClient@1911742]@198bf1a{STOPPING,8<=8<=200,i=0,q=1}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$313/28600488@81af1] Couldn't stop Thread[WebSocketClient@1911742-359761,5,main]
17:08:01.896 [WARN ] [org.jline                            ] - The Parser of class org.apache.karaf.shell.impl.console.ConsoleSessionImpl$$Lambda$1041/19902995 does not support the CompletingParsedLine interface. Completion with escaped or quoted words won't work correctly.
17:08:09.533 [ERROR] [raf.system.internal.SystemServiceImpl] - Halt error
org.osgi.framework.BundleException: Unable to acquire the state change lock for the module: osgi.identity; type="osgi.bundle"; version:Version="3.12.100.v20180210-1608"; osgi.identity="org.eclipse.osgi"; singleton:="true" [id=0] STOPPED [STOPPED]
        at org.eclipse.osgi.container.Module.lockStateChange(Module.java:337) ~[?:?]
        at org.eclipse.osgi.internal.framework.EquinoxBundle$SystemBundle$EquinoxSystemModule.asyncStop(EquinoxBundle.java:156) ~[?:?]
        at org.eclipse.osgi.internal.framework.EquinoxBundle$SystemBundle.stop(EquinoxBundle.java:262) ~[?:?]
        at org.eclipse.osgi.internal.framework.EquinoxBundle$SystemBundle.stop(EquinoxBundle.java:267) ~[?:?]
        at org.apache.karaf.system.internal.SystemServiceImpl.lambda$shutdown$0(SystemServiceImpl.java:71) ~[?:?]
        at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.util.concurrent.TimeoutException: Timeout after waiting 5 seconds to acquire the lock.
        at org.eclipse.osgi.container.Module.lockStateChange(Module.java:334) ~[?:?]
        ... 5 more

In this PR, I now run the commands not in separate threads.
Let us know if it helps.

1 Like

@Lolodomo, sorry missed your last post. Have the new version 2.5.0.201909221101 in test since today. Will let you know.

Happy to report no problems with the new version of the nuki binding :grinning:
Thanks @Lolodomo for your great work!

1 Like

org.apache.cxf.interceptor.Fault: unable to create native thread: possibly out of memory or process/resource limits reached

at org.apache.cxf.service.invoker.AbstractInvoker.createFault(AbstractInvoker.java:162) ~[bundleFile:3.4.5]

at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:128) ~[bundleFile:3.4.5]

at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:201) ~[bundleFile:3.4.5]

at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:104) ~[bundleFile:3.4.5]

at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59) ~[bundleFile:3.4.5]

at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96) ~[bundleFile:3.4.5]

at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308) ~[bundleFile:3.4.5]

at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) ~[bundleFile:3.4.5]

at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:265) ~[bundleFile:3.4.5]

at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234) ~[bundleFile:3.4.5]

at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208) ~[bundleFile:3.4.5]

at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160) ~[bundleFile:3.4.5]

at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:225) ~[bundleFile:3.4.5]

at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:298) ~[bundleFile:3.4.5]

at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:222) ~[bundleFile:3.4.5]

at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) ~[bundleFile:3.1.0]

at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:273) ~[bundleFile:3.4.5]

at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) ~[bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550) ~[bundleFile:9.4.46.v20220331]

at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:74) ~[bundleFile:?]

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) ~[bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600) ~[bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) ~[bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) ~[bundleFile:9.4.46.v20220331]

at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:294) ~[bundleFile:?]

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) ~[bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) ~[bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[bundleFile:9.4.46.v20220331]

at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:90) ~[bundleFile:?]

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) ~[bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) [bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) [bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) [bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) [bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) [bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) [bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) [bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) [bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) [bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) [bundleFile:9.4.46.v20220331]

at java.lang.Thread.run(Thread.java:829) [?:?]

Caused by: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached

at java.lang.Thread.start0(Native Method) ~[?:?]

at java.lang.Thread.start(Thread.java:798) ~[?:?]

at org.openhab.core.io.rest.JSONResponse.createResponse(JSONResponse.java:180) ~[?:?]

at org.openhab.core.io.rest.JSONResponse.createResponse(JSONResponse.java:87) ~[?:?]

at org.openhab.core.io.rest.core.internal.persistence.PersistenceResource.getItemHistoryDTO(PersistenceResource.java:235) ~[?:?]

at org.openhab.core.io.rest.core.internal.persistence.PersistenceResource.httpGetPersistenceItemData(PersistenceResource.java:175) ~[?:?]

at jdk.internal.reflect.GeneratedMethodAccessor323.invoke(Unknown Source) ~[?:?]

at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]

at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]

at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:179) ~[bundleFile:3.4.5]

at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96) ~[bundleFile:3.4.5]

… 50 more

2022-08-18 17:29:02.570 [WARN ] [e.internal.SseItemStatesEventBuilder] - Attempting to send a state update of an item which doesn’t exist: undefined

2022-08-18 17:29:02.591 [WARN ] [org.eclipse.jetty.server.HttpChannel] - /rest/persistence/items/Lichtsterkte_zonnecollectoren

org.apache.cxf.interceptor.Fault: unable to create native thread: possibly out of memory or process/resource limits reached

at org.apache.cxf.service.invoker.AbstractInvoker.createFault(AbstractInvoker.java:162) ~[?:?]

at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:128) ~[?:?]

at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:201) ~[?:?]

at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:104) ~[?:?]

at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59) ~[?:?]

at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96) ~[?:?]

at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308) ~[?:?]

at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) ~[?:?]

at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:265) ~[?:?]

at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234) ~[?:?]

at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208) ~[?:?]

at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160) ~[?:?]

at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:225) ~[?:?]

at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:298) ~[?:?]

at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:222) ~[?:?]

at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) ~[bundleFile:3.1.0]

at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:273) ~[?:?]

at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) ~[?:?]

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550) ~[?:?]

at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:74) ~[?:?]

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) ~[bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600) ~[?:?]

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) ~[bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) ~[bundleFile:9.4.46.v20220331]

at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:294) ~[?:?]

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) ~[?:?]

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) ~[bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[bundleFile:9.4.46.v20220331]

at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:90) ~[?:?]

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) ~[bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) ~[bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) [bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) [bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) [bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) [bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) [bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) [bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) [bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) [bundleFile:9.4.46.v20220331]

at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) [bundleFile:9.4.46.v20220331]

at java.lang.Thread.run(Thread.java:829) [?:?]

Caused by: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached

at java.lang.Thread.start0(Native Method) ~[?:?]

at java.lang.Thread.start(Thread.java:798) ~[?:?]

at org.openhab.core.io.rest.JSONResponse.createResponse(JSONResponse.java:180) ~[?:?]

at org.openhab.core.io.rest.JSONResponse.createResponse(JSONResponse.java:87) ~[?:?]

at org.openhab.core.io.rest.core.internal.persistence.PersistenceResource.getItemHistoryDTO(PersistenceResource.java:235) ~[?:?]

at org.openhab.core.io.rest.core.internal.persistence.PersistenceResource.httpGetPersistenceItemData(PersistenceResource.java:175) ~[?:?]

at jdk.internal.reflect.GeneratedMethodAccessor323.invoke(Unknown Source) ~[?:?]

at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]

at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]

at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:179) ~[?:?]

at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96) ~[?:?]

… 50 more

Hello,
My open tab 3.3 system crashed again after 7 days.
I have started a completely new installation with a new sd card.

Are there any people who can help?
below you can see all the logs.