I’m facing issues regularly with no events shown in logtails. Then my openHAB system in basically inoperative.
syslog and daemon.log contain entries like the ones below, many of them per second. Those logs ate ~15 GB of my sd card although logs we rotated. This led to “no space on disk” errors.
Feb 26 12:17:35 openHABianPi4 karaf[815]: Exception in thread "EventAdminAsyncThread #12" Exception in thread "EventAdminAsyncThread #11" Exception in thread "EventAdminAsyncThread #14" java.lang.NullPointerException
Feb 26 12:17:35 openHABianPi4 karaf[815]: #011at java.util.concurrent.LinkedBlockingQueue.dequeue(LinkedBlockingQueue.java:213)
Feb 26 12:17:35 openHABianPi4 karaf[815]: #011at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:444)
Feb 26 12:17:35 openHABianPi4 karaf[815]: #011at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
Feb 26 12:17:35 openHABianPi4 karaf[815]: #011at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
Feb 26 12:17:35 openHABianPi4 karaf[815]: #011at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Feb 26 12:17:35 openHABianPi4 karaf[815]: #011at java.lang.Thread.run(Thread.java:748)
Feb 26 12:17:35 openHABianPi4 karaf[815]: java.lang.NullPointerException
Feb 26 12:17:35 openHABianPi4 karaf[815]: #011at java.util.concurrent.LinkedBlockingQueue.dequeue(LinkedBlockingQueue.java:213)
Feb 26 12:17:35 openHABianPi4 karaf[815]: #011at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:444)
Feb 26 12:17:35 openHABianPi4 karaf[815]: #011at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
Feb 26 12:17:35 openHABianPi4 karaf[815]: #011at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
Feb 26 12:17:35 openHABianPi4 karaf[815]: #011at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Feb 26 12:17:35 openHABianPi4 karaf[815]: #011at java.lang.Thread.run(Thread.java:748)
Feb 26 12:17:35 openHABianPi4 karaf[815]: java.lang.NullPointerException
First, I thought that it’s related to Amazon Echo Control since it caused similar exceptions. I always updated to the newest version if the problem re-occurred. At least then openHAB did not run out of memory any more.
But the thread pool errors persisted and ended up in the logs seen above.
I came across Openhab 2.5 is blocking after one week of runtime here. The Thread::sleep calls I’m using are rarely called (only on specific events like reboot via voice), but maybe I’m causing thread leaks via wrong use of reentrant locks:
rule "Minimum battery level of devices went below 20%"
when
Item GwarningsSensorsBatteryLevelsMin received update
then
latch.lock
try {
var triggeringGroup = (triggeringItem as GroupItem)
checkMinBatteryLevel.apply(triggeringGroup, 20)
} finally {
latch.unlock
}
end
I think, I noticed that when exceptions occur within the try block, the finally block does not executed. This isn’t exactly the behavior I’d expect from a resource protection block. Is there a better way to safeguard this?
I got the same problem for some time now - after I added lots of new items and a couple of new rules:
May 01 08:56:08 openHABianpi karaf[26383]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
May 01 08:56:08 openHABianpi karaf[26383]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
May 01 08:56:08 openHABianpi karaf[26383]: at java.lang.Thread.run(Thread.java:748)
May 01 08:56:08 openHABianpi karaf[26383]: Exception in thread "EventAdminAsyncThread #830008" Exception in thread "EventAdminAsyncThread #830007" java.lang.NullPointerException
May 01 08:56:08 openHABianpi karaf[26383]: Exception in thread "EventAdminAsyncThread #830006" Exception in thread "EventAdminAsyncThread #830005" Exception in thread "EventAdminAsyncThread #830004" Exception in thread "EventAdminAsyncThread #830003" Exception in thread "EventAdminAsyncThread #830002" Exception in thread "EventAdminAsyncThread #830001" Exception in thread "EventAdminAsyncThread #830000" Exception in thread "EventAdminAsyncThread #829999" Exception in thread "EventAdminAsyncThread #829998" Exception in thread "EventAdminAsyncThread #829997" Exception in thread "EventAdminAsyncThread #829995" Exception in thread "EventAdminAsyncThread #829996" Exception in thread "EventAdminAsyncThread #829994" Exception in thread "EventAdminAsyncThread #829993" Exception in thread "EventAdminAsyncThread #829992" Exception in thread "EventAdminAsyncThread #829991" Exception in thread "EventAdminAsyncThread #829989" Exception in thread "EventAdminAsyncThread #829990" Exception in thread "EventAdminAsyncThread #829988" Exception in thread "EventAdminAsyncThread #829987" Exception in thread "EventAdminAsyncThread #829986" Exception in thread "EventAdminAsyncThread #829985" Exception in thread "EventAdminAsyncThread #829984" Exception in thread "EventAdminAsyncThread #829983" Exception in thread "EventAdminAsyncThread #829982" Exception in thread "EventAdminAsyncThread #829981" Exception in thread "EventAdminAsyncThread #829980" Exception in thread "EventAdminAsyncThread #829979" Exception in thread "EventAdminAsyncThread #829978" Exception in thread "EventAdminAsyncThread #829977" Exception in thread "EventAdminAsyncThread #829976" Exception in thread "EventAdminAsyncThread #829975" Exception in thread "EventAdminAsyncThread #829974" Exception in thread "EventAdminAsyncThread #829973" Exception in thread "EventAdminAsyncThread #829972" Exception in thread "EventAdminAsyncThread #829971" Exception in thread "EventAdminAsyncThread #829970" Exception in thread "EventAdminAsyncThread #829969" Exception in thread "EventAdminAsyncThread #829968" Exception in thread "EventAdminAsyncThread #829967" Exception in thread "EventAdminAsyncThread #829966" Exception in thread "EventAdminAsyncThread #829965" Exception in thread "EventAdminAsyncThread #829964" Exception in thread "EventAdminAsyncThread #829963" Exception in thread "EventAdminAsyncThread #829962" Exception in thread "EventAdminAsyncThread #829961" Exception in thread "EventAdminAsyncThread #829960" Exception in thread "EventAdminAsyncThread #829959" Exception in thread "EventAdminAsyncThread #829958" Exception in thread "EventAdminAsyncThread #829957" Exception in thread "EventAdminAsyncThread #829956" Exception in thread "EventAdminAsyncThread #829955" Exception in thread "EventAdminAsyncThread #829954" Exception in thread "EventAdminAsyncThread #829953" Exception in thread "EventAdminAsyncThread #829952" Exception in thread "EventAdminAsyncThread #829951" Exception in thread "EventAdminAsyncThread #829950" Exception in thread "EventAdminAsyncThread #829949" Exception in thread "EventAdminAsyncThread #829948" Exception in thread "EventAdminAsyncThread #829947" Exception in thread "EventAdminAsyncThread #829946" Exception in thread "EventAdminAsyncThread #829945" Exception in thread "EventAdminAsyncThread #829944" Exception in thread "EventAdminAsyncThread #829943" Exception in thread "EventAdminAsyncThread #829942" Exception in thread "EventAdminAsyncThread #829941" Exception in thread "EventAdminAsyncThread #829940" Exception in thread "EventAdminAsyncThread #829939" Exception in thread "EventAdminAsyncThread #830010" Exception in thread "EventAdminAsyncThread #829938" Exception in thread "EventAdminAsyncThread #829937" Exception in thread "EventAdminAsyncThread #829936" Exception in thread "EventAdminAsyncThread #829935" Exception in thread "EventAdminAsyncThread #829934" Exception in thread "EventAdminAsyncThread #829933" Exception in thread "EventAdminAsyncThread #829932" Exception in thread "EventAdminAsyncThread #829931" Exception in thread "EventAdminAsyncThread #829930" Exception in thread "EventAdminAsyncThread #829929" Exception in thread "EventAdminAsyncThread #829928" Exception in thread "EventAdminAsyncThread #829927" Exception in thread "EventAdminAsyncThread #829926" Exception in thread "EventAdminAsyncThread #829925" Exception in thread "EventAdminAsyncThread #829924" Exception in thread "EventAdminAsyncThread #829923" Exception in thread "EventAdminAsyncThread #829922" Exception in thread "EventAdminAsyncThread #829921" Exception in thread "EventAdminAsyncThread #829920" Exception in thread "EventAdminAsyncThread #829919" Exception in thread "EventAdminAsyncThread #829918" Exception in thread "EventAdminAsyncThread #829917" Exception in thread "EventAdminAsyncThread #829916" Exception in thread "EventAdminAsyncThread #829915" Exception in thread "EventAdminAsyncThread #829914" Exception in thread "EventAdminAsyncThread #829913" Exception in thread "EventAdminAsyncThread #829912" Exception in thread "EventAdminAsyncThread #829911" Exception in thread "EventAdminAsyncThread #829910" Exception in thread "EventAdminAsyncThread #829909" Exception in thread "EventAdminAsyncThread #829908" Exception in thread "EventAdminAsyncThread #829907" Exception in thread "EventAdminAsyncThread #829906" Exception in thread "EventAdminAsyncThread #829905" Exception in thread "EventAdminAsyncThread #829904" Exception in thread "EventAdminAsyncThread #829903" Exception in thread "EventAdminAsyncThread #829902" Exception in thread "EventAdminAsyncThread #829901" Exception in thread "EventAdminAsyncThread #829900" Exception in thread "EventAdminAsyncThread #829899" Exception in thread "EventAdminAsyncThread #829898" Exception in thread "EventAdminAsyncThread #829897" Exception in thread "EventAdminAsyncThread #829896" Exception in thread "EventAdminAsyncThread #829895" Exception in thread "EventAdminAsyncThread #829894" Exception in thread "EventAdminAsyncThread #829893" Exception in thread "EventAdminAsyncThread #829892" Exception in thread "EventAdminAsyncThread #829891" at java.util.concurrent.LinkedBlockingQueue.dequeue(LinkedBlockingQueue.java:213)
May 01 08:56:08 openHABianpi karaf[26383]: at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:444)
May 01 08:56:08 openHABianpi karaf[26383]: at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
May 01 08:56:08 openHABianpi karaf[26383]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
May 01 08:56:08 openHABianpi karaf[26383]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
May 01 08:56:08 openHABianpi karaf[26383]: at java.lang.Thread.run(Thread.java:748)
May 01 08:56:08 openHABianpi karaf[26383]: java.lang.NullPointerException
May 01 08:56:08 openHABianpi karaf[26383]: at java.util.concurrent.LinkedBlockingQueue.dequeue(LinkedBlockingQueue.java:213)
May 01 08:56:08 openHABianpi karaf[26383]: at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:444)
May 01 08:56:08 openHABianpi karaf[26383]: at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
May 01 08:56:08 openHABianpi karaf[26383]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
May 01 08:56:08 openHABianpi karaf[26383]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
May 01 08:56:08 openHABianpi karaf[26383]: at java.lang.Thread.run(Thread.java:748)
May 01 08:56:08 openHABianpi karaf[26383]: java.lang.NullPointerException
I reverted the changes now and see if it becomes more stable.
For me, with the above mentioned configuration my problems disappeared. I have to say, that I reboot my openHAB nightly. Additionally I’ve some monitoring installed which reboots openHAB whenever OOM errors show up in the log. That mechanism could also be applied for EventAdminAsyncThread exceptions.