Logs flooded with Exception in thread EventAdminAsyncThread

[ceedee]

I’m facing issues regularly with no events shown in logtails. Then my openHAB system in basically inoperative.

syslog and daemon.log contain entries like the ones below, many of them per second. Those logs ate ~15 GB of my sd card although logs we rotated. This led to “no space on disk” errors.

Feb 26 12:17:35 openHABianPi4 karaf[815]: Exception in thread "EventAdminAsyncThread #12" Exception in thread "EventAdminAsyncThread #11" Exception in thread "EventAdminAsyncThread #14" java.lang.NullPointerException
Feb 26 12:17:35 openHABianPi4 karaf[815]: #011at java.util.concurrent.LinkedBlockingQueue.dequeue(LinkedBlockingQueue.java:213)
Feb 26 12:17:35 openHABianPi4 karaf[815]: #011at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:444)
Feb 26 12:17:35 openHABianPi4 karaf[815]: #011at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
Feb 26 12:17:35 openHABianPi4 karaf[815]: #011at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
Feb 26 12:17:35 openHABianPi4 karaf[815]: #011at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Feb 26 12:17:35 openHABianPi4 karaf[815]: #011at java.lang.Thread.run(Thread.java:748)
Feb 26 12:17:35 openHABianPi4 karaf[815]: java.lang.NullPointerException
Feb 26 12:17:35 openHABianPi4 karaf[815]: #011at java.util.concurrent.LinkedBlockingQueue.dequeue(LinkedBlockingQueue.java:213)
Feb 26 12:17:35 openHABianPi4 karaf[815]: #011at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:444)
Feb 26 12:17:35 openHABianPi4 karaf[815]: #011at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
Feb 26 12:17:35 openHABianPi4 karaf[815]: #011at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
Feb 26 12:17:35 openHABianPi4 karaf[815]: #011at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Feb 26 12:17:35 openHABianPi4 karaf[815]: #011at java.lang.Thread.run(Thread.java:748)
Feb 26 12:17:35 openHABianPi4 karaf[815]: java.lang.NullPointerException

First, I thought that it’s related to Amazon Echo Control since it caused similar exceptions. I always updated to the newest version if the problem re-occurred. At least then openHAB did not run out of memory any more.

But the thread pool errors persisted and ended up in the logs seen above.

I came across Openhab 2.5 is blocking after one week of runtime here. The Thread::sleep calls I’m using are rarely called (only on specific events like reboot via voice), but maybe I’m causing thread leaks via wrong use of reentrant locks:

rule "Minimum battery level of devices went below 20%"
when
    Item GwarningsSensorsBatteryLevelsMin received update
then
    latch.lock
    try {
        var triggeringGroup = (triggeringItem as GroupItem)
        checkMinBatteryLevel.apply(triggeringGroup, 20)
    } finally {
        latch.unlock
    }
end

I think, I noticed that when exceptions occur within the try block, the finally block does not executed. This isn’t exactly the behavior I’d expect from a resource protection block. Is there a better way to safeguard this?

Best regards,
ceedee

[manswiss (Manuel Schärer)]

I’ve the same Problem:

Syslog and daemon.log flood with Java-Error

After a while when openhab is running (some hours) it begins to fill daemon.log and syslog. Around 5Gb every hour. Here is the first lines when it starts: Feb 3 07:51:43 openHAB karaf[573]: Exception in thread "pool-5-thread-1" java.lang.NullPointerException Feb 3 07:51:43 openHAB karaf[573]: #011at java.base/java.util.concurrent.LinkedBlockingQueue.dequeue(LinkedBlockingQueue.java:217) Feb 3 07:51:43 openHAB karaf[573]: #011at java.base/java.util.concurrent.LinkedBlockingQueue.take(LinkedB…

And another topic with that:

Syslog Errors (100 GB)

My syslog (and deamon.log) is full (many, many GB of Data) with the following example: Feb 11 11:22:48 openHAB karaf[555]: Exception in thread "OH-eventexecutor-615" java.lang.NullPointerException Feb 11 11:22:48 openHAB karaf[555]: #011at java.base/java.util.concurrent.LinkedBlockingQueue.dequeue(LinkedBlockingQueue.java:214) Feb 11 11:22:48 openHAB karaf[555]: #011at java.base/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435) Feb 11 11:22:48 openHAB karaf[555]: #011a…

But nobody can help me. It happens sometimes after some hours, sometime after days or weeks.

[ceedee]

I’ve added the follwing to runtime.cfg:

org.eclipse.smarthome.threadpool:ruleEngine=100

For a month now, I did not see the mentioned errors again.

Hope it helps someone out there!

Best regards,
cd_

[manswiss (Manuel Schärer)]

It not help by me

[Schrott.Micha (Michael)]

I got the same problem for some time now - after I added lots of new items and a couple of new rules:

May 01 08:56:08 openHABianpi karaf[26383]:         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
May 01 08:56:08 openHABianpi karaf[26383]:         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
May 01 08:56:08 openHABianpi karaf[26383]:         at java.lang.Thread.run(Thread.java:748)
May 01 08:56:08 openHABianpi karaf[26383]: Exception in thread "EventAdminAsyncThread #830008" Exception in thread "EventAdminAsyncThread #830007" java.lang.NullPointerException
May 01 08:56:08 openHABianpi karaf[26383]: Exception in thread "EventAdminAsyncThread #830006" Exception in thread "EventAdminAsyncThread #830005" Exception in thread "EventAdminAsyncThread #830004" Exception in thread "EventAdminAsyncThread #830003" Exception in thread "EventAdminAsyncThread #830002" Exception in thread "EventAdminAsyncThread #830001" Exception in thread "EventAdminAsyncThread #830000" Exception in thread "EventAdminAsyncThread #829999" Exception in thread "EventAdminAsyncThread #829998" Exception in thread "EventAdminAsyncThread #829997" Exception in thread "EventAdminAsyncThread #829995" Exception in thread "EventAdminAsyncThread #829996" Exception in thread "EventAdminAsyncThread #829994" Exception in thread "EventAdminAsyncThread #829993" Exception in thread "EventAdminAsyncThread #829992" Exception in thread "EventAdminAsyncThread #829991" Exception in thread "EventAdminAsyncThread #829989" Exception in thread "EventAdminAsyncThread #829990" Exception in thread "EventAdminAsyncThread #829988" Exception in thread "EventAdminAsyncThread #829987" Exception in thread "EventAdminAsyncThread #829986" Exception in thread "EventAdminAsyncThread #829985" Exception in thread "EventAdminAsyncThread #829984" Exception in thread "EventAdminAsyncThread #829983" Exception in thread "EventAdminAsyncThread #829982" Exception in thread "EventAdminAsyncThread #829981" Exception in thread "EventAdminAsyncThread #829980" Exception in thread "EventAdminAsyncThread #829979" Exception in thread "EventAdminAsyncThread #829978" Exception in thread "EventAdminAsyncThread #829977" Exception in thread "EventAdminAsyncThread #829976" Exception in thread "EventAdminAsyncThread #829975" Exception in thread "EventAdminAsyncThread #829974" Exception in thread "EventAdminAsyncThread #829973" Exception in thread "EventAdminAsyncThread #829972" Exception in thread "EventAdminAsyncThread #829971" Exception in thread "EventAdminAsyncThread #829970" Exception in thread "EventAdminAsyncThread #829969" Exception in thread "EventAdminAsyncThread #829968" Exception in thread "EventAdminAsyncThread #829967" Exception in thread "EventAdminAsyncThread #829966" Exception in thread "EventAdminAsyncThread #829965" Exception in thread "EventAdminAsyncThread #829964" Exception in thread "EventAdminAsyncThread #829963" Exception in thread "EventAdminAsyncThread #829962" Exception in thread "EventAdminAsyncThread #829961" Exception in thread "EventAdminAsyncThread #829960" Exception in thread "EventAdminAsyncThread #829959" Exception in thread "EventAdminAsyncThread #829958" Exception in thread "EventAdminAsyncThread #829957" Exception in thread "EventAdminAsyncThread #829956" Exception in thread "EventAdminAsyncThread #829955" Exception in thread "EventAdminAsyncThread #829954" Exception in thread "EventAdminAsyncThread #829953" Exception in thread "EventAdminAsyncThread #829952" Exception in thread "EventAdminAsyncThread #829951" Exception in thread "EventAdminAsyncThread #829950" Exception in thread "EventAdminAsyncThread #829949" Exception in thread "EventAdminAsyncThread #829948" Exception in thread "EventAdminAsyncThread #829947" Exception in thread "EventAdminAsyncThread #829946" Exception in thread "EventAdminAsyncThread #829945" Exception in thread "EventAdminAsyncThread #829944" Exception in thread "EventAdminAsyncThread #829943" Exception in thread "EventAdminAsyncThread #829942" Exception in thread "EventAdminAsyncThread #829941" Exception in thread "EventAdminAsyncThread #829940" Exception in thread "EventAdminAsyncThread #829939" Exception in thread "EventAdminAsyncThread #830010" Exception in thread "EventAdminAsyncThread #829938" Exception in thread "EventAdminAsyncThread #829937" Exception in thread "EventAdminAsyncThread #829936" Exception in thread "EventAdminAsyncThread #829935" Exception in thread "EventAdminAsyncThread #829934" Exception in thread "EventAdminAsyncThread #829933" Exception in thread "EventAdminAsyncThread #829932" Exception in thread "EventAdminAsyncThread #829931" Exception in thread "EventAdminAsyncThread #829930" Exception in thread "EventAdminAsyncThread #829929" Exception in thread "EventAdminAsyncThread #829928" Exception in thread "EventAdminAsyncThread #829927" Exception in thread "EventAdminAsyncThread #829926" Exception in thread "EventAdminAsyncThread #829925" Exception in thread "EventAdminAsyncThread #829924" Exception in thread "EventAdminAsyncThread #829923" Exception in thread "EventAdminAsyncThread #829922" Exception in thread "EventAdminAsyncThread #829921" Exception in thread "EventAdminAsyncThread #829920" Exception in thread "EventAdminAsyncThread #829919" Exception in thread "EventAdminAsyncThread #829918" Exception in thread "EventAdminAsyncThread #829917" Exception in thread "EventAdminAsyncThread #829916" Exception in thread "EventAdminAsyncThread #829915" Exception in thread "EventAdminAsyncThread #829914" Exception in thread "EventAdminAsyncThread #829913" Exception in thread "EventAdminAsyncThread #829912" Exception in thread "EventAdminAsyncThread #829911" Exception in thread "EventAdminAsyncThread #829910" Exception in thread "EventAdminAsyncThread #829909" Exception in thread "EventAdminAsyncThread #829908" Exception in thread "EventAdminAsyncThread #829907" Exception in thread "EventAdminAsyncThread #829906" Exception in thread "EventAdminAsyncThread #829905" Exception in thread "EventAdminAsyncThread #829904" Exception in thread "EventAdminAsyncThread #829903" Exception in thread "EventAdminAsyncThread #829902" Exception in thread "EventAdminAsyncThread #829901" Exception in thread "EventAdminAsyncThread #829900" Exception in thread "EventAdminAsyncThread #829899" Exception in thread "EventAdminAsyncThread #829898" Exception in thread "EventAdminAsyncThread #829897" Exception in thread "EventAdminAsyncThread #829896" Exception in thread "EventAdminAsyncThread #829895" Exception in thread "EventAdminAsyncThread #829894" Exception in thread "EventAdminAsyncThread #829893" Exception in thread "EventAdminAsyncThread #829892" Exception in thread "EventAdminAsyncThread #829891"         at java.util.concurrent.LinkedBlockingQueue.dequeue(LinkedBlockingQueue.java:213)
May 01 08:56:08 openHABianpi karaf[26383]:         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:444)
May 01 08:56:08 openHABianpi karaf[26383]:         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
May 01 08:56:08 openHABianpi karaf[26383]:         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
May 01 08:56:08 openHABianpi karaf[26383]:         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
May 01 08:56:08 openHABianpi karaf[26383]:         at java.lang.Thread.run(Thread.java:748)
May 01 08:56:08 openHABianpi karaf[26383]: java.lang.NullPointerException
May 01 08:56:08 openHABianpi karaf[26383]:         at java.util.concurrent.LinkedBlockingQueue.dequeue(LinkedBlockingQueue.java:213)
May 01 08:56:08 openHABianpi karaf[26383]:         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:444)
May 01 08:56:08 openHABianpi karaf[26383]:         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
May 01 08:56:08 openHABianpi karaf[26383]:         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
May 01 08:56:08 openHABianpi karaf[26383]:         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
May 01 08:56:08 openHABianpi karaf[26383]:         at java.lang.Thread.run(Thread.java:748)
May 01 08:56:08 openHABianpi karaf[26383]: java.lang.NullPointerException

I reverted the changes now and see if it becomes more stable.

[ceedee]

For me, with the above mentioned configuration my problems disappeared. I have to say, that I reboot my openHAB nightly. Additionally I’ve some monitoring installed which reboots openHAB whenever OOM errors show up in the log. That mechanism could also be applied for EventAdminAsyncThread exceptions.

[ssalonen]

This is also discussed in Syslog Errors (100 GB) - #35 by ssalonen

For me, I got the issue only with zulu java and was resolved by migrating to AdoptOpenJDK.