In addition I have a lot of “503 jersey is not ready yet” using PaperUI since I’m on 2.4 (I never had these messages with 2.3). Maybe this is in relation to the warning?
Restarted openhab. But the warning is still there.
But if I got it right (and as this is just a warning) this should be nothing to worry about, should it?
This false positive warning is specific to openHAB 2.4.0. If you didn’t apply the suggested logging configuration changes (while upgrading) you can also suppress it by raising the log level in the Console with:
The false positive warning seems to be fixed with the upgrade to Karaf 4.2.2 which uses Jetty 9.4.12.v20180830 and is available in openHAB 2.5.0-SNAPSHOT Build # 1482 (see PR #834).
I’m concerned this is less of a false positive and more of an actual issue on systems with big configurations. When I upgraded from 2.3.0 to the 2.4.0 maintenance releases (I believe I went up at M4) I noticed that my bindings and things would take a very long time to load when starting OH. According to the logs, all of my rules and items load with no issues. As soon as that happens, the ThreadPoolBudget warning comes up and everything goes sideways. I’ve seen it take as long as 10 minutes after that for my things and bindings to load. This system has a substantial backend, I’m not worried about resources. I’ve read through these threads and applied about 10 different configurations in the runtime.cfg and quartz.properties which fixed some of the issues, but this still happens. Everything eventually loads and the messages calm down. While the errors are flying I can see bindings loading one at a time, random things showing up in the inbox (presumably before the thing configuration loaded), and then the things attaching to them when they finally load. The only thing that comes to mind is that the system is starved for threads in the pool and it’s taking a long time to load everything. Are there any other thread pool configurations that could be added to mitigate this? To note, I’m also seeing the 503 errors noted by adahmen above.
I’ll cut over to 2.5 when I get some time to take the plunge. Inevitably those leaps breaks something and I don’t have time to start getting in the weeks of fixing things right now. I was hoping to do a quick test of any additional thread pool configs before making that leap.
For reference, these are the parameters I’ve configured so far in /etc/openhab2/services/runtime.cfg:
org.eclipse.smarthome.threadpool:thingHandler
org.eclipse.smarthome.threadpool:discovery
org.eclipse.smarthome.threadpool:safeCall
org.eclipse.smarthome.threadpool:ruleEngine
And in /usr/share/openhab2/runtime/etc/quartz.properties (only because I can’t figure out the correct syntax for /etc/openhab2/services/runtime.cfg)
org.quartz.threadPool.threadCount
EDIT: Go down about 10 replies for the longer list.
I’ve tested several different variations from very low numbers (<100) to insanely high (up to 10000) and I’ve come to the conclusion that “it depends on what you have running” is mostly the answer. Right now I believe I have the first 4 and quartz set to 1000 and the second 4 set to 50 and 500 respectively for min/max. The only negative I’ve seen with going to insanely high is more memory utilization. While 1000 is definitively overkill, I’ve not otherwise been negatively effected by having it that high and it ensures I won’t jam up during load. The OH install runs as a VM on an ESXi host with considerable resources so memory usage was not a concern of mine through the testing. I’ve dialed them up and down and the answer has effectively been “as long as you have enough threads, you don’t jam up”.
I bit the bullet last night and pushed to 2.5.0-SNAPSHOT (Build 1486) and while the error message went away, the startup behavior has not changed. I’ve always been a fan of the motto that “if someone spent the time to code the error/warning, there was probably a reason they did it even if that’s not immediately apparent.” At this point, my belief is that I’m in some sort of race condition or thread starvation which is causing the bindings and things to load in a congested manner. I would like to figure out what was creating the original thread warning (and setting me to 8 threads) and manually set that number higher. ESXi handles threads a little differently than a physical host would and I can utilize additional threads through some magic voodoo that they have.
Ive hit this issue and im sure its causing me dramas with bindings not loading. Today the gnu.io unloaded itself magically (i had to reinstall it) and the zwave shut down.
I have this in my runtime.cfg configuration.
My CPU is a Quad core Xeon with 8GB Ram
Has anyone been able to resolve this, it appears to be impacting.
Is anyone else noticing a crazy amount of lag on the 2.5 nightly builds? I have rules firing minutes late. The bus itself even seems to be lagging pretty bad based on logs. I’m wondering if these configs need to be changed based on the changes that have been made over the past few months.
Right, the above issue was fixed in 1618. So, clearly there’s something else in play on your system. On which build did you first notice the degradation in performance?
I noticed it when I went from M1 to 1603 at first. It then was still present on 1607 and 1613. 1621 is “better” but still not as it was. 1621 brought the flag from minutes into the 15-30 second range.
Apparently I had 3 issues. First, yes, 1618 fixed the issue. I went back once I found the other two issues and installed 1617, confirmed the issue, and then upgraded to 1618 and confirmed it was gone.
Issue #2, related to this thread, is that there are more threadpool items which need to be updated on larger systems. I also finally figured out the quartz item I was having issues with above. This is the list I have right now, tailor it to your system needs:
Issue #3 was the Neeo IO transport. There is an issue right now with the one that comes in the regular bundle which requires the two jar files to be put into the addons folder. Apparently something with the IO file was causing other services/bindings to not load properly. This was causing a whole other set of issues which made the system look slow.
I’m familiar with all the above with the exception of the last one (i.e. org.apache.felix.eventadmin.impl.EventAdmin:org.apache.felix.eventadmin.ThreadPoolSize). In my recent testing, I’ve not seen any degradation in state updates and command processing with that one set to the default, even under exceptionally high load (e.g. 2000 state updates per second).
That one was in there as a dice roll. One of the things I did was go through config:list looking for anything with a thread count. I figured it couldn’t hurt to dial it up.
