That is so bizarre. I have no idea why it’s saying it needs 13 threads. I’m running on a pretty powerful host, and I’ve never seen anything like this (and I’ve seen a lot).
When you figure it out, please post here.
I probably should catch the IllegalStateException. If the squeezebox binding can’t get the coverart, it’s not a big deal…
I’ve tried quite a few 2.4 versions, including bleeding edge, but they all exhibit the same behaviour, so I preferred to go back to the more stable M versions.
This does feel a bit like what is happening, but then I could just be going down the wrong rabbit hole…
Still haven’t figured out the magical incantation for the runtime.cfg to get the max threads changed :-(.
I will check my ‘lots of cores’ theory by spinning up a VM with only one core on the same server and install openhab on that with the squeezebox plugin.
@koentav So, this area of code has changed pretty substantially recently. I’m wondering if a synchronization issue possibly has been introduced that is being exploited by your environment.
I ran M5 last week with the same results. M6 only made it into the openhab testing repository a couple of days ago and I had hoped that upgrading would solve the problem.
I went ahead and set up a single cpu VM, with exactly the same setup, debian stretch, M6, zulu-8. And lo and behold it works out of the box. I can control the squeezebox perfectly through openhab, and no mention of jetty problems in the log:
openhab> log:tail
22:27:52.091 [INFO ] [mebuilder.internal.HomeBuilderServlet] - Started Home Builder at /homebuilder
22:27:52.222 [INFO ] [bpanel.internal.HABPanelDashboardTile] - Started HABPanel at /habpanel
22:28:02.432 [INFO ] [rthome.model.lsp.internal.ModelServer] - Started Language Server Protocol (LSP) service on port 5007
22:28:03.321 [INFO ] [thome.event.ItemChannelLinkAddedEvent] - Link 'squeezebox_squeezeboxplayer_f16ede86_000420061c8c_album-squeezebox:squeezeboxplayer:f16ede86:000420061c8c:album' has been added.
22:28:03.346 [INFO ] [thome.event.ItemChannelLinkAddedEvent] - Link 'squeezebox_squeezeboxplayer_f16ede86_000420061c8c_currentPlaylistRepeat-squeezebox:squeezeboxplayer:f16ede86:000420061c8c:currentPlaylistReeat' has been added.
22:28:03.347 [INFO ] [thome.event.ItemChannelLinkAddedEvent] - Link 'squeezebox_squeezeboxplayer_f16ede86_000420061c8c_currentPlayingTime-squeezebox:squeezeboxplayer:f16ede86:000420061c8c:currentPlayingTime' as been added.
22:28:03.349 [INFO ] [thome.event.ItemChannelLinkAddedEvent] - Link 'squeezebox_squeezeboxplayer_f16ede86_000420061c8c_duration-squeezebox:squeezeboxplayer:f16ede86:000420061c8c:duration' has been added.
22:28:03.350 [INFO ] [thome.event.ItemChannelLinkAddedEvent] - Link 'squeezebox_squeezeboxplayer_f16ede86_000420061c8c_artist-squeezebox:squeezeboxplayer:f16ede86:000420061c8c:artist' has been added.
22:28:03.351 [INFO ] [thome.event.ItemChannelLinkAddedEvent] - Link 'squeezebox_squeezeboxplayer_f16ede86_000420061c8c_mute-squeezebox:squeezeboxplayer:f16ede86:000420061c8c:mute' has been added.
22:28:03.367 [INFO ] [thome.event.ItemChannelLinkAddedEvent] - Link 'squeezebox_squeezeboxplayer_f16ede86_000420061c8c_currentPlaylistShuffle-squeezebox:squeezeboxplayer:f16ede86:000420061c8c:currentPlaylistSuffle' has been added.
22:28:03.368 [INFO ] [thome.event.ItemChannelLinkAddedEvent] - Link 'squeezebox_squeezeboxplayer_f16ede86_000420061c8c_title-squeezebox:squeezeboxplayer:f16ede86:000420061c8c:title' has been added.
22:28:03.369 [INFO ] [thome.event.ItemChannelLinkAddedEvent] - Link 'squeezebox_squeezeboxplayer_f16ede86_000420061c8c_control-squeezebox:squeezeboxplayer:f16ede86:000420061c8c:control' has been added.
22:28:03.371 [INFO ] [thome.event.ItemChannelLinkAddedEvent] - Link 'squeezebox_squeezeboxplayer_f16ede86_000420061c8c_volume-squeezebox:squeezeboxplayer:f16ede86:000420061c8c:volume' has been added.
22:28:03.372 [INFO ] [thome.event.ItemChannelLinkAddedEvent] - Link 'squeezebox_squeezeboxplayer_f16ede86_000420061c8c_power-squeezebox:squeezeboxplayer:f16ede86:000420061c8c:power' has been added.
22:28:04.119 [INFO ] [i.dashboard.internal.DashboardService] - Started Dashboard at http://192.168.13.5:8080
22:28:04.123 [INFO ] [i.dashboard.internal.DashboardService] - Started Dashboard at https://192.168.13.5:8443
22:28:04.490 [INFO ] [marthome.ui.paper.internal.PaperUIApp] - Started Paper UI at /paperui
22:28:05.424 [INFO ] [ome.event.ThingStatusInfoChangedEvent] - 'squeezebox:squeezeboxserver:f16ede86' changed from UNINITIALIZED to INITIALIZING
22:28:05.436 [INFO ] [ome.event.ThingStatusInfoChangedEvent] - 'squeezebox:squeezeboxplayer:f16ede86:000420061c8c' changed from UNINITIALIZED to UNINITIALIZED (BRIDGE_UNINITIALIZED)
22:28:05.494 [INFO ] [ome.event.ThingStatusInfoChangedEvent] - 'squeezebox:squeezeboxserver:f16ede86' changed from INITIALIZING to ONLINE
22:28:05.607 [INFO ] [ome.event.ThingStatusInfoChangedEvent] - 'squeezebox:squeezeboxplayer:f16ede86:000420061c8c' changed from UNINITIALIZED (BRIDGE_UNINITIALIZED) to INITIALIZING
22:28:05.626 [INFO ] [ome.event.ThingStatusInfoChangedEvent] - 'squeezebox:squeezeboxplayer:f16ede86:000420061c8c' changed from INITIALIZING to ONLINE
22:28:06.137 [INFO ] [smarthome.event.ItemStateChangedEvent] - squeezebox_squeezeboxplayer_f16ede86_000420061c8c_artist changed from NULL to
22:28:06.148 [INFO ] [smarthome.event.ItemStateChangedEvent] - squeezebox_squeezeboxplayer_f16ede86_000420061c8c_album changed from NULL to
22:28:15.628 [INFO ] [smarthome.event.ItemStateChangedEvent] - squeezebox_squeezeboxplayer_f16ede86_000420061c8c_power changed from NULL to ON
22:28:15.642 [INFO ] [smarthome.event.ItemStateChangedEvent] - squeezebox_squeezeboxplayer_f16ede86_000420061c8c_control changed from NULL to PAUSE
22:28:15.659 [INFO ] [smarthome.event.ItemStateChangedEvent] - squeezebox_squeezeboxplayer_f16ede86_000420061c8c_currentPlayingTime changed from NULL to 0
22:28:15.663 [INFO ] [smarthome.event.ItemStateChangedEvent] - squeezebox_squeezeboxplayer_f16ede86_000420061c8c_volume changed from NULL to 11
22:28:15.668 [INFO ] [smarthome.event.ItemStateChangedEvent] - squeezebox_squeezeboxplayer_f16ede86_000420061c8c_currentPlaylistRepeat changed from NULL to 0
22:28:15.673 [INFO ] [smarthome.event.ItemStateChangedEvent] - squeezebox_squeezeboxplayer_f16ede86_000420061c8c_currentPlaylistShuffle changed from NULL to 0
22:28:15.678 [INFO ] [smarthome.event.ItemStateChangedEvent] - squeezebox_squeezeboxplayer_f16ede86_000420061c8c_title changed from NULL to BBC Radio 4
22:34:43.137 [INFO ] [smarthome.event.ItemCommandEvent ] - Item 'squeezebox_squeezeboxplayer_f16ede86_000420061c8c_control' received command PLAY
22:34:43.167 [INFO ] [arthome.event.ItemStatePredictedEvent] - squeezebox_squeezeboxplayer_f16ede86_000420061c8c_control predicted to become PLAY
22:34:43.173 [INFO ] [smarthome.event.ItemStateChangedEvent] - squeezebox_squeezeboxplayer_f16ede86_000420061c8c_control changed from PAUSE to PLAY
22:34:43.687 [INFO ] [smarthome.event.ItemStateChangedEvent] - squeezebox_squeezeboxplayer_f16ede86_000420061c8c_duration changed from NULL to 0
22:34:45.716 [INFO ] [smarthome.event.ItemStateChangedEvent] - squeezebox_squeezeboxplayer_f16ede86_000420061c8c_currentPlayingTime changed from 0 to 1
22:34:46.718 [INFO ] [smarthome.event.ItemStateChangedEvent] - squeezebox_squeezeboxplayer_f16ede86_000420061c8c_currentPlayingTime changed from 1 to 2
22:34:46.855 [INFO ] [smarthome.event.ItemCommandEvent ] - Item 'squeezebox_squeezeboxplayer_f16ede86_000420061c8c_control' received command PAUSE
22:34:46.875 [INFO ] [arthome.event.ItemStatePredictedEvent] - squeezebox_squeezeboxplayer_f16ede86_000420061c8c_control predicted to become PAUSE
22:34:46.876 [INFO ] [smarthome.event.ItemStateChangedEvent] - squeezebox_squeezeboxplayer_f16ede86_000420061c8c_control changed from PLAY to PAUSE
22:34:47.194 [INFO ] [smarthome.event.ItemStateChangedEvent] - squeezebox_squeezeboxplayer_f16ede86_000420061c8c_currentPlayingTime changed from 2 to 3
I actually quite like this solution, and might just stick with this.
I can try to slowly increase the number of CPUs and see at which point the jetty error shows up…
It looks exactly like the CPU number is the problem, where it pre-allocates (?) half of the CPU number of threads. If another thread is needed and makes the total go over 10 it fails…
with 16 CPUs:
22:48:14.219 [INFO ] [ome.event.ThingStatusInfoChangedEvent] - 'squeezebox:squeezeboxplayer:f16ede86:000420061c8c' changed from UNINITIALIZED (BRIDGE_UNINITIALIZED) to INITIALIZING
22:48:14.222 [INFO ] [ome.event.ThingStatusInfoChangedEvent] - 'squeezebox:squeezeboxplayer:f16ede86:000420061c8c' changed from INITIALIZING to ONLINE
22:48:26.861 [INFO ] [smarthome.event.ItemCommandEvent ] - Item 'squeezebox_squeezeboxplayer_f16ede86_000420061c8c_control' received command PLAY
22:48:26.868 [INFO ] [arthome.event.ItemStatePredictedEvent] - squeezebox_squeezeboxplayer_f16ede86_000420061c8c_control predicted to become PLAY
22:48:26.882 [INFO ] [smarthome.event.ItemStateChangedEvent] - squeezebox_squeezeboxplayer_f16ede86_000420061c8c_control changed from NULL to PLAY
22:48:27.414 [INFO ] [smarthome.event.ItemStateChangedEvent] - squeezebox_squeezeboxplayer_f16ede86_000420061c8c_duration changed from NULL to 0
22:48:28.141 [INFO ] [smarthome.event.ItemCommandEvent ] - Item 'squeezebox_squeezeboxplayer_f16ede86_000420061c8c_control' received command PAUSE
22:48:28.144 [INFO ] [arthome.event.ItemStatePredictedEvent] - squeezebox_squeezeboxplayer_f16ede86_000420061c8c_control predicted to become PAUSE
22:48:28.145 [INFO ] [smarthome.event.ItemStateChangedEvent] - squeezebox_squeezeboxplayer_f16ede86_000420061c8c_control changed from PLAY to PAUSE
with 18 CPUs:
22:45:56.566 [INFO ] [ome.event.ThingStatusInfoChangedEvent] - 'squeezebox:squeezeboxplayer:f16ede86:000420061c8c' changed from UNINITIALIZED (BRIDGE_UNINITIALIZED) to INITIALIZING
22:45:56.569 [INFO ] [ome.event.ThingStatusInfoChangedEvent] - 'squeezebox:squeezeboxplayer:f16ede86:000420061c8c' changed from INITIALIZING to ONLINE
22:45:56.590 [ERROR] [et.http.internal.WebClientFactoryImpl] - Could not start Jetty http client
java.lang.IllegalStateException: Insufficient configured threads: required=10 < max=10 for QueuedThreadPool[common]@8ed1f47{STARTED,5<=5<=10,i=5,q=0}[ReservedThreadExecutor@582e1ee2{s=0/1,p=0}]
at org.eclipse.jetty.util.thread.ThreadPoolBudget.check(ThreadPoolBudget.java:149) ~[?:?]
with 20 CPUs:
22:42:48.659 [INFO ] [ome.event.ThingStatusInfoChangedEvent] - 'squeezebox:squeezeboxplayer:f16ede86:000420061c8c' changed from UNINITIALIZED (BRIDGE_UNINITIALIZED) to INITIALIZING
22:42:48.665 [INFO ] [ome.event.ThingStatusInfoChangedEvent] - 'squeezebox:squeezeboxplayer:f16ede86:000420061c8c' changed from INITIALIZING to ONLINE
22:42:48.698 [ERROR] [et.http.internal.WebClientFactoryImpl] - Could not start Jetty http client
java.lang.IllegalStateException: Insufficient configured threads: required=11 < max=10 for QueuedThreadPool[common]@2f61ff91{STARTED,5<=5<=10,i=5,q=0}[ReservedThreadExecutor@6fd73573{s=0/1,p=0}]
at org.eclipse.jetty.util.thread.ThreadPoolBudget.check(ThreadPoolBudget.java:149) ~[?:?]
at org.eclipse.jetty.util.thread.ThreadPoolBudget.leaseTo(ThreadPoolBudget.java:130) ~[?:?]
Before opening an issue, it would be good if we can solve it by changing one (or more) of the config parameters bny setting it in runtime.cfg? We tried doing that previously, but we either picked the wrong parameter, or didn’t have the syntax quite right for what we put in runtime.cfg.
Yes, but this error is handled more gracefully because I catch the IllegalArgumentException. So, this error doesn’t kill the listener thread, and everything should continue to work properly. Correct?