Fix for Jetty error when running on host with many cores

Setup, Configuration and Use Runtime
,
1

If you’re running openHAB on a host computer with many cores, you may see an error in your log file that looks like this.

16:25:43.381 [ERROR] [et.http.internal.WebClientFactoryImpl] - Could not start Jetty http client
java.lang.IllegalStateException: Insufficient configured threads: required=13 < max=10 for QueuedThreadPool[common]@75f3f957{STARTED,5<=5<=10,i=5,q=0}[ReservedThreadExecutor@16f08868{s=0/1,p=0}]
        at org.eclipse.jetty.util.thread.ThreadPoolBudget.check(ThreadPoolBudget.java:149) ~[?:?]
        at org.eclipse.jetty.util.thread.ThreadPoolBudget.leaseTo(ThreadPoolBudget.java:130) ~[?:?]
        at org.eclipse.jetty.util.thread.ThreadPoolBudget.leaseFrom(ThreadPoolBudget.java:175) ~[?:?]
        at org.eclipse.jetty.io.SelectorManager.doStart(SelectorManager.java:251) ~[?:?]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[?:?]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138) ~[?:?]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) ~[?:?]
        at org.eclipse.jetty.client.AbstractConnectorHttpClientTransport.doStart(AbstractConnectorHttpClientTransport.java:64) ~[?:?]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[?:?]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138) ~[?:?]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) ~[?:?]
        at org.eclipse.jetty.client.HttpClient.doStart(HttpClient.java:241) ~[?:?]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[?:?]
        at org.eclipse.smarthome.io.net.http.internal.WebClientFactoryImpl$2.run(WebClientFactoryImpl.java:258) ~[?:?]
        at org.eclipse.smarthome.io.net.http.internal.WebClientFactoryImpl$2.run(WebClientFactoryImpl.java:1) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
        at org.eclipse.smarthome.io.net.http.internal.WebClientFactoryImpl.createHttpClientInternal(WebClientFactoryImpl.java:244) ~[?:?]
        at org.eclipse.smarthome.io.net.http.internal.WebClientFactoryImpl.access$7(WebClientFactoryImpl.java:242) ~[?:?]
        at org.eclipse.smarthome.io.net.http.internal.WebClientFactoryImpl$1.run(WebClientFactoryImpl.java:218) ~[?:?]
        at org.eclipse.smarthome.io.net.http.internal.WebClientFactoryImpl$1.run(WebClientFactoryImpl.java:1) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
        at org.eclipse.smarthome.io.net.http.internal.WebClientFactoryImpl.initialize(WebClientFactoryImpl.java:209) ~[?:?]
        at org.eclipse.smarthome.io.net.http.internal.WebClientFactoryImpl.getCommonHttpClient(WebClientFactoryImpl.java:160) ~[?:?]
        at org.eclipse.smarthome.io.net.http.HttpUtil.executeUrlAndGetReponse(HttpUtil.java:194) ~[?:?]
        at org.eclipse.smarthome.io.net.http.HttpUtil.downloadData(HttpUtil.java:438) ~[?:?]
        at org.eclipse.smarthome.io.net.http.HttpUtil.downloadImage(HttpUtil.java:401) ~[?:?]
        at org.eclipse.smarthome.io.net.http.HttpUtil.downloadImage(HttpUtil.java:372) ~[?:?]
        at org.eclipse.smarthome.io.net.http.HttpUtil.downloadImage(HttpUtil.java:358) ~[?:?]
        at org.openhab.binding.squeezebox.internal.handler.SqueezeBoxPlayerHandler.lambda$0(SqueezeBoxPlayerHandler.java:419) ~[?:?]

Jetty calculates the number of threads it needs based on the number of cores on the host computer. If the number of needed threads exceeds the maximum number of configured threads, Jetty (and HttpUtil) will throw this error. This error may prevent bindings from being able to download needed content.

You can resolve this problem by changing the number of threads that are allocated to the ESH webclient. The current system defaults are shown below, so this is the point from which you should start increasing. In particular, maxThreadsShared is the parameter that you’ll most likely need to increase.

These configuration parameters should be placed in services/runtime.cfg. An openHAB restart should not be required for these changes to take effect.

org.eclipse.smarthome.webclient:minThreadsShared=10
org.eclipse.smarthome.webclient:maxThreadsShared=40
org.eclipse.smarthome.webclient:minThreadsCustom=5
org.eclipse.smarthome.webclient:maxThreadsCustom=10

Edit: Updating my post to reflect the correct names for later versions of openHAB post sunset of ESH.

org.openhab.webclient:minThreadsShared=10
org.openhab.webclient:maxThreadsShared=40
org.openhab.webclient:minThreadsCustom=5
org.openhab.webclient:maxThreadsCustom=10
8 Likes
2

Thank you so very much, this solved a problem I had with debugging my openHAB docker images on my development server. Basically the same docker image would run perfectly on my Raspberry Pis, but on the much more powerful server in the basement, no luck, always that error message about “Insufficient configured threads.”…

A itty bitty little addition to my docker image, and the world is fine again :smile:

Time for a PR to add that example config to the openhab-docker project I guess.

2 Likes
3

I’m running on a pine 64. I think I have only 4 cores. yet I have the same error.

Can I apply this too?
If so, what should I set the values? (or better how can I -and others- figure out the correct number of threads to set?

4

Glad this fixed the issue. This was a tough one to track down. :sweat:

I’m considering opening an ESH issue to request a better way to deal with this.

5

I don’t see any harm as long as you have sufficient memory. I read somewhere that it’s about 1 MB per thread.

I’m not sure where to tell you to start. I’m running this without issue on a 6 core box with 8 GB running Ubuntu 18.04.

org.eclipse.smarthome.webclient:minThreadsShared=20
org.eclipse.smarthome.webclient:maxThreadsShared=60
org.eclipse.smarthome.webclient:minThreadsCustom=15
org.eclipse.smarthome.webclient:maxThreadsCustom=25
6

Maybe those values should be part of the runtime.cfg as distributed, so new users have a hint as to where they could look.

Honestly, I am not really sure where I would have found those keys if you wouldn’t have pointed them out in your posting…

7

Yeah, they’re not really documented anywhere (except in WebClientFactoryImpl.java) :open_mouth: And I only found them there while trying to troubleshoot a problem someone was having with HttpUtil.downloadImage.

I was thinking that WebClientFactoryImpl could set the values based on the number of cores in the box. That way it would be memory friendly for Pi’s, yet not fail in mysterious ways for more capable servers.

OTOH, there could be a doc that suggests some runtime.cfg parameters for more capable servers. It would include these parameters, as well as threads for rules, etc. Maybe also java memory settings…

8

My pine64 is 4 cores with 2 GB.

this morning I had

free      
              total        used        free      shared  buff/cache   available
Mem:        2036460     1096836       79692       29568      859932      866116
Swap:       1018224           0     1018224

only openhab is running on this system.
I assume that 79692 means I had about 79 mb free; so 60 seems high.

correct?

9

Yes, seems high. Maybe try increasing just the minimums.

org.eclipse.smarthome.webclient:minThreadsShared=20
org.eclipse.smarthome.webclient:minThreadsCustom=15
10

Issue opened.

@hakan Feel free to add any additional comments.

2 Likes
11

Turns out the minimum was not enough, I got a complained that that webclient:maxThreadsCustom was lower then the minThreadsCustom .
So I added

org.eclipse.smarthome.webclient:minThreadsShared=20
#org.eclipse.smarthome.webclient:maxThreadsShared=40

org.eclipse.smarthome.webclient:minThreadsCustom=15
org.eclipse.smarthome.webclient:maxThreadsCustom=40
(Not sure if 40 is too high, seems to work for now)

2 Likes
12

For future reference: in Openhab 3.0 org.eclipse.smarthome.webclient has changed to org.openhab.webclient. So it becomes:

org.eclipse.smarthome.webclient:minThreadsShared=20
org.eclipse.smarthome.webclient:maxThreadsShared=40

org.eclipse.smarthome.webclient:minThreadsCustom=15
org.eclipse.smarthome.webclient:maxThreadsCustom=40

1 Like
13

I believe you meant

org.openhab.webclient:minThreadsShared = 10
org.openhab.webclient:maxThreadsShared = 40
org.openhab.webclient:minThreadsCustom = 10
org.openhab.webclient:maxThreadsCustom = 20

At least, this solved my thread problem in 3.0.0 (dockerized).

1 Like
14

Grrrrrreat!! Was looking for that and didn’t even see the changed package names until I triple-read your post!

15

Stupid question perhaps. If I have more powerful hardware than usual, could this mean that I’m not using as many threads as I could be using? So that the software limits me, so to speak? It would make a difference whether I have 4GB, 8GB, 16GB, 32GB, 64GB or 128GB RAM.

And then another question. How do I determine how many threads I could/would like to use? Do I have to guess and test? Let’s say I have 4GB RAM and use 40 threads. With 128GB RAM, can I then say I am using 200 threads?

So I’m still not sure what exactly I’m setting, why and how? Does this increase performance at all? The fact that I should no longer get the Jetty error would be another important effect.

16

The intent of my original post was to prevent the Jetty failure only. I never considered whether the change would bring any improvement in performance.