[SOLVED] After upgrade to 2.4 stable I'm flooded with java.lang.IllegalStateException

I updated my system (raspi3b) to the OH2.4 stable and I’m now flooded with the following [WARN] Messages approx every minute.
So far I’m unable to locate where it is coming from.

2018-12-29 13:49:44.917 [WARN ] [org.eclipse.jetty.server.HttpChannel] - /rest/sitemaps/Haus/Haus
javax.servlet.ServletException: javax.servlet.ServletException: A MultiException has 1 exceptions.  They are:
1. java.lang.IllegalStateException: ServiceLocatorImpl(__HK2_Generated_6,7,16588668) has been shut down.......

I did delete all entries on the Sitemap “Haus”, but that didn’t stop the message!

I had something similar when upgrading and cleaning the cache and rebooting helped. Hope this will help you as well.

Thanks, actually I’m trying some regressions (removing bindoings…) The restart option is a good one, Thanks

I experienced the same issue after upgrading from 2.4.0 SNAPSHOT #1445 to 2.5.0 SNAPSHOT #1479. My platform is also an RPi 3B, up-to-date OpenHABian.

I updated to 2.5.0 SNAPSHOT #1480 this morning and on the first restart I saw none of these exceptions. However, I have noted that the system is being hammered from time to time by one or two java threads – CPU utilization > 150%.

[Update: Not so fast, see post #8]

1 Like

Moved my installation to an Raspi2 for the moment, since I could get this message to disappear. It is running on the Raspi2 without this problem, I have no clue!

Update:
Stripped down my Raspi3 to a minimal setup and started to put all back in. The messages came back in the end (last thing that was setup was an EchoDot)
Set the log:level to ERROR for org.eclipse.jetty.server stopped the messages, although I do see now CPU Loads of 20-25%!! That doesn’t sound good!

Edit
Removing the EchoDot didn’t help.

Usually that exception is a side effect of the load.

The increased load could be due to openHAB trying to apply addon changes every minute.

Can you post the contents of the following files?

  • /etc/openhab2/services/addons.cfg
  • /var/lib/openhab2/config/org/openhab/addons.config

It may help to delete /var/lib/openhab2/config/org/openhab/addons.config so it is regenerated with correct content (see [SOLVED] OH 2.3 missing ComfoAir Binding in PaperUI).

I spoke too soon, it looks like the problem is not gone with 2.5.0 SNAPSHOT #1480. This morning my logs are again filled with:

2018-12-30 05:59:51.455 [WARN ][ qtp22805078-79][org.eclipse.jetty.server.HttpChannel] - /rest/sitemaps/demo/demo
javax.servlet.ServletException: javax.servlet.ServletException: A MultiException has 1 exceptions.  They are:
1. java.lang.IllegalStateException: ServiceLocatorImpl(__HK2_Generated_10,11,3787572) has been shut down

	at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:88) ~[?:?]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) ~[84:org.eclipse.jetty.server:9.4.11.v20180605]
	at org.eclipse.jetty.server.Server.handle(Server.java:531) ~[84:org.eclipse.jetty.server:9.4.11.v20180605]
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352) [84:org.eclipse.jetty.server:9.4.11.v20180605]
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) [84:org.eclipse.jetty.server:9.4.11.v20180605]
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281) [75:org.eclipse.jetty.io:9.4.11.v20180605]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) [75:org.eclipse.jetty.io:9.4.11.v20180605]
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) [75:org.eclipse.jetty.io:9.4.11.v20180605]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) [87:org.eclipse.jetty.util:9.4.11.v20180605]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) [87:org.eclipse.jetty.util:9.4.11.v20180605]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) [87:org.eclipse.jetty.util:9.4.11.v20180605]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) [87:org.eclipse.jetty.util:9.4.11.v20180605]
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) [87:org.eclipse.jetty.util:9.4.11.v20180605]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762) [87:org.eclipse.jetty.util:9.4.11.v20180605]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680) [87:org.eclipse.jetty.util:9.4.11.v20180605]
	at java.lang.Thread.run(Thread.java:748) [?:?]
Caused by: javax.servlet.ServletException: A MultiException has 1 exceptions.  They are:
1. java.lang.IllegalStateException: ServiceLocatorImpl(__HK2_Generated_10,11,3787572) has been shut down

	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:489) ~[?:?]
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) ~[?:?]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) ~[?:?]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) ~[?:?]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) ~[?:?]
	at com.eclipsesource.jaxrs.publisher.internal.ServletContainerBridge.service(ServletContainerBridge.java:76) ~[?:?]
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865) ~[?:?]
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:535) ~[?:?]
	at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:71) ~[?:?]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) ~[?:?]
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) ~[?:?]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) ~[?:?]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) ~[?:?]
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) ~[?:?]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) ~[?:?]
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317) ~[?:?]
	at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:293) ~[?:?]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) ~[?:?]
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) ~[?:?]
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) ~[?:?]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) ~[?:?]
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219) ~[?:?]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) ~[?:?]
	at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:80) ~[?:?]
	... 15 more
Caused by: org.glassfish.hk2.api.MultiException: A MultiException has 1 exceptions.  They are:
1. java.lang.IllegalStateException: ServiceLocatorImpl(__HK2_Generated_10,11,3787572) has been shut down

	at org.jvnet.hk2.internal.FactoryCreator.getFactoryHandle(FactoryCreator.java:106) ~[?:?]
	at org.jvnet.hk2.internal.FactoryCreator.dispose(FactoryCreator.java:173) ~[?:?]
	at org.jvnet.hk2.internal.SystemDescriptor.dispose(SystemDescriptor.java:526) ~[?:?]
	at org.glassfish.jersey.process.internal.RequestScope$Instance.remove(RequestScope.java:532) ~[?:?]
	at org.glassfish.jersey.process.internal.RequestScope$Instance.release(RequestScope.java:549) ~[?:?]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:319) ~[?:?]
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) ~[?:?]
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) ~[?:?]
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) ~[?:?]
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) ~[?:?]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) ~[?:?]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) ~[?:?]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) ~[?:?]
	at com.eclipsesource.jaxrs.publisher.internal.ServletContainerBridge.service(ServletContainerBridge.java:76) ~[?:?]
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865) ~[?:?]
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:535) ~[?:?]
	at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:71) ~[?:?]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) ~[?:?]
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) ~[?:?]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) ~[?:?]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) ~[?:?]
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) ~[?:?]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) ~[?:?]
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317) ~[?:?]
	at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:293) ~[?:?]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) ~[?:?]
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) ~[?:?]
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) ~[?:?]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) ~[?:?]
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219) ~[?:?]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) ~[?:?]
	at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:80) ~[?:?]
	... 15 more
Caused by: java.lang.IllegalStateException: ServiceLocatorImpl(__HK2_Generated_10,11,3787572) has been shut down
	at org.jvnet.hk2.internal.ServiceLocatorImpl.checkState(ServiceLocatorImpl.java:2288) ~[?:?]
	at org.jvnet.hk2.internal.ServiceLocatorImpl.getServiceHandleImpl(ServiceLocatorImpl.java:629) ~[?:?]
	at org.jvnet.hk2.internal.ServiceLocatorImpl.getServiceHandle(ServiceLocatorImpl.java:622) ~[?:?]
	at org.jvnet.hk2.internal.ServiceLocatorImpl.getServiceHandle(ServiceLocatorImpl.java:640) ~[?:?]
	at org.jvnet.hk2.internal.FactoryCreator.getFactoryHandle(FactoryCreator.java:103) ~[?:?]
	at org.jvnet.hk2.internal.FactoryCreator.dispose(FactoryCreator.java:173) ~[?:?]
	at org.jvnet.hk2.internal.SystemDescriptor.dispose(SystemDescriptor.java:526) ~[?:?]
	at org.glassfish.jersey.process.internal.RequestScope$Instance.remove(RequestScope.java:532) ~[?:?]
	at org.glassfish.jersey.process.internal.RequestScope$Instance.release(RequestScope.java:549) ~[?:?]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:319) ~[?:?]
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) ~[?:?]
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) ~[?:?]
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) ~[?:?]
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) ~[?:?]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) ~[?:?]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) ~[?:?]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) ~[?:?]
	at com.eclipsesource.jaxrs.publisher.internal.ServletContainerBridge.service(ServletContainerBridge.java:76) ~[?:?]
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865) ~[?:?]
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:535) ~[?:?]
	at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:71) ~[?:?]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) ~[?:?]
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) ~[?:?]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) ~[?:?]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) ~[?:?]
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) ~[?:?]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) ~[?:?]
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317) ~[?:?]
	at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:293) ~[?:?]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) ~[?:?]
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) ~[?:?]
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) ~[?:?]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) ~[?:?]
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219) ~[?:?]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) ~[?:?]
	at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:80) ~[?:?]
	... 15 more

I stopped OH2, then removed /var/lib/openhab2/config/org/openhab/addons.confg, and then I restarted OH2 – no joy, my logs are still filled with the multiexceptions.

Maybe you can check what threads generate the load? See Openhab2 java cpu 100%.

1 Like

The CPU hog seems to be:

39   │ features-3-thread-1                                                                           │ WAITING       │ 313802   │ 309070

Full output of threads --list:

Id   │ Name                                                                                          │ State         │ CPU time │ Usr time
─────┼───────────────────────────────────────────────────────────────────────────────────────────────┼───────────────┼──────────┼─────────
1    │ main                                                                                          │ WAITING       │ 2820     │ 2630
2    │ Reference Handler                                                                             │ WAITING       │ 345      │ 330
3    │ Finalizer                                                                                     │ WAITING       │ 320      │ 320
4    │ Signal Dispatcher                                                                             │ RUNNABLE      │ 0        │ 0
10   │ Active Thread: Equinox Container: db1ef208-a1c0-48c2-a8d8-822d3c032718                        │ TIMED_WAITING │ 26369    │ 24970
12   │ Framework Event Dispatcher: org.eclipse.osgi.internal.framework.EquinoxEventPublisher@1deb50e │ WAITING       │ 1397     │ 1140
13   │ Start Level: Equinox Container: db1ef208-a1c0-48c2-a8d8-822d3c032718                          │ WAITING       │ 3293     │ 3170
14   │ Karaf Lock Monitor Thread                                                                     │ TIMED_WAITING │ 451      │ 330
15   │ Karaf Shutdown Socket Thread                                                                  │ RUNNABLE      │ 1        │ 0
27   │ Coordination Timer                                                                            │ WAITING       │ 0        │ 0
28   │ CM Configuration Updater                                                                      │ WAITING       │ 3615     │ 3520
29   │ CM Event Dispatcher                                                                           │ WAITING       │ 135      │ 130
30   │ fileinstall-/var/lib/openhab2/etc                                                             │ TIMED_WAITING │ 398      │ 340
31   │ Thread-12                                                                                     │ RUNNABLE      │ 4        │ 0
33   │ EventAdminAsyncThread #11                                                                     │ WAITING       │ 3004     │ 2610
35   │ EventAdminAsyncThread #12                                                                     │ WAITING       │ 3013     │ 2620
37   │ fileinstall-/usr/share/openhab2/addons                                                        │ TIMED_WAITING │ 11124    │ 6100
38   │ Thread-16                                                                                     │ RUNNABLE      │ 0        │ 0
39   │ features-3-thread-1                                                                           │ WAITING       │ 313802   │ 309070
50   │ Bundle File Closer                                                                            │ WAITING       │ 405      │ 240
51   │ Refresh Thread: Equinox Container: db1ef208-a1c0-48c2-a8d8-822d3c032718                       │ WAITING       │ 18111    │ 17280
57   │ encryption-1-thread-1                                                                         │ TIMED_WAITING │ 146      │ 110
59   │ Thread-18                                                                                     │ RUNNABLE      │ 0        │ 0
60   │ encryption-3-thread-1                                                                         │ TIMED_WAITING │ 127      │ 110
61   │ Thread-19                                                                                     │ RUNNABLE      │ 0        │ 0
62   │ SCR Component Actor                                                                           │ WAITING       │ 0        │ 0
64   │ EMF Reference Cleaner                                                                         │ WAITING       │ 1504     │ 870
65   │ pool-1-thread-1                                                                               │ WAITING       │ 28       │ 20
66   │ pool-2-thread-1                                                                               │ WAITING       │ 3131     │ 2960
67   │ pool-3-thread-1                                                                               │ WAITING       │ 6263     │ 6170
79   │ ServletEventDispatcher: 1                                                                     │ WAITING       │ 9        │ 0
80   │ ServletEventDispatcher: 2                                                                     │ WAITING       │ 51       │ 50
81   │ qtp30497826-81                                                                                │ RUNNABLE      │ 10       │ 10
82   │ qtp30497826-82                                                                                │ RUNNABLE      │ 1        │ 0
83   │ ESH-OSGiEventManager                                                                          │ TIMED_WAITING │ 9480     │ 8990
84   │ Scheduler-6717365                                                                             │ TIMED_WAITING │ 3        │ 0
85   │ qtp30497826-85-acceptor-0@10e82da-0.0.0.0:8443@2685c7{SSL,[ssl, http/1.1]}{0.0.0.0:8443}      │ RUNNABLE      │ 3        │ 0
86   │ qtp30497826-86                                                                                │ TIMED_WAITING │ 2092     │ 2080
88   │ qtp30497826-88-acceptor-0@1e5be06-default@17cf09c{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}          │ RUNNABLE      │ 66       │ 60
89   │ qtp30497826-89                                                                                │ RUNNABLE      │ 1945     │ 1910
90   │ ServletEventDispatcher: 3                                                                     │ WAITING       │ 17       │ 10
96   │ com.google.inject.internal.util.$Finalizer                                                    │ WAITING       │ 0        │ 0
98   │ Scheduler-25539645                                                                            │ TIMED_WAITING │ 168      │ 150
101  │ Thread-34                                                                                     │ RUNNABLE      │ 3        │ 0
102  │ Dir Watcher                                                                                   │ WAITING       │ 0        │ 0
104  │ qtp30497826-104                                                                               │ TIMED_WAITING │ 2033     │ 2010
106  │ openHAB-job-scheduler_Worker-1                                                                │ TIMED_WAITING │ 4623     │ 3880
107  │ openHAB-job-scheduler_Worker-2                                                                │ TIMED_WAITING │ 15959    │ 15030
108  │ openHAB-job-scheduler_QuartzSchedulerThread                                                   │ TIMED_WAITING │ 625      │ 620
114  │ ServletModel-11-114                                                                           │ RUNNABLE      │ 0        │ 0
115  │ ServletModel-11-115                                                                           │ TIMED_WAITING │ 1        │ 0
116  │ ServletModel-11-116                                                                           │ RUNNABLE      │ 0        │ 0
117  │ ServletModel-11-117                                                                           │ TIMED_WAITING │ 1        │ 0
118  │ ServletModel-11-118                                                                           │ TIMED_WAITING │ 1        │ 0
119  │ ServletModel-11-119                                                                           │ TIMED_WAITING │ 1        │ 0
120  │ ServletModel-11-120                                                                           │ TIMED_WAITING │ 2        │ 0
121  │ ServletModel-11-121                                                                           │ TIMED_WAITING │ 1        │ 0
139  │ Timer-5                                                                                       │ WAITING       │ 59       │ 30
140  │ Timer-6                                                                                       │ WAITING       │ 0        │ 0
141  │ pool-8-thread-1                                                                               │ TIMED_WAITING │ 1073     │ 860
142  │ safeCall-2                                                                                    │ TIMED_WAITING │ 2332     │ 2120
145  │ Timer-7                                                                                       │ WAITING       │ 0        │ 0
146  │ Timer-8                                                                                       │ WAITING       │ 336      │ 220
148  │ Timer-9                                                                                       │ WAITING       │ 0        │ 0
149  │ Timer-10                                                                                      │ WAITING       │ 5024     │ 4300
160  │ lsp-1                                                                                         │ RUNNABLE      │ 4        │ 0
183  │ qtp30497826-183                                                                               │ TIMED_WAITING │ 650      │ 620
193  │ Thread-62                                                                                     │ RUNNABLE      │ 36       │ 20
194  │ sshd-SshServer[1e61819]-timer-thread-1                                                        │ TIMED_WAITING │ 200      │ 200
196  │ ESH-httpClient-common-196                                                                     │ RUNNABLE      │ 0        │ 0
197  │ ESH-httpClient-common-197                                                                     │ TIMED_WAITING │ 20       │ 20
198  │ ESH-httpClient-common-198                                                                     │ RUNNABLE      │ 0        │ 0
199  │ ESH-httpClient-common-199                                                                     │ RUNNABLE      │ 130      │ 120
200  │ ESH-httpClient-common-200                                                                     │ TIMED_WAITING │ 36       │ 20
201  │ ESH-httpClient-common-201                                                                     │ TIMED_WAITING │ 175      │ 160
202  │ ESH-httpClient-common-202                                                                     │ TIMED_WAITING │ 111      │ 110
203  │ ESH-httpClient-common-203                                                                     │ TIMED_WAITING │ 101      │ 80
204  │ ESH-httpClient-common-204                                                                     │ TIMED_WAITING │ 29       │ 20
205  │ ESH-httpClient-common-205                                                                     │ RUNNABLE      │ 6        │ 0
206  │ HttpClient@306a00-scheduler                                                                   │ WAITING       │ 5        │ 0
207  │ HttpClient@1b64b26-207                                                                        │ TIMED_WAITING │ 356      │ 320
208  │ HttpClient@1b64b26-208                                                                        │ TIMED_WAITING │ 263      │ 220
209  │ HttpClient@1b64b26-209                                                                        │ TIMED_WAITING │ 148      │ 120
210  │ HttpClient@1b64b26-210                                                                        │ TIMED_WAITING │ 278      │ 270
211  │ HttpClient@1b64b26-211                                                                        │ TIMED_WAITING │ 170      │ 150
212  │ HttpClient@1b64b26-212                                                                        │ RUNNABLE      │ 388      │ 370
213  │ HttpClient@1b64b26-213                                                                        │ RUNNABLE      │ 879      │ 840
214  │ HttpClient@1b64b26-214                                                                        │ TIMED_WAITING │ 129      │ 120
215  │ HttpClient@1b64b26-scheduler                                                                  │ TIMED_WAITING │ 34       │ 30
216  │ pool-19-thread-1                                                                              │ TIMED_WAITING │ 26387    │ 25460
219  │ pool-21-thread-1                                                                              │ TIMED_WAITING │ 64       │ 60
220  │ ESH-thingHandler-1                                                                            │ TIMED_WAITING │ 7577     │ 7210
225  │ ESH-thingHandler-2                                                                            │ TIMED_WAITING │ 7066     │ 6590
229  │ items-1                                                                                       │ TIMED_WAITING │ 926      │ 830
230  │ items-2                                                                                       │ TIMED_WAITING │ 827      │ 760
231  │ items-3                                                                                       │ TIMED_WAITING │ 1043     │ 930
232  │ items-4                                                                                       │ TIMED_WAITING │ 1074     │ 990
233  │ Timer-14                                                                                      │ WAITING       │ 1948     │ 1630
235  │ ESH-thingHandler-3                                                                            │ TIMED_WAITING │ 3759     │ 3460
239  │ SocketListener(JmDNS-/192.168.1.205)                                                          │ RUNNABLE      │ 1818     │ 1750
240  │ JmDNS(JmDNS-/192.168.1.205).Timer                                                             │ TIMED_WAITING │ 447      │ 410
241  │ JmDNS(JmDNS-/192.168.1.205).State.Timer                                                       │ TIMED_WAITING │ 18       │ 10
245  │ pool-27-thread-1                                                                              │ WAITING       │ 461      │ 410
247  │ Timer-15                                                                                      │ WAITING       │ 25       │ 20
249  │ ESH-thingHandler-4                                                                            │ WAITING       │ 1412     │ 1320
250  │ ESH-thingHandler-5                                                                            │ TIMED_WAITING │ 6455     │ 5980
255  │ HttpClient@36839a-scheduler                                                                   │ TIMED_WAITING │ 1        │ 0
256  │ JmDNS pool-23-thread-1                                                                        │ WAITING       │ 142      │ 140
260  │ WebSocketClient@8097821-260                                                                   │ RUNNABLE      │ 0        │ 0
261  │ WebSocketClient@8097821-261                                                                   │ TIMED_WAITING │ 11       │ 10
262  │ WebSocketClient@8097821-262                                                                   │ RUNNABLE      │ 48       │ 40
263  │ WebSocketClient@8097821-263                                                                   │ TIMED_WAITING │ 12       │ 10
264  │ WebSocketClient@8097821-264                                                                   │ TIMED_WAITING │ 58       │ 50
265  │ WebSocketClient@8097821-265                                                                   │ TIMED_WAITING │ 72       │ 60
266  │ WebSocketClient@8097821-266                                                                   │ TIMED_WAITING │ 18       │ 10
267  │ WebSocketClient@8097821-267                                                                   │ TIMED_WAITING │ 14       │ 10
269  │ Thread-94                                                                                     │ RUNNABLE      │ 2160     │ 1040
270  │ ZToolPacketParser                                                                             │ TIMED_WAITING │ 10640    │ 9930
271  │ HttpClient@1004733-scheduler                                                                  │ TIMED_WAITING │ 1        │ 0
272  │ Timer-17                                                                                      │ TIMED_WAITING │ 41       │ 40
274  │ pool-22-thread-1                                                                              │ WAITING       │ 5686     │ 5280
275  │ pool-22-thread-2                                                                              │ TIMED_WAITING │ 2410     │ 2270
293  │ pool-22-thread-3                                                                              │ TIMED_WAITING │ 4731     │ 4360
294  │ pool-22-thread-4                                                                              │ TIMED_WAITING │ 5753     │ 5400
295  │ pool-22-thread-5                                                                              │ WAITING       │ 3400     │ 3290
303  │ pool-22-thread-6                                                                              │ WAITING       │ 4323     │ 4030
307  │ items-5                                                                                       │ TIMED_WAITING │ 1111     │ 950
355  │ Timer-26                                                                                      │ WAITING       │ 0        │ 0
382  │ Timer-27                                                                                      │ WAITING       │ 0        │ 0
383  │ Timer-28                                                                                      │ WAITING       │ 0        │ 0
384  │ Timer-29                                                                                      │ WAITING       │ 0        │ 0
387  │ Timer-30                                                                                      │ WAITING       │ 0        │ 0
388  │ Timer-31                                                                                      │ WAITING       │ 0        │ 0
390  │ Timer-32                                                                                      │ WAITING       │ 0        │ 0
392  │ Thread-119                                                                                    │ TIMED_WAITING │ 4        │ 0
393  │ ZWaveReceiveInputThread                                                                       │ RUNNABLE      │ 1220     │ 1000
394  │ Timer-33                                                                                      │ WAITING       │ 3        │ 0
395  │ ZWaveReceiveProcessorThread                                                                   │ WAITING       │ 516      │ 500
397  │ Thread-122                                                                                    │ TIMED_WAITING │ 1458     │ 1390
398  │ Timer-35                                                                                      │ WAITING       │ 1        │ 0
399  │ DTLS-Sender-0.0.0.0/0.0.0.0:55192                                                             │ WAITING       │ 386      │ 340
400  │ DTLS-Receiver-0.0.0.0/0.0.0.0:55192                                                           │ RUNNABLE      │ 448      │ 420
402  │ ResourcePublisher                                                                             │ WAITING       │ 85797    │ 85490
407  │ sender-ftcazgsqne PING                                                                        │ TIMED_WAITING │ 49       │ 40
408  │ Thread-128                                                                                    │ RUNNABLE      │ 516      │ 490
409  │ sender-ysn4ogdaza PING                                                                        │ TIMED_WAITING │ 51       │ 40
410  │ Thread-130                                                                                    │ RUNNABLE      │ 446      │ 430
412  │ sender-ewjjxo9juh PING                                                                        │ TIMED_WAITING │ 56       │ 40
413  │ Thread-132                                                                                    │ RUNNABLE      │ 341      │ 330
417  │ sender-d48nm6tjt9 PING                                                                        │ TIMED_WAITING │ 50       │ 30
418  │ Thread-137                                                                                    │ RUNNABLE      │ 472      │ 450
429  │ sender-20t811ws7b PING                                                                        │ TIMED_WAITING │ 43       │ 40
430  │ Thread-149                                                                                    │ RUNNABLE      │ 485      │ 460
433  │ sender-vn6lj59u5u PING                                                                        │ TIMED_WAITING │ 58       │ 40
434  │ Thread-153                                                                                    │ RUNNABLE      │ 349      │ 340
436  │ ZWaveNode3Init2018-12-30 06:39:23.277                                                         │ TIMED_WAITING │ 11       │ 10
449  │ MQTT Rec: paho350204602329668                                                                 │ RUNNABLE      │ 52       │ 50
450  │ MQTT Snd: paho350204602329668                                                                 │ WAITING       │ 14       │ 10
451  │ MQTT Call: paho350204602329668                                                                │ WAITING       │ 293      │ 280
452  │ MQTT Ping: paho350204602329668                                                                │ TIMED_WAITING │ 7        │ 0
453  │ ESH-FirmwareUpdateServiceImpl-5                                                               │ TIMED_WAITING │ 5        │ 0
482  │ Google Calendar Event Downloader                                                              │ TIMED_WAITING │ 812      │ 750
483  │ RRD4J Sync-ThreadPool for org.rrd4j.core.RrdSyncThreadPool@3707a1 [Thread-1]                  │ WAITING       │ 10       │ 10
484  │ RRD4J Sync-ThreadPool for org.rrd4j.core.RrdSyncThreadPool@3707a1 [Thread-2]                  │ WAITING       │ 1        │ 0
485  │ RRD4J Sync-ThreadPool for org.rrd4j.core.RrdSyncThreadPool@3707a1 [Thread-3]                  │ WAITING       │ 0        │ 0
486  │ RRD4J Sync-ThreadPool for org.rrd4j.core.RrdSyncThreadPool@3707a1 [Thread-4]                  │ WAITING       │ 1        │ 0
488  │ ESH-persist-Monitor-1                                                                         │ TIMED_WAITING │ 3366     │ 3330
494  │ Timer-38                                                                                      │ WAITING       │ 24       │ 10
496  │ Expire Refresh Service                                                                        │ TIMED_WAITING │ 95       │ 80
498  │ Timer-40                                                                                      │ WAITING       │ 0        │ 0
499  │ Timer-41                                                                                      │ WAITING       │ 0        │ 0
500  │ RRD4J Sync-ThreadPool for org.rrd4j.core.RrdSyncThreadPool@3707a1 [Thread-5]                  │ TIMED_WAITING │ 7        │ 0
501  │ RRD4J Sync-ThreadPool for org.rrd4j.core.RrdSyncThreadPool@3707a1 [Thread-6]                  │ WAITING       │ 1        │ 0
503  │ upnp-main-1                                                                                   │ TIMED_WAITING │ 314      │ 280
504  │ upnp-main-2                                                                                   │ RUNNABLE      │ 1282     │ 1210
506  │ upnp-main-4                                                                                   │ RUNNABLE      │ 55       │ 50
508  │ jupnp-jetty-client-508                                                                        │ RUNNABLE      │ 25       │ 20
515  │ HttpClient@da6bc5-scheduler                                                                   │ WAITING       │ 1        │ 0
516  │ jupnp-jetty-client-516                                                                        │ TIMED_WAITING │ 34       │ 20
517  │ Timer-42                                                                                      │ WAITING       │ 5        │ 0
520  │ jupnp-jetty-client-520                                                                        │ RUNNABLE      │ 17       │ 10
521  │ Automation Provider Processing Queue                                                          │ TIMED_WAITING │ 101      │ 100
522  │ Timer-43                                                                                      │ WAITING       │ 0        │ 0
523  │ Timer-44                                                                                      │ WAITING       │ 57       │ 30
524  │ Timer-45                                                                                      │ WAITING       │ 0        │ 0
525  │ Thread-203                                                                                    │ RUNNABLE      │ 307      │ 240
529  │ Automation Provider Processing Queue                                                          │ TIMED_WAITING │ 165      │ 150
530  │ Automation Provider Processing Queue                                                          │ TIMED_WAITING │ 79       │ 60
537  │ Thread-205                                                                                    │ TIMED_WAITING │ 0        │ 0
540  │ Thread-206                                                                                    │ RUNNABLE      │ 142      │ 130
546  │ ESH-usb-serial-discovery-linux-sysfs-1                                                        │ TIMED_WAITING │ 11503    │ 8030
559  │ weakref reaper                                                                                │ WAITING       │ 0        │ 0
597  │ pool-39-thread-1                                                                              │ WAITING       │ 2766     │ 2710
655  │ ESH-discovery-10                                                                              │ TIMED_WAITING │ 141      │ 140
665  │ pool-43-thread-1                                                                              │ WAITING       │ 51       │ 50
667  │ pool-42-thread-1                                                                              │ WAITING       │ 34       │ 30
677  │ ESH-discovery-11                                                                              │ TIMED_WAITING │ 208      │ 200
681  │ pool-41-thread-1                                                                              │ WAITING       │ 19       │ 10
758  │ qtp30497826-758                                                                               │ TIMED_WAITING │ 53       │ 50
759  │ qtp30497826-759                                                                               │ RUNNABLE      │ 1949     │ 1890
763  │ safeCall-15                                                                                   │ TIMED_WAITING │ 1288     │ 1150
966  │ safeCall-20                                                                                   │ TIMED_WAITING │ 735      │ 660
987  │ ESH-file-processing-10                                                                        │ TIMED_WAITING │ 60       │ 60
988  │ ESH-file-processing-11                                                                        │ TIMED_WAITING │ 40       │ 30
989  │ ESH-file-processing-12                                                                        │ TIMED_WAITING │ 44       │ 40
990  │ ESH-file-processing-13                                                                        │ TIMED_WAITING │ 60       │ 60
1014 │ safeCall-21                                                                                   │ TIMED_WAITING │ 583      │ 490
1196 │ ESH-ruleEngine-17                                                                             │ TIMED_WAITING │ 333      │ 320
1261 │ ESH-common-26                                                                                 │ TIMED_WAITING │ 8        │ 0
1269 │ pool-26-thread-229                                                                            │ TIMED_WAITING │ 39       │ 30
1282 │ pool-26-thread-242                                                                            │ TIMED_WAITING │ 139      │ 130
1303 │ safeCall-22                                                                                   │ TIMED_WAITING │ 520      │ 470
1345 │ ESH-file-processing-14                                                                        │ TIMED_WAITING │ 23       │ 20
1361 │ pool-26-thread-261                                                                            │ TIMED_WAITING │ 42       │ 40
1377 │ ESH-ruleEngine-22                                                                             │ TIMED_WAITING │ 173      │ 160
1425 │ jupnp-jetty-client-1425                                                                       │ TIMED_WAITING │ 5        │ 0
1426 │ jupnp-jetty-client-1426                                                                       │ TIMED_WAITING │ 0        │ 0
1448 │ ESH-ruleEngine-24                                                                             │ TIMED_WAITING │ 176      │ 170
1452 │ safeCall-23                                                                                   │ TIMED_WAITING │ 155      │ 140
1462 │ qtp30497826-1462                                                                              │ TIMED_WAITING │ 0        │ 0
1464 │ sshd-SshServer[1e61819]-nio2-thread-1                                                         │ WAITING       │ 1618     │ 1610
1465 │ Thread-228                                                                                    │ TIMED_WAITING │ 4        │ 0
1466 │ sshd-SshServer[1e61819]-nio2-thread-2                                                         │ WAITING       │ 1344     │ 1310
1467 │ org.apache.karaf.shell.ssh.SshTerminal@badcd6 input pump thread                               │ WAITING       │ 14       │ 10
1469 │ Karaf ssh console user openhab                                                                │ WAITING       │ 1700     │ 1630
1479 │ ESH-ruleEngine-25                                                                             │ TIMED_WAITING │ 46       │ 40
1486 │ pool-26-thread-266                                                                            │ TIMED_WAITING │ 92       │ 70
1497 │ ESH-common-32                                                                                 │ TIMED_WAITING │ 1        │ 0
1508 │ pool-26-thread-267                                                                            │ TIMED_WAITING │ 26       │ 20
1509 │ pool-26-thread-268                                                                            │ TIMED_WAITING │ 13       │ 10
1510 │ pool-26-thread-269                                                                            │ TIMED_WAITING │ 1        │ 0
1513 │ pool-26-thread-272                                                                            │ TIMED_WAITING │ 83       │ 80
1514 │ pool-26-thread-273                                                                            │ TIMED_WAITING │ 10       │ 10
1521 │ pool-26-thread-280                                                                            │ TIMED_WAITING │ 21       │ 10
1522 │ pool-26-thread-281                                                                            │ TIMED_WAITING │ 7        │ 0
1529 │ ESH-persist-24                                                                                │ TIMED_WAITING │ 15       │ 10
1534 │ ESH-discovery-47                                                                              │ TIMED_WAITING │ 0        │ 0
1535 │ ESH-common-33                                                                                 │ TIMED_WAITING │ 0        │ 0
1552 │ ESH-common-34                                                                                 │ TIMED_WAITING │ 0        │ 0
1553 │ ESH-discovery-48                                                                              │ TIMED_WAITING │ 0        │ 0
1554 │ ESH-ruleEngine-26                                                                             │ TIMED_WAITING │ 0        │ 0
1555 │ safeCall-24                                                                                   │ TIMED_WAITING │ 0        │ 0
1561 │ ESH-persist-26                                                                                │ TIMED_WAITING │ 0        │ 0
1562 │ ForkJoinPool.commonPool-worker-1                                                              │ TIMED_WAITING │ 0        │ 0
1563 │ upnp-async-99                                                                                 │ TIMED_WAITING │ 6        │ 0
1564 │ upnp-async-100                                                                                │ TIMED_WAITING │ 7        │ 0
1566 │ Keep-Alive-Timer                                                                              │ TIMED_WAITING │ 0        │ 0
1567 │ job controller 1                                                                              │ WAITING       │ 0        │ 0
1568 │ pipe-threads --list                                                                           │ RUNNABLE      │ 70       │ 70

By the way, thank you for that hint, I have been trying to find a way to relate the PID I get from top with a java thread. This doesn’t do that, but it does provide the information I was after.

[Updated to include output of threads 39]

openhab> threads 39 >/tmp/threads-39-output.log
Thread 39 features-3-thread-1 WAITING
Stacktrace:
sun.misc.Unsafe.park line: -2
java.util.concurrent.locks.LockSupport.park line: 175
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await line: 2039
java.util.concurrent.LinkedBlockingQueue.take line: 442
java.util.concurrent.ThreadPoolExecutor.getTask line: 1074
java.util.concurrent.ThreadPoolExecutor.runWorker line: 1134
java.util.concurrent.ThreadPoolExecutor$Worker.run line: 624
java.lang.Thread.run line: 748

[Further updated with filtered output from threads]

"features-3-thread-1" Id=39 in RUNNABLE
    at java.lang.Object.hashCode(Native Method)
    at java.util.HashMap.hash(HashMap.java:339)
    at java.util.HashMap.containsKey(HashMap.java:596)
    at java.util.HashSet.contains(HashSet.java:204)
    at java.util.ArrayList.batchRemove(ArrayList.java:724)
    at java.util.ArrayList.removeAll(ArrayList.java:694)
    at org.apache.karaf.features.internal.region.AbstractRegionDigraphVisitor.preEdgeTraverse(AbstractRegionDigraphVisitor.java:74)
    at org.eclipse.equinox.internal.region.SubgraphTraverser.traverseEdges(SubgraphTraverser.java:45)
    at org.eclipse.equinox.internal.region.SubgraphTraverser.visitRemainingSubgraph(SubgraphTraverser.java:38)
    at org.eclipse.equinox.internal.region.SubgraphTraverser.traverseEdges(SubgraphTraverser.java:47)
    at org.eclipse.equinox.internal.region.SubgraphTraverser.visitRemainingSubgraph(SubgraphTraverser.java:38)
    at org.eclipse.equinox.internal.region.SubgraphTraverser.visitSubgraph(SubgraphTraverser.java:32)
    at org.eclipse.equinox.internal.region.StandardRegionDigraph.visitSubgraph(StandardRegionDigraph.java:374)
    at org.eclipse.equinox.internal.region.BundleIdBasedRegion.visitSubgraph(BundleIdBasedRegion.java:264)
    at org.apache.karaf.features.internal.region.SubsystemResolveContext.findProviders(SubsystemResolveContext.java:204)
    at org.apache.felix.resolver.Candidates.populate(Candidates.java:210)
    at org.apache.felix.resolver.ResolverImpl.getInitialCandidates(ResolverImpl.java:501)
    at org.apache.felix.resolver.ResolverImpl.doResolve(ResolverImpl.java:390)
    at org.apache.felix.resolver.ResolverImpl.resolve(ResolverImpl.java:378)
    at org.apache.felix.resolver.ResolverImpl.resolve(ResolverImpl.java:332)
    at org.apache.karaf.features.internal.region.SubsystemResolver.resolve(SubsystemResolver.java:257)
    at org.apache.karaf.features.internal.service.Deployer.deploy(Deployer.java:388)
    at org.apache.karaf.features.internal.service.FeaturesServiceImpl.doProvision(FeaturesServiceImpl.java:1025)
    at org.apache.karaf.features.internal.service.FeaturesServiceImpl.lambda$doProvisionInThread$13(FeaturesServiceImpl.java:964)
    at org.apache.karaf.features.internal.service.FeaturesServiceImpl$$Lambda$75/1891285.call(Unknown Source)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

Yes almost half of your load seems to be caused by feature (addon/ui/etc) installation:

Id Name CPU time Total CPU time %
39 features-3-thread-1 313802 46,28%
402 ResourcePublisher 85797 12,65%
216 pool-19-thread-1 26387 3,89%
10 Active Thread: Equinox Container: db1ef208-a1c0-48c2-a8d8-822d3c032718 26369 3,89%
51 Refresh Thread: Equinox Container: db1ef208-a1c0-48c2-a8d8-822d3c032718 18111 2,67%
107 openHAB-job-scheduler_Worker-2 15959 2,35%
546 ESH-usb-serial-discovery-linux-sysfs-1 11503 1,70%
37 fileinstall-/usr/share/openhab2/addons 11124 1,64%
270 ZToolPacketParser 10640 1,57%
83 ESH-OSGiEventManager 9480 1,40%

It would be helpful if you can paste the contents of:

  • /etc/openhab2/services/addons.cfg
  • /var/lib/openhab2/config/org/openhab/addons.config

Maybe they contain invalid contents causing the feature installation logic to continuously try apply updating features causing all the load.

What might also help is raising the log level of the openHAB feature installation logic with:

log:set debug org.openhab.core.karaf

It will print what it is trying to accomplish. Normally you’d only see every minute something like:

15:33:11.125 [DEBUG] [.core.karaf.internal.FeatureInstaller] - Running scheduled sync job
1 Like

Thank you for your quick response, @wborn.

Contents of /etc/openhab2/services/addons.cfg:

# The installation package of this openHAB instance
# Note: This is only regarded at the VERY FIRST START of openHAB
# Note: If you want to specify your add-ons yourself through entries below, set the package to "minimal"
# as otherwise your definition might be in conflict with what the installation package defines.
#
# Optional. If not set, the dashboard (https://<yourserver>:8080/) will ask you to choose a package.
#
# Valid options:
#   - minimal  : Installation only with dashboard, but no UIs or other add-ons. Use this for custom setups.
#   - simple   : Setup for using openHAB purely through UIs - you need to expect MANY constraints in functionality!
#   - standard : Default setup for normal users, best for textual setup
#   - expert   : Setup for expert users, especially for people migrating from openHAB 1.x
#   - demo     : A demo setup which includes UIs, a few bindings, config files etc.
#
# See https://www.openhab.org/docs/configuration/packages.html for a detailed explanation of these packages.
#
package = minimal

# Access Remote Add-on Repository
# Defines whether the remote openHAB add-on repository should be used for browsing and installing add-ons.
# This not only makes latest snapshots of add-ons available, it is also required for the installation of
# any legacy 1.x add-on. (default is true)
#
remote = true

# Include legacy 1.x bindings. If set to true, it also allows the installation of 1.x bindings for which there is 
# already a 2.x version available (requires remote repo access, see above). (default is false)
#
legacy = true

# A comma-separated list of bindings to install (e.g. "binding = sonos,knx,zwave")
binding = amazonechocontrol,astro1,chromecast,expire1,googletts,hue,kodi,lutron,mqtt1,ntp,tradfri,unifi,weather1,yamahareceiver,zigbee,zwave

# A comma-separated list of UIs to install (e.g. "ui = basic,paper")
ui = basic,classic,paper,habpanel,habmin

# A comma-separated list of persistence services to install (e.g. "persistence = rrd4j,jpa")
persistence = mapdb,rrd4j,gcal

# A comma-separated list of actions to install (e.g. "action = mail,pushover")
#action = 

# A comma-separated list of transformation services to install (e.g. "transformation = map,jsonpath")
transformation = jsonpath,map,regex

# A comma-separated list of voice services to install (e.g. "voice = marytts,freetts")
voice = googletts

# A comma-separated list of miscellaneous services to install (e.g. "misc = myopenhab")
misc = gcal1,restdocs,hueemulation,ruleengine

# Surprise, the new authentication DOES NOT WORK!
org.eclipse.smarthome.auth:authentication.enabled="false"

Contents of /var/lib/openhab2/config/org/openhab/addons.config:

binding="amazonechocontrol,astro1,chromecast,expire1,googletts,hue,kodi,lutron,mqtt1,ntp,tradfri,unifi,weather1,yamahareceiver,zigbee,zwave"
legacy="true"
misc="gcal1,restdocs,hueemulation,ruleengine"
package="minimal"
persistence="mapdb,rrd4j,gcal"
remote="true"
service.pid="org.openhab.addons"
transformation="jsonpath,map,regex"
ui="basic,classic,paper,habpanel,habmin"
voice="googletts"

I have set the DEBUG log level for org.openhab.core.karaf and will report here with any further details gleaned from the log.

From the logs, it seems there may be a problem with the googletts binding:

[07:40:21] root@rpi3:~# fgrep 'FeatureInstaller' /var/log/openhab2/openhab.log
2018-12-30 07:40:35.514 [DEBUG][ol-231-thread-1][core.karaf.internal.FeatureInstaller] - Installed 'openhab-binding-googletts'
2018-12-30 07:40:49.384 [DEBUG][ool-21-thread-1][core.karaf.internal.FeatureInstaller] - Running scheduled sync job
2018-12-30 07:41:31.435 [DEBUG][ol-235-thread-1][core.karaf.internal.FeatureInstaller] - Installed 'openhab-binding-googletts'
2018-12-30 07:41:49.388 [DEBUG][ool-21-thread-1][core.karaf.internal.FeatureInstaller] - Running scheduled sync job
2018-12-30 07:42:28.502 [DEBUG][ol-238-thread-1][core.karaf.internal.FeatureInstaller] - Installed 'openhab-binding-googletts'
2018-12-30 07:42:49.393 [DEBUG][ool-21-thread-1][core.karaf.internal.FeatureInstaller] - Running scheduled sync job
2018-12-30 07:43:42.475 [DEBUG][ol-241-thread-1][core.karaf.internal.FeatureInstaller] - Installed 'openhab-binding-googletts'
2018-12-30 07:43:50.040 [DEBUG][ool-21-thread-1][core.karaf.internal.FeatureInstaller] - Running scheduled sync job
2018-12-30 07:44:44.772 [DEBUG][ol-243-thread-1][core.karaf.internal.FeatureInstaller] - Installed 'openhab-binding-googletts'
2018-12-30 07:44:52.484 [DEBUG][ool-21-thread-1][core.karaf.internal.FeatureInstaller] - Running scheduled sync job

googletts is not a binding so it might help reduce the load by removing it from that list. :wink:

1 Like

Odd, I thought it was a binding:

penhab> bundle:list | grep -i google
 21 │ Active   │  80 │ 2.7.0.v20170129-0911   │ Gson: Google Json Library for Java
 22 │ Active   │  80 │ 18.0.0                 │ Guava: Google Core Libraries for Java
 23 │ Active   │  80 │ 3.0.0.v201312141243    │ Google Guice (No AOP)
268 │ Active   │  80 │ 1.14.0.201812300308    │ openHAB Google Calendar
280 │ Active   │  80 │ 2.5.0.201812301123     │ Google Cloud Text-to-Speech

That command shows the bundles. It’s a voice addon bundle. It’s in both the binding and voice addons list in your addons.cfg. Removing it from the bindings list will probably fix the load issues.

Gotcha! I completely missed that, :sleeping:, thanks for your persistence.

1 Like

Note that having an invalid binding in addons.cfg also may cause you to run out of memory eventually…

Edit: Note that I haven’t retested this with the latest Karaf upgrade.

I’ve created a PR so it will log on error whenever features fail to install:

Hopefully that makes it easier to troubleshoot these issues. :smiley: