Can anyone point me towards the docs, or let me know what “qtp” tasks are, and how to identify what they correspond to in OH?
I get the occasional seize up of my server when OH5.0.1 (Linux-x64, dual core, plenty of memory) from the OH process maxing out. In order I’m finding the main consumers of CPU are: OH-RR4J-1, OH-OSCiEventManager, OH-eventexecutor-13-1, and then the numbers drop off a lot. What I am seeing though is a bunch of qtpNNNNNNNN-NNNNN tasks spiking all over the shop, which I feel warrants further scrutiny.
I never did figure it out, it’s been less pervasive recently (or I’ve just not been paying enough attention), I’ll try the thread dump next time I see it.
I’ve attached the thread dump for somebody who knows more than me what’s going on.
While top is reporting 150% CPU usage currently on for the openhab pid, I notice that actually the threads listed here, while they’ve clocked up a history of time, most of the current percentage utilisation seems fairly flag. I tried ordering by %-cpu, and not a lot is happening on a 1 second refresh interval.
Cherry picking a couple of entries that look high at this random moment, I chose: 844933
"qtp1384190402-844933" #844933 [11139] prio=5 os_prio=0 cpu=74194.03ms elapsed=6651.86s tid=0x00007fe000892600 nid=11139 runnable [0x00007fdfaf4e2000]
java.lang.Thread.State: RUNNABLE
at java.math.BigDecimal.valueOf(java.base@21.0.10/BigDecimal.java:1318)
at java.math.BigDecimal.valueOf(java.base@21.0.10/BigDecimal.java:1294)
at java.math.BigDecimal.createAndStripZerosToMatchScale(java.base@21.0.10/BigDecimal.java:5246)
at java.math.BigDecimal.stripTrailingZeros(java.base@21.0.10/BigDecimal.java:3094)
at tech.units.indriya.function.DefaultNumberSystem.isFractional(DefaultNumberSystem.java:616)
at tech.units.indriya.function.DefaultNumberSystem.narrow(DefaultNumberSystem.java:502)
at tech.units.indriya.internal.function.Calculator.load(Calculator.java:102)
at tech.units.indriya.internal.function.Calculator.of(Calculator.java:69)
at tech.units.indriya.quantity.NumberQuantity.<init>(NumberQuantity.java:75)
at tech.units.indriya.quantity.Quantities.getQuantity(Quantities.java:105)
at org.openhab.core.library.types.QuantityType.<init>(QuantityType.java:192)
at org.openhab.persistence.rrd4j.internal.RRD4jPersistenceService.lambda$5(RRD4jPersistenceService.java:810)
at org.openhab.persistence.rrd4j.internal.RRD4jPersistenceService$$Lambda/0x0000000100f337d8.apply(Unknown Source)
at org.openhab.persistence.rrd4j.internal.RRD4jPersistenceService.query(RRD4jPersistenceService.java:535)
at org.openhab.core.ui.internal.chart.defaultchartprovider.DefaultChartProvider.addItem(DefaultChartProvider.java:358)
at org.openhab.core.ui.internal.chart.defaultchartprovider.DefaultChartProvider.createChart(DefaultChartProvider.java:251)
at org.openhab.core.ui.internal.chart.ChartServlet.doGet(ChartServlet.java:307)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:497)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:584)
at org.ops4j.pax.web.service.spi.servlet.OsgiInitializedServlet.service(OsgiInitializedServlet.java:102)
at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)
at org.ops4j.pax.web.service.spi.servlet.OsgiFilterChain.doFilter(OsgiFilterChain.java:113)
at org.ops4j.pax.web.service.jetty.internal.PaxWebServletHandler.doHandle(PaxWebServletHandler.java:334)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234)
at org.ops4j.pax.web.service.jetty.internal.PrioritizedHandlerCollection.handle(PrioritizedHandlerCollection.java:96)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:731)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
at org.eclipse.jetty.server.HttpChannel$$Lambda/0x00000001007c0e50.dispatch(Unknown Source)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.lang.Thread.runWith(java.base@21.0.10/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.10/Thread.java:1583)
What jumped out to me on this one is the reference to ‘chart’. So I closed my Edge session that had my overview page open (not a chart, though I had looked at some earlier) and the CPU usage dived.
QueuedThreadPool integrates with the Jetty component model, implements Executor, provides a TryExecutor implementation (discussed in the adaptive execution strategy section), and supports virtual threads (introduced as a preview feature in Java 19 and Java 20, and as an official feature since Java 21).
For example, a sudden large spike of requests arriving to the server may find the thread pool in an idle state where the number of threads is shrunk to the minimum. This will cause many tasks to be queued up, way before an HTTP request is even read from the network. Add to this that there could be I/O failures processing requests, which may be submitted as a new task to the thread pool.
Jetty is the internal component that provides OH’s internal web server. So, the qtp threads belong to the web server, which does multiple things, including service the UIs, the REST API, web socket events etc.
The thread dump doesn’t show quite the same situation as the screenshot, not threads are marked as “blocked” in the thread-dump (although the definition of “blocked” might vary). I can’t say anything definitive, but it does seem like the problem is in the ChartServlet - or with whatever keeps calling the ChartServlet. I see that multiple threads are in process of creating a chart and adding items, which leads to persistence being queried.
It’s hard to imagine that charts should be “created” very often, so it’s likely that something (probably the UI) keeps calling the servlet repeatedly. I think that’s where the investigation should start at least.
I don’t know who is best equipped to do that, @florian-h05 perhaps?
I’m using a fair bit of the trends on labels, but I would expect it to be more consistently terrible than intermittent, if it was a problem with this page.
The ChartServlet (serving at /chart) isn’t used by Main UI, it’s only used by sitemaps, i.e. Basic UI & mobile apps. Main UI renders the charts and the trendlines itself and gets its data from the /rest/persistence endpoints.
