Just a suggestion… If anyone is familiar with VisualVM, you might be able to use it get some insights into what’s causing the problem.
I did, to
-Xms350m
it just delays the problem, before it crashed about daily, now after two days. I don’t run TR064 binding, also no UI based rules (all rules still in .rules file).
I am on 3.0.1 Release on a RPI4 with 4GB.
I set up the rules back in ui only and setup visualvm so i will report back.
1 hour uptime
2 hour uptime
HeapDump as 7z after 2h runtime
https://drive.google.com/file/d/1tD-2IVU4pQiK9X3HcqAKPwjDbbIruQgg/view?usp=sharing
Uncompressed
https://drive.google.com/file/d/1e1Ggv4cFYPI9TgiR6621zZOiw_HZun5d/view?usp=sharing
ThreadDump as 7z after 2h runtime
https://drive.google.com/file/d/1tLkXiIK7KkUtivuRVLrwqwqmVIj2po3I/view?usp=sharing
Uncompressed
https://drive.google.com/file/d/1EZVjB_J6NC9paa4ZGbyYnTC0qfhGnh79/view?usp=sharing
30 min after set cron time instead of item change
HeapDump
https://drive.google.com/file/d/1slswSyZRouk5h2_Dmd7idc9mjE6S56Ga/view?usp=sharing
Uncompressed
https://drive.google.com/file/d/1QBPKkadHD-59frZANO3mXIVbxhNCf-Fx/view?usp=sharing
ThreadDump
https://drive.google.com/file/d/1tWs1eCVBzWARLNBRA_fTlprZghtJZ5CB/view?usp=sharing
Uncompressed
https://drive.google.com/file/d/1_-qShrLn3Aj-zMQrIq1tkpcm7NMz5Phz/view?usp=sharing
Good night
4 Likes
I had some time to play around with this today. I found something. It may not be the issue you all are having, but it’s definitely a problem. A really big problem.
I created a DSL rule that runs every second. It’s a very simple rule. I created that rule using the UI, and I also created the same rule as a text rule.
When I introduced an error into the rule created through the UI, memory consumption went bananas. To the tune of ~12 MB increase per rule iteration. When I fixed the error, memory stabilized. When I introduced the same error into the text rule, memory consumption remained stable.
For completeness, here’s the DSL text version of my rule. The rule created using the UI has the same body, and also runs every second.
rule "Test One per Second"
when
Time cron "* * * * * ? *"
then
xxxxxxx
var Number n = 1
end
When I save the above text-based rule, this is what I see in the log.
2021-02-14 13:51:27.900 [INFO ] [el.core.internal.ModelRepositoryImpl] - Loading model 'test.rules'
2021-02-14 13:51:28.168 [INFO ] [el.core.internal.ModelRepositoryImpl] - Validation issues found in configuration model 'test.rules', using it anyway:
The value of the local variable n is not used
2021-02-14 13:51:29.486 [ERROR] [internal.handler.ScriptActionHandler] - Script execution of rule with UID 'test-1' failed: The name 'xxxxxxx' cannot be resolved to an item or type; line 8, column 5, length 7 in test
2021-02-14 13:51:30.355 [ERROR] [internal.handler.ScriptActionHandler] - Script execution of rule with UID 'test-1' failed: The name 'xxxxxxx' cannot be resolved to an item or type; line 8, column 5, length 7 in test
2021-02-14 13:51:31.355 [ERROR] [internal.handler.ScriptActionHandler] - Script execution of rule with UID 'test-1' failed: The name 'xxxxxxx' cannot be resolved to an item or type; line 8, column 5, length 7 in test
2021-02-14 13:51:32.353 [ERROR] [internal.handler.ScriptActionHandler] - Script execution of rule with UID 'test-1' failed: The name 'xxxxxxx' cannot be resolved to an item or type; line 8, column 5, length 7 in test
2021-02-14 13:51:33.354 [ERROR] [internal.handler.ScriptActionHandler] - Script execution of rule with UID 'test-1' failed: The name 'xxxxxxx' cannot be resolved to an item or type; line 8, column 5, length 7 in test
When I save the same rule in the UI, this is what I see in the log.
2021-02-14 13:56:25.804 [ERROR] [internal.handler.ScriptActionHandler] - Script execution of rule with UID 'ui-created-rule' failed: ___ xxxxxx
var Number n = 1
The method or field xxxxxx is undefined; line 1, column 0, length 6
2021-02-14 13:56:31.790 [ERROR] [internal.handler.ScriptActionHandler] - Script execution of rule with UID 'ui-created-rule' failed: ___ xxxxxx
var Number n = 1
The method or field xxxxxx is undefined; line 1, column 0, length 6
2021-02-14 13:56:33.110 [ERROR] [internal.handler.ScriptActionHandler] - Script execution of rule with UID 'ui-created-rule' failed: ___ xxxxxx
var Number n = 1
The method or field xxxxxx is undefined; line 1, column 0, length 6
2021-02-14 13:56:34.081 [ERROR] [internal.handler.ScriptActionHandler] - Script execution of rule with UID 'ui-created-rule' failed: ___ xxxxxx
var Number n = 1
The method or field xxxxxx is undefined; line 1, column 0, length 6
FTR I’m on build 2169.
2 Likes
I’ve looked over most of what is on here, there are a few more things for me to go through. Thank you to everyone who provided info. As it sits, I have a few theories and I think I have produced a way to test this all so we can try to get some answers.
As posted above, I created [automation] Make rule engine threading configurable by morph166955 · Pull Request #2182 · openhab/openhab-core · GitHub in an attempt to resolve this. There are three jar files to be replaced to test this. I would GREATLY appreciate anyone with this issue to apply these jars and associated configs listed below. A few things could happen:
- Enabling threadpools totally resolves this (unlikely, but possible). I believe the threadpools will definitely help this situation on lower memory systems. I’m concerned that the Pi4/4GB models are having this issue, I would have expected the memory on those platforms to seriously help this issue.
- Additional debugs added to the code may help identify where the memory leak is happening (more likely).
- Absolutely nothing is resolved and the debugs are worthless (highly unlikely, but it will at least tell us that the issue is not with the thread creation or the addition of [automation] Cache parsed script in order to improve performance by kaikreuzer · Pull Request #2057 · openhab/openhab-core · GitHub in 3.0.1 (highly unlikely that we will have complete worthless failure of this effort).
- Your system becomes completely unstable and you have to revert. I HIGHLY doubt this will happen. I’m running these jars on my system and have been for days with no issues. In the event this happens, just clear cache and it should revert this out and restore stability.
So, how to test.
Step 1: restart OH to make sure you have as stable of a system as possible. I REALLY don’t want anyone doing this on a system that’s been spinning for a while in the event that the updates push you over the edge. Let’s give this the best chance of success
Step 2: download the 3 jars from: Release Automation ThreadPool Test - Rev15 · morph166955/openhab-core · GitHub
Step 3: Login to your karaf console.
$ ssh -p 8101 openhab@localhost
openhab> bundle:list -s | grep org.openhab.core.automation
132 │ Active │ 80 │ 3.1.0.202102071659 │ org.openhab.core.automation
openhab> bundle:list -s | grep org.openhab.core.model.script.runtime
177 │ Active │ 80 │ 3.1.0.202102082247 │ org.openhab.core.model.script.runtime
openhab> bundle:list -s | grep org.openhab.core
128 │ Active │ 80 │ 3.1.0.202102061843 │ org.openhab.core
openhab> bundle:update 177 file:///home/openhab/org.openhab.core.model.script.runtime-3.1.0-SNAPSHOT-threadpool15.jar
openhab> bundle:update 132 file:///home/openhab/org.openhab.core.automation-3.1.0-SNAPSHOT-threadpool15.jar
openhab> bundle:update 128 file:///home/openhab/org.openhab.core-3.1.0-SNAPSHOT-threadpool15.jar
We need to replace the three files in sequence. Your ID numbers may be different than mine. Please be careful to update the specific bundles to match. To note, this (specifically the last one) WILL cause your OH to completely lose it’s mind so as soon as the updates are done, shutdown (do not restart, we have to make some config changes before you start it again) the OH service.
If you get a completely unstable system, you can run “sudo openhab-cli clean-cache” and it will clear all these jars out and load the stock/stable files. Again, we’re not making super major changes here so I doubt you will get a completely unstable system if you update.
Step 4: Shutdown OH if you haven’t…
Step 5: Make the following config changes:
in /etc/openhab/services/runtime.cfg
Add “org.openhab.threadpool:automation=30” to the end of the file.
in /var/lib/openhab/etc/log4j2.xml Add:
<RollingRandomAccessFile fileName="${sys:openhab.logdir}/threadpoolmanager.log" filePattern="${sys:openhab.logdir}/threadpoolmanager.log.%i" name="THREADPOOLMANAGER">
<PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} [%-5.5p] [%-36.36c] - %m%n"/>
<Policies>
<OnStartupTriggeringPolicy/>
<SizeBasedTriggeringPolicy size="16 MB"/>
</Policies>
</RollingRandomAccessFile>
<RollingRandomAccessFile fileName="${sys:openhab.logdir}/threadpoolqueue.log" filePattern="${sys:openhab.logdir}/threadpoolqueue.log.%i" name="THREADPOOLQUEUE">
<PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} [%-5.5p] [%-36.36c] - %m%n"/>
<Policies>
<OnStartupTriggeringPolicy/>
<SizeBasedTriggeringPolicy size="16 MB"/>
</Policies>
</RollingRandomAccessFile>
<RollingRandomAccessFile fileName="${sys:openhab.logdir}/automation.log" filePattern="${sys:openhab.logdir}/automation.log.%i" name="AUTOMATION">
<PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} [%-5.5p] [%-36.36c] - %m%n"/>
<Policies>
<OnStartupTriggeringPolicy/>
<SizeBasedTriggeringPolicy size="16 MB"/>
</Policies>
</RollingRandomAccessFile>
And then further down:
<Logger additivity="false" level="TRACE" name="org.openhab.core.common.ThreadPoolManager">
<AppenderRef ref="THREADPOOLMANAGER"/>
</Logger>
<Logger additivity="false" level="TRACE" name="org.openhab.core.common.QueueingThreadPoolExecutor">
<AppenderRef ref="THREADPOOLQUEUE"/>
</Logger>
<Logger additivity="false" level="DEBUG" name="org.openhab.core.automation">
<AppenderRef ref="AUTOMATION"/>
</Logger>
<Logger additivity="false" level="TRACE" name="org.openhab.core.model.script.runtime">
<AppenderRef ref="AUTOMATION"/>
</Logger>
This hopefully will let us capture any failures or memory/thread issues.
Step 6: Start OH. You may (likely will) have to restart once or twice if your system is acting funny. It will need to recompile the majority of the libraries and this can cause some odd things until everything is cached. You can confirm that the jars worked by logging back into karaf and comparing to:
openhab> bundle:list -s | grep org.openhab.core
128 │ Active │ 80 │ 3.1.0.202102141801 │ org.openhab.core
openhab> bundle:list -s | grep org.openhab.core.automation
132 │ Active │ 80 │ 3.1.0.202102141805 │ org.openhab.core.automation
openhab> bundle:list -s | grep org.openhab.core.model.script.runtime
177 │ Active │ 80 │ 3.1.0.202102141807 │ org.openhab.core.model.script.runtime
Step 7: Monitor! If this works, please let us know. If this fails, also please let us know. I would like to see the three log files created above in either case to validate that the code is working and (if there is a failure) to see what is breaking.
Mark, thanks for checking out this issue. As you say, I think there may be more then one issue causing this problem but as many folks are experiencing it, I’m concerned we may have a real problem on our hands.
So from this, I’m guessing that when the error is run, the system doesn’t ignore the rule and attempts to run it? I’m not super informed about OH3 but from texted based DSL rules experience, I would expect the rule not to load and complain with an error. Was the error purposefully introduced into the rule something that an automatic syntax check should have easily caught?
Correct. It tries to run the rule every second.
The error was introduced intentionally. You can see the rule above, as I added it to my post.
1 Like
Also B R, thank you for putting effort into trying to resolve the issue. Thanks to the folks providing thread dumps ect.
I’m guessing you are aware that Kai introduced thread pooling early on in the OH3 development. I can dig around github and try to find the threads but I think you already are aware of this issue. As I recall, there was a bad problem with the thread pooling and OH3 was immediately patched to remove the thread pooling feature. I fear a regression or bug was introduced some where along the way. I’m guessing you are aware of this history correct?
Duh… so something really simple a simple syntax error check would have easily caught. That rules file would have never loaded. Great diagnostics!!! As you say, this may not be the issue everyone in this thread is having but definitely needs fixed.
Advice to everyone else having an issue to help find the buggy if you have time and inclination to test, remove complex rules, get down to only one simple text rule and one simple UI rule, make them dead simple like Mark’s and run ever few seconds to stress test. Even if you don’t have time to test, carefully check your UI rules for errors
Also please post links to any issues you post on github so we all can play along at home
Great work guys!!! keep it up 
Yes, 100% tracking that and the issue it caused. In theory there is some code in here to help prevent the issues that it caused. Also, as opposed to what was done originally, this is something that can be enabled/disabled by the end user so folks that don’t need threadpools (like myself) can continue to run in single thread mode and all of this code is effectively disabled.
2 Likes
Have done that.
System does not start completely all bundles are installed an active.
logfiles are created
can’t click on rules or script in ui load endless
items etc works
openhab> bundle:list -s | grep org.openhab.core | grep 3.1.0
128 (0x Active x 80 x 3.1.0.202102141801 x org.openhab.core
132 (0x Installed x 80 x 3.1.0.202102141805 x org.openhab.core.automation
177 (0x Installed x 80 x 3.1.0.202102141807 x org.openhab.core.model.script.runtime
automation log is empty
threadpoolmanager.log (1.3 KB)
threadpoolmanager.log.1.log (1.3 KB)
threadpoolmanager.log.2.log (1.3 KB)
threadpoolmanager.log.3.log (1.3 KB)
threadpoolqueue.log (399.6 KB)
threadpoolqueue.log.1.log (213.6 KB)
threadpoolqueue.log.2.log (144.4 KB)
threadpoolqueue.log.3.log (141.9 KB)
the both 132 and 177 keep state install not active so edit rules not possible and they dont run anymore.
clear cache all works normal, will try again tomorrow
Ah ****. I hadn’t even contemplated the version number discrepancy between 3.0.1 and 3.1.0. You need to be running a 3.1.0-SNAPSHOT release for those jars. I gotta see if I can pull the 3.0.1 codebase and apply the diff cleanly and create some jars based on that. SORRY!
Option B for anyone who wants to try this right now, upgrade to a snapshot release.
EDIT: Just to try to get this to the end faster, I’m going to create an ubuntu/OH3.1.0-SNAPSHOT VM to try to recreate this based on @mhilbush info above. If I can put the “bad rule” in along side my jars I may be able to see what’s spinning off and why.
Ok, no problem I setup a test raspberry pi tommorow with the latest snapshot release and will try my rules on it and report back, good night.
@mhilbush I replicated your test rule above on a brand new VM running OH3.1.0-SNAPSHOT S2207. I replicated the error you were able to get. What I’m not getting is your memory build. I am getting a slow build however. I’m using the systeminfo binding as a quick and dirty monitor.
2021-02-14 20:36:37.181 [DEBUG] [e.automation.internal.RuleEngineImpl] - The trigger '1' of rule 'aeefe05726' is triggered.
2021-02-14 20:36:37.196 [ERROR] [internal.handler.ScriptActionHandler] - Script execution of rule with UID 'aeefe05726' failed: ___ xxxxxxx
var Number n = 1
The method or field xxxxxxx is undefined; line 1, column 4, length 7
javax.script.ScriptException: ___ xxxxxxx
var Number n = 1
The method or field xxxxxxx is undefined; line 1, column 4, length 7
at org.openhab.core.model.script.runtime.internal.engine.DSLScriptEngine.eval(DSLScriptEngine.java:135) ~[?:?]
at org.openhab.core.automation.module.script.internal.handler.ScriptActionHandler.lambda$0(ScriptActionHandler.java:62) [bundleFile:?]
at java.util.Optional.ifPresent(Optional.java:183) [?:?]
at org.openhab.core.automation.module.script.internal.handler.ScriptActionHandler.execute(ScriptActionHandler.java:59) [bundleFile:?]
at org.openhab.core.automation.internal.RuleEngineImpl.executeActions(RuleEngineImpl.java:1179) [bundleFile:?]
at org.openhab.core.automation.internal.RuleEngineImpl.runRule(RuleEngineImpl.java:987) [bundleFile:?]
at org.openhab.core.automation.internal.TriggerHandlerCallbackImpl$TriggerData.run(TriggerHandlerCallbackImpl.java:89) [bundleFile:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Memory build:
2021-02-14 20:29:10.861 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 790 to 792
2021-02-14 20:29:11.863 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 792 to 796
2021-02-14 20:29:13.864 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 796 to 806
2021-02-14 20:29:15.865 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 806 to 795
2021-02-14 20:29:32.879 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 795 to 796
2021-02-14 20:29:40.886 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 796 to 799
2021-02-14 20:30:04.899 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 799 to 800
2021-02-14 20:30:11.903 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 800 to 799
2021-02-14 20:30:18.908 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 799 to 800
2021-02-14 20:30:27.914 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 800 to 811
2021-02-14 20:30:31.917 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 811 to 802
2021-02-14 20:31:32.950 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 802 to 803
2021-02-14 20:31:33.951 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 803 to 812
2021-02-14 20:31:35.952 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 812 to 806
2021-02-14 20:31:37.952 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 806 to 808
2021-02-14 20:31:39.953 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 808 to 809
2021-02-14 20:31:43.955 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 809 to 817
2021-02-14 20:31:49.957 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 817 to 818
2021-02-14 20:31:50.958 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 818 to 806
2021-02-14 20:31:51.958 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 806 to 807
2021-02-14 20:32:04.965 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 807 to 808
2021-02-14 20:32:10.969 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 808 to 813
2021-02-14 20:32:20.973 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 813 to 809
2021-02-14 20:32:27.976 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 809 to 810
2021-02-14 20:32:28.977 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 810 to 841
2021-02-14 20:32:29.980 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 841 to 829
2021-02-14 20:32:32.979 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 829 to 830
2021-02-14 20:32:33.980 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 830 to 819
2021-02-14 20:33:08.995 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 819 to 820
2021-02-14 20:33:43.011 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 820 to 821
2021-02-14 20:33:44.013 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 821 to 822
2021-02-14 20:33:46.013 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 822 to 823
2021-02-14 20:34:20.028 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 823 to 824
2021-02-14 20:34:23.030 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 824 to 825
2021-02-14 20:34:35.035 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 825 to 827
2021-02-14 20:34:36.036 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 827 to 821
2021-02-14 20:35:17.053 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 821 to 822
2021-02-14 20:35:19.053 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 822 to 824
2021-02-14 20:35:37.061 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 824 to 825
2021-02-14 20:35:47.071 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 825 to 827
2021-02-14 20:35:48.070 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 827 to 828
2021-02-14 20:36:23.090 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 828 to 835
2021-02-14 20:36:24.090 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 835 to 836
2021-02-14 20:36:26.091 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 836 to 832
2021-02-14 20:37:27.115 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Memory_Used' changed from 832 to 833I’m not sure what Systeminfo reports. Is it total memory used on the box?
Edit: I’m not getting that stack trace either.
I suppose I could try the latest build to see if there’s a difference. I’ll save a backup of the current one in case I want to go back.
Ok, maybe found it. @mhilbush can you load the jar below and see if it fixes the memory leak? If so, I know what part of the code is causing the leak, I just need to figure out how to fix it (my jar just reverts a commit which is still needed but may need to be reworked).
FWIW, I see the same behavior in build 2207.
After a while I start to see evidence that it’s oom.
2021-02-14 17:05:47.734 [WARN ] [ab.core.internal.events.EventHandler] - Dispatching event to subscriber 'org.openhab.core.internal.items.ItemUpdater@49de7fa8' takes more than 5000ms.
2021-02-14 17:05:53.245 [WARN ] [ab.core.internal.events.EventHandler] - Dispatching event to subscriber 'org.openhab.core.io.monitor.internal.EventLogger@3854893e' takes more than 5000ms.
2021-02-14 17:06:03.358 [WARN ] [ab.core.internal.events.EventHandler] - Dispatching event to subscriber 'org.openhab.core.automation.internal.module.handler.ItemStateTriggerHandler@72534e1a' takes more than 5000ms.
Yeah, but I can’t do it at the moment. I’m in the middle of cooking dinner… 
I suppose I can wait… (kidding)
My system has been better on memory with that jar. It goes up but then drops back down and stays in a ~20MB range. If that does fix it, I need to figure out why. Likely something not handling an exception properly and getting stuck. Can you also monitor total cpu thread usage at the same time? That may indicate if we have threads spinning off to nowhere also.