I notice the last time (months) a lot more recompiling of large numbers of items-files when changing a single item or sometimes changing a rule.
I thought recompiling of items in other item-files should only occur when one of the items in the changed item-file is also in the other rule.
Or could it now be the case that every changed item belonging to a group also lets the other itemfiles recompile when there are items from the same group in that other file? I have some large groups spanning items in multiple groups, for instance for persistence settings.
I notice the recompiling because I get a lot of errors of non existing items in rules and lack of responsiveness of openHAB. It can take several minutes (sometimes up to 10) before the system is quiet again.
Who has an idea or explanation? I tried all kind of debug options using the console but could not get a finger behind the issue.
I don’t know of anything that’s changed in how the config files are loaded and parsed. I definitely would expect certain changes to cascade across multiple files.
It definitely used to be the case that any change to any .item file would cause all the rights to reload. I think it was made smarter than that some time ago. I don’t know how smart it is though. And if a Group gets modified or even if it’s in a fine that’s modified, I could see that cascading to multiple files.
If you can find some sort of pattern it might be worth filling an issue.
If you can write down the steps to reproduce it, someone can test it / take a look. Otherwise, afaik, there is no current bug in reloading of items/things.
As for rules, there were recent work (i.e. in the past 2 years) to do with pre-compilation of JS.
It is impossible to write this down. Actually it is simple to reproduce on my machine.
-open an itemfile
-change the name of, or remove or ad a file and save it.
Watch what happens. The utilization of the Windows machine increases and the error messages appear of not found/unrecognized items. Program execution gets slow. Changes in the Sitemap only appears after a long time.
i have the feeling, the longer the items file the more impact the change has. The items are mostly only used in one rule and in the Sitemap. They mostly belong to one group, defined in the top of that file. I think his behaviour has started some month ago, around version 5.1 or a followed milestone or snapshot.
I invested the issue further and it looks the cause is a large items file. I have an items file for my (recently added) MercedesMe binding. It contains more then 100 items. A change cause the “OH-rulesRefresher” from blocking all other activities of openHAB. Other processes time out and cause new errors.
The worst thing that can happen, is making a syntax error in the file. After that (or the correction of the error) it takes very long (5 to 10 minutes) to recover from this situation. All the time the OH-rulesRefresher blocks other rules.
Al the items in the mentioned file are connected to a channel of the binding.
Example:
Number:Dimensionless MercedesChargeMaxSoc "Maximum state of charge" (gMercedes) {channel="mercedesme:hybrid:7a2ea39a6d:W1NKJ5GB1VF658577:charge#max-soc"}
After I found the cause of this issue, I remeber I had similar problems after adding the Modbus binding, also with large item files.
Below an example of how (ttop on Console) it looks after adding two lines to the item file.
Is this an RPi 3 or some other underesourced machine? The key resource to pay attention to is RAM.
I wonder if the YAML format might parse and load more effeciently than the DSL parser. It might be worth a try. You should be able to just select all the Items you want in MainUI and selecty “copy” at the bottom and choose “yaml”. That will copy the full definition of all the selected Items to the clipboard in the new YAML format. You can then paste that to a file as an easy way to convert from DSL to YAML.
It’s what I would expect on a machine that is stuck like you describe. On a healthy system though you should see very few if any threads stuck in a blocked or timed_waiting state.
But it’s the rules, not the Items that are the problem according to this. When you change the Items, the rules need to be reparsed/compiled too often, especially if there are “Member of” triggers.
Change of programming language and environment (VScode) to end up at perharp finding it did not solve anything. At this moment not the way to go for me. But thanks anyway for the suggestion.
Yes my machine is a very efficient low power machine.
Device Name HomeControl
Processor Intel(R) Atom™ x5-Z8350 CPU @ 1.44GHz 1.44 GHz
Installed RAM 4,00 GB (3,81 GB usable)
Storage 115 GB SSD VID:D6 58K722
Graphics Card Intel(R) HD Graphics (114 MB)
System Type 64-bit operating system, x64-based processor
and I am happy with it.
I will try to find out if splitting up the files in smaller portions solves my issues.
The solution to this kind of issues should not always be the use of more powerfull machines. We see where this leads to. The smallest programs now are several gigabytes (up to hundreds like Adobe Acrobat) large and need a very fast and power hungry cpu. In my time we had programs in the size of Megabytes doing about the same on a small Celeron processor!
But Rich, thanks for your information. Always appreciated.
A 64bit-processor running on a 64-bit OS takes about 2x2 the amount of RAM of a 32/32 bit system, so yours is the x86 equivalent of a 1 GB RPi 3.
Which is known to be a problem since OH 5 introduced 64 bit Java.
The main point is still valid, though.
Compile frequency has increased a lot (although I can’t tell either which situations changed).
Plus, compilation is still single threaded. If it was multithreaded, this would improve the situation a lot as about any system has 4+ processor cores, all but one which are unused.
To be clear: it is not about compiling a single rules file on multiple cores in parallel.
It is rather about compiling every rules file in parallel rather than to serialize this process as openHAB did since day 1 and still does.
Thank you Markus for your insights. I’ll consider, when it get too annoying, to replace my OH machine to something more powerful. The other thing is, when everything is running, it only runs on less then 25% of the CPU power. Javaload is only 5%.
If you want to know if the problem is the size of the files, number of Items, or something else then yes “this is what I come up with”.
If you don’t to do experiments (especially something so easy) to find the source of the problem, I can’t/won’t help you. No one is saying you have to stay using YAML forever.
It’s not. But an RPi with only 1 GB RAM is not and has not be sufficient to run OH and has not been for quite some years. Even with the expanded need for RAM with 64-bit, 4 GB should be plenty. You could verify by checking top to see if SWAP is being used.
This then continues for all rules. What could be the reason the ‘reloadAllModelsOfType rules’ gets always triggered? And why does it think the “modelChanged MODIFIED”?
The whole process started when adding an item to the mercedes.rules file.
[INFO ] [.model.core.internal.ModelRepositoryImpl] - Loading DSL model 'mercedes.items'
....
...
[DEBUG] [.core.model.rule.jvmmodel.RulesRefresher] - Item "MercedesVehicleFeatureCapabilities" added => rules are going to be refreshed
[DEBUG] [odel.script.jvmmodel.ScriptItemRefresher] - Item "MercedesVehicleFeatureCapabilities" added => scripts are going to be refreshed
With Things and Items, on every reload, the entire file needs to be read, parsed and then compared against all existing Things / Items. Only then, the changed Things/Items can be identified and updated.
With Rules, there is no way to tell whether a rule is the same or not, so in addition to everything in the file being re-parsed, all the previous rules from that file must be deleted, and the ones just loaded re-added.
