This problem massively hinders the development of Blocky scripts (modify - run - modify - …). If there will be no solution for 32-bit operating systems on the RaspberryPi (in many cases switching to 64-bit won’t be a solution due to RAM limitations), many Blockly users will either look for other hardware or another HA solution.
Being the blockly maintainer I want throw my 2 cents in:
- @florian-h05 have been putting a load of work and hours and if not days of work into it.
- I myself have developed more than a 100 blockly rules that I develop actively.
Let’s state again:
- Blockly rules DO NOT run slower in general
- The rules only take around 14 secs on my machine when run at the first time. Though this might be a bit annoying even to me it is neither a deal breaker nor a nightmare and doesn’t question everything.
- I even had a setup that was proven to be fast even the first time when I was using the external library but we could not find out what the difference was which drove us nuts during the investigation and finally after changing the setup it effect was gone which was very sad.
… and, yes, still I love to have it solved.
However, we are trying to solve the problem as good as we can but let me express that tone in some of the above comments too me very much exaggerate the impact of what it really is and doesn’t really motivate to find the issue…
Thanks, @florian-h05 and all the others who help tracking down the issue.
7 Likes
I do appreciate the efforts to resolve the issue and I can confirm the extent of the delay when first running a Blockly script (in my case about 16 seconds on a Raspberry Pi 4 Model B Rev 1.1 with 4 GB RAM), but I strongly disagree with this being just a minor annoyance.
I’m sorry, I don’t understand. Are you programmatically disabling the rule?
If so, why not use a ‘But only if’ instead?
only run during daytime
1 Like
One aspect that I think is important hasn’t been touched yet:
Users are just migrating to OH4 and should consider the above stated consequences when choosing their setup.
My conclusion of the above discussion is that it’s now better to switch to 64 bit if possible (e.g. Raspi with 4GB or more).
I would like to pull in @mstormi and ask him what he thinks about this and if we should change the docs and readme accordingly. He always recommended not to use 64bit but I think this has to be changed.
And it’s not just a problem about startup time of rules but about the fact that Graal.js is used in an unsupported way as wirthi stated above.
It is not acceptable to use a component in an unsupported way in openHAB (remember that some community members even refuse to help users with ‘unsupported’ openHAB configurations: the same reasons for not helping there apply here).
If fixing the problem means no support for 32-bit OS on Raspberrry Pi (or 32-bit OS on ARM in general - not sure about that), make it absolutely clear in the openHAB 4 release notes, that some Raspberry Pi variants aren’t supported anymore due to RAM restraints.
Should we start a poll to collect information to get an idea what the minimum RAM size for a usable 64-bit openHABian is? It might be as easy as asking the 64-bit users to make some specific openHAB REST API calls. How to interpret the results is, of course, another question.
JM2C
As discussed during our session this morning with Florian, I have put together all results there and recommend making up your mind with regard to my results. I would also appreciate if someone else would verify my testing results exactly the way I did it.
It is not acceptable to only to make demands and complain if you have not yet made a single contribution.
Keep in mind that openHAB is an open-source project — if something is not acceptable for you you can always try to make it better and open a PR. If you come up with a good solution, I will have a look at it. Until then, I will ignore your unqualified comments.
As you have probably realised, openHAB 4 was released end of July, and after the release we cannot adjust the release notes — it‘s very unlikely that anyone reads the release notes weeks after upgrading.
Anyway, we (the developers) will have a look at the possible options and finally decide together what do.
That in fact might be a good idea, however I would wait until we have decided on a solution.
I would guess that 2GB RAM should be enough for most openHAB installations — on my installation on a 64-bit Debian VM openHAB has a total of 1,96 GB of JVM memory (heap and non-heap) available, with currently 1,2 GB commited and only 642 MB actually used (I have 870 Items, 80 rules and 60 Things.)
Well, to me it sounds like ‘shut up or contribute’ - wouldn’t this make a good slogan for openHAB? All the best to openHAB, I am out and will donate my resources elsewhere.
Don’t take it like „shut up or contribute“, take it like „Do not demand, but contribute“.
You can report issue or open feature request and everyone will be fine, but don’t take to right to judge something like:
4 Likes
Frankly I didn’t catch the whole story as I haven’t been involved before.
So there’s a problem with some Blockly rules if on Raspi but only on 32 bit and we all shall upgrade Raspi OS to 64bit because GraalVM only works fine with that ?
Given most parts of native Raspi OS are still 32bit, that sounds a little sketchy to me and not as if that’s the last word on this.
For one, this will not affect many people (some Blockly users only).
Second, there’s 64bit openHABian available so anyone really affected can just reflash his box with that and reimport his config.
So real impact is sparse and a fix is available. I’d say that on global (all-OH) level, that is not a major nuisance. I didn’t catch what the root cause is but I have full trust in @florian-h05 and @stefan.hoehn that they’ll keep trying to identify and fix it.
So it’s not a sufficient let alone good reason to change the overall recommendation.
You might want to amend it with some hint on that for the time being.
1 Like
No, all rules using Javascript with the helper library.
But it’s true that it’s too early to give a clear recommentation. It’s more my current personal conclusion to switch to 64bit as I’m just installing a new system for OH4 anyway.
Blockly “compiles” to ECMA Script 2022, which is run by the JavaScript Scripting (GraalJS) add-on.
GraalVM itself does not support 32-bit, however GraalJS is running on 32-bit up to some version (in fact keeping 32-bit support is holding me back from upgrading GraalJS to a newer version).
With Blockly switching from NashornJS to GraalJS, several users reported that the initial loading of scripts takes really long, which is because the helper library is injected into the script.
Therefore this issue affects all JS Scripting and Blockly users (as @Larsen already said), which of course is not all openHAB users.
Running on 64-bit, no matter whether on GraalVM or OpenJDK/Zulu, seems to really improve the performance of GraalJS (thx to @stefan.hoehn for doing some tests, see [jsscripting] GraalVM's optimised compiler or find another solution · Issue #15600 · openhab/openhab-addons · GitHub).
Given the fact that Jython might go away at some point in time and Rules DSL is IMO outdated (@rlkoshak if you have a better idea how to word this, this idea is really welcome), I feel that both JS Scripting and Ruby Scripting are promoted as go-to solutions when creating new rules.
After veryfying the performance difference between 32-bit and 64-bit and measuring the RAM usage of 64-bit and having a look at the other possible solutions, I would rediscuss this.
I could also imagine having a split solution, that recommends different OSes for different models.
BTW: 64-bit would also allow the installation of InfluxDB 2 on Raspberry Pi.
So only initial loading takes longer on standard openHABian, not execution? That has been true for rules DSL ever, too.
I still would not call that a major nuisance to justify a new general recommendation.
Remember this is only on startup. That
should be neligible on a system designed to run 24x7. I also keep wondering if the statements on GraalVM are fully correct and final. Cannot believe they retroactively drop support for the still most common Raspi setup.
As said you could complement the docs with a statement to try 64bit if on 4+ GB but I would not be recommending that as of today.
Kinda what I was thinking
remember this? (circa 2018) I know you remember Markus, you posted in it
Im not sure, I am only a stupid user with extended knowings but the last comment remember on this Openhab 4.0.x rules need a lot of memory - #99 by Wikibear That is exectly why I’m running now a Pi 4 with 4GB. I have roughly 95 rules and Openhab needs more memory before upgrading from 3.4 to 4. With each rule an my system was overloaded. I mean that a raspberry Pi 3 isn’t optimal if you have so many rules and byside other services like zigbee2mqtt, zigbee Broker, Grafana and InfluxDB. The OS Swap most, because 1GB Ram isn’t enough. At the moment my memory goes up with other services to 1,7GB RAM on Rasp 4.
I want to share only me experience. After switching to a Pi 4 Openhab is much better. Of there is a power user like me, Raspberry Pi 4 is recommend instead of a Raspberry Pi 3.
We come over exact to this point that 24/7 use isn’t really a problem, only on low systems get problems with rules caching. Yes, and it’s pain in a** if you develop a Blockly rule… Every test need 20 sec and that drive me totally crazy. I will say, I that new guys to openhab aren’t really happy if the need so many time for developing a rule…
1 Like
In general a 20 second delay for a first run is neglectable. That is true for most rules but there are some rules where such a delay is not wanted such as door bell to play a sound, etc.
I just want to elaborate what @Wikibear pointed out:
If you professionals write a new rule or make changes for an existing rule you probably write the code, make one test, have one error you fix it and you are done.
All the others have tens of errors and the log file is showing up just a few of them. You have a LOT of iterations of fixing code and running the script. Sometimes you just forgotten a “;”. After you corrected the code another simple error where you missed closing parenthesis pops up. After your code is syntactically correct you run your script and realize that the result is not the one you expect. So you need to add/change your code again and very often you realize that your first approach is too complicated and have a new idea to make it better. So you start again from the begining or somewhere inbetween.
In THIS particular case it is NOT neglectible, it is really an annoying situation to wait in half an hour of code fixing 30 times 20 seconds. This is no complaint, I just wanted to give you the view from non-professional coders.
2 Likes
I think that’s premature. We don’t know yet if what is being proposed will improve the situation and if so, by how much. We have pretty good indications that it will work but we shouldn’t go about changing long made recommendations until we have gathered all the necessary info.
If moving the GraalVM JRE does improve things, we will need coordinate a number of changes across a number of repos.
When we know, we will make this known somehow.
Just for clarity, the “there” is the issue @florian-h05 posted above.
There is an issue with all JS Scripting rules where the first run can take a significant amount of time because of the injection of the helper library (any libraries really). There is some evidence that the rules themselves run slower than the other rules languages as well.
The current hypothesis is that if we run on GraalVM instead of OpenJDK we should see a significant improvement in both of these, definitely the first. However, GraalVM does not have compiled versions for 32-bit ARM.
I would say it’s a sizable minory, if not the outright majority.
No, that’s how I’d put it and I’ve been promoting those two that way for some time now, especially warning against the use of Jython.
It is also when editing. If it takes 20-30 seconds every time you hit save for that first execution of the rule to start, that’s more than just a minor annoyance.
You can compile a 32-bit version yourself if you want, I think. I never tried it and don’t know if there is some library they depend on that only has 64-bit support. But as far as I can tell, GraalVM has never released 32-bit ARM builds. It’s not retroactive, they’ve just never supported it in the first place. I don’t think that SBCs like RPi is really on their radar as a typical user.
4 Likes
@florian-h05 , @rlkoshak , what do you think of the following “workaround”, which helps at least to “solve” the first run delay after a restart and after changing a rule. These delays are annoying especially for sensitive rules like play door bell sound or a MainUI widget which starts a runRule action, etc.
(However, this workaround does not solve the 20 seconds delay while editing/saving a rule).
The idea is as follows:
- tag all “sensitive” rules
- start a rule at systemLevel 80 which does a runRule() of each rule tagged:
var RuleManager = osgi.getService("org.openhab.core.automation.RuleManager");
var javaRules = osgi.getService("org.openhab.core.automation.RuleRegistry").getByTag("cache");
var jsRules = utils.javaSetToJsArray(javaRules);
jsRules.forEach(r => {
if (RuleManager.isEnabled(r.getUID()) {
console.log("Info: Rule",r.getName(),"(", r.getUID(),") will be cached");
rules.runRule(r.getUID(), {'key0': r.getUID()});
}
});
As we do not really want that the code in our tagged rule is executed we just need to make sure the rule knows it is started just for caching reason:
if ((typeof key0 !== 'undefined') && (key0 == ruleUID)) {
//do nothing as we just want to enforce a cache
} else {
//original script goes here
}
For rules which need to be selectively cached there will be a simple list-item widget which dos a runRule action of the edited rule.
Before implementing this I thought better assk experts if they might have concerns or better ideas.
So that’s what you were working on. 
There are some caveats to this approach. The rules being called need to handle event not being defined (if they are UI rules, I’m not certain for file based rules) or else your log is going to become loaded with errors.
But it seems like it should work.
If you have rules triggered by Thing status changes or Item state changes, this period can be inundated by events. It might be prudent to wait for runlevel 100 for things to calm down, or perhaps even a little bit after (via a timer). I created Delay Start [4.0.0.0;4.9.9.9] for just this reason; it disables “noisy” rules at startup then reenables them sometime after. I had a bunch of sensors that were ponding a rule during startup as they all nearly simultaneously report their new states when the Things come online.
You can use the streaming (or what ever it’s called in JavaScript) to streamline (pun intended) the code here.
Also, you’ll probably want to call the rules twice, once with the flag to consider the rule conditions and again to not consider the conditions. Rule conditions can be scripts too and if you don’t consider them, that code won’t be executed and therefore won’t be cached. But if you do consider them, the script actions might not be executed if the conditions aren’t right and those won’t be cached.
jsRules.filter( rule => RuleManager.isEnabled(rule.getUID()) )
.forEach( rule => rules.runRule(rule.getUID(), { 'caching': true }, true );
.forEach( rule => rules.runRule(rule.getUID(), { 'caching': true }, false );
I have too loops there because a rule calling another rule is not thread safe. When the same rule is called too fast I still see the Multithreaded exception. I’m assuming by looping twice there will be enough time between the two calls to avoid that.
Definitely use a more meaningful variable name than "key0".
There is another approach that would also work for manually triggering the rule which is what you’d want to do after modifying it usually. If it’s important the rule run fast that first trigger after startup, it’s still important after modification.
When the rule is triggered, as of OH 4 at least, event is always defined (including time based and system started triggers). If event is undefined that means the rule was triggered by the user pressing the play button or it was called from another rule. Therefore, in the called rule you can use:
if(this.event === undefined) {
console.info('Caching');
}
else {
}
You can see examples of this approach in many of my rule templates. There I just check the config of the rule and the Items the rule uses when manually triggered. I’d recommend this approach though as it will work both when triggered from a rule and when the rule is run manually.
Though, that won’t help for rules that are normally run this way but that’s a less common use case. Those rules can look for variable passed from the calling rule if it needs to do something different when called from another rule or manually.
Note, the above if statement (or equivalent) needs to be added to all the scripts in a given rule.
I’m not sure this will work for Blockly users though without using the inline script block (or a special Blockly library with a new block deployed to the marketplace too. There is no undefined block so there’s no way to test if event exists. Or maybe there is a way I don’t see?
Maybe instead we could add an “exists” option to the “contextual info” block. That would be useful beyond just this one use case. I might file an issue for that after thinking about it.
It would be best if this were deployed as a rule template. Then end users can just install it and configure it. The only configuration parameter would be the name of the tag so it would be a simple template overall.
As a template it would look something like (note this is a copy/paste/edit, it likely contains typos):
uid: cache-rules
label: Cache Rules
description: Runs rules tagged with the configured tag at system startup to avoid the initial load delay when they are triggered later on.
configDescriptions:
- name: tag
type: TEXT
label: Tag
required: true
description: Rules with this tag will be called by this rule at startup.
triggers:
- id: "1"
configuration:
startlevel: 100
type: core.SystemStartlevelTrigger
conditions: []
actions:
- inputs: {}
id: "3"
configuration:
type: application/javascript
script: >-
var RuleManager = osgi.getService("org.openhab.core.automation.RuleManager");
var javaRules = osgi.getService("org.openhab.core.automation.RuleRegistry").getByTag("{{ tag }}");
var jsRules = utils.javaSetToJsArray(javaRules);
jsRules.filter( rule => RuleManager.isEnabled(rule.getUID()) )
.forEach( rule => rules.runRule(rule.getUID(), { 'caching': true }, true );
.forEach( rule => rules.runRule(rule.getUID(), { 'caching': true }, false );
type: script.ScriptAction
1 Like