Blocky/JS unusable (showstopper delay on 1st run): please advice for alternative solution

[Dami1 (Damien)]

Hi,

I’m currently running OpenHab on the followings:

• Hardware: Raspberry Pi 3 Model B Rev 1.2
• CPU: armv71, 4 cores Cortex-A53, 600 MHz
• RAM: 1Gio
• storage: high speed SDHC card 32Gio
• OS: “Raspbian GNU/Linux 11 (bullseye)” (openhabian)
• Java Runtime Environment: 17 (openjdk-17-jre-headless:armhf 1:17.0.9+9-1~deb11u1+rpt1)
• openHAB version: 4.0.4-2

I got a lot of push button devices, e.g. small remote with 4 buttons which I use to trigger actions using scripting. E.g. one button trigger will send a ON / sleep 1s / OFF to an item controlling a contact wired to the garage door contact (simulating a press to a push-and-release button). Another button trigger will send similar “on” on to a contact to open the portal. Another to send “on”, wait for XX secs so it’s almost opened, send ON / sleep 1s / OFF again to stop it so it will be left opened and won’t close automatically, etc.
Today my rules are in written in blocky (which is great btw).

I’m experiencing a “show-stopper” huge delay when pressing a button, thus running the associated rule, for the 1st time after either openhab has been restarted, the rule has been updated, or similar.

After searching, I realized that this is due to the load of the helper library the first time the rule is updated / openhab restarted / etc. And that it won’t be fixed (which honestly desperates me)

This delay is definitely a show stopper for me. Regularly, when pressing the remote’s button to open the garage door, nothing happens (or so it seems) so my wife press the button again, gets angry saying “your stuff does not work” ! Well, joke aside, I got a lot of triggered based rules with zigbee buttons or remote which cannot suffer such a delay that, from user experience, is happening randomly. It is not thinkable to run all rules once after each openhab restart as I saw as resolution suggestions!
To me this delay is simply unthinkable. If something long would have to be done for rules to be executed normally, it should be done at start time, or right after the rule is updated. Not at use time. Anyway.

So I consider two solutions: giving up on openhab and moving to another solution ; all my items are zigbee based with zigbee2mqtt so I got no technical issue with that.
Or re-writing all my rule scripts in another language than blocky / javascript that would not have this delay issue.

This is why I’m writing that: what would you advice as an alternative, if any exists with similar capabilities, that won’t have this unbearable delay ? I’m ready to learn a new language if that is similar in effort than giving up on openhab and re-installing all my automation in another tool (basically my decision will be made on that particular effort!)

Thanks for your answer

[sl4m (Simon)]

Dear Damien, i don’t have a solution but i do have the same issue. I was hoping that this is a bug which will be fixed.
I am restarting my openhab server regularly (because the UI is freezing), after this all rules are delayed by roughly 30s. This is annoying because my wife and the kids are pushing buttons and nothing is happening (exactly the same like Damien mentioned ) but there are also rule like movement alarm video capture of the outside camera. These rule will not work for the next alarm.
I ended up buying myself a new Rasperry pi 4 with 4gb RAM, hoping the UI freeze are because of my old Rasperry pi 3b with 1gb RAM. Hoping that i dont have to restart openhab that frequently anymore. But still, opening a rule or doing a small adaption to rule or script means that some rules will not work properly the next run…
I heard that only the 32bit image is affected by this. Would it make sense (or even possible) to use a 64 bit image for the Rasperry pi 4b?

[scheuerer]

Please read this here:

Blockly rules after upgrade to OH 4.0.1 very slow on the first run

Kinda what I was thinking remember this? (circa 2018) I know you remember Markus, you posted in it

There a some new technical limitation now with OH4 and changes how rules need now external resources for the first time.

But we have some experts here which are involved in this topic. But longer time nothing heard about this…

[Dami1 (Damien)]

Thanks for your answer @sl4m,

Well, as far as I understand, the helper library would still have to be loaded the 1st time the rules are executed, even on a Raspberry 4 with 4 Gio of RAM. Probably faster, but still with a delay (I would prefer that the developers do not decide for users that this delay is “not a big problem” and think about solution, such as automatically loading the helper library at start time and after a rule update, not at rule use time).

Would it make sense to buy new hardware / re-install / move current config to it ? Yes, if I’m guaranteed that the delay issue would be fixed, meaning that the still existing 1st time delay will be reduced to something like less than 1s.
So that’s the point: will 1st time execution of rules be drastically reduced, something like below 1s, on a Raspberry 4 ? I doubt that but I’m open to users who did this change’ experience feedback !

I started on OpenHAB years ago and I think it’s a great software, but if I cannot fix this issue, which makes basically all the cool stuff on home automation appear to randomly fail, I’ll move to something else. I started to have a look, home assistant seems promising.
Or if I can fix that within openhab, e.g. using other scripting language that would not be too difficult to use and not have this issue, that’s another option.

[Dami1 (Damien)]

Thanks for your answer @scheuerer,

I’ve read a bit the post you referenced.
Some statements in it make me crazy, honestly …
" I’d say that on global (all-OH) level, that is not a major nuisance. "
" That should be neligible on a system designed to run 24x7 "
" In general a 20 second delay for a first run is neglectable "
No it’s not ! When you press a button and expect an instant action, a 20 second delay is not neglectable at all, it’s “this stupid software does not work” !
That’s the reason why I’ll give up on OpenHAB and move to another solution if I do not find a way to fix it, definitely NOT neglectable!

But I also found some hope in some other statements
" I want to share only my experience. After switching to a Pi 4 Openhab is much better. "
" Blockly “compiles” to ECMA Script 2022, which is run by the JavaScript Scripting (GraalJS) add-on. GraalVM itself does not support 32-bit "
=> does that mean that switching to a Raspberry 4, 64 bits, would trigger the use of GraalVM and completely remove this showstopper “helper library loading” at 1st rule execution time ?

It is not clear, actually.
So that may be my next question: will switching to a Raspberry 4 (or 5 now it’s out) with 4 Gio or RAM FIX this issue for good ?
Is anyone capable of being crystal clear on that ?

If yes, then I’ll buy this new hardware and keep OpenHAB. I would also advice to clearly say in OpenHAB docs that blocky/JS is incompatible with Raspberry 3 hardware, BTW. When asked for advice, I’m currently discouraging to go to OpenHAB because of that issue.

If no, my initial question still stands: is there an alternative language I could consider for rules that would NOT suffer from this showstopper issue ?

Thank you again, thanks for the time you spend answering, really appreciated.

[scheuerer]

@rlkoshak, I’m sure you find the right answers, for all thoose statements and questions…

[rlkoshak (Rich Koshak)]

@Dami1

And that it won’t be fixed (which honestly desperates me)

What makes you think it won’t be fixed. There is an issue open and some activity has taken place. There is preliminary evidence that running on 64-bit Java or the GraalVM JRE improves the performance by 10X.

@Dami1

Or re-writing all my rule scripts in another language than blocky / javascript that would not have this delay issue.

Doesn’t have to be all. Just those that need to run fast every time.

Though adding some system started triggers and logic to the existing rules to prevent errors seems like a whole lot less work. I’m not sure why that approach is “unthinkable”.

@sl4m

I ended up buying myself a new Rasperry pi 4 with 4gb RAM, hoping the UI freeze are because of my old Rasperry pi 3b with 1gb RAM.

The use of SWAP and high system loads can also cause rules to be delayed when running.

@sl4m

But still, opening a rule or doing a small adaption to rule or script means that some rules will not work properly the next run…

Manually trigger the rule after you save it or add a system runlevel trigger to the rule and get that first delayed run over with right there rather than waiting for a time when you need it to run fast.

@sl4m

Would it make sense (or even possible) to use a 64 bit image for the Rasperry pi 4b?

It’s even possible on an RPi 3, though RAM will be even more of a problem. The current recommendation is to use 64-bit Java (which requires 64-bit OS) or GraalVM. If you do please report if it improves this problem to [jsscripting] GraalVM's optimised compiler or find another solution · Issue #15600 · openhab/openhab-addons · GitHub.

@Dami1

Probably faster, but still with a delay (I would prefer that the developers do not decide for users that this delay is “not a big problem” and think about solution, such as automatically loading the helper library at start time and after a rule update, not at rule use time).

At this time the only viable solution is to use some other rules language. Rules DSL also experiences a delay on the first run for different reasons. jRuby does not appear to have this first run delay.

But again, I don’t know where you get the idea that no one thinks this is a problem and is ignoring it.

@Dami1

Yes, if I’m guaranteed that the delay issue would be fixed, meaning that the still existing 1st time delay will be reduced to something like less than 1s.

There are too many variables. No one can guarantee something like that.

@Dami1

Some statements in it make me crazy, honestly …
" I’d say that on global (all-OH) level, that is not a major nuisance. "
" That should be neligible on a system designed to run 24x7 "
" In general a 20 second delay for a first run is neglectable "
No it’s not ! When you press a button and expect an instant action, a 20 second delay is not neglectable at all, it’s “this stupid software does not work” !
That’s the reason why I’ll give up on OpenHAB and move to another solution if I do not find a way to fix it, definitely NOT neglectable!

At the risk of being offensive: Opinions are like assholes. Everyone has one and most of them stink.

Those are statements from one person. No one person, including me, can speak for all of OH.

OH is an all volunteer community effort.

@Dami1

=> does that mean that switching to a Raspberry 4, 64 bits, would trigger the use of GraalVM and completely remove this showstopper “helper library loading” at 1st rule execution time ?

No but running using GraalVM instead of OpenJava seems to improve performance by 10x. Though a similar improvement comes from running on 64-bit Java too.

@Dami1

So that may be my next question: will switching to a Raspberry 4 (or 5 now it’s out) with 4 Gio or RAM FIX this issue for good ?

We don’t know the answer to this question. It seems to provide a 10x improvement but not everyone whose tried it see that improvement (though other things can cause the problem so it’s hard to say why). No one can guarantee any problem will be fixed for good.

On an x86 based Intel VM the initial delays are sub one second for me at least. I never personally noticed this problem until it was reported on the forum. The main developers of the JS Scripting Addon and Blockly mainly use RPis 4 and they didn’t really notice this problem until it was reported, but they run 64-bit. So that is some evidence that 64-bit makes a huge difference.

@Dami1

If yes, then I’ll buy this new hardware and keep OpenHAB. I would also advice to clearly say in OpenHAB docs that blocky/JS is incompatible with Raspberry 3 hardware

That’s among the discussions going on once we have enough evidence that switching to 64-bit actually does fix the problem.

@Dami1

If no, my initial question still stands: is there an alternative language I could consider for rules that would NOT suffer from this showstopper issue ?

Depending on how the rule is written and the machine Rules DSL often has less of a delay.
jRuby reports they do not have such a delay.
Groovy does not have this delay.
The various Java Rule add-ons that are on the marketplaces do not have this delay, or at least haven’t reported this delay.
Simple UI rules do not have this delay.
Nashorn JS does not have this delay.

I do not know if there is a difference between UI rules and file based JS Scripting rules. File based may not have the same delay.

[sl4m (Simon)]

Hello Rich, thank you for youe detailed answer!

Thats a great idea, didnt think about this before:

@rlkoshak

Though adding some system started triggers and logic to the existing rules to prevent errors seems like a whole lot less work.

I will update to a Raspberry pi 4 and 4gb RAM and 64bit openhabian tomorrow. Will report back if it improved the initial rule performance.

[Dami1 (Damien)]

Thanks @rlkoshak for your detailed answers !
I must say I did not expect such good answer, thanks a lot. As you said, “OH is an all volunteer community effort.” and so, your support here all is outstanding !!

@rlkoshak

What makes you think it won’t be fixed. There is an issue open and some activity has taken place. There is preliminary evidence that running on 64-bit Java or the GraalVM JRE improves the performance by 10X.

Well, that’s what I understood from discussions on this – this issue being considered as “not a big deal” – my apologies ! your remark “there is an issue open” is music to my ears

@rlkoshak

Doesn’t have to be all. Just those that need to run fast every time.

Yes, sure. It’s just that in my case, all the rules I created should be executed instantly. So here, “all rules” is a specific statement for my specific use of OpenHAB.

@rlkoshak

Though adding some system started triggers and logic to the existing rules to prevent errors seems like a whole lot less work. I’m not sure why that approach is “unthinkable”.

As a matter of facts, my first reaction to this solution was a bit too hasty. e.g. on the rule that respond to a button of the zigbee remote to open the garage door: I cannot think about schedule it to be executed at system startup and have my garage door opened then! But after my first reaction, reading more carefully the other post where @Oliver2 suggested to add a test in rules, and if executed at system startup do nothing – just be executed so that the helper libraries are loaded and next calls have normal respond time. That would require to update my rules, but is definitely a clever workaround idea. I’ll probably try that next and keep you posted. As you pointed, it’s a less work try than the other solutions.

@rlkoshak

At this time the only viable solution is to use some other rules language. Rules DSL also experiences a delay on the first run for different reasons. jRuby does not appear to have this first run delay.

But again, I don’t know where you get the idea that no one thinks this is a problem and is ignoring it.

Again, my apologies !

@rlkoshak

At the risk of being offensive: Opinions are like assholes. Everyone has one and most of them stink.

Those are statements from one person. No one person, including me, can speak for all of OH.

OH is an all volunteer community effort.

Wasn’t that a line from “Inspector Harry” ?
I understand your reaction and apology: I may have been too offensive myself (and provocative… I apology for that too. I did not hope an answer, actually, and certainly not one so fast and so complete.)
I should have written something like there are indeed use cases where this delay issue is not a problem, and there are use cases where it definitely is (such as mines).
For users who don’t have such use cases, it’s not worth spending time and efforts on this. For others yes. How many users like me ; is it worth the time ? None can answer, as you said, no one can speak for all of OH.

@rlkoshak

@Dami1

If yes, then I’ll buy this new hardware and keep OpenHAB. I would also advice to clearly say in OpenHAB docs that blocky/JS is incompatible with Raspberry 3 hardware

That’s among the discussions going on once we have enough evidence that switching to 64-bit actually does fix the problem.

At some point in time, I’ll buy a new raspberry 4 and switch. I’ll keep you posted on what is the impact on this delay issue for blocky/JS rules execution. Probably not right away (I actually already have a raspberry 4 used as a TV box, running kodi. I’ll probably switch this one to a raspberry 5 when the kodi dedicated distribution I use (namely librelec) supports it, and get this rasp 4 for OH then).

I think that I’ll first try to implement the system startup workaround, I’ll keep you posted on this too.

[rlkoshak (Rich Koshak)]

@Dami1

: I cannot think about schedule it to be executed at system startup and have my garage door opened then!

Add

if(event !== undefined || event.type == 'WhatEverEventTriggersTheRuleNormally') {
    // rule code goes here
}

You can log out event.type to see what “WhatEverEventTriggersTheRuleNormally” is.

I believe the ability to test event for undefined was added to blockly, though it may have been in 4.1. In that case use two inline script blocks, one with the if statement at the top of all the blocks and the closing curly bracket in one at the bottom of the blocks.

[sl4m (Simon)]

Dear all, today i switched from RPI 3b+ 1gb RAM with 32 bit Openhabian (99% RAM memory used) to a RPI 4 4gb RAM with 64 bit Openhabian (56% RAM memory used). Both Openhab 4.0.4.
First of all it was a nightmare because zigbe2mqtt lost the connection to most my devices. Also zigbee switches behind wall switches … so i had to remove wall plugs in order to re-pair devices
But after i restored all backups things are runing way smoother.
To answer the rule performance question: With the RPI3b+ i had a delay of 20-30s for the initial operation of a rule. With the RPI4b (and 64 bit) it is 4s. Till now i also had no UI freezes anymore, so that i think i dont have to reboot the server that often. Which means i dont have a problem with delayed rules anymore!

To sum it up: Yes switching to a new system is work, but i think its worth it.

[Dami1 (Damien)]

Thank you so much both for your inputs.
@sl4m : good to know that with RPI4 I can expect a drop from 20/30 s I got today to 4s. Still, 4s seems a bit high for my use cases, while much better than 30s anyway!

I also spent a bit of time and came to a workaround solution that I believe is working, while still keeping all 100% with blocky.
I’ll put details in there later when I have time to and when I completed further tests to validate it works in all situations.

[Dami1 (Damien)]

OK, so this is how I did fix this issue.
I did that before @rlkoshak suggested to test the event, so I may revisit the solution to remove the use of the item’s metadata later. Anyway, it works like that:

1. I created a dedicated item called “Load Scripts and Rules”
   

=> this item is used to

• have a button to press to trigger the 1st, “blank”, execution of scripts and rules that have the JS helper library load time issue – when a rule is updated I can either run it or press this button and let it work

• Trigger that at start time, which is the true resolution action

• Share information between several scripts/rules using this item’s metadata

2. I created a script that is intended to run all scripts and rules “blank” (not actually doing something, but only for the scripts/rules to be executed and have the helper library loaded if needed).

This script is executed by a rule when the previous button receives “ON”

This script is executed by a rule after openhab service has started (level 100)

3. what’s in this script content:

pretty simple, actually.
a loop on all scripts or rules IDs that need to be executed once at start time. Currently I put only 3 of them (see example below) but I’ll extend to almost all my scripts/rules – all that needs to be executed without the 30s delay.

The idea is to put the string “load” in the special item’s metadata namespace named after the script/rule ID.
It’s the way to share information between this script and the ones that are executed by it.

4. I updated all scripts/rules to test the string in the special item’s metadata namespace named after the script/rule ID, so if it’s “load” it does nothing and update the namespace value to “run” for next run to do the stuff.
   => I probably can remove that and implement the “event” test, as suggested by @rlkoshak – I’ll check that particular idea later.

example of a rule updated with that test:

[Dami1 (Damien)]

Adding precision: the event idea does not work, as it is defined in the execution of the 1st script (the one that does a loop to execute all others) but is not in the others.
I could probably change the design and create a rule for each script/rule to be executed when the system reaches level 100 (end of openhab start) and then, test the event that would probably be defined, but it would be more work than the current design, so I keep things as it is now.

Thanks all for valuable answers/support/information I found in this forum.

[rlkoshak (Rich Koshak)]

This is a pretty clever approach. I think there might be a slightly cleaner way to do it this approach without the Item metadata.

When you call a rule from another rule, you can pass data to the called rule. That’s what the “with context” part of the block does. Instead of setting metadata, which is permanent and survives a restart which means it will eventually need to be cleaned up, you could pass a flag in the context when you run the other rule and use the “get context attribute” in the called rules.

I don’t have easy access to screenshots at the moment but from the calling rule you’d click on the gear for the “create empty dictionary” and add something like “init = true”.

Then in all the called rules you’d use “if not get context attribute init” to see if this rule was called by the initializing rule. I think that approach would be a little cleaner than using metadata because in the long run you’ll have to add logic or have to remember to remove rules from the Item metadata if you ever want to remove a rule from this initialization behavior. It also means you don’t have to do as much work in the called rules as all you have to to is test for init. If init isn’t defined, the if condition will pass and the body of the rule will run (that’s why the “not” is in there). You don’t need the else or the blocks to reset the metadata.

Why not just have the one rule and have it triggered both with the Item and the system start level? Besides commanding OFF to the item_load_scripts_and_rules Item action are they any different?

If you use the context to pass data, you also don’t need the item_load_scripts_and_rules Item any more. You could have just the one rule that is triggered by the system runlevel which, when you need to initialize the rules you can run manually without commanding an Item.

@Dami1

Adding precision: the event idea does not work, as it is defined in the execution of the 1st script (the one that does a loop to execute all others) but is not in the others.

I use this idea in my rule templates so I know it works. Show what you tried.

The fact that it doesn’t exist is kind of the point. You need to test whether event === undefined which is what I do in my templates but I don’t know if Blockly supports directly.

It looks like testing with event It turns out it doesn’t matter. I just ran a test, and in OH 4.1 M3 at least “get context attribute ‘event’” has something meaningful in it for manual and runRuleAction triggers. So you could use

if get context attribute "event" != "manual" and get context attribute "event" != "runRuleAction"
    // code goes here

I also confirmed that the following works when it doesn’t exist:

if contextual info event type != null
    // code goes here

It looks like the event type is undefined when the rule is triggered with a system runlevel trigger. So you could cover all the bases with:

if get context attribute "event" != null and get context attribute "event" != "manual" and get context attribute != "runRuleAction"
    // code goes here

With that if, the “code goes here” will only execute when the rule is not triggered with a system runlevel event, manually, or run from another rule.

Experiment to see what the event context info is under all the triggers you may need but I think these three would be sufficient to cover all the bases. They will let you initialize the rule:

• by manually running the rule
• by adding a system runlevel trigger (note that with a system runlevel trigger the rule will run immediately upon being saved, saving you the extra step
• by running the rule from another rule

You don’t have to command an Item or go somewhere else to initialize the rule as well as creating a system run level rule to run them at system started.

One thing I like to do is instead of just skipping the rule action I actually perform some tests on the rule or configuration the rule depends upon.

[Dami1 (Damien)]

Hi Rich,

@rlkoshak

This is a pretty clever approach. I think there might be a slightly cleaner way to do it this approach without the Item metadata.

When you call a rule from another rule, you can pass data to the called rule. That’s what the “with context” part of the block does. Instead of setting metadata, which is permanent and survives a restart which means it will eventually need to be cleaned up, you could pass a flag in the context when you run the other rule and use the “get context attribute” in the called rules.

Yes indeed, I did not know this way of passing information from one script calling another one, but that is much better than using the metadata on an item. Using metadata works but it’s certainly slower, as it stores and retrieve the information from somewhere in the disk – metadata being persistent.

So, the “load scripts and rules” script executed at level 100 after openhab service is started became:

and in each script the test to run it “blank” and do nothing but load the helper library in memory became:

or, simpler, as you suggested:

=> there is no “loading script” log message anymore, but now we do know it’s working, that is not needed anymore.

That’s better, lighter and faster. Thanks for the tip!

@rlkoshak

If you use the context to pass data, you also don’t need the item_load_scripts_and_rules Item any more. You could have just the one rule that is triggered by the system runlevel which, when you need to initialize the rules you can run manually without commanding an Item.

Yes, the item_load_scripts_and_rules is not actually needed (no use of metadata requiring an item). At startup, the “load scripts and rules” script is executed on event “level 100 reached”.
We can keep it to have a matching button in some UI page to manually trigger the “load scripts and rule”, or not, that’s whenever we like.

@rlkoshak

@Dami1

Adding precision: the event idea does not work, as it is defined in the execution of the 1st script (the one that does a loop to execute all others) but is not in the others.

I use this idea in my rule templates so I know it works. Show what you tried.

The fact that it doesn’t exist is kind of the point. You need to test whether event === undefined which is what I do in my templates but I don’t know if Blockly supports directly.

Well, yes it works – what I meant is: it’s not applicable here. When a script is executed from another script, the event is not defined. In clear: when the “load scripts and rules” script is executed by the rule “start level 100 reached”, there is an event defined in it. Then it runs all scripts/rules we need to run “blank” and in them, the event is not defined. We could imagine test “if event is not define” then run “blank” but … some of these scripts are atomic actions that are also called from other scripts… therefore must “do something” in that case, where the event won’t be defined either.
The event idea is therefore not the right one to use in that use case. The run with context definitely is!