Java223 Scripting - Script with Java on openHAB [5.0.1.0,6.0.0.0)

Add-on Marketplace Bundles
, ,
41

I think you misunderstood me - I haven’t criticized the locks, I’m rather pointing out that after the initialization, it seems that multiple Python scripts can run at once (the initialization part should be extremely quick except the first time), and with no tools to actually handle synchronization from the scripts (and potentially as @rlkoshak points out - that it’s too difficult for most to handle), this seems a bit risky to me. Maybe the entire script execution should be synchronized, like it is with JavaScript, because that’s the only way to make this “safe”.

With Java, as was the original topic here, where proper synchronization tools are available, and maybe a higher level of knowledge can be expected, I think the situation is different.

42

It is the same way as in java script. It was inspired by the java script addon.

43

I’m thinking of this lock:

It will conditionally synchronize on the script engine instance itself, if the instance implements Lock. That means that effectively all execution with this engine is serialized (assuming that a similar mechanism is implemented everywhere in Core where a script is launched - I haven’t checked that), and it seems this applies to the Python engine as well as JavaScript:

So, the initialization lock isn’t strictly relevant to this, because it doesn’t enforce that only one script can run at the script engine at the time, but this locking mechanism does. This means that there’s no need for synchronization/locks to prevent concurrent access between invocations of the same script engine. It’s not fully watertight though, because you can still access objects from Core that might need synchronization/locks, and there’s still no way to do this (or maybe there is a way to invoke Java synchronization objects via JSR223 - I wouldn’t know).

But, for the most part, I think this will work without too many issues, the Core APIs mostly make defensive copies before handing out mutable objects.

Potential scriptengine deadlock
44

Just FYI:

1 Like
45

On Concurrency

In OpenHAB/JSR223

Concerning concurrency in scripting (rule) engines I created rules_overview.md: elaborate about concurrency support in rule engines by dilyanpalauzov · Pull Request #2565 · openhab/openhab-docs · GitHub , stating that for GraalVM (JavaScript and Python) only a single rule or transformation can be executed at any time. Is this correct?

For clarification, if automation/jsr223/seven.py (or seven.js) installs seven rules and automation/jsr223/three.py (or three.js) contains three rules, does the implements Lock in org.openhab.automation.pythonscripting.internal.PythonScriptEngine (and org.openhab.automation.jsscripting.internal.OpenhabGraalJSScriptEngine) ensure that at most one out of ten rules are executed at once, or does it mean, that one of the seven rules is executed, and at the same time one of the three rules can be executed? That is, does splitting one file in automation/jsr223/.(js|py) into many, independent files, each of which installs a rule, increase the parallelism?

Locks in threads, explicitly started from a automation/jsr223/.java file

The current statement is that for the files automation/jsr223/.java allowInstanceReuse cannot have side effects, as these files are executed only once. But these files can install rules, contained in the jsr223/.java files, and rules can be executed in parallel. For me it is unclear, when a jsr223/.java file installs and thus contains a rule, is then allowInstanceReuse relevant for the rule inself, but irrelevent for the jsr223/.java file?

Locks in Rules

As an example for the above, I have a DSL rule, which changes the color of a sitemap item, depending on how long a device is on:

Text item=e_power labelcolor=[e_color=="yellow"="yellow",e_color=="gold"="gold",e_color=="olive"="olive",e_color=="orange"="orange",e_color=="red"="red",e_color=="maroon"="maroon"]

I have the items DateTime e_last_change, String e_color, Number:Power e_power.

var eLock = new java.util.concurrent.locks.ReentrantLock
var Timer timer
val ir = org.openhab.core.model.script.ScriptServiceUtil.getItemRegistry

rule "E Power Change"
when
  Item e_power changed
then
  if (previousState != NULL && Math.abs(e_power.getStateAs(Number).intValue - (previousState as Number).intValue) > 10) { // the device is on, if it uses 60 W
    val prev = (previousState as Number).floatValue < 60
    val cur = e_power.getStateAs(Number).floatValue < 60
    if ((!prev && cur) || (prev && !cur))
      e_last_change.postUpdate(new DateTimeType)
  }

  eLock.lock
  try {
    if ((e_power.state as Number).intValue < 5) {
      if (timer !== null) { timer.cancel; timer = null }
      if (e_color.state != NULL) e_color.postUpdate(NULL)
      return
    }

    timer = createTimer("Set E color", now.plusMinutes(4)) [|
      eLock.lock
      try {
         val last_mod = (e_last_change.state as DateTimeType).getZonedDateTime(ZoneId.systemDefault())
         var b = ZonedDateTime.now
         for (StringType c: newArrayList(new StringType(''), new StringType('yellow'), new StringType('gold'), new StringType('orange'), new StringType('maroon'))) {
           b = b.minusMinutes(4)
           if (last_mod.isAfter(b)) {
             if (timer !== null) timer.reschedule(now.plusMinutes(1))
             e_color.postUpdate(if (c == '') NULL else c)
             return
           }
        }
      } finally {
        eLock.unlock
      }
      if (e_color.state != "red") {
        e_color.postUpdate("red")
        timer = null
      }
    ]
  } finally {
    eLock.unlock
  }
end

The above starts a thread from a rule and then loops and sleeps in that thread, changing the color in each iteration. I do not remember the exact reasons, but I had to use locks to get this right. If I convert the above to a Java223 rule (automation/jsr223/e_color.java), provided that allowInstanceReuse is not relevant for automation/jsr223/*.java, do I still have to use locks, irrespective of allowInstanceReuse setting?

Another example for possibly required lock in a rule - Rules · openhab/openhab1-addons Wiki · GitHub - “If a rule triggers on UI events it may be necessary to guard against concurrency.” … in DSL. How about in Java223?

Removing allowInstanceReuse option from Java223

I think the multi-threading considerations in transformations and rules are all the same for all JSR223 rule engines, with the exception of GraalVM (Python and JavaScript), which are single-threaded. For all the other rule engines users have dealt so far fine with multi-threading/locking. For this reason I suggest removing the allowInstanceReuse option. If multi-threading in rules requires clarifications to get right, I think the right place is to describe this at JSR223 Scripting | openHAB or Rules - Other Technologies | openHAB. Honestly, having a choice between writing OH rules in many languages, who is at the same time going to choose Java and have difficulties with concurrent programming? The option allowInstanceReuse is only meant for those people, who choose Java, write code, which shares values between executions, and at the same time cannot programme threads.

My current understanding, based on the current README.md on github (last changed by me), is that allowInstanceReuse allows sharing values between invocations, but only when a value is shared between invocations, there can be multi-threading issues, so in this case - when allowInstanceReuse offers advantages - allowInstanceReuse should not be used. If this option is not removed, having an example in the add-on documentation, explaining when a problem with allowInstanceReuse can happen, and how this problem can be solved (no allowInstanceReuse or demonstrated thread synchronization) would be very good.

Loading automation/lib/java files

The log is erroneous. I changed it to
Received ‘{}’ for path ‘{}’ - ignoring (wrong extension or unused event kind)

I see no changes uploaded on github. In the case, where a user puts a file at automation/lib/java/u.java this file should not be ignored, until the newly created file is changed, but instead read. Currently it is ignored.

Inconsistent logging of syntax errors

I agree that openhab-core logs inconsistently ScriptExceptions (syntax errors) - sometimes logs and sometimes does not log. For transformations in channels I have opened ChannelTransformation: log exact syntax errors in transformations by dilyanpalauzov · Pull Request #5043 · openhab/openhab-core · GitHub. It does add special handling of ScriptException from TransformationException, but also says, that syntax errors in transformations, which are not from a rule engine, and originate e.g. from the JSONPATH transformation, are still not logged.

About dependencies for transformations vs scripts

Changing a lib dependency SHOULD force recompilation of script that declares depending on this library. It works for “standard” script (i.e. I tested a script created from the GUI in the “script” section, and it runs every time with updated lib), but it doesn’t work for a transformation script.
I debugged a little, and I found that transform scripts are not registered in the dependency tracking system, at all. …
I don’t use other JSR223 services, but by reading code I’m pretty sure it won’t work for them either.

I filled at Transformation with ScriptEngine do not have dependecy change and do not recompile on dependency change · Issue #5046 · openhab/openhab-core · GitHub that in case of JavaScript(GraalVM) changing dependencies of a transformation does not update the transformation, until the source code of the transformation is updated.

Here a different test:

I create automation/lib/java/u.java with content

public class U {
    public static String v() {
        return "R";
    }
}

and under automation/jsr223/m.java I put:

import java.util.*;
import org.openhab.automation.java223.common.InjectBinding;
import org.openhab.core.automation.Action;
import org.openhab.core.automation.Trigger;
import org.openhab.core.automation.module.script.rulesupport.shared.ScriptedAutomationManager;
import org.openhab.core.automation.module.script.rulesupport.shared.simple.SimpleRule;
import org.openhab.core.automation.util.TriggerBuilder;
import org.openhab.core.config.core.Configuration;
import org.slf4j.LoggerFactory;

public class M {
    @InjectBinding(preset = "RuleSupport", named = "automationManager") ScriptedAutomationManager automationManager;

    public Object main() {
        automationManager.addRule(new SimpleRule() {
            @Override public List<Trigger> getTriggers() {
                return Collections.singletonList(TriggerBuilder.create().withId("trig1").withTypUID("core.ItemStateChangeTrigger").withConfiguration(new Configuration(Collections.singletonMap("itemName", "r"))).build());
            }

            @Override public Object execute(Action module, Map<String, ?> inputs) {
                LoggerFactory.getLogger("logger1").warn("U is " + U.v());
                return null;
            }
        });

        return null;
    }
}

When item r changes, a message is logged: U is R. Afterwards I change in u.java R with T. When item r changes again, the old message U is R is repeated, instead of the new one U is T.

Private-Package: freemarker.ext.ant,freemarker.ext.rhino

After Exclude directories freemarker/ext/{rhino,ant} from the bundle by dilyanpalauzov · Pull Request #9 · dalgwen/openhab-addons · GitHub the Java223 jar file does not include the directories freemarker/ext/ant/ and freemarker/ext/rhino/. But org.openhab.automation.java223-5.0.1.jar:META-INF/MANIFEST.MF contains Private-Package:freemarker.ext.ant,freemarker.ext.rhino,…. I tried utilizing org.apache.maven.plugins/maven-dependency-plugin to avoid in first place extracting the two directories under bundles/org.openhab.automation.java223/target/classes/freemarker/ext/rhino/ (or ext/ant/), but I had no success with this.

46

Core already implements locks that prevent the same rule from running more than one thread at a time. However, this doesn’t apply to timers or rules that call other rules. This applies to all the languages.

I don’t know if it applies to script transformations though.

GraalVM JS requires additional locking to also block execution of timers at the same time as other timers from that same rule or the that same rule iteself from running at the same time. Maybe python has a similar limitation? I do not know, but I don’t think it does.

But from what I know in JS, in either of your scenarios, all ten of the rules could be running at the same time. But if a timer or the same rule is triggered while it’s already running, the lock will prevent it from immediately running until the lock is released. This lock exists in the JS Scripting add-on and happens outside of the rule script.

Splitting them into separate files will not increase parallelism as the separate rules are already running in parallel to the maximum extent possible. All the locking is done in the context of a single rule.

It looks like you need to prevent the timer from running at the same time as the rule. That is indeed not a use case handled by core, so the lock would be needed to prevent that, or choosing a different approach that doesn’t require the timer.

In GraalVM JS, you would not need the lock in this case because the add-on will prevent the timer from running at the same time as the rule already.

I can’t speak to GraalVM Python.

This comes from the fact that when rules are run directly instead of being triggered (e.g. from a MainUI Widget’s action or from another rule) none of the locks are applied. This means it does become possible for two instances of the rule to be running at the same time. When that happens in GraalVM JS, a multithreaded exception will be thrown. Adding locks inside the rule won’t help you though because this exception gets thrown before the rule can start running and they attempt to acquire the lock.

But you can work around this limitation by having a triggering Item instead of calling the rule directly.

47

GraalVM JS requires additional locking to also block execution of timers at the same time as other timers from that same rule or the that same rule iteself from running at the same time. Maybe python has a similar limitation? I do not know, but I don’t think it does.

But from what I know in JS, in either of your scenarios, all ten of the rules could be running at the same time. But if a timer or the same rule is triggered while it’s already running, the lock will prevent it from immediately running until the lock is released. This lock exists in the JS Scripting add-on and happens outside of the rule script.

Is my reading of the above correct, that implements Lock in openhab-addons/bundles/org.openhab.automation.jsscripting/src/main/java/org/openhab/automation/jsscripting/internal/DebuggingGraalScriptEngine.java at main · openhab/openhab-addons · GitHub and openhab-addons/bundles/org.openhab.automation.pythonscripting/src/main/java/org/openhab/automation/pythonscripting/internal/PythonScriptEngine.java at main · openhab/openhab-addons · GitHub have implications only for timers and other threads started from within a rule?

Or is my reading of openhab-core/bundles/org.openhab.core.automation.module.script/src/main/java/org/openhab/core/automation/module/script/internal/handler/ScriptActionHandler.java at main · openhab/openhab-core · GitHub more correct, that for each JS-transformation, JS-rule, or JS-script, first a lock for GraalVM/JS is obtained, then the (compiled) code (e.g. a JS(|input + "inline transformation"):%s) is executed, and then the lock is released? That is: at most one transformation at a time is executed.

48

I cannot speak to the code, only from experience and past discussions with @florian-h05 and others as they implemented these locks. I filed at least two issues (maybe more) to try and eliminate the MultithreadedExceptions as much as possible.

I’m far enough away from Java programming to know what the Lock interface really means. But if I were to guess, and based on the names of those methods, how the lock works largely depends on whether there is only one instance of that class for all rules, or, as is my understanding, there is one instance per rule.

I wouldn’t expect the ScriptActionHandler to be involved with transformations at all. A Script Action is a part of a UI rule.

And again, it depends on whether there is one scriptEngine for all or one per rule.

49

No. There is no parallelism to be had using these languages, they don’t support it/can’t handle it. Your above example means that one in 10 can run at the same time, which probably sounds terrible, but don’t forget that your browser does the same for all the JavaScript it runs (with the exception of “Web workers”), and so does a lot of other software written in these languages. The clue is that the tasks can’t “block” (sleep or wait for answers from external sources), if they don’t “block”, they will usually finish very quickly, within milliseconds, so that 10 rules can still run within half a second.

I don’t know the answer to this, but I’ll add that createTimer() isn’t exactly creating a thread. The timer functionality is something created specifically by OH, and I think it works so that when the timer fires, the “rule thread” actually executes the rule. Each rule in OH has a dedicated thread that runs all Actions (and potentially Conditions too - I don’t remember).

50

The way it used to work was timers pulled a thread from a separate pool from the rules. I do not know if that’s still the case.

51

I was wrong there then, I was really convinced that the timer would trigger a run in the rule-dedicated thread.

Ok, I should probably look at the code before saying anything more on the subject :wink:

This sounds like something that should be “refined”… but it might be difficult if the script has to wait for the execution of the other script to return.

52

This lock is supposed to protect all use of the ScriptEngine itself, regardless of source, so that nothing that calls the ScriptEngine can do so in parallel. If this is what actually happens depends on whether the use of the lock is implemented everywhere where the ScriptEngine is called, but if it is, it applies to everything, regardless of “source” and any other locks that might exist. The “single rule thread” applies independent of this lock.

The lock applies to the ScriptEngine instance, and actually hadn’t contemplated that there might be multiple instances of the same ScriptEngine, but a quick look in the code actually indicates that it probably is. There is a ScriptEngineManager that handles instances of the different script engines, but it looks like transformations might not use it, and instead create its own :open_mouth:

That said, I know too little about JSR223 to know what the implications of this are. It might not pose a problem.

53

That doesn’t surprise me. When you have a JS Script transformation (I don’t know about the other languages), there is just that one transformation no matter how many different places you use it. If you apply the transformation as a profile in half a dozen different Item Channel Links, there is still just the one instance of that transformation.

Obviously it’s really likely that two links might want to call this profile at the same time which would trigger the dreaded MultiThreadedException. So it makes sense that transformations would be locked different from rules, espeically since they are accessed and called different from rules.

54

In case of Java223, if I create a sitemap with three transformations

sitemap b label="B" {
  Text item=x label="U1a [JAVA(|return \"A1q\";):%s]"
  Text item=x label="U2d [JAVA(|return \"A2q\";):%s]"
  Text item=x label="U3d [JAVA(|return \"A3q\";):%s]"
}

Java223ScriptEngineFactory.createScriptEngine(String scriptType) is executed three times and the Java223ScriptEngine constructor is also executed three times.

Likewise, if I create two .java files under /etc/openhab/automation/jsr223/, Java223ScriptEngineFactory.createScriptEngine(String scriptType) is executed twice, and the constructor of Java223ScriptEngine is also executed twice.

However, if I create one file openhab/automation/jsr223/m.java installing two rules, as in this example, where I had to add .txt otherwise the website does not allow me to upload it with .java extension, m.java.txt (1.8 KB) Java223ScriptEngineFactory.createScriptEngine(String scriptType) is executed once, the Java223ScriptEngine constructor is executed once, and there are two rules installed!

So regarding the creation of ScriptEngines: one instance of the scripting (rule) engine is created for each transformation and for each file under automation/jsr223 – irrespective of how many rules the files from automation/jsr223/.java install.

55

For Java. But you are pointing to GraalVM JS and GraalVM Python. And in those cases is there just the one or is there one per rule?

It is perfectly reasonable that different automation add-ons work differently in this reguard.

56

I’m lacking knowledge to assess the full picture, I don’t know if different instances of ScriptEngine really are wholly different instances inside the JSR223 implementation and whether running multiple is safe. Neither do I know how expensive they are to create, or how much memory they consume, but I would imagine that these are fairly expensive (and slow) to create.

As such, one script egine per transformation might be excessive. I’d think that transformation would usually be very quick to process.

57

Which is how it appears to be implemented. At least I know that if I set a variable in a transformation applied in one link, that variable will still have that value on another link, implying that there is just the one context for all usages of the one transformation.

Based on the above, it seems that Jav223 does the opposite as JS Scripting and it creates a separate Script Engine for each transformation but reuses the same one for the rules. :person_shrugging:

58

The reuse for rules follow the design of OH core, since these script engines are handled by ScriptEngineManager, only one of each type is created. The transformations seem not to use ScriptEngineManager - thus the different behavior.

I suspect that this behavior isn’t dictated by the scripting add-on, but by Core, so there is probably one engine for each transformation in JS and Python as well, and these will then use different locks so they are allowed to run in parallel. But, as long as the instances are truly independent of each other, and all “context” is independent, executing those in parallel probably isn’t a problem. I would be curious to know the “cost” of creating all these instances though.

59

My reading is that each openHAB Automation Add-On exposes a method createScriptEngine() for the org.openhab.core.automation.module.script.ScriptEngineFactory interface. Then this method is executed, whenever openhab-core is in the mood to call it, and the Add-On does not call the method itself. So openhab-core mandates, how many ScriptEngine instances are created by invoking createScriptEngine the right amount of times.

The implementations of createScriptEngine() in Python(GraalVM) and JavaScript(GraalVM) are return new PythonScriptEngine(…) and return new DebuggingGraalScriptEngine<>(new OpenhabGraalJSScriptEngine(…)). So when openhab-core says createScriptEngine() in these two cases new objects are created, likewise in Groovy.

60

All this is how I read it too, but the difference is that transformations call ScriptEngineFactory.createScriptEngine() directly, while rules get them from ScriptEngineManager, that makes sure that only once instance exists (in its realm).