Rules and return codes

Yes, I thought of that too, but it would mean that conditions would have to be written “using different rules” than actions - maybe not ideal? Also, as I said, actions can also return values, although rarely used.

Absolutely - I’m just “brainstorming”.

Not necessarily true, rules have a generic “configuration” container where you can put key/values of your choice. But, to get the UI to support it is another matter… not that it’s difficult in itself, but it would make the UI “lean quite far” to suite a particular add-on.

tl;dr: Give me a flag to detect when the wrapper is enabled, and I’m happy. I’m cautious about shipping with the wrapper enabled though as it is a breaking change.

Yes.

The result of the last line executed becomes the implicit return value. I’ve always thought that was weird, but it’s how it’s always worked since the Experimental Rules Engine was first introduced way back in OH 2.1 or 2.2 (I remember having a discussion with @ysc about ti when I first started experimenting with it).

There is precedence for this as this is how the return for a Rules DSL lambda function is determined too.

But you just have to be careful to make sure that all the last lines of a condition that could occur evaluate to a boolean.

For example, here’s an actual one of mine:

console.loggerName = 'org.openhab.automation.rules_tools.Debounce';
// ~ TODO ~ TODO ~ TODO ~ TODO ~ TODO ~ TODO ~ TODO ~ TODO
// Rework when `event` always exists and check the event.type
if(this.event !== undefined && this.event.itemName !== undefined){
  const item = items.getItem(this.event.itemName);
  if(item.isUninitialized) {
    console.debug('Debounce for Item', this.event.itemName, 'is blocked for state', this.event.itemState);
  }
  !item.isUninitialized;
}
else {
  console.trace('Debounce passed conditions');
  true;
}

That condition works when the wrapper is turned off. I must modify it to the following when the rapper is turned on.

console.loggerName = 'org.openhab.automation.rules_tools.Debounce';
// ~ TODO ~ TODO ~ TODO ~ TODO ~ TODO ~ TODO ~ TODO ~ TODO
// Rework when `event` always exists and check the event.type
if(this.event !== undefined && this.event.itemName !== undefined){
  const item = items.getItem(this.event.itemName);
  if(item.isUninitialized) {
    console.debug('Debounce for Item', this.event.itemName, 'is blocked for state', this.event.itemState);
  }
  return !item.isUninitialized;
}
else {
  console.trace('Debounce passed conditions');
  return true;
}

Don’t get me wrong, I much prefer having an explicit return. But making the wrapper configurable adds all kinds of headaches. You can never know from within the rule what to do.

As long as the wrapper can be turned off, there will have to be two versions of rule templates, or we need to abandon those who turn it off or turn it on and say “too bad, this script won’t work for you anymore unless you disable the wrapper.” Or abandon the use of rule conditions entirely. I don’t like any of those choices.

As long as this is user configurable, I don’t see how the upgrade tool can help. That only runs during an upgrade, but the user can enable or disable the wrapper at any time.

Only applying the wrapper if it’s an action would be the least disruptive approach compared to what’s been mentioned thus far. However it would mean that the conditions would only get the raw Java event and have additional restrictions that do not exist for Script Actions. I don’t like that one bit. I think we need a bigger reason than this to justify that.

But my biggest limitation is lack of a reliable way to detect in the script whether the wrapper is enabled or not. Maybe the wrapper can add a variable when it calls the script, or openhab-js utils can have a little function to call to tell us whether the wrapper is enabled or not? Maybe I can access the add-ons settings from a registry or API call? I can work with that.

Would it be enough to see if event.raw !== undefined? Is that always guaranteed to be there when the wrapper is enabled?

It occurs to me that I need that info in my script actions too for my templates because it’s not enough to just check that the openhab-js version > 5.14.0 since the wrapper can be turned off. And to maintain backward compatibility and avoid deprecation warnings I need to get the event variables a different way depending on whether the wrapper is on or not. For example, this is what I currently do in Debounce:

var triggerType = null;
var newState = null;
if(this.event !== undefined) {
  if(helpers.compareVersions(utils.OPENHAB_JS_VERSION, '5.14.0') < 0){
    console.debug('Running on version before 5.14.0');
    triggerType = event.type.toString();
    newState = event.itemState.toString();
    itemCommand = event.itemCommand.toString();
  } 
  else {
    console.debug('Running on 5.14.0 or greater');
    triggerType = event.triggerType;
    newState = event.receivedState;
    itemCommand = event.receivedCommand;
  }
}

Clearly that won’t work since the wrapper can be disabled, but if I have a flag I can test instead of version numbers that overall approach will still work. I’d do the same for my conditions. Having the return statement in the code isn’t a problem. It’s only a problem when OH tries to execute it.

No matter what, some way to determine if the wrapper is enbled from within a script is something I think I really do need beyond just this conditions issue.

With that, my only concern is that shipping with the wrapper enabled would be a breaking change. It’s easy to just turn it off, and we can document that in the release notes so it’s not a huge breaking change. Or we can ship it off by default until OH 6. Or something in between.

And this was my point above, somewhere this logic must exist, and it should be possible to refine so that it only adds the “implied return” where appropriate, as long as it doesn’t reside in the script engine (Graal) itself.

Although such an approach would work for you, I think it’s way too “advanced” for the average user to deal with. I think it would be really preferable if the script didn’t have to sort this out.

A whole different approach could be to introduce some kind of “directive” in the script itself that enables the wrapper. I don’t know what would “fit” in JS syntax, but it could be @Wrapped, // wrapper: on or anything of the sort. It would be easy to parse using regex from Java and determine if it should be wrapped or not. If the wrapper was off by default (without a directive), there would be nothing breaking at all. The configuration option could also be removed.

The only downside to this approach is that users must know about the directive to benefit from the wrapper.

edit: Maybe the most “JS way” would be to have the directive be:

"use wrapper"; 

No, it’s somewhere in core. Back then I was dealing with Nashorn JS. GraalJS wasn’t a thing back then.

I think all the other rules languages work the same way too for conditions. I know it does for Rules DSL.

The average user doesn’t really have to sort that out though. All they have to do is make a choice. If they want the wrapper they have to change their conditions by adding return statements. If they don’t want to do that they can disable the wrapper.

The only thing we have to decide is whether we surprise them in OH 5.1 release by enabling the wrapper by default.

No one except rule template authors will be trying to write backwards compatible code so, having a flag is only for us. It lets us maintain one template for multiple versions of OH for longer, reducing maintenance over all (with a slight increase in complexity in the code itself) and makes it easier for users of the templates since they don’t need to move to a new version of the tempalte, just update and regenerate (thanks to the wonderful work you put in on that front).

I don’t see how that’s any less of an imposition on the end users than a flag. But again, most users don’t have that much to worry about and there isn’t a lot of complexity here. It’s only those who need to maintain backwards compatibility with their code and be able to support all JS Scripting users with one template that need to be concerned whether the wrapper is enabled or not inside the rule itself.

Everyone else either has the wrapper enabled or disabled and writes their rules accordingly.

That depends a bit. What if you need to enable or disable the wrapper (which is a global configuration) because of some other things you want to use, a script you want to copy or whatever. Then you have to redo everything, and then when you later change your mind, you might have to redo it again. I think that, fundamentally, having this as a global switch is problematic.

Again, the directive was just a suggestion, but it would allow per-script behavior. If the directive had an off/on indication, the default could even change e.g. with the release of 6.0, and still allow scripts where the directive is explicit to continue to work.

Can an single add-on register two different script “languages”? E.g., “ECMAScript (ECMAScript 262 Edition 11)” and “ECMAScript-NoWrap (ECMAScript 262 Edition 11)”

That also fixes the template issue on a individual rule basis without any new configurations. It gives users a chance to upgrade any impacted rules in the same way as major script language changes by going one rule at a time and ensuring that all code is compatible before saving with the new “version”. I assume it also makes it easy to start adding a depreciation warning to the NoWrap versions if that is deemed necessary in the long run.

I don’t think so, the script “language” registration is already quite the mess. But, even if that was possible, how would it help? How should the user indicate which “language version” to use?

I didn’t mean that it was just in Graal, but that it was in the ScriptEngine, whatever that is. To me it seems like that’s the case, here is what Google has to say:

The ScriptEngine.eval() method in Java’s Scripting API returns the value that results from the execution of the script. This return value is an Object.

Key aspects of the return value:

  • Completion Value:

The eval() method returns the “completion value” of the script. For simple expressions, this is the direct result of the evaluation. For scripts with multiple statements, it typically returns the value of the last evaluated expression or statement.

  • Type Conversion:

The specific type of the returned Object depends on the scripting language and the value produced by the script. For example, a JavaScript number might be returned as a java.lang.Number (e.g., Integer, Double), a string as a java.lang.String, and a JavaScript object might be returned as a specialized object like ScriptObjectMirror (in Nashorn) or a Map representing its properties.

  • null or undefined:

If the script does not explicitly return a value or the last statement does not produce a meaningful result, the eval() method might return null (in Java) or undefined (if the underlying scripting engine’s concept of “no value” maps to undefined).

  • Exceptions:

If an error occurs during script execution, eval() will throw a ScriptException.

This bug also indicates that this is done within the ScriptEngine:

https://bugs.openjdk.org/browse/JDK-7181664

The script language is explicitly configured in every script action:

conditions: []
actions:
  - inputs: {}
    id: "7"
    configuration:
      type: application/javascript
      script: >

And easily configurable (for managed rules at least):

Yes, but this whole construct is a “house of cards” that barely holds together. I’ve created an issue on the subject in the past, I’d say that more functionality shouldn’t be attempted to be packed into that particular parameter. But, as I said above, the configuration section can include anything, so it could have a specific “wrap” parameter if desirable. But, it would only make sense to the JavaScript add-on:

https://github.com/openhab/openhab-core/issues/4825

The whole „implicit“ return stuff in conditions in kind of shady, so IMO would mean nothing bad if one needs to explicitly return in these actions.

What about always enabling it then? Sure, it is a breaking change but if we clearly communicate it, avoid confusion by giving no choice, then it should be possible to get along with it. Adding return isn‘t that complicated and if we provide the right query for the developer searchbar, it should be easy to locate all conditions.
Rules can now also be regenerated from templates thanks to @Nadahar.
Also don‘t forget that the pure JS event object isn‘t available without the wrapper.

You can check ScriptConditionHandler in core, the condition is just the plain script that is passed to ScriptEngine::eval or CompiledScript::eval.
Given that ScriptEngine::eval allows for stuff like eval(„10 > “), I think it is expected that the „implicit“ return works. As long as the global script ends its execution with a boolean value, it is considered (at least it seems to me) to have returned this (you cannot really return from a global script, hence the error if return is used without wrapper).

Just seen that Nadahar googled this, and I agree with him on the house of cards wrt to script identifiers (@Nadahar I have seen your issue, wanted to answer).

It’s not specific for conditions, it’s from the ScriptEngine specification itself as far as I can understand, and will apply to anything sent to eval(). The difference between Actions and Conditions is merely that Actions accept a null return value, while conditions don’t. So, you have the same fundamental problem with return values of Actions, they will “return something” according to those (shady) evaluation rules. But, most of the stuff that use the return values of Actions in OH isn’t in use (I’m not sure anybody still here actually know how to use it), so this won’t be noticed. The same “challenge” exists though.

Yes, that’s pretty much what I’ve concluded too - that this behavior is built into the scripting engine.

When it comes to just enabling the wrapping, this isn’t really my business at all, but I as a user would be frustrated by things breaking. That is in fact the main reason that I don’t use JS scripting myself, it breaks too much for me just with the different ECMA versions etc.

edit: This further confirms that this behavior is in the ScriptEngines themselves:

I think the “directive” approach would be the best (e.g. include "use wrapper=true" in the script) of the ideas that have been discussed this far. If the directive is missing, the default would apply, and that could change with e.g 6.0. But, anything written with "use wrapper=false" would then continue to work even after using the wrapper became the default, and those that wanted to could start using it with 5.1 by including "use wrapper=true". Those that wasn’t aware of the change and are angered by the change, can then “fix” all their scripts by adding "use wrapper=false".

The only “downside” I can see with this is that the wrapper isn’t enabled by default from 5.1. But, whether that’s an up- or downside depends entirely on your perspective…

Please create a seperate topic to discuss this specific case further.

Thanks!

So I see these options going forward:

  1. Always enable the wrapper, though that would be a breaking change and require manual adjustment
  2. Conditionally enable the wrapper based on the usage of some keywords like return, const, function, class, let (this is a new idea). Though with that approach I have to adjust the event object conversion to work without the wrapper
  3. Revert the change, then there is no return keyword etc., which would be a loss IMHO. In complex scripts, return would be a great way to early exit execution, and class and function declaration are also nice

I won’t go down the path of adding another specific MIME type.
And I’m not too sure about the directive „use wrapper“ to enable it.

It would have to be always enabled everywhere though. Not just conditions.

Then I can just check for the version number of openhab-js and if it’s >= 5.14.0 use one way and if < use the old way. I thought that was how it was working before you told me above there is a setting.

There will be much frustration among the users as rules that have worked for years suddenly need a change. That’s going to happen eventually, but it usually goes over better when there is a bit of a deprecated period giving users time to migrate.

I’ve actually been unsuccessful with that so far. Whether or not a rule has a condition, let alone whether it has a script condition is not something I’ve yet been able to create a query for in the developer sidebar nor in a rule. I’m about to just pull in the whole JSONDB file and search through the parsed JSON using JSONPATH.

  1. Eventually I like this as the end goal. I’m mainly wary of just jumping to that out of the gate and there might be pushback to release such a large breaking change in a point release by whom ever polices that.

  2. We have to be careful here because all of those are legit even without the wrapper when they are inside a function or other context like if statement or what have you. The mere presence of those keywords is not enough to determine if a script is asking to use the wrapper or not.
    For example, the following is a working managed script without the wrapper.

var myFunction = () ={
  console.info('do some stuff');
  return true;
}

var foo = 7;
if(event.itemState == 'ON') {
  const bar = 9;
  let baz = foo + bar;
}

It the wrapper were enabled above, there would be problems because I use event.itemState instead of receivedState or newState.

  1. I don’t think this is a real option. The benefits of the normalization between file based and managed code are too great.

There is also option 5. (I’m counting the directive as option 4) Keep it globally configurable but provide/document a way for a given script to detect if the wrapper is enabled or not. That gives the end users that time to adjust before we eventually move to 1. Providing some sort of tool they can use to detect which rules have conditions that need to be fixed like the deprecation warning on event, that would be even better. I know I would benefit from such a tool. I’ve enough rules that going through them one-by-one looking for conditions is tedious.

I think checking event.raw !== undefined is going to be enough to detect, but it’s early morning here, I’ve not looked at the code to confirm. Having something a little more deliberate like utils.wrapperEnabled() which is less likely to get forgotten and changed later would be better.

I agree; the wrapper is an excellent more forward.

A lot of noise was made in 4 → 5 about this magnitude of breaking change really being restricted to major version upgrades and not even milestones. I’d rather not see this shelved until 6. So I think it’s best for everyone to see one of the mitigation factors worked out.

Forgive me I’m missing something here, but this only fixes some small portion of the issues, correct? It addresses the return issue (depending on the structure within the script), but you can’t change whether the rule code uses let or var declarations after detecting the wrapper within the code, and nearly every event reference would then have to be conditional.

The main issue, initially was just the condition return concern. So the proposal to enable/disable the wrapper based one keyword detection would work if only applied to condition scripts, right?

A condition script that is complex enough to have it’s own subfunction with a return statement is, I think, a rare enough edge case that this might be the most robust solution for now. Then a depreciation warning can be added now, and a full breaking change of always using the wrapper on conditions (and so requiring a return statement) can be slotted for 6.

Can we let users construct/create a wrapper themselves within their script as they see fit, instead of being forced inside one beyond their control?

For normal users they will either have the wrapper on or off.

If it’s on, they are required to use return in conditions, have the option to use let and const everywhere, and need to access the new event properties. Existing users of managed JS Scripting will need to modify their rules. But they are only required to change their JS Scripting conditions. They just get warnings for the event Object and use of var instead of let or const isn’t an error.

If it’s off, they are required to avoid the use of return anywhere except a locally defined function, can only use let and const inside a subcontext, and must use the raw Java event Object, just like has been the case up to now.

It’s either or. There is no reason they will need to know if the wrapper is enabled or not inside their rules themselves. They just need to know whether they have it enabled and write/modify their rules accordingly.

The main concern for these users is providing a reasonable transition path to using the wrapper eventually. It would be nice if there were a way to move rules one-by-one to using the wrapper but that doesn’t appear to be feasible. So, when the users are ready, they will have the option to toggle the wrapper on and then they need to update their scripts accordingly. It would be nice if there were a tool or something to tell the users what needs to be changed. The deprecation warnings in the event wrapper Object handles that already. It’s really only a tool to help identify the managed rules with conditions that need to be updated.

One who needs to write rules that are backwards compatible with versions of OH 5.0- or those who need to write a rule that works whether or not the wrapper is enabled does need to worry about this inside their rules. And indeed in those cases they will need logic to go down different code paths depending on whether the wrapper is enabled or not. And they will not be able to use let and const in the global context at all because it breaks when the wrapper isn’t there. They will have to continue to use var or wrap their script action in a self executing function like is suggested for transformations. But using var is just bad practice, not something that actually causes a problem.

But template authors are usually more capable coders, so this isn’t that huge of an imposition on them. And they can choose to just support running with the wrapper or without and not both. But for those of us who want to support all JS Scripting users with our templates we must be able to detect when the wrapper is there.

This is the problem that caused me to raise this whole stink in the first place. It has a huge impact on me and I want to avoid a similarly large impact on my template users (meaning they have to know the status of the wrapper settings and install the correct rule template based on that setting).

But I don’t have a huge concern about the regular users as long as we provide an upgrade path that isn’t “you have to change all your rules now in 5.1”. They have the ability to turn the wrapper off until they are ready to transition their rules (if ever).

Part of the reason for the wrapper is to make it so that we can provide a pure JavaScript event Object to all JS users and enable JS coding best practices (e.g. let and const). Right now this is different between .js file users and managed users where the managed users get the raw Java event and .js file users get a nicely wrapped and enhanced JS version of the event Object.

Over all, managed JS rules and file based JS rules are different and incompatible environments. The wrapper normalizes them both so we no longer have “if you are in the UI you use event.itemState but if you are in a file you use event.receivedState” and other similar differences which makes it harder to share code, duplicates a bunch of the documentation, and overall adds complexity to those users least able to handle it (i.e. managed file users).

The wrapper exists outside the script in the add-on. It’s basically what calls the script in the first place. So there really would be no way for users to implement this themselves. But yes, you can put your script inside a wrapper using a self calling function. But that doesn’t solve all the problems we need solved by using the wrapper in the first place. You still get the raw Java event and file based users get the enhances JS event.

What about just adding a wrapper blacklist as an advanced option in the add-on settings?

It’s not quite as user friendly as some sort of option right in the rule configuration itself, but would allow for complete rule-by-rule customization.

Rule templates would have to specify in their pages that users have to add the UID of the rule they create from the template to the blacklist if necessary, but that’s not too onerous a step even for beginner users (or even if there is a way to test for the wrapper, add a log warning about adding the rule to the blacklist).

Unfortunately, in my experience, that is exactly the sort of step that will trip up a lot of users.