[automation] provision of host objects to automation scripts

Currently the automation framework in Openhab allows scripts to get at objects from OH itself, via the scriptExtension.importPreset. This works by asking a scripting engine to inject values into the scripts (e.g. the defaults, plus the scriptExtension object itself). Then when the scriptExtension object is called, it askes the scripting engine to inject more things.

This seems both complex, and also kind-of backwards, as I would expect the control-flow to operate such that the script requests things that the host provides, so a pull mechanism, rather than a pull-to-initiate-a-push? I want to raise this in the context of two items:

  • OH3 - is this planned to change?
  • Javascript module support and the conflict of this system vs. modules.

I am considering implementing a pull-based system on top of the current code, such that a script writer can write:

const rulesupport = require('@runtime/rulesupport');
rulesupport.Foo.doThing()

Whilst I believe that this is a much better and idiomatic experience for the JS scriptwriter, it feels like I’m fighting against the current code.

I therefore wanted to open this topic to discuss this and potentially understand the direction of this area of scripting.
@5iver this one’s probably for you…

ScriptExtensions can be used to add certain objects to the context of a script or to share objects between scripts. The ScriptExtensions group together objects into presets and I see their functionality as useful in preventing namespace congestion, versus dumping all of the objects into each script. I do not share your opinion that this mechanism is complex or backwards.

I do not plan to change anything in regards to the ScriptExtension mechanism, but a few presets will likely be modified and more added when implementing the scripting API. I am not aware of all the plans of the core maintainers, but I am also not aware of any of their current plans that would specifically affect this functionality.

Could you please provide more details on what conflict you are seeing between ScriptExtensions and JavaScript module support?

Maybe I’m missing something, but I don’t see much benefit in this. How would your changes be used in a Jython script?

I’m elated to see others interested in progressing the OH automation engine! :partying_face:

Maybe I could be clearer on the part of the mechanism that I find complex. Currently the module support I’ve build for JS mirrors how NodeJS does it, which is that every library that is referenced by a script causes the runtime to go out and find the library, load and cache it, then return the symbols to script (this may happen recursively). These libraries are cached in ‘module’ instances which link to each other as dependencies.

As I mentioned, I have considered switching the GraalJS runtime to this approach for script extensions, so that a script can explicitly import/require the symbols from a specific grouping. When I think about how I would do this, it would make sense to put the symbols/objects into a ‘module’ instance and return it to the script (e.g. same as other lib references). The challenge arises with the current mechanism - which is that the script calls out to scriptExtension, which then asks the ScriptEngineFactory to import the symbols. It does not return the symbols. The best way that I can see to achieve this right now is:

  • store the scriptExtension objects in the ScriptEngineFactory, keyed by containing ScriptEngine
  • when I get a request to require a special/runtime lib (I prefixed above with @runtime, but whatever), then I would need to:
    • create a new module instance with the name of the runtime module being looked up, and store in the ScriptEngineFactory, per ScriptEngine (assuming no transitive deps, and no multithreading)
    • lookup the associated scriptExtension that I’ve stored for the requesting ScriptEngine (lookup in the Factory)
    • explicitly call the importPreset method on the specific scriptExtension
    • when scopeImports is then called, capture the symbols provided and copy them into the module (not the engine)
    • return the module to the runtime

So it’s possible, but it seems much more complex than it should be - for example if importPreset returned (or also returned) the symbols then everything would be straightforward.

As for Jython, I’m a bit confused. I thought that it was pretty much the same as JS, and that you’d want to support the import/module system. For example, this is how I think it should work:

import runtime.rule_support //first I import a library
...
rulesupport.Bar.baz() //then I reference things from the library

But what I currently see is:

scriptExtension.importPreset("RuleSupport") // would be nice to use a standard import...
...
Bar.baz() //where did Bar come from?

So I think that the systems are the same, and that I’m suggesting to make the framework code able to support the native module system for the guest language. I cannot comment on whether it’s possible to achieve this with Jython, that would depend on the ability to intercept the import call from the guest runtime (which is precisely how I acheive it in JS).

Using any scripting language, after this…

scriptExtension.importPreset"RuleSupport")

… the RuleSupport preset objects are now available in the script’s context, as if they were individually imported. In your script, you can also create presets and include whatever objects you’d like in them. These custom ScriptExtensions can also then be accessed in other scripts. They can even be used to share values across scripts, even of different script languages. If you do make changes, all of the existing functionality will need to be maintained.

What you are proposing sounds like a very large effort, may not even be technically possible, would require special handling for all scripting languages, and currently works perfectly well for all scripting languages. Respectfully, I do not see the benefit of changing this functionality. There is a LOT of other work to be done!

The reason that I’m even suggesting this change is that it’s bitten me quite a number of times as I’ve built JS libraries. The most common one is the items object, although it’s not the only one. This is a common term to use as a variable name, or as an import (and there is even a library in openhab-scripters based on specific item support, which will become problematic once we have modules there). It’s very easy to end up referencing the wrong object, but it’s hard to understand why, because these are injected by magic. This is also likely to cause problems with 3rd party libraries if there are any naming collisions because they expect to run with some level of isolation, however the objects are globally reference-able (I would expect this to also be a problem with Python). I guess it wouldn’t be a problem if they were uniquely prefixed in some way.

I agree that attempting to use the standard module system in Jython may be a large piece of work. I’m not suggesting that it’s attempted or even worth it, just that the framework would allow it if desired. The only real blocker is the lack of a single-line return statement in ScriptExtensionManager.importPreset - it would be a tiny change that I’d be happy to submit a PR for?

(I have realised that I could implement this a simpler way by copying the ScriptExtensionManager functionality into the ScriptEngineFactory (allowing the providers to be injected directly), which feels like a direct workaround to the lack of return statement.)

FYI I had a quick look at Jython; hooking into the import system to dynamically provide symbols is supported, and it actually how Jython imports Java classes! It would need a hook provided to the runtime which would be routed to the ScriptExtensionManager for it’s pieces. Looks fairly straightforward. It’s under the section titled ‘Import Hooks’ on https://jython.readthedocs.io/en/latest/chapter7/.

Not necessarily something worth doing right now, but shows that this is possible & supported.

So I’ve implemented as cleanly as I can with what is currently available. All works but isn’t pretty. In case you’re interested, the code is here. I’m happy with the trade-off of one piece of dirty Java for many pieces of clean JS.

I just got a Jython addon working. I have your two PRs to review and a pile of PRs to review in the helper library repo. So, I’m a bit jammed up ATM, but I’ll take a look when I can. It’s great that you’re contributing to automation! Maybe you can find something of interest in this backlog? It would be really good to get the Windows issues resolved before the 2.5 release.

Another thought… I plan to include the Jython helper libraries in the addon too. You may want to consider doing the same in the graaljs addon.

I just got a Jython addon working

You mean extraction of Jython to a separate addon? Or using Graal? Or pulling in things via import?

Maybe you can find something of interest in this backlog?

I’m happy to take a look - although I’d need to ask some questions on the issues as there isn’t enough detail in all of them atm for me to understand exactly what they need. I’m guessing that it’s possible to comment on the cards but I don’t have access to? Also it’s probably unlikely I’ll be able to tackle the windows issues as I haven’t used it for years, but the others I can look at.

I plan to include the Jython helper libraries in the addon too.

You mean the actual python code? Hmm, I hadn’t thought of that, although the CommonJS module support that is in the plugin also supports loading modules directly from java resource paths so it would be trivial to do.

I’m unrolling it from the standalone jar. Had to move AbstractScriptEngineFactory out of internal and made a couple other changes in OHC to get it working.

They were just notes to myself, so not much detail. :smiley: I’ll take look at the permissions. Probably best to just get a PM going here in the forum.

Yes. Jython will look in the directory containing the jython.jar for libraries, but I’m also setting python.home and python.path in the addon.