Namespaces & Packaging for Scripting APIs

jpg0 · December 5, 2019, 12:54am

@5iver I wanted to start a topic to discuss the namespacing for included APIs for scripting. Currently what I have for JS is a little different to that for Python, and I wanted to ensure that we choose the best organisation for both (and potentially more) languages.

There are a few areas where I think we need to decide:

Module locations

I alluded to this in your Jython PR, but I think we need to align on the best approach here. I believe that there is nothing to discuss re: scripts (3 directories, all distinct), only modules/libs. I think that there are a few questions to be answered, and some assumptions that I have:

Assumptions:

We want to allow 3rd party scripts to be installed, without modification.
We want to bundle core modules with the OSGi bundle.
We want to allow users to add to the locations being searched for modules (don’t know about overriding completely).

Questions:

Do we want to allow users to ‘override’ scripts provided by core (or community) with their own version, without modifying core or consuming scripts?
Where do 3rd party libs go?

My views:

We should allow overriding, as specified above. It makes debugging much simpler for users, and they can return to the default state much more easily.
I would suggest that 3rd party libs can go in any of the folders! This means that core or community contributions can include their own dependencies. But I’d be happy for a single folder to be the suggested place.

Module namespacing
I think that there are two main namespaces that will be provided: the ‘scripting API’ and the ‘imported host objects’. As part of the Jython PR, these are under core and core/jsr223/scope I believe.
Currently in the JS code I have built, it’s slightly difference (but generally easily changable). Firstly, some relevant points about how JS (CommonJS, specifically) differs from Python here:

JS doesn’t really encourage ‘submodules’ or a hierarchy like Python (or Java). It’s possible, but awkward, prone to problems, and poorly supported by tooling.
JS does perform top-level imports pretty much the same way, and allows either importing all symbols or a subset of those available
A ‘module’ returned from an import/require call in JS is just an object, so ‘submodules’ can be emulated as child objects. This means that instead of require('top/child') you would require('top').child
JS can request either relative: require('./foo') or absolute require('foo') imports.

Now currently the code I have provides modules like this:

to get API sections: require('ohj').items or require('ohj').triggers etc.
to get host objects require('@runtime/Defaults') or require('@runtime/RuleSupport') etc

Our approaches are obviously different. My views on how we should align:

I don’t like my mixed import-style approach, nor my naming
I think we should discourage (but still provide) access to the host objects, so I like your sub-namespacing. I noticed that the Python style guide suggests using leading underscores for native modules (C/C++) with an accompanying higher-level module, it feels like we have a similar situation here, so could possibly use this.
I don’t like the ‘jsr223’ segment you have in the module path, it’s not much help for the average user (personally, I think we should drop it entirely, how many users care which jsr proposed scripting languages in Java, or even what a jsr is?!)
I’m torn on ‘core’. On the one hand it’s already there, and it refers to ‘core’ libraries. On the other hand, maybe something either OH-specific or ‘special’ should be used for this namespace, or at least for the host objects namespace. Also ‘core’ already exists both on PyPI and NPM.
I also don’t like the ‘scopes’ sub-namespace; I don’t think it describes what it is, which is a collection of objects imported from OH. Below I suggested that we use ‘_host’, but we can probably do better.
I believe that it’s ideal to be able to import specific host objects in the import statement.

If we stick with core, I’d suggest that we go with:

Python

import core.items // import a piece of the API
- or import items from core
import core._host
- or import events from core._host // the default scope
import core._host.RuleSupport
- or import automationManager from core._host.RuleSupport

JS

require('core').items
- or let { items } = require('core')
require('core')._host
- or let { RuleSupport } = require('core')._host
require('core')._host.RuleSupport
- or let { automationManager } = require('core')._host.RuleSupport

Anyway, I wanted to start thread to get the conversation going.

rlkoshak · December 5, 2019, 5:29pm

I think that now is the time to be having these discussions. It’ll be much easier to change these sorts of things now that there are not that many users overall (though more than you might think) than later.

I don’t have a strong opinion about module locations. I don’t fully understand all the nuances here so don’t have any useful opinions on the matter.

As for naming, a lot of the names and directory structure I think is left over from legacy. The root source of the HL goes back to OH 1.x I believe so we are living with some code and decisions made years ago. As such, even though the number of users is small relative to Rules DSL, there are still a lot of users that would be impacted by a change in any of the paths and name changes.

I completely agree. I think the only argument against changing it (IIRC there was a discussion about that a year or so ago) is because of the legacy. I believe it’s not really even correct any more and I know I at least have been using “Scripted Automation” instead of JSR223 in my posts to this forum and issues on GitHub.

Honestly, now that I’ve spent some time with it, I find the whole directory structure to be hard to use. Maybe, now that we’ve got an OSGI install for Python, we can bring the personal folders up a bit. To write Rules DSL, I have a path under $OH_CONF that is one deep: $OH_CONF/rules. For Python I have a path that is four deep for scripts and a wholly separate four deep path for modules. Manually navigating around the code on the command line or even within VSCode is a little frustrating, at least for me.

How important is it that the languages be kept separated? Logically it makes sense to keep them separated but practically, I wouldn’t expect many users to be using more than one language at a time, or if they are it is because they are using something from the Community libraries. It feels like a potentially unnecessary layer of directory structures to impose on the users (it makes perfect sense for the libraries though). I’m only talking about the “personal” folders here, not the overall structure of the HL.

Ideally it would be great if, from a user’s perspective, we could use $OH_CONF/automation/modules and $OH_CONF/automation/scripts or something like that. What happens under those folders is up to the user. Again, I’m only talking about the “personal” folders.

That’s my two cents worth.

I’m not torn on this. I never liked core. It’s too generic. It should have an OH specific name IMHO. I think this is another one of those legacy things and changing it would probably be some effort. to change though.

As a user and someone who helps other users who have less experience programming these are my main observations. I don’t really have an opinion on the rest of the topics raised. And I’ve not had time to test out the OSGI bundle yet (sorry Scott). I might change opinions or have further ideas once I do.

jpg0 · December 6, 2019, 3:40am

You’re completely correct, I didn’t even think of that. Both the latest JS & Python bundles ship their own version of the language, and encourage skipping the javax.script APIs altogether. All the more reason to remove it.

I completely agree, actually. Maybe being a dev I’m using to navigating complex directory structures so I didn’t notice it so much. But now you say it, it is a pain, because of both the depth, and the many language folders which (like you) I only use two. I’d certainly support your suggestion of the two directories, that would be sufficient for me.

If changing it is feasible, I’d certainly opt to change it.

I guess that all of this points to the fact that it would be great to change all of this, but not if the overall impact is too high. My feeling is that it should be changed, and now is the (only) time to do it. My reasoning is:

This is been very clearly marked as ‘experimental’ for a reason; if we regard it as locked-down, there was no point in marking it as experimental.
The changes in JS are likely going to mean rewrites anyway
- Whilst moving to ES6 is in theory backwards-compatible with ES5, the different engines (Nashorn v. GraalJS) mean that it’s not as soon as a script has used a Nashorn-ism, which is most of them (certainly the scripts in openhab-scripters)
- Introduction of a module system changes quite a lot, as previously the code just looked up the (absolute file) location of the lib, then effectively inlined it. Library locating, scope differences, module syntax changes all mean that backwards compatibility sits somewhere between impossible and infeasible.

On the topic of naming, we can mostly preserve backwards compatibility, by (temporarily) allowing usage via the old namespace. This is pretty trivial in JS, and I’d expect so in Python too.

mjcumming · December 6, 2019, 9:16pm

We did have a discussion about this on github

jpg0 · December 6, 2019, 10:51pm

Thanks for the link! Looking at that discussion, it seemed to be primarily about repo layout and the experience of those interacting with the repo. As some people may have expected, that translated to filesystem layout on OH instances. Possibly few people realised that this could have translated to how scripts and modules are written and referenced, and potentially even compatibility with 3rd party code.

5iver · December 11, 2019, 5:59pm

It appears that what you are referring to as “Scripting APIs” are not the same as what I have been referring to. The new rule engine is awesome, but it was developed with UI rules in mind, and it needs some effort to improve it’s almost complete lack of user support for scripted automation. This is why the helper libraries were created. They are very good at sandboxing and then testing functionality that should be built into OH. Once proven, this functionality needs to be moved into Java.

To improve the NGRE, we should build out the layers of abstraction between the user and the rule engine through ScriptExtensions, Actions, and a new Scripting API (in Java) that can be used by scripts launched with any flavor of ScriptEngine. This is what I have meant by the Scripting API replacing the functionality of the helper libraries. Language specific libraries will no longer be required for creating triggers and rules, accessing Actions, manipulating metadata, Items and Things, etc.

Completely refactoring the helper libraries at this time would not be worth it, and our efforts should be focused on implementing the helper library functionality in Java. Please don’t take me the wrong way… I’m thrilled that you are interested in scripted automation. I think it is just a matter of combining our efforts or we will be stepping all over each other.

What you have put together has diverged significantly from the Jython and JS helper libraries and I’m beginning to doubt that it will be able to be integrated, which is a shame. Early on, I tried to lead you down the path of developing the Scripting API, rather than expanding the functionality of the JS HLs. What you have now is a completely separate implementation of helper libraries specifically for JavaScript.

I disagree. For OH3, as I’ve mentioned previously, I plan to rename the /automation/jsr223/ directory to /automation/scripts/ or potentially remove it completely. Any references to jsr223 in the docs should be replaced with scripted automation.

And Community too! Potentially! This will be driven by how the ExtensionService is implemented in OH3. I’m starting to think it would be best to package the libraries separately, possibly as individual bundles. There are pros and cons to both, but I do not want to discuss this here (better in GH).

Either way, by packaging the libraries, rule templates can be distributed with Scripted Actions and Conditions that make use of those libraries. Scripted automation is included in the UI (REST API) rules. One of our main goals should be to advance OH automation should be to facilitate custom library and script writers in their efforts to build rule templates for use in UI rules, so that OH has a robust archive of rule templates that are easily shared and collaborated on. The helper libraries are a big step forward, but the Scripting API will be even more useful.

No. Core and Community scripts and libraries are not to be modified except through a PR. In order to develop them, they need to be first copied to Personal. These libraries have to be standardized so that they can be reusable and supportable. Core will fade into the Scripting API, but Community will remain included in a custom ScriptEngineFactory, a single separate bundle, or as individual bundles available through an ExtensionService.

The HL repo has examples of how to write custom scripts and the use of the libraries. Looking to the future, a fraction of users will use this information. A smaller fraction will develop rule templates that contain scripts using custom libraries. However, these rule templates will be shared through an ExtensionService and used by the vast majority of OH users in their UI rules.

It depends on what is using them. If they are for personal use, the libraries should go in /automation/lib/personal/, or wherever the user wants, with a symlink back to this directory, or with an entry in EXTRA_JAVA_OPTS to add it to the python.path. Third party libraries used by Community scripts or libraries go into the Community package. Look at the Esper and Autoremote packags for examples.

https://openhab-scripters.github.io/openhab-helper-libraries/Python/Reference.html#package-and-module-locations

What do you mean by this? I’m assuming you are using terminology from the ECMAScript spec and are referring to ScriptExtensions?

I’m not following your meaning… I think. The Jython bundle only adds Jython, along with the Core and Community libraries, as an OSGi bundle. Other than the OGSi aspect, it all behaves the same as if Jython and the libraries were manually installed. I’m not sure what you mean by provided namespaces.

Assuming you mean ScriptExtensions, we should not discourage their use. This is the mechanism that will provide most, if not all, of the scripting API. It also allows scripts to store and share objects. What do you mean by “your sub-namespacing”?

When there are breaking changes, I will be renaming core.jsr223. But I’m in no hurry and have not decided what to rename it to. This may be the only core Jython HL that will not be able to be pulled into the Scripting API.

I suggest you read some of the history of the helper libraries…

Originally, they were in a package named openhab, which caused a lot of confusion and namespace ugliness. Core was chosen by consensus and aligns with the naming used by OH and a lot of other software, including Python. I don’t see this changing, but your welcome to bring it up as an issue in GH, which is where any feature requests for the helper libraries should be discussed.

I don’t understand what you mean by this. BTW, Python imports look like…

from package.subpackage.module import function

I think you need some more time with Python ! As for the modifications to ScriptExtensions, I do not see a need for these changes and will respond in GH during my review.

In the last couple months, you’ve had a flurry of development around scripted automation and the helper libraries (thank you!), but my time is extremely limited. Your GraaljsScriptEngineFactory (still needing review) includes CommonJS, which is a step in a direction that I am not yet convinced is needed for the helper libraries. I feel it is very important to keep the helper library functionality as similar as possible between languages in order to ease their absorption into OH as the scripted automation functionality is expanded. The libraries that you are implementing have diverged so much from the Jython HLs, that I’m having trouble imagining them fitting in the helper library repo. The current JS HLs are awaiting JDK9 and ES6 support. The GraaljsScriptEngineFactory could expedite this. Either way, and as I’ve said mseveral times before, the triggers.js and rules.js libraries will be built out to match the Jython versions, or a Scripting API will be built out for them… I’m not sure which will come first!

NO!!! A week before a major release is a horrible time for this, unless you are planning to not have any developer input! You should all be out testing!

There have been module and script name changes, and the directory structure is all new and heavily thought out. However, I will be making some changes in OH3 and will open an issue in GH to discuss them when the timing is better. Changing the directory structure is a huge undertaking, starting with getting consensus!

How so? Both bundles make use of the javax.script APIs. This is our only option until we have an implementation of GraalVM.

In less than a week, the NGRE will no longer be labelled experimental! There have also not been any breaking changes since S1310, back in July 2018. Users may think of it as experimental, but the devs do not treat it as such. There will be a big focus on automation for OH3.

jpg0 · December 11, 2019, 10:25pm

Thanks for giant response. Some of what you say I agree with, some I think are misunderstandings between us, some I don’t agree with.

I do agree that we need more Host (Java) Scripting APIs. There are a number of reasons that I have been putting my time into the JS code rather than Java code however:

The interop between JS and Java is not seamless. Only certain types of collections work. JS equality comparisons don’t work the same way as Java. You can have non-null Java objects that report as null in JS. All this gets really confusing when attempting to write a script, which is why I’ve wrapped all the Java in JS, so that standard JS idioms work. You may be able to work around this if you build the same compatibility layer in Java, to export Java objects to look like JS ones, but that’s seems like a pretty daunting task.
This is why I suggested that we shouldn’t encourage direct access to the host objects/java scripting API.
I regard a module system as an essential foundation for JS before we start exposing this API to it. Jython already has a module system built into it. JS does not. The code in the helper libraries now is by necessity terrible in that it needs to manually locate and inline other files.
I think you already know that I’m also again the current push-style mechanism of the ‘importPreset’ stuff. I guess this is paired with the whole module system above.

Great!

Agree.

I think there is a misunderstanding here. This is exactly what I’m suggesting: that the mechanism to override is to copy to ‘personal’! But what I’m saying is that that is all - that at this point the ‘personal’ version effectively replaces the core one. I think currently in the Python implementation that there are now two, rather than one hiding another.

I want to point out that for 3rd party library support, this matters. There are a number of 3rd party libraries that I’ve used in my scripts. One example is ‘cronstrue’ which converts cron expressions ot human-readable form. In most of them, they import other libraries. How these are loaded is standard, but will only work if we design it to. If we put all 3rd party scripts in a folder which cannot be loaded with a simple require('module_name') then 3rd party scripts will not work. It’s a very simple thing to do, but I want to point out that it does matter (I don’t know whether it matters for Python).

I was not aware of this terminology, I just made it up, but yes!

Re:namepsaces, I’m talking about the raw, imported host objects / script extensions, and the HL APIs. I now see your view that these should replace the HL APIs entirely over time. I was saying that Script extensions I thought was a child namespace of the HL APIs.

Re: naming ‘core’, I’m happy to accept that this is chosen term. I agree that ‘openhab’ is potentially worse. I do want to point out that the module name and ‘directory for OH-provided-HL-APIs’ are distinct concepts however, but I’m not sure they are treated that way.

As for bringing the JS code back in the direction you want, I’d certainly be happy to look at moving things into script extensions. I believe that most of the new HL I’ve written can be ported fairly easily (except the ‘fluent’ stuff, but I’d separate this out into an optional JS-only piece), and made compatible with the Jython HLs. But I don’t think it’s reasonable to do this with no module system in JS. I looked at ES6 modules, we don’t get them with Graal (only their nodeJS runtime, not designed for embedding). I guess a good direction for me would be to get the JS foundation in (which I regard as ES6, modules, script-extensions in modules), then I would be happy to work on the extensions that work for both existing Jython & JS.

They only do this because that it what the OH APIs require. My point is that they don’t need to. Either way, we should wait for Graal APIs.