[WIP] NGRE/JSR223 Documentation refactoring

Hello interested reader

I wanted to start this ages ago, but i simply had time yet to start with it in my own environment.
Now i have done some things with jsr223 and have an impression of it.

I think we should give this area a complete rework, which is the reason for pinging some persons who have had already contact with jsr223:
@5iver, @lewie, @rlkoshak

Of course we can use the already existing parts in the docs.

Info:
I will update this start post or make it a wiki post, so we can archive the current state from time to time.

What

I want to collect ideas and requirements here.
Ideas of how we can do and describe things and requirements for the contents that have to be included.

How

In a first step i would like to focus on the “vanilla implementations”.
Since we are talking about the official documentation, i would like to avoid promoting special libraries/repositories for now.
I know there is some change going on anyway, so we may change the docs representation of the libraries too at a later point.

I am also pulling in rich, as he has already witten a bunch of threads about the ngre in general.
Since all is depending on the same codebase i think it is just straight forward to see this as one topic.

Invitation

Of course everyone is welcome to help us here with ideas, suggestions, feedback and contributions.
Your help is highly appreciated.

Current state of the TODO list:

  • Rename JSR223 into something readable
    • Suggestions: ?!
    • Mention the jsr223 spec of course
  • General Rules Article/Area
    • Add a generic rules article and describe existing rule implementations
    • Add some pros an cons for rules-dsl, automation modules
  • JSR223 Area (New name of course)
    • Prerequesites
      What is needed
    • Installation
    • Global introduction for parts that are valid for all jsr223 implementations
    • Javascript,
    • Jython
    • Groovy

Additional thought (not sorted):
Maybe we should leave out PaperUI for the first attempts, since we are facing some changes in the UI world currently.
Include som stuff from User enhancements

What are your thoughts and ideas?

2 Likes

I’ve been holding off on some of this and stopped my Next Gen threads for now largely because it’s a bit of a moving target. Some open questions/things in work that I think need to be completed before we go too far down this path:

  • there are a few important bugs that are in process of being fixed, the big one that comes to mind being the broken System started triggers
  • the separate helper libraries are under the process of becoming merged into one central library
  • I believe the plan is to distribute the helper libraries so they can just be used, no extra download and update necessary
  • similarly, I believe the plan is to distribute the Jython jar file (I don’t know what Groovy needs) so all three languages are equal full class citizens in the OH ecosystem
  • while the underlying engine is the same, creation and management of Rules will look and therefore require very different documentation for GUI created Rules and text created Rules.

Hopefully Scott can provide some insight into whether I’m correct in these statements and a timeline: already done, likely part of 2.5 release, definitely part of 2.5 release, not until 3.0 or beyond.

For example, if the libraries and needed jar files get installed as part of the Experimental Rules Engine package or separately install-able as it’s own OSGI bundle (or whatever) then the Installation section of the docs becomes moot. Similarly, if we distribute the helper libraries (in which case they cease being helper libraries and become “the way to do things”) the docs could look pretty different as we won’t have to document a whole lot of stuff beyond “use this function in the library”.

Anyway, let’s just assume we would be documenting what exists right now.

It might be helpful to include a little bit of a generic theory of operation section explaining how OH overall works. We would include information like how OH is event based, how events flow to/from Rules, common gotchas that will be true no matter the language (e.g. sendCommand won’t necessarily change the state of the Item immediately), stuff like that. Given the questions we see here in the forum, some of these basic fundamentals about how it all works are missing. This may be what you intended by “how does it work” but I want to state it outright in case it wasn’t.

I also think we need to have two types of documentation. We need reference docs, which is frankly all we have now with Rules DSL. But what is missing and what I think we will need is some sort of tutorial that gradually introduces Rules programming concepts to the users. We should note have to reproduce the underlying language’s basics though links to tutorials and reference docs will be needed. But something that looks a lot like your Time of Day translation tutorial where we start very simple (log out something to show it works) and gradually build up a Rule with links and help to teach the users how to use the reference documentation might be useful. As we build up the Rule(s) in the tutorial we introduce important concepts like the different rule triggers, useful Actions, working with Items and Groups, timers, etc. We don’t need to put everything in the tutorial, that’s what the reference docs is for. But we want to give the users, particularly non-programming users, a hands on path they can follow and do to understand important Rules concepts and get them used to looking for answers in the reference guides.

If we can hold the user’s hand a bit we might have a better first experience for some users (some won’t read the docs no matter how good they are). It’s just a thought.

To reiterate in bullet form, with a few more ideas added:

  • overall theory of operation for Rules in OH
  • reference docs separate from tutorial
  • per language design patterns? Some I imagine would be universal as they have to deal with how OH works
  • when it becomes available, where and how to find and use rule templates (maybe that will eliminate the need for design pattern docs)

I strongly suspect that the overall way that users create their Rules is going to be very different once we have Rule Templates and the GUI Rules builder. Many/most average users will click together their Rules, sometimes even very complicated Rules using the GUI and importing and using Rule Templates. Advanced users like us will become Rule Template writers and there will become an ecosystem akin to bindings for Rules Templates as well. If this does happen, it will only be the very technically advanced users who will be writing .py/.js files by hand. At that point the reference guide will largely become a guide for developers. We are a long way from that but it’s where I can see this going.

I think almost all we need in terms of reference information, at least for Jython and perhaps for JavaScript are available in various places. We should be able to consolidate that into the official docs and that all by itself would be an improvement. The step by step tutorials need to be written, but you’ve already a big jump start on that.

I feel like I’m rambling so I’ll stop here.

3 Likes

Thanks for the extended input rich.

You are fully right.
We definetely should not put a high amount of work in finished articles for special “under construction” areas.
But we can for sure add them to a table of contents with a “do later” label.

I have also thought about the tutorial part, but i hadn’t yet a good idea how to handle this properly.
The Migration tutorial may give us some parts, but it aims clearly on people with a bit rule experience.

Also I had another idea today and started investigating.

Currently jsr223 is a bit isolated in the docs.
It has its area and nothing more.
Most of the other docs contents is referring to rules-dsl examples.
So i am looking for a way to change this in the future.

My idea is to provide rule examples in different languages.

It could look like this

rulesTabs

and i started my investigation how that could be possible with our vuepress website.

Reference:

Of course this is an idea in an early state which can be adapted, if someone has a better one. :slight_smile:

2 Likes

I definitely like the idea. I’m a little wary though as we will have four times the code to keep up, but I think it’s important not to privileged any of the languages.

Given that we don’t want to privledge any of the languages over the rest, I’d suggest where ever we have them listed we use alphabetical order. The somewhat random order in your example implies an order of preference I think which I don’t think we want to imply.

I’ve also been holding off in communicating about NGRE and scripted automation. There are a few more things I’d like to wrap up, but getting close! In regards to naming, I prefer using Scripted Automation rather than JSR223, as it’s really javax.script and this works for when/if GraalVM is implemented. I will be submitting a PR to remove/rename the /automation/jsr223/ directory, but that would be for 3.0.

I fixed this a few months back :slight_smile:, but have yet to merge the Jython helper library portion, as it will be included with an upgrade to support the 2.5 and the ESH reintegration. However, the StartupTrigger will only be available with helper libraries, until an official one is written.

My plan is to implement a Scripting API (probably 3.0) and migrating/expanding the Core Actions, which will eliminate the need for most of the helper libraries. Distributing libraries is basically last on the list. However, with consolidation of the libraries, I also plan to extend the Groovy and JS libraries to match the functionality of Jython. JS libraries will benefit a lot from OH support of JDK9. There also appears to be a recent issue with Groovy, which will need to be corrected before its libraries can be done.

WIP! Hopefully ready in time for 2.5. The Scripted Actions/Conditions for Jython and Groovy are already in the snapshots…

The raw API is ugly and NOT easy to use. I suggest directing beginners to the helper libraries, as the raw API will be a big turnoff. This will improve with the scripting API and Core Actions, but I think any tutorial should be kept to a minimum ATM, and suggest to use the helper libraries and their documentation. The common piece that might work in the official docs is the Action content, which is what you would need for scripted Actions/Conditions in a UI, so using the ‘events’ object, ScriptExtensions, thing actions, binding actions, and language specific tutorials like if/then/else, iterables, etc. If work is going to be done now, it needs to be weighed against the upcoming changes, which will quickly make them obsolete and IMO not worth the effort. I suggest focus on the helper library documentation, and pull some of the generic info out of them to put into the official docs.

I’ll be getting some posts out soon!

3 Likes

I completely agree and I meant to bring that up. I do think a new name is in order.

1 Like

All I can say is I agree with everything you’ve said. Thank You! :+1:

@5iver, Nashorn is deprecated since Java 11. In order to use GraalVM we have to follow the JAVA version used in openHAB. Is there possibly already a rough schedule when openHAB will be switched to Java 11?

A small request, please name JavaScript as what it is - "JavaScript ". “ECMAScript” is the term for the standard that only geeks like us know. ECMAScript (ECMA 262) is the standardized language core of JavaScript. Not to confuse beginners, so please use: “JavaScript (ECMA - 262 Edition 5.1)”. Or does the term JavaScript have such a bad reputation? :wink:

@Confectrician, I love the Tab Design. It makes the comparability very clear.

You can tell me when and where the current JS helper libraries should be added. Or do you do this in the course of the upcoming preparations?

I think it makes sense to wait a bit for some progressbefore we have a look at the libraries.

I just read in https://www.graalvm.org/docs/getting-started/ that GraalVM is based on OpenJDK 8!

Has anyone tried to estimate whether the conversion effort would be small enough to switch from JSR223 to GraalVM already now in openHAB 2.5 or 2.6?

I only mean, because the expenditure which we start now, would build then already on the future-safe platform.

What do you mean?

No schedule, but it’s close… https://github.com/openhab/openhab-distro/issues/768.

Not so small :smile:! Here’s an example of what you can get from the javax.script.ScriptEngineFactories when JS, Groovy and Jython are present…

getEngineName getEngineVersion getLanguageName getLanguageVersion getMimeTypes getExtensions getNames
Oracle Nashorn 1.8.0_201 ECMAScript ECMA - 262 Edition 5.1 [application/javascript, application/ecmascript, text/javascript, text/ecmascript] js [nashorn, Nashorn, js, JS, JavaScript, javascript, ECMAScript, ecmascript]
Groovy Scripting Engine 2.0 Groovy 2.4.12 [application/x-groovy] groovy [groovy, Groovy]
jython 2.7.1 python 2.7 [text/python, application/python, text/x-python, application/x-python] py [python, jython]

I used a combination of getLanguageName and getLanguageVersion in the ScriptTypeModuleProvider. There is no dynamic way to identify the ScriptEngineFactory as compatible with JavaScript, so that it could be used in the name. I brought this up in several times when discussing the PR in GH. It could be changed so that the names aren’t generated dynamically, if it is really that important. Personally, I prefer seeing the actual name and version, so that I know specifically what is supported. For example, if we just say JavaScript, someone could try to use classes in their script.

I’m not sure of the level of effort, but the introduction of GraalVM would be a significant change that I expect would have to go into a major release. I do not know yet if it is something that is technically feasible or would be approved, but it’s about time to create an issue to start the discussion. Like javax.script, GraalVM can be used for much more than automation, so the discussion could take a while. From what I’ve picked up, it sounds like we will go from 2.5 to 3.0, which would leave very little time for it, and there is already a lot of work to do within automation before then.

I know what ECMAScript means. Dynamic reading is already best! There are many, many more important things! :wink:

The integration would not mean that all other facets that GraalVM can offer have to be done at the same time. It would also correct the later conversion to Java 11/12, if JSR223 has to be replaced.

Of course, it would only make sense if GraalVM could be made usable relatively easily in openHAB 2.5 and few classes would have to be adapted. Thank you for your estimation, you are much deeper in the story.

1 Like

Maybe we should cut out the graalvm topic from here, since it isn’t that related to the documentation.
I will prepare a little summary at the weekend and then we can have a look at things that could be done/prepared already.

2 Likes

In this context I think JDK11 and GraalVM are for openHAB 3. If I remember correctly from one of the heated topics Kai also indicated that openHAB 2.x would stay on JDK8.

I think it would be good to indicate for any of the given ideas whether you think it is for openHAB 2.x or 3.x.

1 Like

Several people reacted positively to this post. There may be some info in it that could be used in the docs.

3 Likes

We should definetely think about the generic rule presentation in the docs too, while organizing the jsr223 refactoring.

This is something we have to improve.
We should not say xtend is bad or jsr is better, but we should have a rules area that explains the possibilities and then afterwards shows more information about the different ways of doing rules.

I am also thinking of extracting the rules articles out of the configuration section and moving it into its own docs section.
Rules is such a big topic and you cant explain it with one side per rule techinque.

2 Likes

I agree. I wonder if a hierarchical organization makes sense. We have a root article that talks about what Rules are for and some of that generic stuff that I’ve mentioned above. Then under that we link to separate pages for Rules DSL and each of the JSR223 languages.

That way we privilege none of the languages over the others and it becomes apparent even to newer users that there is more than one option.

I’m ambivalent as to whether we should also mention third party options here. Part of me thinks it would be helpful and part of me thinks it’s not our job.

I had something similar in mind as wrote

Currently, there is no Concepts> Rules section. We could make one and put the general discussion and explanation of the differences between the rule engines in there. The Configuration Guide> Rules and JSR223 sections could be combined and use the fancy tabs as in Jerome’s example. IMO we should put emphasis on the new rule engine, with UI first and then put the DSL at the end. After 3.0, the DSL section can be tagged as legacy or moved to another area which includes steps for migrating .rule files. The New User Tutorial> Working with Rules and Script section could focus on working with rules in the UI, as that is what will be easiest for a beginner.

I really like the tabs! In looking at the docs, there is a wall of information that I think the tabs could help organize in a way that it is easier to locate what you are looking for. For example, the sidebar for the Installation Guide could just be Overview, OS and Hardware, with tabs in each section. This is too general for this discussion, but those tabs could really help the docs. :slightly_smiling_face:

Sorry for the delay, prio was on other topics.

I have edited the first post with an updated TODO list.

Feedback welcome.
I think i will setup a jsr-refactoring branch in the docs repo already, so we can do some improvements in a first run.