Configuration of openHAB

(Udo Hartmann) #201

Really the whole code? Every snippet of code I saw until now, was blown up. This one is far more better to understand (at least for me…)

(Rich Koshak) #202

One should note that JavaScript doesn’t have annotations like Python does so the rules in that language will be a little more verbose but not much more than Rules DSL. I don’t know what Groovy Rules look like.

One should also note that Rules defined through the UI and those fully text based can all work together and interact. No longer will we be faced with either/or choices for Rules development.

(Scott Rushworth) #203

Well, I left off the imports, but they would be used by all rules in the script. Here’s a thread…

Note, ‘System started’ will work after this PR, but I’ll need to make an update first. Actually, I got Jython 2.7.1 working too, which will also require an update to the modules.

It does, but not in ES 5.1, which ships with JDK8. JDK9 has some of ES 6, but it does include support for classes. I recently got OSGI working in a JS script, but we’ll need classes for ScriptExtensions. I’ve been slowly migrating the Jython helper libraries to JS and Groovy, with the intention of putting them all in the OH-Jython-Scripters organization (after a rename)… preparing for the addition of Jython and Groovy to Paper UI from this PR. Hopefully soon, I will have some bundles for the installation of Jython and Groovy… possibly with the core helper libraries.

With Nashorn currently deprecated in JDK11, it’s doubtful we’ll see the @, which arrived in ES7. :slightly_frowning_face: But with GraalVM, there will be many more things supported. :slight_smile:

(Lolodomo) #204

With this discussion, for the first time ever I am anxious of what could be the future of openHAB ! I hope we could keep the flexibility which is the major advantage of openHAB.
I hope Kai will enter in the discussion soon. Now we need to be reassured.

(Rich Koshak) #205

One thing to watch for that I encountered with Lewie’s library is you have to exclude triggersAndConditions.js from being imported into your Script Actions or else you get duplicate symbol errors. It makes sense since the UI does all that stuff for you.

See post 192 and post 199.

When I post I tend to limit myself to the as-is state of things. But it’s good to know that annotations might be coming to JS one way or the other.

(Lucky) #206

Nope but we do have decorators. If JS is one of the future languages we’re going to use, in that case, I’d prefer Typescript

The key here is backwards compatibility.

(Rich Koshak) #207

Right now we have Nashorn because that is what Java ships with and it’s the default in the current PaperUI NGRE because it doesn’t require any further actions on the part of the user to install something else. But, that has been deprecated and like Scott indicates, GraalVM will eventually come on the scene soon. At that point theoretically any language supported by GraalVM will potentially be usable as a Rules engine. I say theoretically because one never knows what problems might arise, as like what happened when someone tried to get JRuby to work with JSR223 Rules about a year or so ago.

I’m not sure I understand the comment. What I was saying is that Rules defined through PaperUI (or what ever GUI) and that get stored in the JSONDB (for now) can “see” the Rules defined wholly in text using JSR223 type syntax, and visa versa. And since one of the things we should be able to do in both types of Rules is call other Rules that gives us some interesting capabilities, as well as giving us a middle step for those who wish to migrate one way or the other.

(Lucky) #208

Ahh. Got me confused by your post before this one. I thought you were referring to DSL based (files) and the new way interacting together. Not sure why it wouldn’t though, given a proper bridge.

(Rich Koshak) #209

If Scott is successful being them to run on JSR223 and GraalVM, with some minor modifications and additions it should be true for Rules DSL too.

(Nico Bille) #210

So… I hope I will not be off topic here and I can explain some personal opinions on these things.
I can read code - a little bit - and I can write some rules. And when not @rlkoshak was always helpful (yes, it’s been some time).
I ilke textfiles for the ease of “copying them away” (backup) and for mass editing.
But what I wanted to explain was my journey with iobroker. There is an openhab-adapter for iobroker and because of their visualisation I wanted to try that. In the end I haven’t used the vis, but other things:
iobroker’s adapter have for the most an autodiscovery and will import all things they can find through this routine. As the KNX-adapter for OH2 was in developement I tried their adapter. It had an importoption through the ETS-Projectfile. So I had all my KNX-Items in iobroker. homematic-things where discovered and hence in the system. Same with some other adapters. So all the things/items I had in iobroker was nice.
What helped me then was that as ruleenginge you had the possibility to just make little javascripts to make your automationrules. And because it’s so rare I write these things they had blockly, an editor where you can click javascriptscripts together. I had made neat little automations in a very fast time.
Then I discovered they had also node red integrated and tried that. Was usable for many things. But wait, I discovered there are openhab-modules in node red… And then I could migrate back to openhab and have node red as rule engine and the OH2-KNX-binding was ready, too.
iobroker has also many flaws but what I wanted to point out is how easy it was to have all my items of the different technologies in the system and with blockly, node red and all items availible how fast I could my home “smarter” (come on, one buzzword is ok?). The effort I had to make was a fractions of what to do on OH2.
As @Udo_Hartmann stated “less complexity” - yes please.
The differentiation between the things and items has some logic, but I have the feeling 99 % of the time it’s just doubling the work.
Having things editible through the webinterface in a pleasent and efficient way is a good thing for quick little edits and I would like to be able to do that because having some device with a browser availible is amost everytime given.
I am absolutely pro config files but I have also no Idea about the runtimedatabase and how much effort is needed to make this and that.
I just wanted to show what things I liked on another system and what things may be attracting other and new users and tell some points which may be worth to consider in the discussion.
If this has been discussed elsewhere please just ignore…

(Ian Macdonald) #211

I am going to jump in here, I have been using openhab since the V1 days and went through the confusion about what I can and can’t do in paperui, then reverted to doing as much as I can in text files so I can build rules. I will admit i read about half the thread before I started skimming to the end so apologies if this has already been said.

Lets take a step back and have a look at how other large projects are doing this kind of thing. In the systems engineering/devops world things are evolving towards infrastructure as code. In the Chef world this means a ruby DSL that describes how a service is deployed and configured. Puppet, Ansible and Salt all do it a little differently, but ultimately the goal is to be able to get the code into a source code repository eg git a complete description of how something is deployed, have it is versioned, and have it regression tested. The bonus is you can share your chef examples with other users in the community. This is also how most software development is done. It all comes back to a text. So I think having a text based way to deploy OpenHab is important for the reasons i stated above for infrastructure as code.

Lets put that aside for one second and agree that we also need a way for non technical users or users who just don’t care about versioning, sharing, repeatable deployment, and just want a gui to click on. I think one of the challenges of openhab is that it is a relatively complex thing to install, requires knowledge of linux, how to install rpm/deb packages, even installing using openhabian on a raspberry pi has a reasonably high bar. So our user base will tend to be on the more technical level to start with. If openhab wants to become more mainstream then that is an area that needs to be worked on. Lets just assume openhab need to pass the spouse test, so being able to use openhab by a spouse who didn’t install it should be a goal that we have. In my opinion openhab fails the spouse test at the moment, my spouse is a software developer and I would never dream of inflicting paperui on her. Hab pannel is getting closers but she still will not use it. So the TDLR is that we need a UI that works for management and configuration.

So how have other companies solved this? The first example i am going to use is Tenable’s Security Center. They split the UI from the Core and the only way to interact with core of the product is via the API, but you can write your own interactions using what ever language you like as long as you use the REST based API. If you don’t want to do that then you use the provided UI that talks to the API. They have replaced the UI completely a couple times, the first was in Flash, then the more recent version is HTML 5. The second example is Amazon AWS, again everything is via an API, if you want to deploy using infrastructure as code eg git based text files you can use tools like Teraform to configure and deploy your complete service from the network, security, loadbalancers, all the way to webservers. If you want to do it manually most of the AWS APIs now have a web GUI that allows you to configure it.

So here is my suggestion, lets focus on creating an API for creating, updating, and deleting the types of objects we need, eg rules, items, things, persistences, bindings. Then build an abstraction layer that allows you to do everything via text files, or yaml, or json, or the existing openhab DSL. If you don’t like the yaml then there is nothing to stop you from writing your own app that uses json. Then the GUI people can build their version that does the same thing using the same APIs. It really shouldn’t matter how the data is persisted in the core app as long as it can manipulated consitently via the API. I haven’t looked at the openhab APIs in detail so this much of this may already exist, so all that has to be done is build an app that works with the existing text files.

Lets look at the rules file as an example. Is there a way to upload a rules files to openhab via the API? I am guessing no since I can’t edit a rules file via PaperUI but willing to be corrected on this. Having a simple way to upload an existing items,things,configs,rules would be a kind of V1 API type thing. Then V2 would would add more complex APIs that allow you build via something like a GUI. The V1 API would give everyone the same thing they have right now with some wrapper code that that you run to perform an update. If you use github you could even publish using a webhook on merging a pull request. The V2 API would require a more complex app that parses the rules and translates them into the V2 API calls to manipulate the data stored in the core app.

Using an API also gives you the advantage of being able to query the core for auto discovered things and create textual representations of them if you choose, in essence this could be the way to backup all the data stored in core. The Web GUI might use the APIs to export it to a single yaml file and give you a graphical way to import only the sections you want, again the actual format shouldn’t mater as long as the app you are using knows how to backup the data then import it again.

With an API based solution all you need to do is write or convince a developer to write an app that does what you want, be it GUI or Text.

What do people think, could this work for us?

(Rich Koshak) #212

But OH isn’t a company. Companies have resources and market forces that apply that will not apply to a purely FOSS perked line this.

OK, we’re done. Now what? What you describe concerning having a REST API to the core with a UI that only works through that API Is how OH already works.

Which is exactly what was proposed. The bulk of the thread you skimmed concerned arguments about details and ways to make that happen.

OH but it does because OH itself can generate config through automatic discovery and that config needs a way to come back out of OH in a useful and source control friendly manner.

This is the crux of the problem.

For NGRE, yes. For Rules DSL no because the part of OH that handles this is legacy OH 1.x so it only can read the .rules files. Adding such a rule via the REST API isn’t supportable at this time. And it probably doesn’t make sense to add it and instead focus on adding support for Xtend syntax rules to NGRE abd/or providing a migration path from the legacy Rules DSL to NGRE.

Which is also another major point of contention discussed above because some users have a configuration management that very much depends on the file watcher.

Nothing you’ve proposed so far is new and has been discussed at length above. You are basically describing what David proposed that so many here argued against for various technical reasons.

(Ian Macdonald) #213

Your right, i missed some stuff in my skimming :slight_smile:

This rest is not in response to @rlkoshak, just a general comment, and I appreciate @rlkoshak valuable comments in this thread.

My main concern is that Davids proposal seems to be a UI based approach to the detriment of a text based solution. GUIs tend to have limitations when you trying to represent code graphically, I see crontab format being discussed in this thread and the challenges in getting a gui that can represent all the options available.

How will we help users debugging a rule in the openhab forum with the new implementation, 4 or 5 screenshots that show how everything is set?

What I see being proposed is similar to the way habpanel is works, eg widgets and javascript for more advanced stuff, but I am concerned that I might accidentally loose my habpanel configs or adjustments I make on the fly and forget to back up the config. That has happened to me at least once.

Changing to a different rules syntax will mean all the current knowledge base of how to do things will become void. I still see references to OH 1.X ways of doing things when I search for how to solve stuff. So I am sure the OH 2.X stuff will stick around for a while causing confusion.

A language converter would be nice, but i suspect it will end up being too complex given the current development constraints, so will be dropped. Moving from OH1 -> OH2 was painful for me, I suspect this change will also be as complicated and mean learning a new language. I have been using openhab for a while and only relatively recently did I work out that openhab rules are not the same syntax as java ( it does look very similar so I had just assumed it was since openhab is written in java) and instead use Xtend, and then how to do bitwise operations in Xtend. So having good documentation on how to do it will be vital, with lots of examples :slight_smile:

I would strongly suggest that we have a way to support power users that want to serialize and load their data files in a way that can be controlled via source control. I would really like it if an entire OH system could be cold booted from text files. I have had to rebuild my entire install from scratch multiple times. If i could control everything via a text file then this would have a been a simple exercise. Instead I ended up removing everything then discovering it via the inbox in paperui, set the name, set the location, etc so it was back the way it was at the start from paperui point of view.

while I see the desire to remove the OH1 code, I don’t think that justifies the desire to get rid of the of a way to configure everything via editable text files. This is a very powerful feature for the power/advanced users. I think it would be worth investing in a UI/extension that replicates and extends the watch file functionality via a standard APIs.

(Sebastian) #214

Also currently we have a mixture of runtime cache and configuration which imho has to be resolved, too.

Also there are some similarities things are not quite the same. David proposed a triggered import/export from the jsondb which implies that the jsondb will always be the source of truth.
@imacdonald312 proposed that all configuration goes through the same API and that there is no configuration cache (besides cached runtime information) which is the master. This would mean that all ways to configure openhab are equal since the contained information is everywhere the same.
The information master is the API-call that sets the corresponding information.

(Christian W) #215

Yes, but you will also need a storage for the API configured configuration - currently the json db.
Otherwise this system would not benable to restart/ reboot. The API calls of a REST API are one way, OH can’t ask back ‘how was item xy setup’ it needs to rember this by itself.

(Sebastian) #216

This obviously does not work with REST as you pointed out correctly. There has to be some configuration API internally of openhab which then can implement custom serializers. On restart openhab then has to request the configuration through the API making it load the config from database/file/whatever.
This also implies that there is a way to persist configuration through the API.

(David Graeff) #217

Guys, don’t argue. See Directions for openHAB 3.

Just ask the Architecture Council what is possible. Present your solution to the problem in a more or less detailed whitepaper, with examples or whatever. I’m not part of the AC, so nobody must be afraid of me being vetoing any textual config suggestions (although I never did, but that’s another story).

Cheers, David

(Kai Kreuzer) #218

Oh yes, guys, please DO ARGUE! The AC is not meant to disrupt any discussions, creative thinking and suggestions - rather the opposite. If there are good ideas on how to improve stuff, the AC is meant to help to say “yes, we do it”, so that such discussions do not end up in a void and nobody knows how to move forward with it.

So I think it is very important that @David_Graeff brought up the question of configuration in openHAB as this is indeed a pain point. Clearly there are different opinions on what is needed and I think we all agree that we need to support power users with textual and beginners with UI options (and @David_Graeff does not question this afaik).

I have to admit that I didn’t follow all posts in this pretty long topic - would anyone be able to give a summary of potential different options or maybe even create a small poll to see what options might be most relevant to further pursue in the discussion?

(David Graeff) #219

Kai, you really didn’t read this thread, otherwise you wouldn’t have suggested this :smiley: That was the most heated discussion I have ever seen in this forum, but quite interesting as well. The forum summarize button works quite well and condenses the discussion into a 20min read.

Basically I said something like:

  • The current textual config syntax will probably change (yaml, etc) to get rid of XTend/XText for thing/item/sitemap files
  • File watchers for thing/item files are a bad idea, especially with thing/item files as one source of truth. A user triggered import/export with the jsondb as the only source of truth will be the technically cleaner solution. That would downgrade item/thing files to pure “Imports”. On conflicts jsondb will win and not other way round like it is now.
  • An export mechanism would write out what is stored in jsondb.

People are then worried that:

  • the order of things/items would not be preserved
  • the file association is lost (nobody wants all things dumped into a single big file)
  • comments are lost

My solutions are:

  • We store a “storage-association” in thing and item objects
  • We store a “storage-order” in thing and item objects
  • We have an “annotation” field for all jsondb serialized DTOs where we keep the comment that is directly above a thing / item in a yaml (or differently formatted) file.

Cheers, David

(Ssalonen) #220

Even though I have read the whole thread one thing remains unclear for me in the import/export solution…

Does it only “append” things & items? What if one removes a thing from the xx.things and then re-imports? Or is the storage association which is used to detect than xx.things no longer has the thing, and as such it should be deleted from runtime?