Configuration of openHAB

[rlkoshak (Rich Koshak)]

@hakan

Maybe this would keep the flood of questions in the forum a bit in check.

in my experience, nope, it wont. There is nothing you can do to prevent that. As evidenced by this thread and the several others, change, even the discussion of change, is like kicking an ant hill. The users become agitated and very active, some of them sting.

We just need to be prepared for it. And preparing for it will include pre-releases that those who are active on the forum can play with ahead of the release not just to find bug but to learn about how to use it and learn ahead of time those parts that might cause users problems, warning message from apt about big changes would be nice, complete docs, etc. This needs to occur before the changes appear in the snapshots I think, at least in part. As we saw with MQTT 2, once it shows up in the snapshots users will find it and start to complain. If we can get ahead of it we can write “my experience postings”, tutorials, and the like which some users will be able to find and we can direct the rest.

@marcel_erkel

Markus’ point is that this will render all the current examples and solutions in the forum and documentation usesless as these will no longer be valid.

Removing Xtend from openHAB will also obsolete any DSL based rules that people have written. In my opinion this will be the biggest pain point for most current users.

There was plenty of discussion and agreement from all on one of these threads that there will be some sort of migration aides developed. Kai himself also mentioned that the Xtend Rules will be ported to JSR223/NGRE so we can continue to use our existing Rules for a time at least.

Assuming everything that I’ve read that has been said on this topic gets implemented, the transition to this new version will likely be as easy or easier than than the transition from OH 1 to OH 2. That’s not to say that it won’t be completely pain free and easy. But it will be manageable.

Someone has already written a script that will port .items files to JSONDB (with some assumptions, search the forum). That script could be expanded, perhaps using some of the code in OH that already handles them. Something like that can handle the Items and Things. Rules will eventually need to be ported to NGRE/JSR223 but I don’t think that one day we will wake up and all of a sudden everything has to be changed immediately. At least I for one will be strongly pushing strongly for this.

@yves

I understand that, and I don’t like it, yet might be ok for some if that is the only way to stay with configs, that is only once every x days/weeks.

Yet I know that Markus and I spend a good deal of time on this forum helping people deal with exactly this problem. Though the bigger problem is this combined with the file watcher queuing up the loads for multiple file changes (i.e. if you change a file and save, OH starts reloading, then change another file and save it queues up that load to occur after the current one finishes).

@yves

That should indeed be turned of until the openhab system is ready.

The issue has been open for over a year. There has been much discussion on it. There is apparently no simple fix to this. It appears to require a near complete rewrite of the file parser/loader.

@yves

No idea why,

I would guess it is like what I describe above. The file watcher starts loading files when it sees a file that has changed, it starts loading files, then more files change so it queues up another file load process. It all depends on timing.

@marcel_erkel

Really, how often do you need to restart openHAB? It should be running 24/7/365. A reboot every now and then shouldn’t be a big problem if it takes a little longer.

But the extended load times greatly exacerbate the file load queueing problem I just described which is a more serious problem.

@hakan

The main, “productive” Installation which must not break lest the family kills me? only if I buy new hardware (which is far too often, but this is another topic )

For some they are seeing memory leaks which have not yet been found and fixed. These users really have little choice but to restart their production OH frequently (weekly, daily).

@marcel_erkel

This problem seems to be an issue only on the Pi.

Its most noticeable on an RPi because the files take so long to load and parse. But RPi is one of the most common platforms to deploy and it is the recommended hardware (if the user has no other preference). So if its a problem on the RPi, IMHO it is a serious problem that needs to be addresses.

@job

But why isn’t it simply possible to create the jsondb entries from the the current text files? On some kind of trigger the jsondb is updated with the text files contents.

I think that is exactly what is proposed with the import/export function David has been talking about.

@Spaceman_Spiff

But how can I support it if it only seems to have negative impacts. It seemingly doesn’t support such simple things like a configuration file.

I think at the end of the day it comes down to this. Kai and company set up OH with this central database. David is not going to sign up to change this. Kai and company who set it up like that in the first place are not going to sign up to change this.

So unless and until someone steps up to do the significant amount work necessary to restructure OH internals, this central database of serialized Java objects is fixed. It isn’t going anywhere. And any discussion about how we will move forward will have to conform to this constraint.

I will submit that David has presented a solution that conforms to this constraint. If anyone who feels strongly about changing this architecture please step up and contribute. It’s not something the current developers are willing to change at this time.

So, without a volunteer to change the underlying architecture and within the constraints of the JSONDB Java Serialized Objects, what can be done to support all the requirements we have set forth? Given these constraints, I think David is making a good faith effort to support as many use cases as possible.

This is an open source project. I don’t think it is fair to blame him for not being willing to take on a change of this type when none of us is willing to set up and do it ourselves.

[Ira (Ira Emus)]

I remember clearly being at developers conference when the features of the new version were announced. The head of development started by saying, “We’ve given you what you need, not what you asked for.” Sadly in that case while the upgrade was exactly what was stated, for some of us it was magic, any walls we had previously found were gone and finding new ones was difficult at best, while for others, they just said, “that’s to complicated” and lived in the past.

At this point I think it would serve the developers best if they just ignored this. If there isn’t already an architect or a group architect that has a well thought out plan that’s being followed, this product has no future and something else will step in to take it’s place. If you know where home automation will be in 10 years, then you belong on the team leading this project, otherwise this is all for naught.

Or, that’s my opinion anyway.

Linux has been around for a long time, but until it got a GUI interface it was just an interesting exercise. Useful for servers, embedded stuff and hobbyists, but not really going anywhere, once there was a GUI and it could replace a Mac, then it started to make inroads. OpenHAB has the same problem possible because HA is essentially still undefined. We each know what we want, but do we know what HA needs. Partly it needs a universal API so anything can talk to anything. I chose Zwave and now I’m stuck there unless I want to dump a lot of money into something else.

This thing needs to be wife friendly, and at the moment there is no way my wife could use this. She can use the app on her iPhone to turn things one and off and that makes her happy, but the way it is now she could not likely be taught to add a new device.

Ira

[yves (YvesHanoulle)]

Maybe it would help to get a link or a short statment of the current state?

you totally lost me

[yves (YvesHanoulle)]

Yes it is a mixture of cache + configuration + runtime data.
And that is the correct and wanted behavior.

I find that a strong statement, I have always heard that it’s a bad thing to mix configuration and run time data, and cache typically is in many different places.

I hear that you want that, and yes it makes things simpler for development.
Yet it’s usually the support people that have a totally a different view on that.

[yves (YvesHanoulle)]

The issue has been open for over a year. There has been much discussion on it. There is apparently no simple fix to this. It appears to require a near complete rewrite of the file parser/loader.

I understand. And I agree that there are many problems related to it.
yet this seems to be the bigger problem then loading takes long.
Once this is fixed, we can see if there still is another problem to fix with the loading.

I’m a big believer in fixing one thing at the time and setting priorities in fixing.

Let’s not try to fix everything together, this usually creates bigger problems and less simple design.

[luckymallari (Lucky)]

@yves

I’m a big believer in fixing one thing at the time and setting priorities in fixing.

I think that’s one of the reasons we have this discussion.

@yves

Let’s not try to fix everything together, this usually creates bigger problems and less simple design.

I agree but at some point we need to have some agility to address things in a timely manner. Otherwise it will become a cat and mouse chase between bugfixes and new features.

[yves (YvesHanoulle)]

I would guess it is like what I describe above. The file watcher starts loading files when it sees a file that has changed, it starts loading files, then more files change so it queues up another file load process. It all depends on timing.

I doubt that, as really all files are downloaded together.
I have noticed this behavior also (or especially) when I restart the system as then I’m usually behind an SHH connection to the system.

But why isn’t it simply possible to create the jsondb entries from the the current text files? On some kind of trigger the jsondb is updated with the text files contents.
I think that is exactly what is proposed with the import/export function David has been talking about.

Except that json does not keep config in same order, so it’s not usuable for version control.

[yves (YvesHanoulle)]

I agree but at some point we need to have some agility to address things in a timely manner. Otherwise it will become a cat and mouse chase between bugfixes and new features.

My main reason to not mix stuff, and to keep things smaller is to actually deliver faster and have more agility. That is what I’m teaching teams big and small for +10 years.

People keep thinking that when grouping features, we deliver faster, that is just not true.

[ChrisW (Christian W)]

@yves

Except that json does not keep config in same order, so it’s not usuable for version control.

You would put the exported files into version control, export service should put it in a somehow sorted order.

[luckymallari (Lucky)]

JSON also does not support comments. Something to keep in mind.

[mstormi (Markus Storm)]

@rlkoshak

Though the bigger problem is this combined with the file watcher queuing up the loads for multiple file changes (i.e. if you change a file and save, OH starts reloading, then change another file and save it queues up that load to occur after the current one finishes).

We really need to be more precise in distinguishing the various issues here.
David says Xtend has to go because it’s lame, resulting in lame startup.
While his claim on files processing may be true, it’s not the key issue.
Don’t get misguided. Exchanging Xtend config processing by something fast - eventually even at the expense of losing the ability to do configuration in files - won’t mitigate the problem of lengthy startup.
That one is caused by non-deterministic order of files processing. I remember it’s filed as ESH #1896 and @henning was about to take on it although now that Dt. Telekom has shutdown their support he likely no longer will so someone else has to jump in.
Note replacing Xtend config processing essentially won’t make this problem go away.
It is accompanied and aggravated by the fact that some rules take longer to process than they should because of the lame parser on specific code such as ESH#6805 on primitives use, and there’s also ESH#6410 which is causing unneeded rule compilations on items changes which happens on bad startup order, so it’s probably related to lame startup as well.
But once again, note that removing Xtend from config processing won’t help against either of these.
But fixing these issues would speed up startup a lot, and it would show Xtend isn’t the root of problems, i.e. removing it isn’t the holy grail either.

And neither of these has anything to do with the problem of OH rules compilations to queue up on overlapping rules changes which I feel is the most urging one.
There’s no issue filed for this I’m aware of but it’s obviously unrelated to Xtend/config processing as well as to startup speed.

[luckymallari (Lucky)]

Runtime state would be extremely beneficial as it follows a more proven event-driven approach. The hardest part is to change all bindings to implement the listener unless we add the listener as an overridable method in the base class

[rlkoshak (Rich Koshak)]

@mstormi

David says Xtend has to go because it’s lame, resulting in lame startup.

And it’s one of the reasons we can’t move to a later Java. 1.8 is pretty much end of life but my understanding is it’s the xtend libraries that is keeping us back right now.

@mstormi

Note replacing Xtend config processing essentially won’t make this problem go away.

its not a problem with JSR223 or NGRE rules though as far as I know. Maybe @5iver can correct me on this if I’m wrong.

I’ve never seen the problem in my experiments at least.

It doesn’t appear to be a problem with the jsondb stuff either. While watching events.log everything appears to load on an orderly and sane manner.

Unless Ive just been lucky, which is certainly a possibility, what is being proposed does indeed appear to address at least some of these problems. It’s honestly one of the reasons I’ve been trying to get NGRE usable from a docs perspective.

[5iver (Scott Rushworth)]

@rlkoshak

its not a problem with JSR223 or NGRE rules though as far as I know. Maybe @5iver can correct me on this if I’m wrong.

Currently, the NGRE and JSR223 have no ties to Xtend. But the existing Rues DSL rules files will need to be parsed and fed into the NGRE as Rules, Triggers and Actions, and this will likely be done through the use of Xtext/Xtend. The tricky part is the Action, which doesn’t have much capability without JSR223. Since Xtend is a JVM language, it’s possible that it could be compatible with JSR223. If it is, we might be able to utilize this to simplify the parsing of the Action portion of the rules files, and spit out Xtend and use it as a scripted Action. I’ve spent some time on this and haven’t been successful… but also haven’t proved it to be impossible!

As far as JSR223 being affected by the clunky startup… yes, this is an issue. In the Jython helper libraries, there is a startup delay script which prevents any other script from starting until AutomationManager is available in the context of the javax script engine. As for the JSON rules created in Paper UI, I haven’t done any testing on whether or not they suffer from slow startups, but I’d be surprised if they aren’t.

[Spaceman_Spiff (Sebastian)]

Could we all please step back and differentiate the topic of discussion a bit.
There were some mixed concerns about the creation of rules (automation) while my initial intention with this topic was to discuss how configuration (e.g. I have these devices) could and should be.

---

@David_Graeff

Very short sighted, probably because you don’t use auto-discovery. openHAB can negotiate session tokens with services via auto-discovery and you will never see those “passwords”. That’s how oAuth based services, ikea Tradfri or the Hue bridges work. Except if you want to create the tokens yourself and enter them in your config files each time.

If they get renegotiated each time why even serialize them? There definitely is the need for runtime data - but as the name suggests it is data that is valid during runtime.
I also never said there is no need for auto discovery nor did I say that I don’t want to persist values that are found through auto discovery. Please pay attention and read posts thoroughly.

@David_Graeff

Yes it is a mixture of cache + configuration + runtime data. And that is the correct and wanted behavior. And the reason why your vision of jsondb->yaml-config-files cannot work straight away. But it does work with a dedicated IO service.

I definitely is not the correct behavior. It maybe was the quick and easy way to implement (let’s just dump all objects with this serializer) but it sure is not a consistent way to do things - otherwise we wouldn’t have this discussion.

@Ira: Please note nobody wants to get rid of the gui - we just want a consistent way to configure openhab. Just like in the linux world where most gui applications just create the appropriate config files in the background. Also your wife will never add things but just use your installation (no insult intended). Hooking up a TV is very easy, too (just plug the cables in the port that fit) but there is a really high chance that you had to do and your wife just wants to watch tv just as she just wants to use your automation system.

@rlkoshak

I don’t think it is fair to blame him for not being willing to take on a change of this type when none of us is willing to set up and do it ourselves.

Nobody is blaming him for being willing to change things - change is good and needed. But if his changes remove key features for some users it is important to step up and voice concerns.
I think it is more that users have made valid points in this thread yet they seem to get ignored for the sake of “progress” and ease of change.

[David_Graeff]

This is my final post on this matter as everything has been said.
I try my best to summarize.

@Spaceman_Spiff

Could we all please step back and differentiate the topic of discussion a bit.
There were some mixed concerns about the creation of rules ( automation ) while my initial intention with this topic was to discuss how configuration (e.g. I have these devices) could and should be.

This is the same in openHAB and must be discussed together.

@mstormi

Note replacing Xtend config processing essentially won’t make this problem go away.

It will. I’ll explain in a moment.

@mstormi

And neither of these has anything to do with the problem of OH rules compilations to queue up on overlapping rules changes which I feel is the most urging one.

Why removing XTend will resolve all our problems

What does it mean to remove it:

• No sitemaps
• No DSL Rule files
• No .thing files
• No .item files

Are there replacements planned

• Sitemaps: Yes, they will be stored in jsondb and will receive a REST endpoint to be editable by GUI and console as well as any other OH API compliant client.
• DSL-Rules: Yes, NGRE/JSR223. Multiple scripting languages supported.
• Thing/Item files: An IO Service with Import/Export pattern. Multiple import/output formats possible. Multiple import/export trigger mechanisms could be implemented.

Will there be a grace period where both technologies will coexist?

Probably not.

Will there be aids to migrate to the new technologies?

Yes.

• Things/Item files will be externally converted to one of the supported Import formats. Free standing comments will be lost. Comments that “annotate” a thing/item will be kept and also exported again. The export IO service will support different storage-associations (aka file-names if the destination is a file system).
• Sitemap files will be read in one last time in Xtend and then stored internally in jsondb from there on.
• DSL-Rules: They can probably be auto-converted to Groovy which can be run by JSR223. Auto-conversion is not perfect, so complex rules need manual adaption. Nobody so far has written anything. If that doesn’t change until the big break, then the recommendation will be: Don’t update.

Solving startup and file reload problems

Rules:

DSL-Rules are compiled. And they need to be recompiled each time a referenced “item” changes. The heuristic is not perfect and rules are sometimes recompiled out of order. This fact cannot be fixed as suggested by some users. Recompiling takes some time.

JSR223 scripts are not compiled and they do not suffer from the same problem (there is some ahead of time compilation for some chunks of the code, but you will only notice this as a small hickup during the first run of a (changed) script).

There is one current outstanding, easy to fix issue: The JSR223 engine should only start its job as soon as every other bundle of OH is ready. And yes that is possible in current OH to archive, I’m doing that in the Hue Emulation service as well (which need to start last, so that all Items are already available).

Things/Items:

With an explicit Import pattern we will not run into the file-watcher disaster and queuing up of changed file parser requests. An import will be transaction that completed as soon as all data is fed into the system and all related rules are restarted (not recompiled!). While a transaction is running, there cannot be another one interfering.

Conclusion / Effects for the user

• The user need to learn a new .thing/.item/sitemap file format.
• The import triggers want to be configured once. (E.g. saying yes/no to directory watching, periodic checking, import during startup).
• All documentation regarding .thing / .item files are obsolete from this point on. Which is fine, because you don’t need that anymore because it is so darn easy to use openHab then. And most questions arose out of the fact that .thing / .item files are complicated to use at in the first place.

All said.

My personal attitude

I don’t know about you guys, but I want a stable home automation solution. And some old OH1 technology is blocking me here. I have the same needs as you: Add bindings, configure my installation as easy and comfortable as possible and then create automation rules and add intelligence to the system.

People who are contributing are also steering how a product is used. I see the need and benefits of editing configuration in a textual way (find&replace, batch editing, cloning etc). Physical text files are not required though.

Will mentioned changes drive some people away from openHAB?: Yes probably. But purely for stubbornness reasons, not for technical ones. And those people will come back later on. Not only because OH will be a stable fast solution by then but also because our competitors are not doing better in any way.
I expect that we gain about 10x more people at the same time, because it will be maybe 10 clicks to add your Hue bulbs, tag your items, and export them for Alexa usage. Another 10 clicks to add your auto-discovered DIY IoT device to Alexa. Common use cases must be easy and they will be, just saying.

Cheers, David

[Spaceman_Spiff (Sebastian)]

@David_Graeff

This is the same in openHAB and must be discussed together.

I strongly disagree. They just happen to use the same form of technology and you therefore decide to discuss them together. But ultimately these are different topics which have different requirements thus requiring different solutions.

@David_Graeff

Thing/Item files: An IO Service with Import/Export pattern. Multiple import/output formats possible. Multiple import/export trigger mechanisms could be implemented.

Still an export is only a snapshot at a certain time and thus will not solve the problems of the two truths. Also I am still not understanding how the export will remember the file structure I created in the first place.

@David_Graeff

I don’t know about you guys, but I want a stable home automation solution. And some old OH1 technology is blocking me here. I have the same needs as you: Add bindings, configure my installation as easy and comfortable as possible and then create automation rules and add intelligence to the system.
People who are contributing are also steering how a product is used. I see the need and benefits of editing configuration in a textual way (find&replace, batch editing, cloning etc). Physical text files are not required though.

But what you don’t seem to have in mind is the ease of maintenance. You always assume that there is access to the machine running openhab while this is not the case.
Also physical files are the only reason why textual configuration is much more mighty than editing in a predefined GUI can ever be. If we loose physical files there is no need for textual configuration at all.

@Kai:Is this the new form of governance where there is one dev pushing features effectively breaking openhab usage for others even though valid concerns were raised beforehand?

[David_Graeff]

@Spaceman_Spiff

Still an export is only a snapshot at a certain time

Exactly. And it therefore solves the problem of the two truths.

@Spaceman_Spiff

still not understanding how the export will remember the file structure

I forgot to mention that in my post. There’s probably a “storage-order” field next to the “storage-association”. The idea is of course to keep the export as stable as possible to guarantee small diffs for small changes.

@Spaceman_Spiff

@Kai:Is this the new form of governance where there is one dev pushing features effectively breaking openhab usage for others even though valid concerns were raised beforehand?

I’m sorry but this shows your absolute inability to read. The sitemap change was approved by Kai, the new rule engine was approved by Kai and the .thing/.item file replacement with another syntax is approved as well. You are so much against change, that’s unbelievable. And you do not even think of a solution (that works!) yourself. Unbelievable. Really.

[mstormi (Markus Storm)]

@David_Graeff

No DSL Rule files

Oh dear. Now you’re finally telling the full truth. I have been fearing you would raise that, too. So your vision is even much more impacting than we thought.
Sure if we remove/replace the rules engine that would ‘fix’ the problem of overlapping rules processing, too. Just like dropping OH altogether would ‘fix’ all of my OH problems.
But that ‘fix’ now would be at the expense of all the work all the people have put into creating their rules!
To date, we have been discussing about config data only and the impact of replacing that.
Then again, to replace rules as well has much more impact on users and existing installations as removing the existing rules engine would invalidate all rules any OH user ever wrote, too.
You even want that to happen as a ‘big bang’, i.e. without any grace period and backward compatibility, i.e. no chance of going back if things don’t work.
You effectively want to force anybody to rewrite his rules to NGRE/JSR223 just because of your personal crusade on Xtend.
You don’t spend thoughts on alternatives, just claim they don’t exist.
Sorry but I don’t believe those claims of yours. It’s pure (and biased) guessing of yours that this will bring us more users than it will make us loose (not to factor in that our position is to keep the existing interfaces so we wouldn’t loose anyone).
You’re not convincing when you indicate there’s no way to modify the existing code such that it stops compiling rules when it notices the source changes. You’re not convincing there’s no way to build a non-Xtend parser that does NOT change the current input format(s).

@David_Graeff

@Kai:Is this the new form of governance where there is one dev pushing features effectively breaking openhab usage for others even though valid concerns were raised beforehand?

I’m sorry but this shows your absolute inability to read. The sitemap change was approved by Kai, the new rule engine was approved by Kai and the .thing/.item file replacement with another syntax is approved as well. You are so much against change, that’s unbelievable. And you do not even think of a solution (that works!) yourself. Unbelievable. Really.

Ouch.
First - don’t get personal. I’m stating that as a moderator.
Second, you’re wrong. Any approval to add alternatives such as NGRE does not contain/imply the approval to drop any existing solution.

I already stated that in my last post but will do again:
Noone is against positive changes - additions such as PaperUI-NG are welcome.
But everybody except you is against dropping proven functionality and interfaces because that will invalidate knowledge and a LOT of work people have put into OH, and it’ll force them to learn and put work into migrating their installations.
That indeed is a governance question and mustn’t be decided in favor of some single developer’s vision which is in heavy contrast with the majority of people.
(@kai I guess it’s time for you to step in).

[Udo_Hartmann]

As a long term user (since OH1.0…) just my two cents:

I’m still running my home through OH1.8

1. Because OH2 has suffered solutions for some bindings which I need.
2. In case of knx, I had to wait more than two years for a working solution.
3. It’s configuration is far more complex than in OH1 (!) and it’s still not working as reliable as in OH1
4. There is less stuff integrated in my Test installations of OH2.5, but even with more RAM it takes ages to restart OH2, where OH1.8 is starting very fast. Both on XenVM, same host, same VM configuration (1GB, 1CPU, 10GB hd, debian stretch64, current Java8)
5. OH2 is less reliable (I repeat, with a less complex configuration).

Although I’m pretty familiar with rules DSL, I’m willing to learn a new language, but PLEASE, when changing formats and languages, don’t make it more and more complex every time (and simultaneously lose some functionality!), please make it more easy, especially for non-programmers. Hello world in rules DSL:

rule "hello world"
when
    Item my1stItem received command // or another trigger
then
    my2ndItem.sendCommand("Hello World")
end

How’s the JSR232/Jython/whatever preferred language looking? By the way: an easy to learn language like VisualBASIC would be very nice, at least for the execute part.

I’m not a fan of GUIs when the GUI is “One field per parameter, One page per channel” and so on.
I’m ok with a GUI which provides a text-like configuration (like demonstrated by David far above…)

There has to be an option to do mass changing (like changing dozens of group membership for Items in an items file at once). In a text file, it’s so easy to change all appearances of a single word, let’s say I want to change the word “kitchen” to “Küche”. When defining a mass of similar items, in a text file I’ll do a copy&paste and change a detail of each item. PLEASE make it easy for users!

I’m fine with changes, when there is a real benefit for both users and developers, but for users this would be

1. less complexity
2. less complexity
3. less complexity