Configuration of openHAB

Development
212

But OH isn’t a company. Companies have resources and market forces that apply that will not apply to a purely FOSS perked line this.

OK, we’re done. Now what? What you describe concerning having a REST API to the core with a UI that only works through that API Is how OH already works.

Which is exactly what was proposed. The bulk of the thread you skimmed concerned arguments about details and ways to make that happen.

OH but it does because OH itself can generate config through automatic discovery and that config needs a way to come back out of OH in a useful and source control friendly manner.

This is the crux of the problem.

For NGRE, yes. For Rules DSL no because the part of OH that handles this is legacy OH 1.x so it only can read the .rules files. Adding such a rule via the REST API isn’t supportable at this time. And it probably doesn’t make sense to add it and instead focus on adding support for Xtend syntax rules to NGRE abd/or providing a migration path from the legacy Rules DSL to NGRE.

Which is also another major point of contention discussed above because some users have a configuration management that very much depends on the file watcher.

Nothing you’ve proposed so far is new and has been discussed at length above. You are basically describing what David proposed that so many here argued against for various technical reasons.

213

Your right, i missed some stuff in my skimming :slight_smile:

This rest is not in response to @rlkoshak, just a general comment, and I appreciate @rlkoshak valuable comments in this thread.

My main concern is that Davids proposal seems to be a UI based approach to the detriment of a text based solution. GUIs tend to have limitations when you trying to represent code graphically, I see crontab format being discussed in this thread and the challenges in getting a gui that can represent all the options available.

How will we help users debugging a rule in the openhab forum with the new implementation, 4 or 5 screenshots that show how everything is set?

What I see being proposed is similar to the way habpanel is works, eg widgets and javascript for more advanced stuff, but I am concerned that I might accidentally loose my habpanel configs or adjustments I make on the fly and forget to back up the config. That has happened to me at least once.

Changing to a different rules syntax will mean all the current knowledge base of how to do things will become void. I still see references to OH 1.X ways of doing things when I search for how to solve stuff. So I am sure the OH 2.X stuff will stick around for a while causing confusion.

A language converter would be nice, but i suspect it will end up being too complex given the current development constraints, so will be dropped. Moving from OH1 → OH2 was painful for me, I suspect this change will also be as complicated and mean learning a new language. I have been using openhab for a while and only relatively recently did I work out that openhab rules are not the same syntax as java ( it does look very similar so I had just assumed it was since openhab is written in java) and instead use Xtend, and then how to do bitwise operations in Xtend. So having good documentation on how to do it will be vital, with lots of examples :slight_smile:

I would strongly suggest that we have a way to support power users that want to serialize and load their data files in a way that can be controlled via source control. I would really like it if an entire OH system could be cold booted from text files. I have had to rebuild my entire install from scratch multiple times. If i could control everything via a text file then this would have a been a simple exercise. Instead I ended up removing everything then discovering it via the inbox in paperui, set the name, set the location, etc so it was back the way it was at the start from paperui point of view.

while I see the desire to remove the OH1 code, I don’t think that justifies the desire to get rid of the of a way to configure everything via editable text files. This is a very powerful feature for the power/advanced users. I think it would be worth investing in a UI/extension that replicates and extends the watch file functionality via a standard APIs.

214

Also currently we have a mixture of runtime cache and configuration which imho has to be resolved, too.

Also there are some similarities things are not quite the same. David proposed a triggered import/export from the jsondb which implies that the jsondb will always be the source of truth.
@imacdonald312 proposed that all configuration goes through the same API and that there is no configuration cache (besides cached runtime information) which is the master. This would mean that all ways to configure openhab are equal since the contained information is everywhere the same.
The information master is the API-call that sets the corresponding information.

215

Yes, but you will also need a storage for the API configured configuration - currently the json db.
Otherwise this system would not benable to restart/ reboot. The API calls of a REST API are one way, OH can’t ask back ‘how was item xy setup’ it needs to rember this by itself.

216

This obviously does not work with REST as you pointed out correctly. There has to be some configuration API internally of openhab which then can implement custom serializers. On restart openhab then has to request the configuration through the API making it load the config from database/file/whatever.
This also implies that there is a way to persist configuration through the API.

217

Guys, don’t argue. See Directions for openHAB 3.

Just ask the Architecture Council what is possible. Present your solution to the problem in a more or less detailed whitepaper, with examples or whatever. I’m not part of the AC, so nobody must be afraid of me being vetoing any textual config suggestions (although I never did, but that’s another story).

Cheers, David

2 Likes
218

Oh yes, guys, please DO ARGUE! The AC is not meant to disrupt any discussions, creative thinking and suggestions - rather the opposite. If there are good ideas on how to improve stuff, the AC is meant to help to say “yes, we do it”, so that such discussions do not end up in a void and nobody knows how to move forward with it.

So I think it is very important that @David_Graeff brought up the question of configuration in openHAB as this is indeed a pain point. Clearly there are different opinions on what is needed and I think we all agree that we need to support power users with textual and beginners with UI options (and @David_Graeff does not question this afaik).

I have to admit that I didn’t follow all posts in this pretty long topic - would anyone be able to give a summary of potential different options or maybe even create a small poll to see what options might be most relevant to further pursue in the discussion?

2 Likes
219

Kai, you really didn’t read this thread, otherwise you wouldn’t have suggested this :smiley: That was the most heated discussion I have ever seen in this forum, but quite interesting as well. The forum summarize button works quite well and condenses the discussion into a 20min read.

Basically I said something like:

  • The current textual config syntax will probably change (yaml, etc) to get rid of XTend/XText for thing/item/sitemap files
  • File watchers for thing/item files are a bad idea, especially with thing/item files as one source of truth. A user triggered import/export with the jsondb as the only source of truth will be the technically cleaner solution. That would downgrade item/thing files to pure “Imports”. On conflicts jsondb will win and not other way round like it is now.
  • An export mechanism would write out what is stored in jsondb.

People are then worried that:

  • the order of things/items would not be preserved
  • the file association is lost (nobody wants all things dumped into a single big file)
  • comments are lost

My solutions are:

  • We store a “storage-association” in thing and item objects
  • We store a “storage-order” in thing and item objects
  • We have an “annotation” field for all jsondb serialized DTOs where we keep the comment that is directly above a thing / item in a yaml (or differently formatted) file.

Cheers, David

220

Even though I have read the whole thread one thing remains unclear for me in the import/export solution…

Does it only “append” things & items? What if one removes a thing from the xx.things and then re-imports? Or is the storage association which is used to detect than xx.things no longer has the thing, and as such it should be deleted from runtime?

221

In my very first proposal I saw an import process as a replacement process. JSONdb would only contain what is specified in thing/item files.

In later proposals this is more like an import. So only adding / modifying.

But actually you can have both. Just imagine there are two triggers. One will append/modify, the other will remove/modify/add.

222

Summaries might be/are biased, so to get the full picture I’d recommend reading through.
There are some interesting ideas in this thread and it would not be just to condense them to something they are not.

3 Likes
223

I tend to agree with Sebastian. It will be painful but there are a number of unique perspectives here that will be hard to summarize.

I think David presented the summary of his position pretty well, though I think part of what riled up a bunch of users was his going in position that perhaps we don’t need text config at all and just use the GUI.

I’m sure I’ll forget some things but in my mind the requirements that have come out of this whole thread are (sorry for not providing attribution, it’s just too much to go back and find who posted what). Note some of these are contradictory:

  • there shall to be one source of truth in the configs
  • the internal JSONDB shall be the one source of truth
  • the external file configs shall be the one source of truth
  • there shall be a migration path between text configs and internal configs
  • there shall be a way for OH to write out configs to capture automatically discovered Things for instance
  • text configs shall preserve order so small changes do not result in large diffs when using source control
  • there shall be a way to organize entities so related things are colocated in the same file/directory/category/whatever
  • there shall be a way to automatically import text configs to support certain automated deployment use cases
  • the text configs shall not be automatically imported to avoid the file poll
  • the text configs shall change to a more standard formal like YAML/XML/JSON
  • the text configs shall remain unchanged in format
  • the UI shall support file like operations (copy/paste/edit) to manually add entities
  • the internal database shall not mix config data with runtime data
  • the JSONDB shall remain and remain unchanged
  • the JSONDB shall be removed and replaced with something else
  • all Xtend based stuff shall be removed and replaced with something else
  • some/all Xtend based stuff shall be retained
  • comments in text configs shall be preserved when exporting that config back out of OH

For each of the above there is at least one user who feels strongly that the requirement should be met and has good reasons for their belief.

I think if I had to make a tl;dr, developers want a text based config that allows them to work with OH like any other development platform using source control, repeatable automated deployment, and the like. This is at odds with the current JSONDB internal database which has a mix of runtime and config based stuff and no way to get stuff out of it in a way that supports this approach.

One thing that David has treated as a fixed quantity is that JSONDB shall remain and JSONDB shall be the “single source of truth” for OH config. But based on your comment on the Architecture thread that mentioned this, I don’t think that it is fixed. Replacing JSONDB in whole or in part I think absolutely is on the table. Perhaps we can find more common ground if that is taken as a consideration. Can we meet more of the requirements listed above (and any I missed) with a different approach to how OH stores it’s internal config and internal state?

@David_Graeff, assuming that you could make JSONDB work however you want it to work, would this change anything about your approach? Don’t worry for the time being about who would do the work to change out the DB. Let’s just brain storm how you would design the “single source of truth” storage and how that might change anything recommended above. I get the feeling that a lot of the restrictions you placed upon yourself stem from treating JSONDB as a given. What if you didn’t?

@Spaceman_Spiff, can you elaborate more about how you would design OH’s internal config strorage down to the next level?

Anyone else please feel free to join in.

Let’s try to be constructive, friendly, and cooperative. No one is going to unilaterally make any decisions on this topic.

2 Likes
224

If I could over night change OHs code base then @Spaceman_Spiff suggestion is very reasonable. The runtime data (thing properties, channels, object references etc) is stored in fast binary fashion. The configuration in separate files. But the configuration would still be written to by openHAB in some cases, like auto discovery or when a binding stores thing configuration (like session tokens etc).

But that’s a only a dream considering how openHAB internally works.

225

If we can get everyone on board perhaps it is a dream we can make a reality. We have the opportunity now to present a case for why the way it internally works is holding us back or causing other problems. I’d hate to come up with a suboptimal solution just because we didn’t consider all the options.

Perhaps I’m taking this statement beyond what was intended but I read Kai’s bullet point from Directions for openHAB 3

  • Simplify switching from UI to textual configuration and vice versa - maybe through a new database format, by export mechanisms or whatever.

to indicate that major changes or replacing the JSONDB is not outside the range of consideration.

1 Like
226

Yes Kai is brainstorming here, I read I like you as well. But that subsystem was just replaces in oh 2.2. Why not first address persistence and sitemaps. An import/export service like I have proposed it is written in a weekend. And this one conversion tool of that other guy already allows to import and export items for example. So migration wouldn’t be the issue.

227

Thanks @David_Graeff and @rlkoshak for your summaries.

I am indeed open to ANY solution and my bullet point that you mentioned above was indeed referring to this discussion here - whatever the outcome is.

Maybe some additional infos:

  • Originally, we used MapDB as the internal database, which wrote binary files. Users complained that if the database broke for some reason, they had no chance to look into it and manually fix it. That’s why @chris at the time took the effort and implemented the JSON DB instead, so that the files are human readable and potentially editable by the users (only while the system is shut down). But this was really driven by the requirement that people wished a “new format” and JSON being very popular among developers was pretty much agreed upon by most. You can actually use JSON as a configuration file format and not use DSLs at all. I personally always preferred the very compact and concise syntax of the DSLs, though and never liked writing JSON by hand.
  • The internal architecture is done in a way that it is not at all coupled to any DSL nor database format - everything is fully pluggable and different implementations can be provided (just like the MapDB vs. JSON options). So in theory, we can add further options and do not necessarily have to remove existing ones (we only have to think about the rising complexity of documenting/supporting all those options).
  • Wrt exporting the database to DSL format, we have an according issue/discussion since quite a while in the issue tracker: https://github.com/eclipse/smarthome/issues/4817. It’s just that nobody has implemented it so far…
  • From my personal pov, I would actually LOVE to get rid off the Xtext code (while not necessarily changing the file syntax, this would be another question). The reason is that it is pretty complex code for which we hardly have any maintainers. Xtext upgrades are always painful, the build is much more complex due to code generation, etc. What we would lose though is the LSP support, so the VS Code plugin would need to be improved to cater for this. So any option that would allow us to remove Xtext, would help on the maintenance side (the most difficult part to remove are probably the rules as Xbase is interpreted, but that’s again a different discussion).
1 Like
228

This sounds like the multiple truth problem which came up in the thread a couple of times.
Imho the problem is that there is an internal database which not only stores runtime data (otherwise the file could just be deleted) but it also stores configuration data. It is essential that there is a differentiation between runtime cache (e.g. to improve startup speed) and configuration.
Maybe a ConfigProvider and a CacheProvider so everything is still pluggable?

From what I have read in the thread nobody actually wants Xtext code. It just happens to be the technology which is currently used. Removing Xtext for config files (only) could also be a smaller task since it’s quite easy to write a parser for the file format (I wrote an config checker in python which checks for logical errors a while ago). Creating and writing to the files is a whole other story though.

It’s because json is actually a machine-interchange format which just happens to be readable :wink:.
The current format is indeed very compact, but as a trade-off the structure is different for items, things, sitemaps, persistence, etc…
I therefore think it’s best to use a universal file format which can be used for all these configuration, even though it means that it will be more explicit and less compact.
As I pointed out in my earlier post yaml is the way to go because when done correctly it even provides the flexibility to the user to split configuration across multiple files as he wants (e.g. moving certain sections on any level to another file, not just merging files on a top level together). It also is human editable and allows serialization/loading of custom classes.

1 Like
229

The JSON DB is not about any runtime data, it is pure configuration. And if you do not use any DSLs, they are the one and only truth in your system.

Removing Xtext for config files (only) could also be a smaller task since it’s quite easy to write a parser for the file format

Well, as mentioned, you are losing the LSP feature with it, which is imho very helpful. And having a custom parser can also be complex as soon as you start doing changes to the file format. With Xtext, it is actually pretty easy to add additional properties as all you need to do is to change the grammar.

As I pointed out in my earlier post yaml is the way to go because when done correctly it even provides the flexibility to the user to split configuration across multiple files as he wants

Would be cool if you’d come up with a POC for a yaml syntax and a parser. As pointed out above, we could easily add this as a second (experimental) option, so that people could give it a try how it feels.

1 Like
230

I see, but if I use another source of configuration there is the need to keep them in sync. As a user however I only maintain one source of configuration so this will always be the cause for problems.
Imho caching of configuration should be delegated to a ConfigProvider or at least not be persisted (e.g. if done manually).

I’d happily provide something, but I am neither fluent nor good in Java.
I could provide a POC in python however if that would be fine, too?
It would be just a converter to the current *.items etc files

231

Not strictly true. Markus has valid concerns about the disruption that will occur when/if all the current users of OH are forced to recreate their entire configuration and Rules again. I’ve not voiced them as much but I have such concerns too. It’s a thorny problem. I do believe the majority of commentors in the thread do agree that we are OK with it going away, as long as we have tools and aides to help us transition to the new format.

I like this idea. If we have some devs willing to try it out we can prototype it and see how it goes.

1 Like