Even though I have read the whole thread one thing remains unclear for me in the import/export solution…
Does it only “append” things & items? What if one removes a thing from the xx.things and then re-imports? Or is the storage association which is used to detect than xx.things no longer has the thing, and as such it should be deleted from runtime?
Summaries might be/are biased, so to get the full picture I’d recommend reading through.
There are some interesting ideas in this thread and it would not be just to condense them to something they are not.
I tend to agree with Sebastian. It will be painful but there are a number of unique perspectives here that will be hard to summarize.
I think David presented the summary of his position pretty well, though I think part of what riled up a bunch of users was his going in position that perhaps we don’t need text config at all and just use the GUI.
I’m sure I’ll forget some things but in my mind the requirements that have come out of this whole thread are (sorry for not providing attribution, it’s just too much to go back and find who posted what). Note some of these are contradictory:
there shall to be one source of truth in the configs
the internal JSONDB shall be the one source of truth
the external file configs shall be the one source of truth
there shall be a migration path between text configs and internal configs
there shall be a way for OH to write out configs to capture automatically discovered Things for instance
text configs shall preserve order so small changes do not result in large diffs when using source control
there shall be a way to organize entities so related things are colocated in the same file/directory/category/whatever
there shall be a way to automatically import text configs to support certain automated deployment use cases
the text configs shall not be automatically imported to avoid the file poll
the text configs shall change to a more standard formal like YAML/XML/JSON
the text configs shall remain unchanged in format
the UI shall support file like operations (copy/paste/edit) to manually add entities
the internal database shall not mix config data with runtime data
the JSONDB shall remain and remain unchanged
the JSONDB shall be removed and replaced with something else
all Xtend based stuff shall be removed and replaced with something else
some/all Xtend based stuff shall be retained
comments in text configs shall be preserved when exporting that config back out of OH
For each of the above there is at least one user who feels strongly that the requirement should be met and has good reasons for their belief.
I think if I had to make a tl;dr, developers want a text based config that allows them to work with OH like any other development platform using source control, repeatable automated deployment, and the like. This is at odds with the current JSONDB internal database which has a mix of runtime and config based stuff and no way to get stuff out of it in a way that supports this approach.
One thing that David has treated as a fixed quantity is that JSONDB shall remain and JSONDB shall be the “single source of truth” for OH config. But based on your comment on the Architecture thread that mentioned this, I don’t think that it is fixed. Replacing JSONDB in whole or in part I think absolutely is on the table. Perhaps we can find more common ground if that is taken as a consideration. Can we meet more of the requirements listed above (and any I missed) with a different approach to how OH stores it’s internal config and internal state?
@David_Graeff, assuming that you could make JSONDB work however you want it to work, would this change anything about your approach? Don’t worry for the time being about who would do the work to change out the DB. Let’s just brain storm how you would design the “single source of truth” storage and how that might change anything recommended above. I get the feeling that a lot of the restrictions you placed upon yourself stem from treating JSONDB as a given. What if you didn’t?
@Spaceman_Spiff, can you elaborate more about how you would design OH’s internal config strorage down to the next level?
Anyone else please feel free to join in.
Let’s try to be constructive, friendly, and cooperative. No one is going to unilaterally make any decisions on this topic.
If I could over night change OHs code base then @Spaceman_Spiff suggestion is very reasonable. The runtime data (thing properties, channels, object references etc) is stored in fast binary fashion. The configuration in separate files. But the configuration would still be written to by openHAB in some cases, like auto discovery or when a binding stores thing configuration (like session tokens etc).
But that’s a only a dream considering how openHAB internally works.
If we can get everyone on board perhaps it is a dream we can make a reality. We have the opportunity now to present a case for why the way it internally works is holding us back or causing other problems. I’d hate to come up with a suboptimal solution just because we didn’t consider all the options.
Yes Kai is brainstorming here, I read I like you as well. But that subsystem was just replaces in oh 2.2. Why not first address persistence and sitemaps. An import/export service like I have proposed it is written in a weekend. And this one conversion tool of that other guy already allows to import and export items for example. So migration wouldn’t be the issue.
I am indeed open to ANY solution and my bullet point that you mentioned above was indeed referring to this discussion here - whatever the outcome is.
Maybe some additional infos:
Originally, we used MapDB as the internal database, which wrote binary files. Users complained that if the database broke for some reason, they had no chance to look into it and manually fix it. That’s why @chris at the time took the effort and implemented the JSON DB instead, so that the files are human readable and potentially editable by the users (only while the system is shut down). But this was really driven by the requirement that people wished a “new format” and JSON being very popular among developers was pretty much agreed upon by most. You can actually use JSON as a configuration file format and not use DSLs at all. I personally always preferred the very compact and concise syntax of the DSLs, though and never liked writing JSON by hand.
The internal architecture is done in a way that it is not at all coupled to any DSL nor database format - everything is fully pluggable and different implementations can be provided (just like the MapDB vs. JSON options). So in theory, we can add further options and do not necessarily have to remove existing ones (we only have to think about the rising complexity of documenting/supporting all those options).
From my personal pov, I would actually LOVE to get rid off the Xtext code (while not necessarily changing the file syntax, this would be another question). The reason is that it is pretty complex code for which we hardly have any maintainers. Xtext upgrades are always painful, the build is much more complex due to code generation, etc. What we would lose though is the LSP support, so the VS Code plugin would need to be improved to cater for this. So any option that would allow us to remove Xtext, would help on the maintenance side (the most difficult part to remove are probably the rules as Xbase is interpreted, but that’s again a different discussion).
This sounds like the multiple truth problem which came up in the thread a couple of times.
Imho the problem is that there is an internal database which not only stores runtime data (otherwise the file could just be deleted) but it also stores configuration data. It is essential that there is a differentiation between runtime cache (e.g. to improve startup speed) and configuration.
Maybe a ConfigProvider and a CacheProvider so everything is still pluggable?
From what I have read in the thread nobody actually wants Xtext code. It just happens to be the technology which is currently used. Removing Xtext for config files (only) could also be a smaller task since it’s quite easy to write a parser for the file format (I wrote an config checker in python which checks for logical errors a while ago). Creating and writing to the files is a whole other story though.
It’s because json is actually a machine-interchange format which just happens to be readable .
The current format is indeed very compact, but as a trade-off the structure is different for items, things, sitemaps, persistence, etc…
I therefore think it’s best to use a universal file format which can be used for all these configuration, even though it means that it will be more explicit and less compact.
As I pointed out in my earlier post yaml is the way to go because when done correctly it even provides the flexibility to the user to split configuration across multiple files as he wants (e.g. moving certain sections on any level to another file, not just merging files on a top level together). It also is human editable and allows serialization/loading of custom classes.
The JSON DB is not about any runtime data, it is pure configuration. And if you do not use any DSLs, they are the one and only truth in your system.
Removing Xtext for config files (only) could also be a smaller task since it’s quite easy to write a parser for the file format
Well, as mentioned, you are losing the LSP feature with it, which is imho very helpful. And having a custom parser can also be complex as soon as you start doing changes to the file format. With Xtext, it is actually pretty easy to add additional properties as all you need to do is to change the grammar.
As I pointed out in my earlier post yaml is the way to go because when done correctly it even provides the flexibility to the user to split configuration across multiple files as he wants
Would be cool if you’d come up with a POC for a yaml syntax and a parser. As pointed out above, we could easily add this as a second (experimental) option, so that people could give it a try how it feels.
I see, but if I use another source of configuration there is the need to keep them in sync. As a user however I only maintain one source of configuration so this will always be the cause for problems.
Imho caching of configuration should be delegated to a ConfigProvider or at least not be persisted (e.g. if done manually).
I’d happily provide something, but I am neither fluent nor good in Java.
I could provide a POC in python however if that would be fine, too?
It would be just a converter to the current *.items etc files
Not strictly true. Markus has valid concerns about the disruption that will occur when/if all the current users of OH are forced to recreate their entire configuration and Rules again. I’ve not voiced them as much but I have such concerns too. It’s a thorny problem. I do believe the majority of commentors in the thread do agree that we are OK with it going away, as long as we have tools and aides to help us transition to the new format.
I like this idea. If we have some devs willing to try it out we can prototype it and see how it goes.
I was just going to say the same thing. I think it’s important to distinguish between xtext and the grammar. There’s no linkage between the two other than the current implementation. It seems everyone is ok with getting rid of xtext. It’s the file format (i.e. the current grammar definition) that we would like to be preserved.
Indeed those topics did come up. I didn’t want to put them in as requirements because they really are bugs in the current OH more than a requirement for how the future OH should do things. But indeed, any new config file system would need to not reintroduce or preserve these problems.