Personally I don’t care how it works. I’m a fairly new user and my needs are simple, I’m comfortable with text file and once I figured out that Items and rules beliong in text files and things belong in PaperUI I did fine setting up what I needed to please my wife.
But, if you expect this product to grow, you need to make it so my wife can make changes which means it all has to be through a GUI. The way it is now and if you keep requiring text files, 99 of every 100 people who come here will move on.
Making text files an option so that the people stretching the edges can keep stretching makes perfect sense, but home automation should be for everyone and openHAB needs to realize that or something else will come along.
How it works underneath should not be a concern of the users because if it works, the majority of the users will not know or care. If you think it matters, become a developer and show the team a better way, isn’t that what open source is about.
Hm but that is how OH2 is build internally, it allows multiple ways to access the internal database (jsondb), but that has to happen in a structured way. The console and the rest interface are using the correct interface. And the next logic step is to make the file inputs also use that interface. Please accept that technical fact.
You are probably just used to the OH1 way of thinking. But this is OH2.
This topic is turning into the same conversation like in the other discussion. There’s a poll in the beginning, saying that 90% of people like GUI and textual ways.
6% GUI only and the rest textual only.
The poll unfortunately doesn’t differentiate between people who want a textual interface for copy/paste, search/replace, cloning or real text files.
Nonetheless. OH and also the developer environment must become more newby friendly (but stay powerful of course). Please ask yourself what compromises you are willing to offer for that goal.
I think you are still not understanding what I am proposing. The yaml file is a drop in replacement for the jsondb. But with the yaml file structure it is possible to move configurations into another yaml file or while it still can be used with the yaml parser. This provides the flexibility for proper configuration.
Openhab does auto discovery and detects a z-wave device and links some items. This is what gets created
Note that the openhab.yaml is still valid and can still be written to by the parser.
Openhab registers the file changes and loads everything back.
A part of the gui configuration became now manual configuration.
–> Benefit of db & gui but still allows full manual configuration
With a good concept there is no compromise needed!
I think David’s point is you cannot have static read only text configs and automatic discovery of Things. That is a hard break. Since removal of automatic discovery of Things is not up for discussion we require OH to be able to write this information to disk.
This is the choice. Either get rid of automatic discovery, in which case you may as well go back to OH 1.8, or come up with a more reasonable way to handle the fact that configuration information can come from both the user and OH itself.
See the Hue binding. You configure just the bridge, and for the light and control Things, auto-discovery works just peachy. I believe this would also hold true for MQTT things via the Homie convention. So, in my mind, it is possible to find proper sweet spots between all-text and all-GUI. This is also the way I would expect my dream HA system to work.
The “static read only text configs” provide only just the minimum necessary for the rest of the system to work. Which does not preclude us die-hard greybeards from fully defining all attributes of the Things ourselves, of course
And where do those automatically discovered Things get saved? The binding doesn’t go out and rediscover the Things every time OH restarts. It discovers them, you accept them out of the inbox, and they get saved.
Unless you are wanting to have all the Things be rediscovered every time, in which case that makes it impossible to set any configuration parameters on discovered Things because there is nowhere for those parameters to get saved.
Yes, but tl;dr that will not work. Let me elaborate.
I got you so far and I now understand why you think that this would be possible.
Yaml, Json, XML doesn’t matter. All those markup languages have “external reference” syntax, not just yaml. So in theory, we can do that today with just json and json-schema. But that is actually unnecessary if you watch an entire directory anyway, isn’t it?
You imagine that openhab.json would continue to be the runtime storage and your side-files are now “manually maintained”.
And that is where you are lacking technical implementation details. Json DB as we call it is actually just a serialized copy of an in-memory state (java objects).
The file system is only used as “backup” and is read once on start-up to fill the in-memory state.
And it can only serialize the exact state of the jave objects, no artificial file boundaries are considered.
This is fundamental to openHAB and it can’t be changed over night, nor should it be changed. In-memory is always faster then to wait for the filesystem to confirm a write transaction. Currently the jsondb file is written out every 5 seconds (if a change happened in that cycle).
Discovered things can simply be saved to .things files. If you can save them to a database, then you can save the same data to a flat file.
Currently they get saved to a database which also contains application runtime state. Mixing application runtime state with application configuration is in my opinion a very bad idea.
How are you going to do a clean start? By first manually editing the JSONDB and removing everything that has to do with state? I don’t know if that is even possible without either missing something or accidentally breaking something.
Why do we like Docker so much? Because it packages everything the application needs and allows you to basically run everywhere, but also because it gives us the possibility to start with a clean sheet each and every time we run the container. Someting goes wrong with the container, we simply throw it away and start fresh. We know that the new container instance will run because the last time we started with a new container it also ran. We start from the exact same baseline.
If I start openHAB with a JSONDB that contains runtime state information from a previous instance then I can never start from a known working baseline because that baseline has been contaminated.
Why do we even store state to disk? As soon as openHAB is stopped we must consider the state invalid because the world around openHAB has changed and we cannot guarantee that the state that openHAB persisted is still valid. It’s the same as the Wasp in the Box principle when the box is opened. Until the box is closed and we see or hear a life sign of the wasp inside the box we must assume the fact that the wasp is no longer in the box.
The only benefit I can see why we need to persist runtime state information is for debugging. I had to edit the JSONDB once because for some reason removing a binding using PaperUI didn’t go well and parts remained in the database. This would not have been necessary if the JSONDB had been just an in-memory database. In that case a simple restart would have sufficed.
Storing to disk is slow and doing this every 5 seconds (if a change happened in that cycle) causes unnecessary wear on the the SDcard used by many SBC’s.
When storing a file to disk there is always the chance of data corruption. And guess what, we’re storing runtime state to the same file that contains our carefully crafted application configuration. We’re taking that risk up to 12 times a minute, 720 times per hour, 17280 times per day, which leads to 6.307.200 times each year. That worries me.
Why not? He already included a storage association in his design study. Simply save the Thing to file and I can set the file attributes to read-only as a hardening measure. So I don’t think that is what he meant.
I said “'nuff said” because I’m getting tired of this UI only tunnel vision. Everything needs to be done via the UI, even to the extend that VS Code is now integrated into the PaperUI-NG design study. Why?! Microsoft releases new versions of VS Code every month or so. There are two or three openHAB releases per year. The included version of VS Code is already old by the time a new openHAB version is released. It’s not exactly DRY (or better DRO; Don’t Repeat what Others can do better ).
There are so many great editors out there already.
But I think I’m going to step out of this discussion for a while and see how things develop, and if needed, build my own tooling around this (should I still feel the need).
I am not talking about external reference but rather custom classes which do load and merge the other files.
This is fundamental and why this concept works. !!load_from_file items.yaml is not an external referance but a class that gets constructed with the external filename. It is then used to merge the manually created configuration with the automatically created one.
The class gets serialized to !!load_from_file items.yaml with a custom resolver but actually does contain the file contents because these get loaded in the constructor.
I am fully aware that the files on the disk are always just a serialized copy of in-memory state.
But this is no problem.
Let’s play a load and a dump through.
openhab does serialize to/from openhab.yml and it contains a gui-configured thing “GuiThing”
a custom thing “ManualThing” is defined in mything.yml
in openhab.yml there is a class registered to load mything.yml
there is a file-watcher in place for openhab.yml
if there was a dump to openhab.yml the consequent file events will be ignored
Openhab.yml is loaded. Custom class with external files is constructed and loads configuration for ManualThing. Configuration is then merged in memory and contains configuration for “GuiThing” and “ManualThing”.
Since all things/items/etc. have unique names dumping is easy, too.
First, check if changed thing is in the custom class. If yes, don’t serialize these changes and show warning message. This step is optional.
Serialize data to openhab.yml.
The external, manual files get serialized to !!load_from_file mything.yml so there is no need to know about file boundaries.
It is possible to create configuration through the GUI, but a manual configuration will always have more priority then a configuration through the GUI. There is only one source of truth. Automatic loading and dumping still works.
Good - so we agree that this approach is valid and possible.
Tbh I only think this would affect configuration and serialization.
But this is the development section and we’re here to discuss concepts. And when introducing new concepts I expect that there are changes to be made.
While these changes may not be “low hanging fruits” they solve the problem of two truths while providing the possibility to configure through GUI and textual files. And this is the consens how everyone here wants to configure openhab.
openHAB is prepared to use different storage providers. That is the easy part to switch to yaml.
But everything after that means a lot of work. Things would need to remember their storage association etc. Because the storage service doesn’t know any meta data about the objects its going to serialize.
But as I said, have a look at the code. Personally I’m happy with the internal state database as it is so will not do anything in this regard. I’m more interested in a fast start-up and a good UI. And currently .thing/.items files are parsed via XTend/XText which pulls in a gigantic 3prd party library that is slow and evolves slow (we can’t update to newer java versions at the moment because of that) and I just want to get rid of that mess and instability that it inflicts.
I was under the impression we are talking about configuration not about storage of items/things.
It is perfectly fine to have only one way of doing so ( and this is the status quo as you keep mentioning that everything gets loaded from jsondb).
Nobody wants the keep .thing/.items files nor the XTend libraries. The only dependency here is the yaml-parser and configuration would be done using the yaml syntax.