I see the developer perspective, of course. It is about maintainability, basically.
I have this suggestion, two folded:
Thing files as input
Use Thing/Item files for a “first-start” input only, by for example copying those files into an “input/” directory.
Works on start as well as while OH is running.
OH consumes those input files and removes them on success (every single line could be imported and didn’t cause a conflict) and let them sit there on failure (maybe creating a thing-file.thing.failed log file).
Sync back to Thing files
Via a command (openHAB console, REST API [Paper UI]) openHAB can store its current state back to Thing/Item files. Via parameters openHAB can be instructed to split Things to multiple files. Comments for Things are restored, global file comments are restored (if a generated file matches an input/ filename).
Would this approach be viable?
The important part is, that Thing/Item files are not viewed as constant definition files like they are now. Every input should go through the openHAB API and stored internally in however openHAB like it do to for performance and conflict-avoidance reasons.
Wasn’t there a mentioning that the current implementation is based on libs that do allow for reading but not for writing back to .things ? It’s an idea I like but it would probably be a pretty large change[addition] possibly noone is willing to take on.
For the maintenance standpoint that would be a huge advantage though as Thing/Item file handling is fully decoupled from openHAB.
If at some point it is dropped from the official distribution (and probably made available as external bundle) it can be maintained by all those who see this very valuable for their setup. Could be made available as addon under the “misc” section, and people could even provide similar bundles that write Yaml, XML, MS word documents for Thing/Item inputs/outputs.
@ David_Graeff
A would have to think a bit longer about your suggestion, but at a first glance I see the benefits of this approach. It would solve a lot of the use cases, where I would currently just edit the text file with a few extra step. I would not mind those steps, and they could probably be automated.
If the “exported” files are a 1:1 representation of the internal openHAB configuration and if they maintain the order, those files could be stored in a source code management system. That would allow openHAB to use an internal format, that is best suitable, without the need for making it editable by humans. For example, without knowing if its fits the internal needs of openHAB, I could imagine some sort of embedded SQL database (e.g. SQLite) with all the power these would bring
It would also solve the problem of a migration path from the old to the new configuration, even if the current files would have to meet certain formatting standards.
Blockquote
I don’t believe everything can be seen.
Certainly not, and I agree that any approch should provide enough flexibility. But I am apposed to drop requirements that someone states at an early point, just because they seem to cause problems.
I’m late to join this great discussion, but I would like to start by thanking @David_Graeff for starting this, as it is much needed.
In my opinion, this would in general be a good thing, although I would still use my editor of choice to edit configuration files. The most significant added capability for me would be the ability to configure things/channels/items using the UI for a new binding that might not yet be well documented, then have the textual form of its configuration going forward.
Why would I insist on using my editor(s) of choice you ask. It’s simply because I am most productive with the editor I have used for the past 40 years. I didn’t start in the computer world with anything close to a GUI, I have always been most comfortable working from the command line. I realize that is not the case for the majority of people, but there is a sizable minority out there who feel the same way.
My biggest gripe about using a GUI for configuration is that it is easy to forget some checkbox, radio button or other configuration item, particularly when configuration for something is complex and I haven’t used the GUI configuration tool in a relatively long time. I can currently place comments in textual configuration files as reminders to my future self, which I find to be a very high value capability.
I have to agree that the current syntax of textual configuration files is quite arcane – in the extreme – and, as a result, error prone. A more structured textual configuration syntax, e.g., JSON, YAML, XML (blech!), … would be a major step forward, but, IMO, it would have to support comments, both in-line and block. Also, when the source of a change or a new configuration item results from my edits, order would have to be maintained, otherwise maintaining my configuration in a source control system would be damn near useless.
The above is nothing more than my opinion, don’t take my opinion as my view that there is no other way to do this, only that these are the aspects of configuration that are most important to me.
Again, thank you for initiating this discussion, David!
In response to David’s original question - sign me up for the “one UI” system that can edit my items, things, bindings, sitemaps, etc. I’m at a point where most of my stuff is openhab 1.x and I’m trying to get past my mental roadblock to understand 2.x. If there was just one way to do things and I could see/edit that file then I think I would get understanding sooner.
I tried VSCode last year and failed miserably. I never got it to link between VSCode and my OpenHABIAN system. I should try that again.
Mine is Insteon. Also Expire - but as David said that’s coming in 2.x soon-ish.
I like all the talk lately about “what do the users want” and “how can we make things better” so I summarized my experience here if anyone would like to read way too much about my progress.
I think that if we have already A a lot of problems with the current format could be gone.
B could be worked on seperate and would not give much upgrade problems.
Then later we can see if we still need C.
If we do, going from A +B to see, will be a lot easier for upgrade (as they are both computer readibel and testable files)
yes that would be more work in the end
yet work that is much safer and easier for upgrades.
and at this moment I think it’s too early to know if we really need C.
For completeness, we need to have well-defined documentation on how to reset the system and re-load everything from the .things and .items file generated in your “Sync back to Things files” scenario above.
No where am I saying I want to get rid of the ability to manually create Things. And even today you can manually create Things that go into Jsondb. You can create them. manually through:
PaperUI
Habmin
curl and a properly formatted json
manually editing the JSONDB
None of these are really optimal though so we are talking about what we need to put in place to make getting these manually created Things into the Jsondb better.
I was guessing the problem was caused by the xtend libraries. That issue makes it seem like the whole text config loading is basically FUBAR, which we kind of already knew.
The reason there are two ways to do things is because of the backwards compatibility.
It’s a good list.
Since of it is already supported by the db, others are not. but it’s at heart something to take into consideration.
But I also agree with David. In order to have a transition path for beginners who have out grown the uis to a more text based config, both have to interact. This is going to mean some compromises somewhere
@vzorglub has kindly done so. It never occurred to me to do it before now. Thanks Vincent!
It addressed my biggest concern which is having a migration path from one approach to the other.
The NGRE Rules might be some trouble but I think we can work it out. (it’s a can of worms we don’t need to open here).
That’s a standing requirement for everything we do. Sometimes we miss the mark but it’s anyways a requirement.
just want to add my 2ct from a user perspective (although most points have already been mentioned)
So my point of view is a let’s call it “advanced user view”. I have no idea about openHAB internal software architecture and its development but I’m using OH since 1.6 I think and still love today to do everything possible with text based config files.
Version control: yes
Human readable: yes
Human friendly: no
I get it that the currently used text files (things, items, …) are limited and can’t be adapted to support all the new features that are already there or will come up in the future.
As long as there still would be a way to do a text based config that is more human friendly than the current JSONDB implementation I would absolutely be happy with this, because …
100% agree
So, this would be my requirements for more human-friendliness
(I know some of them might be quite big wishes but you asked for them )
As an advanced user I would like to have the control on where things and items are stored to structure my system according to my needs
E.g. maybe I want to group my things by technology (hue, zwave, …) and my items by room
As an advanced user I would like to use a text editor with syntax and semantics checking to reduce the risk of nok config files
For JSON(DB) this is already possible with e.g. VSC, but I would also like to see an addon like the current openhab extension to check the semantics.
and I add:
As an advanced user I would like to have config files that are not changed by the system when there is no actual config change to get proper version control diffs
What does that mean: I’m just starting with gaining some experience with JSONDB but what I don’t like is that it seems there are config settings and other things (let’s call runtime information) mixed in this database. E.g for the Zwave binding there is a property “zwave_lastheal” for each thing that gets updated every night after the heal. So every day the version control system reports the JSONDB file as changed although the config hasn’t changed really. I don’t know if there are more examples that use the JSONDB like that.
This surely is intended as the architecture is today but when talking about an official way for text based config that should be separated in two different files/databases/whatever
As an advanced user I would like to have a config text file format that is “human friendly” to be able to edit it with a text editor easily
Although I know there is more information in the JSONDB than in the *.things file it feels much more complex to edit or even write something like this
"network:pingdevice:192_168_178_50": {
"class": "org.eclipse.smarthome.core.thing.internal.ThingImpl",
"value": {
"label": "Handy",
"channels": [
{
"acceptedItemType": "Switch",
"kind": "STATE",
"uid": {
"segments": [
"network",
"pingdevice",
"192_168_178_50",
"online"
]
},
"channelTypeUID": {
"segments": [
"network",
"online"
]
},
"label": "Online",
"description": "Gibt an ob das Gerät aktuell online oder offline ist.",
"configuration": {
"properties": {}
},
"properties": {},
"defaultTags": []
},
{
"acceptedItemType": "Number",
"kind": "STATE",
"uid": {
"segments": [
"network",
"pingdevice",
"192_168_178_50",
"latency"
]
},
"channelTypeUID": {
"segments": [
"network",
"latency"
]
},
"label": "Pingzeit",
"description": "Gibt an wie lange ein Ping in Millisekunden an das Gerät dauert.",
"configuration": {
"properties": {}
},
"properties": {},
"defaultTags": []
},
{
"acceptedItemType": "DateTime",
"kind": "STATE",
"uid": {
"segments": [
"network",
"pingdevice",
"192_168_178_50",
"lastseen"
]
},
"channelTypeUID": {
"segments": [
"network",
"lastseen"
]
},
"label": "Zuletzt gesehen",
"description": "Gibt Zeit/Datum an wann das Gerät zuletzt gesehen wurde.",
"configuration": {
"properties": {}
},
"properties": {},
"defaultTags": []
}
],
"configuration": {
"properties": {
"hostname": "192.168.178.50",
"refresh_interval": 60000,
"refreshInterval": 60000,
"retry": 1,
"timeout": 5000
}
},
"properties": {
"uses_dhcp_listen": "no",
"dhcp_state": "No access right for port 67. Bound to port 6767 instead. Port forwarding necessary!",
"presence_detection_type": "",
"arp_state": "Disabled",
"uses_ios_wakeup": "Yes",
"icmp_state": "System ping feature test failed. Using Java ping",
"uses_arp_pings": "no"
},
"uid": {
"segments": [
"network",
"pingdevice",
"192_168_178_50"
]
},
"thingTypeUID": {
"segments": [
"network",
"pingdevice"
]
}
}
}
Actually… Think like you are an openHAB newbie. In Json I assign “label” to “Handy Sebastian”. But there is no self explanatory ordering in the Thing file line and we keep adding stuff (tags, semantic tags, groups, location, properties, configuration) with more and more strange syntax (“@”, square brackets, round brackets, quotation characters).
I think I like the idea of using yaml as a successor file format, that follows my idea from above (reading it as soon as it hits an input/ directory and outputting it, when asked). Like in:
# My awesome comment for the next Thing
network:pingdevice:192_168_178_50:
label: Handy
configuration:
hostname: 192.168.178.50
refresh_interval: 60000
refreshInterval: 60000
retry: 1
timeout: 5000
The following still holds true for my idea:
Files are used as batch input and configuration means. (They are not watched permanently)
Files are one way to perform and restore a backup. Restoring the jsondb would restore also the runtime information though and might still be the preferred way (because it also contains sensitive information like pairing ids etc).
Absolutely. I’m not stuck with the current config files and agree they are neither self explanatory nor will we have enough different kinds of brackets with new features coming.
I like your example and also prefer yaml to json syntax. Also the reduced information compared to the current JSONDB format.
Here I’m not sure what to think about this idea.
I think I don’t like the part of removing successfully parsed files. Doesn’t this mean you have to copy your files between different locations each time you want to perform a config update?
But I like the idea of considering the config files just another kind of UI (did I get this right?)
So jumping on the bandwagon I can image the following setup:
JSONDB will be the single source of truth for the config (and also the runtime data like mentioned above)
JSONDB can as is be edited via GUI, REST, KARAF
JSONDB shouldn’t be edited by hand (-> “real” database could be used if beneficial)
textual configuration is possible via yaml configuration files in a well defined directory structure (this is the replacement for current DSL config files)
this yaml files contain only config information, no runtime information
by some trigger signal (file changed or rest call e.g.) this files get loaded into openHAB
if the parser finds a difference between yaml file and JSONDB the DB is updated
another trigger will do the opposite: compare DB with yaml file. If there is a difference detected the user is faced with a message like “config_whatever.yaml was changed via GUI. Do you wan’t to update your yaml configuration? → YES or NO”. If yes download config from DB back to yaml file
Just some brainstorming, no idea if this in-line with other users UseCases or even feasible with the openHAB design
I really like that yaml format. But let’s dive it a bit further. Effectively we still support two formats: the internal storage (JSON DB) and the input/backup format which could be yaml. Why don’t we then provide an option to support reading and watching those yaml files, since the core kind of support it already. This would satisfies the needs of advance user.
The restriction still holds that what the original yaml file the user created is not static; OH is free to write additional info to that file. So essentially, the same as what we have to day with the .things file. However they are in yaml format and they can be changed by OH.
Here are some thoughts (Some parts have been mentioned by others).
The current jsondb files don’t seem to be designed to be edited by humans. (The fact it contains db in the name should have been a hint). This doesn’t directly mean it isn’t usable, but might need some changes if it would be the replacement for openHAB 1 config files. Here are the points:
The json files contain redundant data. For a database this is not uncommon as sometimes it’s more efficient. But for human’s this only adds problems, because if you forget to change all you will get errors or worse it will not give any feedback. For example see uidsegments in channels it also contains references to thing id.
The files currently contain state data specific for the system, that is not specifically useful for the user. For example it contains all channels of a thing. You can leave them out of course, but then you can not add them in PaperUI without adding them in the json files. (This is also a known problem when a binding is updated and gets new channels, you than need to read the thing).
It’s a format that can be easily read by computers, but there is no defined order. Therefor the system can store them in any order it likes. This is a problem if you want to manually compare changes, because if the order changes for whatever reason it’s difficult to compare version.
Files that can be changed by users need a lot of additional checks to give feedback to the user in case something is wrong. I don’t think that is currently the case with the json files.
So I would not say current jsondb files are ready to be a replacement of the openHAB 1 configuration files…
In general having configuration files that are editable by users and can be written by a system are very hard to develop. It’s like having generated code that can be edited by a user. Because when the system reads these files it needs to keep track to the structure in the file in order to keep the structure. This is also very likely why openHAB 1 configuration files can only be read and not saved. If the configuration files can contain comments complexity increases. Compilers become much more complex when they need to keep comments. I’m not saying it should not be done, but just to remind that it’s really not trivial to make it happen. It helps to have a format that has this out-of-the-box with libraries supporting it.
There is another complexity. If users can store configuration in their own named files the system needs to track which configuration is in which file. Because if something is changed by the system it must be stored again the same file. Also new configurations made in a ui need to be stored in a file and probably the user wants to select what file. Therefor the system must get knowledge about these files. Currently that is not in the code. It would also mean this information needs to become available in the user interface.
Yes yaml files might look a like the new nice shying thing, but I’ll just quote balloob (about yaml configuration in HA): ... configuration.yaml will not go away, but just solely relying on it is preventing new features of being built...
So I think we should really think about how configuration files should look.
But HA has exactly the same problem. They have an internal state and a bunch of yaml files, created by the user, that the system cannot write back to.
I agree. But they are suitable for git backups, because only changes appear in diffs.
That’s why I suggested a “storage” assignment for Things and Items.
The comment can be stored in the Thing, just another field. I wouldn’t allow free floating comments. They cannot be restored easily.
That is our current problem that we want to solve! We have two sources of truth (“storage” that is read on start-up) then. We want exactly one, the internal one that is most efficient for the core.
Other inputs are really just temporary inputs.
I like the summary, except this part. A user will be confused if the file is processed on a change, but not on boot-up. A REST api call / openhab console command to read from the files is possible though. We wouldn’t need a separate “input/” directory then.
I’m also using a private repository with Git to track my changes. However I think if the interfaces of openHab will be cut down to only “next-gen” things, like PaperUI (all-in-one management, like creating rules/things/items through a GUI, but you will somehow have the ability to manually edit all of them, plus Z-Wave Network tools (like in Habmin - hope I’m right, currently I have no Z-Wave devices so I just assure it from the forum)) and HabPanel, the installation and usage of openHab will be much easier (mainly for new users).
That is our current problem that we want to solve! We have two sources of truth (“storage” that is read on start-up) then. We want exactly one, the internal one that is most efficient for the core.
From what I understand the current problem exists of two seperate issues.
the current input files are not good computer readable
I have a feeling that this has been discussed before, but there have been too many posts. But how about changing the back-end from using JSON DB (a single file now I believe) to using a number of yaml files, one for each binding. For the next couple of releases, OH can read the .things file at start up once as you mentioned, and if everything is deemed okay, the equivalent yaml file will be created in a specific folder. Changes in Paper UI is persisted to the YAML file. And manually updated yaml files will be reloaded by OH.
This approach allows advance user to gradually moves away from .things file while not feeling hand-cuffed to the UI. After couple of releases, the support for .things files can be retired.
Great thread and big thanks for splitting off from other thread as that thread had a lot of really great ideas and some terrific discussions but had started on a sour note, with a not so great title and at times had elements of trolling
I want to say that I have been running 2.3 stable and had read a lot about some of the new changes but recently upgraded to 2.4 stable and was very impressed by all the improvements. (great job everyone!) The Paper UI has be improved quite a bit and it truly is impressive.
With these changes to Paper UI, I can almost envision a user interface that could eliminate a lot of the complexities of text based file configuration. A lot of the discussion in this thread has centered around users asking that their use case gets considered in a next generation openHAB. I’d like to talk about what the new version could be like if we were to embrace the idea and stop worrying about keeping everything backwards compatible. I know my opinion is not popular with everyone and I’ll take my licks for speaking my mind.
I feel like the forum is populated by many folks who have been using openHAB for awhile. (some new users) The forum may not be fully representative of new user’s opinions. I also want to state that I do know how to program in several languages so my opinion isn’t born from lack of programing ability. (AutoLisp anyone?)
Consider this, most of us picked openHAB as the platform for our home automation system because we liked having very great control over every aspect. But if the same exact system was presented to me without the need to hand write configurations and the ability to control my configuration with the exact same functionality or even more… I’d be all over it!
If openHAB had always been that way, you wouldn’t miss your text configuration files. Consider also, many use cases for text configuration files involve the ability to track changes, compare old versions and so forth. I believe a lot of these desires come from the fact that it’s disturbingly easy to screw up the configuration. To the point that most advanced users like to have roll back versions so they can experiment while learning and go back to old version if it screws up. This smacks of a band aid for an underlining problem
What if the new version didn’t allow misconfiguration? What if a new version enforced perfect configuration and made everything easier and more powerful? And faster?
100% agree
)