I’m also using a private repository with Git to track my changes. However I think if the interfaces of openHab will be cut down to only “next-gen” things, like PaperUI (all-in-one management, like creating rules/things/items through a GUI, but you will somehow have the ability to manually edit all of them, plus Z-Wave Network tools (like in Habmin - hope I’m right, currently I have no Z-Wave devices so I just assure it from the forum)) and HabPanel, the installation and usage of openHab will be much easier (mainly for new users).
That is our current problem that we want to solve! We have two sources of truth (“storage” that is read on start-up) then. We want exactly one, the internal one that is most efficient for the core.
From what I understand the current problem exists of two seperate issues.
- the current input files are not good computer readable
- we have two sources
Is this correct?
Yes exactly. Something like yaml can solve problem 1, so that openHAB can read and also write those files.
I have a feeling that this has been discussed before, but there have been too many posts. But how about changing the back-end from using JSON DB (a single file now I believe) to using a number of yaml files, one for each binding. For the next couple of releases, OH can read the .things file at start up once as you mentioned, and if everything is deemed okay, the equivalent yaml file will be created in a specific folder. Changes in Paper UI is persisted to the YAML file. And manually updated yaml files will be reloaded by OH.
This approach allows advance user to gradually moves away from .things file while not feeling hand-cuffed to the UI. After couple of releases, the support for .things files can be retired.
Great thread and big thanks for splitting off from other thread as that thread had a lot of really great ideas and some terrific discussions but had started on a sour note, with a not so great title and at times had elements of trolling
I want to say that I have been running 2.3 stable and had read a lot about some of the new changes but recently upgraded to 2.4 stable and was very impressed by all the improvements. (great job everyone!) The Paper UI has be improved quite a bit and it truly is impressive.
With these changes to Paper UI, I can almost envision a user interface that could eliminate a lot of the complexities of text based file configuration. A lot of the discussion in this thread has centered around users asking that their use case gets considered in a next generation openHAB. I’d like to talk about what the new version could be like if we were to embrace the idea and stop worrying about keeping everything backwards compatible. I know my opinion is not popular with everyone and I’ll take my licks for speaking my mind.
I feel like the forum is populated by many folks who have been using openHAB for awhile. (some new users) The forum may not be fully representative of new user’s opinions. I also want to state that I do know how to program in several languages so my opinion isn’t born from lack of programing ability. (AutoLisp anyone?)
Consider this, most of us picked openHAB as the platform for our home automation system because we liked having very great control over every aspect. But if the same exact system was presented to me without the need to hand write configurations and the ability to control my configuration with the exact same functionality or even more… I’d be all over it!
If openHAB had always been that way, you wouldn’t miss your text configuration files. Consider also, many use cases for text configuration files involve the ability to track changes, compare old versions and so forth. I believe a lot of these desires come from the fact that it’s disturbingly easy to screw up the configuration. To the point that most advanced users like to have roll back versions so they can experiment while learning and go back to old version if it screws up. This smacks of a band aid for an underlining problem
What if the new version didn’t allow misconfiguration? What if a new version enforced perfect configuration and made everything easier and more powerful? And faster?
That is one concern but not the only one. In some circumstances one will not be able to use automatic discovery and have dozens or hundreds of Things or Items to create. this almost will always be massively more effecient with direct access to the text with copy and paste.
1 Like
Also, I’d like to add to this discussion some things I’ve learned by my involvement thus far. Several developers have mentioned that contributors work on what they want to work on. OpenHAB lacks a certain development path because different contributors develop parts of the system they want to develop, many for personal reasons. Other parts of the system get left in an older form or format because no one wants to take on fixing them or messing with that feature. Point is…
If David or anyone else decided they wanted to build a new user interface that didn’t allow text file configuration and used an interface for configuration instead, I don’t think any of us could stop them. (admittedly this would be a difficult task with out changing the underlying architecture of the application) In the same way, if someone wants to create a version which only can be configured with text files, no stopping them. The fear is that if you continue to use an older version, at some point development on your older version will cease. Well, not if people really like it and continue to develop it. If text files are so popular among users, your numbers should be vast enough to include some talented programmers.
So why do the old way of doing it users (text files) want to use the newer version? New features, its better, faster, more capable. So why do the developers who contribute want to do things a new way? Because the application is evolving, getting better, faster, easier to use. Text file configuration is error prone and difficult to learn for new users. Text files are a barrier to entry for many users. Text files may limit what can be done configuration wise.
Wow great point Rich, I think this is why these types of discussions are excellent to have because so many use cases exist. An example of such a case as you mentioned would be MQTT devices which are not capable of auto discovery? MQTT seems really popular and an important part of the platform although I know very little about it.
I bring my experience with OH and PaperUI.
Started with OH1, then moved to OH2 and started development of the new BTicino/OpenWebNet binding mainly to add discovery, not supported by the 1.x binding (not yet integrated in the official release).
Most users of my binding did not know about OH2, they just wanted to integrate their BTicino setup in an easy way, including link to advanced modern UI like Alexa/Google Assistant.
As a matter of fact, many adopted OH2 just for adding voice interaction, making it the real “killer app” or reason to move to OH.
(no official integration with voice assistants exists from BTicino yet).
Most of them have no programming experience at all, no linux experience, most of them think configuring text files following a strict syntax (different for each file!!) is equal to programming.
Most of them where attracted by the promise of having a full UI + autodiscovery based system.
Reality is that most of them ended up with manual file configuration, with a lot of pain, complains and support needed, just because PaperUI cannot fulfill the promise.
Reading these post here I think the risk is to stick with an almost unuseful PaperUI and stick with file configuration as the main UI almost for all, even those who did not wanted it at first, just to satisfy a bunch of programmers around.
In my opinion newbie and everyday users will go away and follow other solutions if PaperUI is not chosen as the main configuration system for OH2 and is is not improved A LOT.
Competition is tough, even in the area of Open Source smart home systems…
Let’s not forget the mass of users to satisfy just few programmers who could survive anyway…
Regular users need: simple UIs, clear user manuals, tutorials, wizards and do not care about making “git-enabled backup of their system with diffs”, if an easy-to-use backup system (like the one you find for ADSL modems) is already in the box.
Keep it simple, keep a basic user centered design > OH2(3?) will attract many many more users, for a long long time.
Bye
M
1 Like
If openHAB had always been that way, you wouldn’t miss your text configuration files.
Wrong. I looked very specific for a text bases system.
In everything I do, I look for text based systems to put in a version control system.
Consider also, many use cases for text configuration files involve the ability to track changes, compare old versions and so forth. I believe a lot of these desires come from the fact that it’s disturbingly easy to screw up the configuration.
yes to track changes.
yet not only to see configuration screw ups.
To the point that most advanced users like to have roll back versions so they can experiment while learning and go back to old version if it screws up.
just one part
creating copies for multiple configurations in another.
As said before, I want to run on two different environments to test openhab versions, not mine versions.
In my case, I get my kids interested in programming, because they like to play around with their room. For me it’s important that when they lean about programming, they learn the advantages of modern programming, hence source control.
Once you have multiple people working on a system, you need a source control system, so you can see what other people did. (including openhab itself)
This smacks of a band aid for an underlining problem
What if the new version didn’t allow misconfiguration? What if a new version enforced perfect configuration and made everything easier and more powerful?
I don’t believe in that. An interface can never follow the speed of development of an API.
In a system like OpenHab you can’t expect all developers working on a binding, to also update paperUI. Yes you can make contracts etc around what can be changed, etc and then comes a new binding with a new type of variable, that PapeerUI is not ready for. You don’t want to block the first people experimenting with the new binding waiting for paperUI to be ready.
In some cases you will need to try a few options to know what is acceptable and not and that makes it a lot harder to do that in sync. If that person creating the new binding can make the changes in PaperUI, then it’s not a problem, yet then you are limited the numbe rof people being able to create a new binding.
The way I see it, @David_Graeff asked for ideas to develop a system where more people are happy using it.
What I and a few others are asking for, is a system that helps new users with the paper Ui and allows them to go further if they want, once they are more experienced.
or go further in one option and stick to paperUIt for other cases that they don’t need to go further.
We are not asking to forget about the first time user.
The problem with the current system is that first time users can work with paperUI and when then want to go further, they are stuck or at least they have to start almost over and learn openhab from a different angle with a different syntax and that is bad.
1 Like
I agree PaperUI must improve to cover more use cases than today and that this will be beneficial in attracting more users. That’ll happen anyway, but it takes time.
But at the same time I heavily disagree to omit or even just undervalue the ability of manual and automated text based input just for the sake of “keeping a couple of programmers happy”.
The ability to control each and every aspect of your home automation in full detail is the heart and core of openHAB and #1 reason why people use it. While not everybody is willing to admit, we all know GUI based SW cannot provide all of it.
I like @David_Graeff’s idea to provide an text import mechanism (rather than to take .things as the source of truth) and to complement that with an export mechanism so additions via Paper UI can also be propagated back to files.
IMHO we should continue to use the current file format as I see no real benefit in changing that - but with any new format whatsoever EVERY user would have to adapt his setup and configuration mechanisms.
The discussion around JSONDB is useless IMHO as it is and would continue to be THE source of truth, but it’s the backend, i.e. you can backup/restore it but it’s never meant to be changed/edited directly, only through frontends such as PaperUI and the .things importer.
4 Likes
Would you be happy with the proposed solution? That is:
- Have Things/Item files sitting in a directory, probably the same like now
- No file watches but a dedicated “openhab-cli import” command.
- A write-back mechanism (openHAB will touch your files, and replace all Things that it knows of) by issuing “openhab-cli export”. Things will have a “storage” association (aka filename). If they haven’t got one, they are stored in things.things.
The file format would change to yaml. (Otherwise we can’t have the write back)
Cheers, david
2 Likes
I think that from a user perspective, the file watch should be kept.
I, for one, don’t want to open my console, log in ssh, and enter the command every time I edit a file.
I want the file updated and “imported” as soon as I hit save in VSCode (or any other editor)
I want the feedback in the logs there and then.
1 Like
And why don’t you use my proposed paper UI vscode text interface with auto correction then? That is part of this change of course.
I’m addressing people that want to conserve openHABs internal state as text files (although with openHAB 2 it stopped working like that). I’m not addressing people that use text files because of copy and paste, cloning etc. That is done via the Webinterface.
You will have to pry my emacs out of my dead fingers first 
1 Like
That’s the reason why I think some kind of export/sync trigger would be better. So the user has control whether his files are modified by openHAB runtime or not.
Agree, I also don’t want to open a terminal and ssh into the system. But what’s about sending the “trigger” via some REST call? This could be integrated into some kind of “next gen VS code plugin”. So you just have to push a “publish changes” button.
I mean file watches would also be great, but if there are some limitations or issues implementing them I could perfectly live with this approach.
A side effect would be, that in my current workflow I sometimes want to save some changes I’ve made in a config file (esp. rules) but don’t want them to be applied to the runtime immediately. This would be solved by a publish trigger.
1 Like
Hmm - no. Input and output in different formats is a bad idea and in contrast with industry standards.
Remind you that every output in turn quickly needs be reused as another input.
We would quickly end up with yaml being the only format (to have the same problems as HA user have with it) and still everybody would need to learn the new format, rebuild scripts he uses for generation etc. Please try to think of it more from the user perspective.
We could keep using the existing format if you were to implement an export routine for it (shouldn’t be too hard, should it ?)
I’d also second keeping the file watcher.
Noone (at least no user) wants to conserve it as the internal state (developers may care about that, users don’t). But users are keen on keeping it as an input format available to them.
With im/export this file format would no longer represent the internal state (btw does it now that we have JSONDB ?). The DB to keep the internal state would be JSONDB and it’s respective representation in RAM.
It doesn’t need to be yaml. But it need to be a format where we can use a predefined library for. The core should be slimmer in next-gen not bigger because of custom this-and-that.
That was the goal. Having the same input and output format. And removing XTend from the core completely (this bloated piece of utterly slowness that blocks us from going to new java versions and so on …).
A conversion tool would convert .thing files to .yaml or whatever the format of choice is then.
I have followed this discussion for some time and from my point of view, it is now too much about file formats and physical storage. Before discussing how information is stored and what kind of textual formats should be used we should define the functional requirements more from an end user’s perspective.
As this discussion shows we have different types of user types. We have the users with no or little IT (or programming) background and we have quite a lot of users that are software developers. From my long-time experience of developing tools for software developers, I have learned that most developers prefer text files and want to use their preferred editors 
.
When I work with openHAB I use a combination of Paper UI and text files. I use Paper UI for the things definitions because I like the auto-discovery and the definition of things in text format is sometimes a bit complicated especially when you have bridges.
For the definition of items I am using the text format and VSCode. This is much efficient then “clicking” the definitions together because I can use copy/paste, mass changes etc.
In addition, we have some configurations that currently can only be done in text files, like persistence, sitemaps and more complex rules.
Independent of the way the configurations are edited we need the following features:
- Versioning of configurations: sometimes it is necessary to go back to an earlier version or I want to compare two versions. I am now using git for my config files and also for the JSONDB
- Import/Export of configurations: I have a production and a test environment and I want to exchange my configs. With the help of exported configs it is also much easier to set up a new installation from scratch.
- Copy/Paste especially for item definitions
- Find/Replace e.g. to perform name changes for items
As we have different user types we probably need different types of configuration editors or frontends:
- A simple frontend for new or inexperienced users. Probably we need something like wizards in the frontends that helps the user also with the creation of items (similar to the simple mode) and with the creation of a sitemap and persistence definitions
- A frontend for experienced users. Maybe similar to the current Paper UI but with support versioning, export/import, copy/paste of item definitions and at the end the possibility to define all configuration options (persistency, sitemaps) in the same frontend.
- Textual configuration. To make it easier we can try to embed an editor into the browser-based fronted, e.g. with the help or Monaco or Theia. The text format should be user-readable and user-maintainable.
It is also important that all configuration options are available in all formats. Right now tags can’t be defined in Paper UI which probably is a big problem for new users who want to use e.g. Google Home. If openHAB gets new features in the future it may be a problem to make all these functions configurable in text format if have to define a syntax for all these options. Therfeore a more generic format like YAML or TOML could be an option
I would like to have this configurable. When I modify a sitemap I like that it is automatically reloaded when I save it. But for items and rules it would sometimes better to explicitly reload the definitions. Especially when you modify items names and also need to modify rules resp. sitemaps at the same time.
3 Likes
Well, yours as a developer. But for sure not a user’s, and most to read this thread are users.
Users all know the current format and don’t want to have to learn anything new. It will create a vast chaos as users will mix and match old and new format resulting in lots of need and efforts to be spend on support and still a lot of frustration on all sides.
I guess you see the conflict, but you don’t have an idea how much of a chaos this is going to produce. Remember all existing knowledge in people’s heads is based on this and any web or forum search will produce lots of information that only applies to the old format.
So there’s two approaches, the developer and the user preferred one, i.e. either yours or to implement the export without a library. I strongly vote against changing to any new input format.
And I don’t understand why we need a library for that to implement (sure it’s easier/quicker but not required).
Well if all Xtend is used for is to parse the text input files then you can remove it AND keep the file format (yes you would need to implement an alternate input parsing routine).
Or keep it. ‘Utterly slowness’ is no argument then because it only applies when im/exporting (and that’s a rare action).