I’m doing everything in text and I plan to stay on it.
All my configuration is in a source control system.
so I can do diffs to see differences why something works and does not.
I have two pine 64 where I use Openhab on.
One is the production system, the other is the test system.
They both use the same configuration.
Last week my production system had 2.4 M5 on it.
I turned it off
I turn the test system on also 2.4 M5
Download the latest scripts to see if everything still works
I upgrade to 2.4 release
This becomes my life system.
If it’s works for a few days (or at least works better I consider it my production system.
After a week or so, I upgrade my previous production system also to 2.4 Release version.
If it teh upgrade gives me problems or something goes wrong in whatever way, or I run out of time to get the upgrade working, I turn of the test system and I turn on the old production version and my house is up and running.
Version 1 and 2 have two different IP adresses.
Compare it to older CI products, you could change everything in the UI, yet you had no trace of configuration changes, and the changes could not be linked to other changes in code.
I consider it and many with me) that all changes should be in a code repository, for code, documentation and configuration.
Although I have not been programming in a few years, I keep the habit of doing all important work in a source control system.
Even my books are written in a source control system (using leanpub to republish new version)
And for clarity, when I make changes to my configuration, I make them on another computer, upload to the source control system and download them on the openhab server.
(which triggers a reload of the config files)
I can make these changes anywhere I am in the and my family can download them using a button I have defined in the openhab sitemap. this way I can fix things when I’m not near the openhab server.)
Sigh. Please read the documentation article about JSON DB and you will find out that Paper UI stores its config in human readable, pretty formatted json for exactly this reason: To store it on revision control systems. Only changed lines will appear in a Diff. ON top of that: It also creates backup files if during a write an outage happens.
That is your own system, as far as I know that does not work to have a test system.
(as it would be the same JSON DB for both systems which is what I want to avoid, as upgrade can change the json db, I want to stay independ.
Older CI systems did the exact same thing, yet many people moved away from that, to have ci independant from internal DB configuration.
Anyway, you asked how people used it and why they would or would not use it.
I told you how I use the system and plan to keep using it.
To understand requirements. The ultimative goal is of course to get rid of both independent, non-interacting ways and only have one way. As far as I understand you, you have some Things that differ in your Test and Production environment.
So would it help, if you could assign a Thing to a storage-context other than default, which would make openHAB to create another, separate json db file?
First I too use source control in a similat fashion as described by yves.
In I find it much simpler to use a good text editor, I am used to, instead of somer browser interface. After editing in the browser I woukd probably review the changed in s text editor and a visual diff tool. I just feel more confortable with tools O am proficient with and can see changes at a glance.
Big changes can be done using tools like sed or awk, which was a big help for me e.g. when migrating to the KNX 2.x binding,
Editing text files can be done, when needed (in fact, just today), using ssh and e.g. joe from my phone, where screen size would probably be to small for a full grown web interface
One little thing though… Having everything in one big file (the huge jsondb file) would fail my requirement of granularity.
When experimenting with, say, the Hue binding, I tend to remove all irrelevant things and bindings from my experimental system, hacking on the hue files all I want, and afterwards commit those changes to the git repository.
The same goes for Somfy Tahoma, where you seriously don’t want to have multiple systems connecting to the upstream servers, as that will in the best case simply stop the main installation from working (“login denied, too many attempts”), and in the worst case, cause the Tahoma servers to block your account.
In both cases, I would have successfully killed my roller shutter integration, which would seriously reduce the Acceptance Factor of She Who Must Be Obeyed
As far as I understand you, you have some Things that differ in your Test and Production environment.
No. These are exactly the same.
It’s not to test my configuration, it’s to test the new versions of openhab.
(sometimes my configuration, yet that hardly happens these days, maybe when I start playing with a new binding, yet usually I can tunr that binding off, so it’s not really an issue.
The ultimative goal is of course to get rid of both independent, non-interacting ways and only have one way.
that is your goal.
My requirement as a user is different.
You want to have users use paperUI with manual changes to fill in your json db
and take it from there.
I don’t want any UI to change my configurations at all.
You prefer one system. As a user who has written quite some software in the past and who is coaching a lot of teams on how to be effective, I don’t believe in that.
Yes, because you’re unlikely to replicate the power of Visual Studio Code, and its really focused on disk-based editing. In VSC you can edit everything – things, items, sitemaps, rules, etc… if you still have to switch tools and edit files for anything, its a duplication to have a different way for anything else.
IMO, JSON DB is the wrong technology. Its having a hammer, and using it, rather than asking what the problem is and what the right tool is. There’s a reason a slew of projects (like Spring, as an example) have started to move to YAML from JSON – because JSON isn’t really human-editable. Its human-readable, but its a giant pain in the ass to edit by hand. Putting everything in a single monolithic file just magnifies the problem. (Its a bad architecture for YAML, as well).
If you said “what about storing each individual thing in its own JSON file” (or YAML) I’d agree with it. But if any editing is being done outside an API that is enforcing correctness of structure, a monolithic solution isn’t right.
Alternately, have PaperUI store it in a single file, but have the system parsing them parse any files in that directory, so an “advanced” user can move chunks of it out to other files for organizational purposes. But personally, I think if I name a Thing “Living Room Sconce Dimmer”, a file like living_room_sconce_dimmer.json or living_room_sconce_dimmer.yml being created would be preferable.
IMO, though, the only downside really to the manual method of setting things up is a lack of documentation for doing so with VS Code, because VS Code makes it so easy to set up items. The learning curve is steeper than it needs to be. And, I think Xtend is just a disaster when it comes to a scripting environment (largely because there aren’t any editors that properly do syntax checking, so its just hit or miss if something is going to work).
I guess it’s a good thing the JSONDB implementation doesn’t do that then . It uses a number of different files for different tables…
PaperUI is not in any way linked to this - the database is in the backend - not the UI. There’s not so much scope for changing this since the DB is an ESH interface - JSON DB is just one implementation, but they all need to comply with the same interface. Given that the data stored in the database can be anything, the implementation can’t really make assumptions about what it is and start splitting “things” (that it knows nothing about) into different files…
In my experience common sense isn’t that common. And if we just wait around for the users to get better we will ultimately fail. We have the power to change ourselves but not others.
This post and the little template are just two attempts we have made to try and help the users help us to help them. We are always open to further ideas. It’s not like we are ignoring the problem.
But we have our own constraints and will push back on ideas that exceed them.
I think the target is well defined. Perhaps it is not documented well enough. We would welcome any advice anyone has, and even better content, to address that. Then at least the group of uses who do read the docs will benefit.
If I’m talking to someone in the software industry I agree completely.
When talking to someone outside the industry, there is no difference between the two. Both activities are “programming” in their minds.
This is one of those places where I think it is better to go with the flow instead of introducing and describing new concepts that ultimately do not matter from a practical perspective.
I should amend my statement. My understanding is the goal to have one officially promoted and recommended way. I believe that many of the legacy ways will remain supported as long as feasible. But the documentation for the how to use them will become less prominate and will eventually become a subculture within OH, mind of like JSR223 is today.
OH too bad a concept for these three types of interfaces. It’s working furiously to improve the beginners interface (PaperUI) but it’s not there yet.
The Expert UI (text files) are the most mature but at some point something must be done to replace this. The current formats for the files are not aging well as new features are added and they use a proprietary format that doesn’t have the party libraries for reading and writing. At some point these files add they exist today will have to be dropped. Probably in an OH 3 time frame if I were to guess. I believe this is what David is referring to. It is frankly dumb to have two completely different file formats to define Things et. al. But maintaining legacy support sometimes forces us to do dumb things, at least for awhile.
OH has a very good REST API which is how all the web based UIs work.
You said it so much more concisely.
All of what you say is true but it lacks context. In OH 1.x ALL we had were text configs. There were no UIs, no automatic discovery, etc. All we had were .items, .rules, .sitemaps, .persist, and one big openhab.cfg. We Only had yves’ expert UI.
OH 2 is an effort to build that beginner’s UI. And to achieve that it took completely rearchitecting the core and introducing concepts like Things.
The Most important thing to understand is that OH 2 is not done. We do not yet have parity between the old way and the new way yet and we need to get there. Most of the complaints you list all stem from the the fact that OH just isn’t done. For example, I know the Android app is undergoing changes to let it support HABPanel. I have great expectations for HABot as a UI. But once we do get to parity and everything that can or needs to be done can be done through one of the admin UIs, the old ways will have to change. They are not sustainable. I’m not saying that we need to get rid of text based configs. But the . text files as they currently exist will have to change. We cannot and should not maintain two completely separate configuration subsystems for OH.
Which can be done with JSONDB.
This process can continue unchanged with the only difference being which folders you check into and out of your source control. Instead of /etc/openhab2 you would control /var/lib/openhab2/jsondb.
I completely agree, that is why I do it with JSONDB. There is nothing special about it. It is just a series of JSON formatted text files. In fact, because the JSONDB format includes more information than you get in .things files (for example) the diffs and history is actually more useful than what you get from the .things files.
Pretty much all of this can continue, though there might be a problem because, at least for now, JSONDB cannot be edited while OH is running. So your button would need to shutdown OH, pull the changes, then restart OH.
@David_Graeff, this limitation is I think a long term problem that will need to be addressed. So far it is the only limitation I’ve been able to discern between the old text configs and JSONDB. Someone at some point will also need to write something to parse through the old .text files and convert them to JSONDB (perhaps through the REST API?) to ease the transition of users who are sitting on thousands of Items (for example).
To repeat what David said, JSONDB is a text file. You can do all of this with it.
Would David’s suggestion of having the ability to assign Things to a storage-context address this?
Everyone needs to realize a few things.
We cannot continue to maintain the current format of the text config files. They are in a proprietary format and there are no libraries for readers or writers and they are based on a sub language whose future (IMHO) is in doubt.
We cannot continue to maintain two COMPLETELY separate storage mechanisms for this sort of configuration. It is wasteful, will introduce errors, and introduces a whole lot of work that the project cannot sustain.
The text config files as they exist now will go away at some point.
So I’ll repeat David’s question slightly differently.
What would do you need the JSONDB to support in order to make it usable in your situation? So far I’ve seen two things which JSONDB doesn’t currently support:
be able to edit it while OH is running
be able to separate it into separate files instead of one big file for all Things (for example)
What else?
Speak up now because you have the attention of a developer who is actually willing to work on it. Don’t lose this chance!
I think everyone pretty much agrees. BUT the text configs as they exit right now are based on Xtend too. We can’t drop the one without dropping them all.
If I’m talking to someone in the software industry I agree completely.
When talking to someone outside the industry, there is no difference between the two. Both activities are “programming” in their minds.
This is one of those places where I think it is better to go with the flow instead of introducing and describing new concepts that ultimately do not matter from a practical perspective.
I disagree. You scare people(non IT people) away by calling it development
People are already used to configuring their systems
windows, phones etc
The Expert UI (text files) are the most mature but at some point something must be done to replace this.
I don’t say that It should not be changed, I can imagine it has to be updated.
yet I do think you need such an expert UI
All my configuration is in a source control system.
so I can do diffs to see differences why something works and does not.
Which can be done with JSONDB.
From what I understand that’s not the same way as I do it.
From what I understand PaperUI is the interface that updates and read the jsondb and we should not update the json.
At least that is what I understood months ago when moving to 2.0
And while people are creating a ui like paperUI, I understand that.
Updating two interfaces (aka simple and expert) at the same time I would not advice
This process can continue unchanged with the only difference being which folders you check into and out of your source control. Instead of /etc/openhab2 you would control /var/lib/openhab2/jsondb.
that is not what I want. I guess I am not clear.
I want to have 1 system that is exact the same as the other.
I swap system by turning them on and off
I do want the configuration on both systems be exact the same, except for the Openhab version.
that way I can test,
It is just a series of JSON formatted text files
except json is not really human friendly in my opinion
Pretty much all of this can continue, though there might be a problem because, at least for now, JSONDB cannot be edited while OH is running. So your button would need to shutdown OH, pull the changes, then restart OH.
Which is a huge issue.
One of the reasons why I like openhab, is that it did not need to be restarted in the first place.
That really helps, for my children /partner to understand and use openhab.
I’m not sure your remark about long term problem is on this or on the upgrade part
which is something I agree, yet is for me something that could only be done when 2.0 is a lot further
The text config files as they exist now will go away at some point.
I understand they might need to change
We were asked the requirements. As a user I’m saying that like I see now the way the json is be implemented, does not seem to be that replacement for me.
Speak up now because you have the attention of a developer who is actually willing to work on it. Don’t lose this chance!
I’m doing it, and it feels that when I say it, I’m told, that’s wrong that is not a requirement
I really believe you need the two systems.
I do agree that you need a system that is easy/easier to maintain
yet I also think it’s to early to drop that old way of working
For me it feels like 2.0 is all about adding a simple user interface like paperUI and whatever is needed to support that
When that is “finished”, have a new look to work on 3.0 where you can think about reworking the text file to have a better expert file system.
Don’t drop the expert text system, while working on the simplir system.
that also help on setting priorities, which is for me the biggest bottleneck for all projects.
all projects want more that they have time and people for.
If I could do that, on a per-Thing-basis, so I can have all my hue-things in one jsondb, all tahoma-things in another, and so on (or whatever logical grouping I might have need of), the this would be fine.
If the items and rules would follow that distinction with storage contexts, it would be even better.
The service definition files (e.g. “influxdb.cfg” and so on) already are in other files, right?
You mean years ago? We’ve come a long ways since OH 2.0 was released and except for the problem where you cannot edit the file while OH is running there is nothing wrong with editing it by hand. I just copied/pasted/edited a couple of dozen new MQTT Things by hand in the JSONDB with no issues what so ever. It’s just a text file.
What exactly is it about JSON formatted text files located in /var/lib/openhab2/jsondb is different from Xtend formatted text files located in /etc/openhab2 that prevents you from having exactly the same configuration on both machines?
I run in Docker so pretty much every time I restart OH I’m running it on a new “machine”.
I agree, it’s a big problem that must be addressed.
That’s not a requirement. That’s “I don’t like it”. What would need to be changed to make it a replacement. Because there will be a replacement and right now JSONDB is the only candidate.
This might be because what was asked was what features would JSONDB need to have to be an acceptable “expert ui” and your answer is “I need an expert UI”. And given the scenario you outlined, I could not see anything that would make JSONDB unsuitable as it exists today, let alone with new features, except for the before mentioned inability to edit while OH is running.
The biggest problem that need to be solved is to have a migration path from the beginner configuration to the expert configuration. We don’t have that now and we will never have that with the text based configs in their current form. We need a way for users to get started using the UIs and as they become experts they can seamlessly migrate to the “expert ui” without recreating everything again. And at the moment, the prime candidate for managing that migration path is to make improvements to JSONDB so it can become the expert UI. Does this mean making it easier to edit them by hand (e.g. the restart problem)? Does this mean building a new UI or sub UI in PaperUI? Does this mean being able to break them up a bit more?
I am under the assumption that this is exactly what we are talking about. But if we wait until 3.0 comes around it will be too late.
No one is talking about dropping anything for the foreseeable future. But the writing is on the wall that they will have to change at some point. How they change can be influenced in part by discussions like these. Besides, pretty much all of the developers are expert users themselves. They are not going to shoot themselves in the foot by getting rid of anything prematurely.
It’s hard to set priorities in an opensource project because ultimately each and every developer who chooses to contribute sets their own priorities except in the rare cases where a company pays the developer to contribute (OH has precious few of those, if any). That’s why we have new features like HABot coming down the line without support in the UIs to configure it, for example.
Seems like once you do it for one it would be simple enough to do it for all. Personally, this is one of the many things I think needs to happen for NGRE to lose the “Experimental” from it’s name.
Yes, under $USERDATA/config/org/openhab/<addon>.config.
You mean years ago?
That is why I said :" At least that is what I understood months ago when moving to 2.0"
In these years I did not learn that was changed. thanks to this discussion I did
you cannot edit the file while OH is running there is nothing wrong with editing it by hand.
This still makes it not usable for me art this moment.
Willing to wait.
What exactly is it about JSON formatted text files located in /var/lib/openhab2/jsondb is different from Xtend formatted text files located in /etc/openhab2 that prevents you from having exactly the same configuration on both machines?
When you originally wrote it, I understood it as one machine/etc/openhab2
the other machine at /var/lib/openhab2/jsondb
That’s not a requirement. That’s “I don’t like it”.
I originally wrote: My requirement is that I want to completely configure it from files that I save in a source control system. That was not I don’t like json
With what I learn I would add:
I want to download text files from my own source control system when the system is running and it reads them when they are changed
I would like to make clearer that these should be files that are not changed by upgrades of openhab.
So that when I have one version of openhab 2.4 and openhab 2.5 running it keeps working (I’m not talking about new bindings that would not work in the older version
This might be because what was asked was what features would JSONDB need to have to be an acceptable “expert ui” and your answer is “I need an expert UI”.
1 I want to add, that is how it felt, that does not mean it was intended like that.
what it made me feel like that, was the fact that I felt pushed to paperUI, where I never want to go for a UI, I want text based. This has to do with the fact I have been told multiple times that json as it was used, could not be changed manually by users, only programmatically. (combine that with the fact that I find json totally not people friendly, I considered it logically, and I no longer questioned that. )
(even if we use json as a configuration at multiple projects with a client)
The biggest problem that need to be solved is to have a migration path from the beginner configuration to the expert configuration. We don’t have that now and we will never have that with the text based configs in their current form. We need a way for users to get started using the UIs and as they become experts they can seamlessly migrate to the “expert ui” without recreating everything again
I agree that is helpful
I’m not sure it can be solved in the same version as the paperui that makes it a great beginner UI
you also need a conversion to help people do the move. If you don’t too many people will stay behind that need to be supported, when you make the move easy, you only need to support the move, not the old way of working, so this support will die
I am under the assumption that this is exactly what we are talking about.
Yes this conversation make sit very clear that we are not so far apart
But if we wait until 3.0 comes around it will be too late.
To discuss yes. I do think that architecture can be changed very late, if you have enough tests to prove everything still works.
Not everyone who wrote a binding might have written enough tests, I have no idea how much this is enforced during reviews at openhab.
Yes, under $USERDATA/config/org/openhab/<addon>.config
This is a folder/ files where I not so long ago read that I should not change. Is this wrong now?
If this is no longer true, and there is a tool that upgrades older .cfg files to new config files, we already have one place less to maintain.
Yet I assume that might give problems with OH1 bindings
I believe that is the case which actually caused a problem during this last upgrade cycle for people who ignored the “experimental” part and are using the Experimental NGRE. From what I’ve gathered thus far, the format for the Rules JSONDB changed but the upgrade did not touch the files. As a result the old files were no longer working.
This is the opposite side to the same problem and it needs to be addressed. I’m not sure what the best answer would be. Hopefully this is just growing pains for the NGRE but we cannot rule out the chance that a change in format may be needed in the future as well. And that needs to be handled better than having all the Rules just disappear like they did.
But so far, the formats for Things, Items, and Links have remained unchanged since late OH 2.0 time frame.
This is something I’ve been fighting for months. I think a bunch of old timers are remembering all the pain there was for a few months back in the day when we were using MapDB instead of JSONDB, and MapDB was slow, stored everything in an opaque binary blob, and became corrupted easily. When MapDB was replaced with JSONDB to fix these problems and address the concerns that many of us had, myself include, to keep a history of changes in source control and have the ability to edit them by hand. But people still see those old postings and old timers still think that we have MapDB running all of this stuff and make statements that are not true.
I’ll be the first to admit that JSONDB as it exists today is no panacea and there are many reasons one may not want to use it. But most of the reasons people site for not wanting to use it (it can only be edited using PaperUI, it can’t be edited by hand, it can’t be backed up or checked into source control) do not apply. I think I called this FUD in a separate thread.
Next Gen Rules Engine. It is sometimes referred to as the Experimental Rules Engine. This is eventually going to replace the Rules DSL as the default Rules Engine. A conversion tool or some such will be created to ease with the transition. I think they will somehow graft the Xtend based Rules syntax into the engine so it can run along side the others. I’ve started some docs for it at Experimental Next-Gen Rules Engine Documentation 1 of : Introduction.
It’s not ready for prime time but it is going to be pretty awesome when it matures. If you want to use the “expert ui” version of it now, it’s JSR223 Rules. The NGRE is the Beginner’s UI version of it that will let users build Rules in PaperUI. And it will support all the things people miss from Rules DSL like “real languages”, libraries, Rule templates, etc. Someone started a NodeRed like UI for it as well.
This is where the configuration gets created when you configure an addon through PaperUI (etc). All of your .cfg files in /etc/openhab2/services get moved to those files. I see no reason why we can’t edit those files by hand, though the same caveat about avoiding editing them while OH is running might apply. I admit I’ve not played with them that much.
Indeed, OH1 bindings will always be a problem until we have viable 2.x versions for all the major bindings. But what’s a major binding? I’m sure your list is different from mine. I think we will be stuck dealing with those problems for quite some time. But for me, I’m actually down to just HTTP and Expire as 1.x version bindings.
You don’t understand. For a reliable system configuration need to be pulled from one source of truth. We have two at the moment. If jsondb knows about a thing and your beloved text file knows about the same one, you get a conflict exception and undefined randomness happens.
JSON as a format don’t need to stay. It can also be yaml or xml. Doesn’t matter for the backend. But it need to be machine readable on start-up and writable for syncing the internal state.
I’m not sure if this will be possible without some workarounds. If openHAB and the user stores the file nearly at the same time, who wins? The user would need to create a “lock” (either by a .lock file or by locking via the UI).
Actually .thing files changed a lot and OH got stricter over time and files broke for many users (that didn’t stick to the syntax). jsonDB stores a version number and can be auto migrated.
Not necessarily paper UI. The REST interface. For the moment without a locking mechanism of jsonDB files this is the only correct way if you want to avoid OH1 technology. In theory there could also be a file watcher on the system that watches .thing files and pushes changes to the REST API.
And both get a replacement (as HTTP2 and as a timer framework feature).
openHAB has a service called “Configuration Admin” that reads all .cfg files in and stores its state periodically to .config files. The state can be influenced through files, the openHAB console and the rest interface (web UIs, android, iOS).
Based on observation, and not extensive observation mind you, OH only writes to the JSONDB files based on some event. For example when I create a new Thing and hit that bit check mark icon or when OH shutsdown or the like. From what I’ve seen, even the Things in the inbox don’t get written to the JSONDB until you accept it.
So how much effort is necessary and worth it to solve the file locking problem? It would only be a problem is there is more than one user administering OH at the same time in two different ways. Is this a common occurrence?
I admit I don’t like to leave a race condition problem like this lurking in the code, but the common use case wouldn’t run into it. And since only experts would be doing this we can rely on documentation to tell them to avoid it.
Just a thought. I don’t know the code itself, just what I’ve observed.
I have thought about something like that as a way to ease the transition to JSONDB for those with extensive setups. Maybe it’s worth thinking more grandly.
I know and am excited, especially for the timer framework.
. It uses a number of different files for different tables…
