Next generation design : Ideas & Discussions

Development Misc
13

If I’m talking to someone in the software industry I agree completely.
When talking to someone outside the industry, there is no difference between the two. Both activities are “programming” in their minds.
This is one of those places where I think it is better to go with the flow instead of introducing and describing new concepts that ultimately do not matter from a practical perspective.

I disagree. You scare people(non IT people) away by calling it development

People are already used to configuring their systems
windows, phones etc

14

The Expert UI (text files) are the most mature but at some point something must be done to replace this.

I don’t say that It should not be changed, I can imagine it has to be updated.
yet I do think you need such an expert UI

All my configuration is in a source control system.
so I can do diffs to see differences why something works and does not.
Which can be done with JSONDB.

From what I understand that’s not the same way as I do it.

From what I understand PaperUI is the interface that updates and read the jsondb and we should not update the json.
At least that is what I understood months ago when moving to 2.0

And while people are creating a ui like paperUI, I understand that.
Updating two interfaces (aka simple and expert) at the same time I would not advice

This process can continue unchanged with the only difference being which folders you check into and out of your source control. Instead of /etc/openhab2 you would control /var/lib/openhab2/jsondb.

that is not what I want. I guess I am not clear.
I want to have 1 system that is exact the same as the other.
I swap system by turning them on and off
I do want the configuration on both systems be exact the same, except for the Openhab version.

that way I can test,

It is just a series of JSON formatted text files
except json is not really human friendly in my opinion

Pretty much all of this can continue, though there might be a problem because, at least for now, JSONDB cannot be edited while OH is running. So your button would need to shutdown OH, pull the changes, then restart OH.

Which is a huge issue.
One of the reasons why I like openhab, is that it did not need to be restarted in the first place.
That really helps, for my children /partner to understand and use openhab.

I’m not sure your remark about long term problem is on this or on the upgrade part
which is something I agree, yet is for me something that could only be done when 2.0 is a lot further

The text config files as they exist now will go away at some point.

  1. I understand they might need to change
  2. We were asked the requirements. As a user I’m saying that like I see now the way the json is be implemented, does not seem to be that replacement for me.

Speak up now because you have the attention of a developer who is actually willing to work on it. Don’t lose this chance!

I’m doing it, and it feels that when I say it, I’m told, that’s wrong that is not a requirement

I really believe you need the two systems.
I do agree that you need a system that is easy/easier to maintain
yet I also think it’s to early to drop that old way of working

For me it feels like 2.0 is all about adding a simple user interface like paperUI and whatever is needed to support that

When that is “finished”, have a new look to work on 3.0 where you can think about reworking the text file to have a better expert file system.

Don’t drop the expert text system, while working on the simplir system.

that also help on setting priorities, which is for me the biggest bottleneck for all projects.
all projects want more that they have time and people for.

15

If I could do that, on a per-Thing-basis, so I can have all my hue-things in one jsondb, all tahoma-things in another, and so on (or whatever logical grouping I might have need of), the this would be fine.

If the items and rules would follow that distinction with storage contexts, it would be even better.

The service definition files (e.g. “influxdb.cfg” and so on) already are in other files, right?

16

You mean years ago? We’ve come a long ways since OH 2.0 was released and except for the problem where you cannot edit the file while OH is running there is nothing wrong with editing it by hand. I just copied/pasted/edited a couple of dozen new MQTT Things by hand in the JSONDB with no issues what so ever. It’s just a text file.

What exactly is it about JSON formatted text files located in /var/lib/openhab2/jsondb is different from Xtend formatted text files located in /etc/openhab2 that prevents you from having exactly the same configuration on both machines?

I run in Docker so pretty much every time I restart OH I’m running it on a new “machine”.

I agree, it’s a big problem that must be addressed.

That’s not a requirement. That’s “I don’t like it”. What would need to be changed to make it a replacement. Because there will be a replacement and right now JSONDB is the only candidate.

This might be because what was asked was what features would JSONDB need to have to be an acceptable “expert ui” and your answer is “I need an expert UI”. And given the scenario you outlined, I could not see anything that would make JSONDB unsuitable as it exists today, let alone with new features, except for the before mentioned inability to edit while OH is running.

The biggest problem that need to be solved is to have a migration path from the beginner configuration to the expert configuration. We don’t have that now and we will never have that with the text based configs in their current form. We need a way for users to get started using the UIs and as they become experts they can seamlessly migrate to the “expert ui” without recreating everything again. And at the moment, the prime candidate for managing that migration path is to make improvements to JSONDB so it can become the expert UI. Does this mean making it easier to edit them by hand (e.g. the restart problem)? Does this mean building a new UI or sub UI in PaperUI? Does this mean being able to break them up a bit more?

I am under the assumption that this is exactly what we are talking about. But if we wait until 3.0 comes around it will be too late.

No one is talking about dropping anything for the foreseeable future. But the writing is on the wall that they will have to change at some point. How they change can be influenced in part by discussions like these. Besides, pretty much all of the developers are expert users themselves. They are not going to shoot themselves in the foot by getting rid of anything prematurely.

It’s hard to set priorities in an opensource project because ultimately each and every developer who chooses to contribute sets their own priorities except in the rare cases where a company pays the developer to contribute (OH has precious few of those, if any). That’s why we have new features like HABot coming down the line without support in the UIs to configure it, for example.

Seems like once you do it for one it would be simple enough to do it for all. Personally, this is one of the many things I think needs to happen for NGRE to lose the “Experimental” from it’s name.

Yes, under $USERDATA/config/org/openhab/<addon>.config.

1 Like
17

You mean years ago?
That is why I said :" At least that is what I understood months ago when moving to 2.0"
In these years I did not learn that was changed. thanks to this discussion I did

you cannot edit the file while OH is running there is nothing wrong with editing it by hand.
This still makes it not usable for me art this moment.
Willing to wait.

What exactly is it about JSON formatted text files located in /var/lib/openhab2/jsondb is different from Xtend formatted text files located in /etc/openhab2 that prevents you from having exactly the same configuration on both machines?

When you originally wrote it, I understood it as one machine/etc/openhab2
the other machine at /var/lib/openhab2/jsondb

That’s not a requirement. That’s “I don’t like it”.

I originally wrote: My requirement is that I want to completely configure it from files that I save in a source control system. That was not I don’t like json

With what I learn I would add:

I want to download text files from my own source control system when the system is running and it reads them when they are changed

I would like to make clearer that these should be files that are not changed by upgrades of openhab.
So that when I have one version of openhab 2.4 and openhab 2.5 running it keeps working (I’m not talking about new bindings that would not work in the older version

This might be because what was asked was what features would JSONDB need to have to be an acceptable “expert ui” and your answer is “I need an expert UI”.

1 I want to add, that is how it felt, that does not mean it was intended like that.

what it made me feel like that, was the fact that I felt pushed to paperUI, where I never want to go for a UI, I want text based. This has to do with the fact I have been told multiple times that json as it was used, could not be changed manually by users, only programmatically. (combine that with the fact that I find json totally not people friendly, I considered it logically, and I no longer questioned that. )
(even if we use json as a configuration at multiple projects with a client)

The biggest problem that need to be solved is to have a migration path from the beginner configuration to the expert configuration. We don’t have that now and we will never have that with the text based configs in their current form. We need a way for users to get started using the UIs and as they become experts they can seamlessly migrate to the “expert ui” without recreating everything again

I agree that is helpful
I’m not sure it can be solved in the same version as the paperui that makes it a great beginner UI

  • you also need a conversion to help people do the move. If you don’t too many people will stay behind that need to be supported, when you make the move easy, you only need to support the move, not the old way of working, so this support will die

I am under the assumption that this is exactly what we are talking about.
Yes this conversation make sit very clear that we are not so far apart

But if we wait until 3.0 comes around it will be too late.
To discuss yes. I do think that architecture can be changed very late, if you have enough tests to prove everything still works.
Not everyone who wrote a binding might have written enough tests, I have no idea how much this is enforced during reviews at openhab.

NGRE

No idea what is NGRE. Searching in the documentation gives a link to https://www.openhab.org/docs/developer/contributing/contributing.html#sign-your-work
which sounds strange.
(and googling gives non openhab related organisations)

Yes, under $USERDATA/config/org/openhab/<addon>.config
This is a folder/ files where I not so long ago read that I should not change. Is this wrong now?

If this is no longer true, and there is a tool that upgrades older .cfg files to new config files, we already have one place less to maintain.
Yet I assume that might give problems with OH1 bindings

18

:+1:

I believe that is the case which actually caused a problem during this last upgrade cycle for people who ignored the “experimental” part and are using the Experimental NGRE. From what I’ve gathered thus far, the format for the Rules JSONDB changed but the upgrade did not touch the files. As a result the old files were no longer working.

This is the opposite side to the same problem and it needs to be addressed. I’m not sure what the best answer would be. Hopefully this is just growing pains for the NGRE but we cannot rule out the chance that a change in format may be needed in the future as well. And that needs to be handled better than having all the Rules just disappear like they did.

But so far, the formats for Things, Items, and Links have remained unchanged since late OH 2.0 time frame.

This is something I’ve been fighting for months. I think a bunch of old timers are remembering all the pain there was for a few months back in the day when we were using MapDB instead of JSONDB, and MapDB was slow, stored everything in an opaque binary blob, and became corrupted easily. When MapDB was replaced with JSONDB to fix these problems and address the concerns that many of us had, myself include, to keep a history of changes in source control and have the ability to edit them by hand. But people still see those old postings and old timers still think that we have MapDB running all of this stuff and make statements that are not true.

I’ll be the first to admit that JSONDB as it exists today is no panacea and there are many reasons one may not want to use it. But most of the reasons people site for not wanting to use it (it can only be edited using PaperUI, it can’t be edited by hand, it can’t be backed up or checked into source control) do not apply. I think I called this FUD in a separate thread.

Next Gen Rules Engine. It is sometimes referred to as the Experimental Rules Engine. This is eventually going to replace the Rules DSL as the default Rules Engine. A conversion tool or some such will be created to ease with the transition. I think they will somehow graft the Xtend based Rules syntax into the engine so it can run along side the others. I’ve started some docs for it at Experimental Next-Gen Rules Engine Documentation 1 of : Introduction.

It’s not ready for prime time but it is going to be pretty awesome when it matures. If you want to use the “expert ui” version of it now, it’s JSR223 Rules. The NGRE is the Beginner’s UI version of it that will let users build Rules in PaperUI. And it will support all the things people miss from Rules DSL like “real languages”, libraries, Rule templates, etc. Someone started a NodeRed like UI for it as well.

This is where the configuration gets created when you configure an addon through PaperUI (etc). All of your .cfg files in /etc/openhab2/services get moved to those files. I see no reason why we can’t edit those files by hand, though the same caveat about avoiding editing them while OH is running might apply. I admit I’ve not played with them that much.

Indeed, OH1 bindings will always be a problem until we have viable 2.x versions for all the major bindings. But what’s a major binding? I’m sure your list is different from mine. I think we will be stuck dealing with those problems for quite some time. But for me, I’m actually down to just HTTP and Expire as 1.x version bindings.

2 Likes
19

Actually. Yes I can. See GitHub - microsoft/monaco-editor: A browser based code editor.
Microsoft provides the VS code editor component and it can connect to an autocomplete openHAB instance (GitHub - TypeFox/monaco-languageclient: Repo hosts npm packages for monaco-languageclient, vscode-ws-jsonrpc, monaco-editor-wrapper, @typefox/monaco-editor-react and monaco-languageclient-examples).

You don’t understand. For a reliable system configuration need to be pulled from one source of truth. We have two at the moment. If jsondb knows about a thing and your beloved text file knows about the same one, you get a conflict exception and undefined randomness happens.

JSON as a format don’t need to stay. It can also be yaml or xml. Doesn’t matter for the backend. But it need to be machine readable on start-up and writable for syncing the internal state.

I’m not sure if this will be possible without some workarounds. If openHAB and the user stores the file nearly at the same time, who wins? The user would need to create a “lock” (either by a .lock file or by locking via the UI).

Actually .thing files changed a lot and OH got stricter over time and files broke for many users (that didn’t stick to the syntax). jsonDB stores a version number and can be auto migrated.

Not necessarily paper UI. The REST interface. For the moment without a locking mechanism of jsonDB files this is the only correct way if you want to avoid OH1 technology. In theory there could also be a file watcher on the system that watches .thing files and pushes changes to the REST API.

And both get a replacement (as HTTP2 and as a timer framework feature).

openHAB has a service called “Configuration Admin” that reads all .cfg files in and stores its state periodically to .config files. The state can be influenced through files, the openHAB console and the rest interface (web UIs, android, iOS).

20

Based on observation, and not extensive observation mind you, OH only writes to the JSONDB files based on some event. For example when I create a new Thing and hit that bit check mark icon or when OH shutsdown or the like. From what I’ve seen, even the Things in the inbox don’t get written to the JSONDB until you accept it.

So how much effort is necessary and worth it to solve the file locking problem? It would only be a problem is there is more than one user administering OH at the same time in two different ways. Is this a common occurrence?

I admit I don’t like to leave a race condition problem like this lurking in the code, but the common use case wouldn’t run into it. And since only experts would be doing this we can rely on documentation to tell them to avoid it.

Just a thought. I don’t know the code itself, just what I’ve observed.

I have thought about something like that as a way to ease the transition to JSONDB for those with extensive setups. Maybe it’s worth thinking more grandly.

I know and am excited, especially for the timer framework.

21

Hm, afaik it writes periodically (every 5 mins or so?) and on Thing changes it accumulates for a bit and then writes as well. But Thing properties are changed quite often (at least I’m doing that in my bindings to visualise state that would get lost in the log).

22

Next Gen Rules Engine.

I have been avoiding it for now, for two reasons.

  1. I understood it was not ready fro prime tiùe

  2. so far, I can do everything I want with the current rules.
    Yes the language takes some use, yet I think it’s also easier to understand for non programmers.

Better testing for rules would be usefull

OH1 bindings

I have +20 anel hut’s in my house. Very happy with the devices (and would not be able to replace them now) yet it seems that the original developer has lost interest (or at least has no interest in moving it to openhab 2.)
I don’t know enough java to do more then look at the code and spot some small errors.

23

The (old) XTend rule engine is slower and more resource intensive then the new one.
I mean there is a reason why there is a new one, right :slight_smile:

1 Like
24

You don’t understand. For a reliable system configuration need to be pulled from one source of truth.

That I understand very well. That was already taking into my consideration.
Like I said , I don’t have paperUI install, that is one of the reasons
I’m using only the text files as way to change the system.
nothing else, because I want only one truth

JSON as a format don’t need to stay. It can also be yaml or xml. Doesn’t matter for the backend. But it need to be machine readable on start-up

And the current files do just that. except from what I understand, OH developers don’t want to keep that format
(and I have no objection to that, I never had.)

Not necessarily paper UI.

Like I said in most of the discussion I felt pushed for paperUI, not for the json and I think this is the second time someone mentions REST api
(REST has it’s own version problems if I remember correctly some of the discussion we had a few months ago at a client)

openHAB has a service called “Configuration Admin” that reads all .cfg files in and stores its state periodically

when I change a cfg file, it seems to be reflected almost instantly in openhab.

25

The (old) XTend rule engine is slower and more resource intensive then the new one.
I mean there is a reason why there is a new one, right

Usually by the time new software is finished that is faster, the hardware has improved faster.
:slight_smile:

26

Yup there is a file watcher installed, that reads a modified file again.

You cut off my sentence, how rude :wink: The second part was important. It need to be writable as well. There is no code in openHAB to write to those custom syntax thing files. But how should OH persist Thing changes then (like auto discovered Things or Thing property changes caused by bindings).

27

Since we sort of doing a survey on the potential of taking away the advance usage scenarios, I’d like to chip in another voice to retain the ability to hand-craft the .things file. It gives advance users lots of control on how to configure things without having to touch the UI. As other has said, the .things file can be checked in version control.

All my devices/services are configured manually. The only things not possible yet are the ZWave devices; @chris thank you for all the hardwork you put into the binding, is this function possible yet? I imagine if we can configure the .things file manually, the move from 2.3 to 2.4 woudn’t have required user to re-add all the items.

28

If it does write the jsondb on a schedule then we do have a problem. But why would it do that? I don’t see the point. It’s not creating a backup for all of those writes. It Durant make sense to me to dump to the file when nothing has changed. It just increases the likelihood of a write failing and corrupting the files.

I don’t see how the benefits or weight the risks.

It sure seems reasonable to cache up a bunch of changes before dumping then to file. There is a small risk of losing a change in a power failure but I’m sure the backup and write of the file is expensive.

This cannot be emphasized enough. OH users on an RPi can see rules file load times in the tens of minutes to over an hour. And the problem compounds of you change a file while it’s still loading from a previous file the changes get queued up and the RPi and OH can be brought to its knees for hours.

The faster and with less resources issue isn’t just a nice to have upgrade. It’s a very significant problem for a large portion of OH users. A problem that must be fixed.

And if I understand correctly, the problem lies in code we don’t own so a fix is not necessarily under our control.

In general the developers of OH have really good reasons for decisions to make charges like these.

I don’t think we are talking about that at all. I think we are talking about keeping the advanced usage scenarios while also:

  • unifying, or maybe normalizing the configs is a better term
  • providing a way to trasition beginner users who start in the UIs to what Yves is calling “the expert UI”, i.e. something programmers would be more interested in
  • providing a config format that can be both read and written by OH

The .things file cannot remain as it is. We are talking about ways to support something the ability to hand craft Things (I still don’t see the attraction personally).

And once again. JSONDB IS A TEXT FILE. You can check it into source control. I do this now.

No, it would have just required you to go and manually change all your Things instead of letting the software do that for you with autodiscovery. If I understand correctly, you had to remove and readd the Things because the new binding requires more or different fields than the old one did.

1 Like
29

Yes - this is possible with ZWave in 2.4.

30

Even if we now have automatic discovery with OpenHAB 2, I still like to do it manually. I once used network item auto discovery and what happened? It hardcoded the devices IP addresses - worst way to do it, because they will change. I’d rather like to use hostnames, and that’s what I (only?) can do manually, so I stick to manual configuration. Just as one example, it’s the details that matter :slight_smile:

1 Like
31

Only if “we” excludes the ESH developers. See major reprocessing on items addition · Issue #6410 · eclipse-archived/smarthome · GitHub for a premature analysis. I’m waiting for @kai to jump the bandwagon. Yes it’s a severe problem.

32

108 messages multiplied by an average of 10 minutes reaction time is around 1080 minutes or the equivalent of 18 hours. A lot of additional documentation, improvements and PRs could have been created. Just imagine …

2 Likes