Next generation design : Ideas & Discussions

(marcel_erkel) #161

I’ve read all comments and there are many to which I would like to leave a comment, but it’s simply too much so I’ll stick to a few that I remember.

If openHAB had always been that way (configuration exclusively via a GUI interface), I would have never used it. And to be clear, I don’t keep my configuration files in a source control system yet. It is something I may do in the future, but currently I don’t. And yet, I still prefer text files over a GUI for the simple reason that you can simply copy/paste lines, comment out the original line, make small changes to the copy, undo those changes simply by removing the copy and uncommenting the orignal line. Suppose you had used a GUI and changed five items/things/channels/links/whatever and you realize that you want to undo the second change. There’s no easy way to do that, you must clicker-the-click through the GUI again to remove all the changes you’ve made to that second item and hope that you still remember how you setup that seconds item 3 months ago. No jsondb backup is going to save you there.
So this has nothing to do with how disturbingly easy it is to screw up your configuration when editing text files (openHAB must just log an error when it fails to parse the config file and nothing bad will happen). You can just as easily (even more easily in my opinion) screw up your configuration when using a GUI, however, when using a GUI it is much more work to unscrew it all so the consequences when using a GUI are much more severe.

Actually, you’re wrong here as well. We are not the ‘old way of doing it users’, a term which I find pretty offending, as in we’re conservative, we’re against change, we’re the ones holding back progress of openHAB. That’s not the case at all. Try configuring Google Home via the GUI. It cannot be done. So we’re actually the ‘bleeding edge way of doing it users’. We go where no GUI user has gone before.

From what I’ve read from your comments, you’re not an openHAB developer, yet you’re making assumptions that developers who contribute want to do things in a new way. I understand David prefers a GUI config over text based config, but David isn’t the only developer working on openHAB. I also think you’re too new in the community to state as fact that that is what the developers want. And no, text files do not limit what can be done configuration wise. After all the JSONDB is a text file.

I understand that for new users a GUI is a lot more comfortable to start out with because it gives guidance when setting up the system. However, for many power users all this clicker-the-click becomes boring real fast.

Do I understand you correctly here that you want to parse the yaml files using JavaScript in the frontend (PaperUI-NG)? That to me seems like a really bad idea. That would mean that every GUI e.g. Habmin, Habplus, PaperUI-NG++, or whatever GUI someone comes up with would need to reinvent the wheel. And worse, what if they don’t like your config file format and choose another one. It also rules out the possibility to make changes to config files via ssh and trigger a reload using the REST API (e.g. using curl).

As a general remark for PaperUI-NG, I don’t see the need to include VS Code (or some other text editor). The normal VS Code fits me just fine and is available and maintained by Microsoft for all major OSes. It just seems unnecessary work to me, but feel free to add it if you must :wink:

I don’t agree here and from what I read others are also not married to the current text config format. I would actually be very happy with e.g. YAML files and I think lipp_markus sums it up quite nicely. The current files have too many inconsistencies. Of course there needs to be tooling to do the conversion from the old format to the new.

So… now that I’ve shot down everyones ideas let me propose one for you to shoot at:

So if there really is a need to get rid of the Xtend based config files because they make the startup of openHAB slow, make it impossible for openHAB to use a newer version of Java, don’t allow the use of standard libraries, are at the limits of [{<()>}] or whatever brackets there are more, or whatever other problems they cause, why not create a tool that imports text based configuration files (this can be the old Xtend based .things, .items, etc. files or new YAML, TOML, or … files) and communicates with openHAB using the REST API (just like PaperUI does)? Using the REST API also solves the db locking issue and no need to restart openHAB when making config changes.
It could may be also export the configuration from a running instance (just the static config, not the runtime config that openHAB now also adds to the JSONDB, that does not belong in static config files in my opinion). By sorting all the attributes in alphabetical order it will be easy to do diffs between different versions.

Would that not solve the problem? The users that prefer text config files can use them and internally there is one source of truth in the JSONDB which can be exported and stored in e.g. Git.

Once everyone is happy and even Markus :wink: has converted his Xtend based config to some more modern format it could be taken in by openHAB or ESH to become part of the core.

(Andrew Rowe) #162

I joined this forum Oct 2018. I see you’ve been here a little longer so I apologize because apparently something I said was not to your agreement. I read your post several times including quotes to things I posted but because I am new to openHAB, I couldn’t understand your examples.

I am sorry. You are correct, I am not an openHAB developer. If you are, or if I have offended any other openHAB developer by making a very overly broad statement, I must apologize because, you are correct, I have absolutely no knowledge what so ever what would motivate any openHAB developer to contribute code which works any which way be it old, new or otherwise.
I have however, developed software on my own and the overly broad statement I made was based on the principle that as a developer, my goal has always been to write code which is as robust, fast and efficient as possible. I am always looking for ‘new’ ways of achieving those goals.

At this point, what I’m beginning to wonder is if this project has aspirations of having a larger user base including none technical users or if it would prefer to remain the realm of those capable of navigating its complexities. I assumed (perhaps wrongly) that a shared common goal of the community was to try to increase the user base to include users who would be unable to configure using text files, perhaps I’m wrong.

If I am wrong that would be a shame because I have recently made an effort to contribute (though not as a developer) to the new user documentation with what I thought was the shared goal of attracting new users. Part of the reason I did so was comments by other users including yourself.

Well if you are wondering Marcel, that number is at least one. Me. I have three merged pull requests so far and counting. Although for the record I don’t believe I ‘complained’ about anything yet.

I would like to point out that the first word in the first post of this thread by the original poster is:


The title of the thread is next generation design. My statements that you quoted (thanks, I’m flattered) were naively asking readers to ‘imagine’ a next generation design for openHAB, where text configuration never existed. A thread like this obviously attracts a little more hard core crowd.

On the face of it, this project seems to encourage people to contribute, If this project attracts talented developers who are then not allow to explore the next step in the evolution, perhaps they will spend their time elsewhere. That to me would be a pity

(Rich Koshak) #163

I believe this is David’s current proposal. But of course, Kai left his comment teasing an nupcoming announcement that may make much of this thread OBE. But my understanding is what you describe is exactly where we ended up.

This is the whole reason why OH underwent a nearly complete rewrite between OH 1.x an 2.x. This is why Things, Channels, and PaperUI were introduced in the first place.

Yes, it is absolutely a goal of OH to increase the user base and make all of this simpler and easier to do.

But it is also a goal of OH to not abandon its existing loyal users. To not be opinionated in how things MUST be done. And to provide an environment that both tech newbees and hard core developers can be productive in.

This whole thread is solely focused on the latter. But that does not take one iota away from the work that has occurred and continues to occur for the former.

The fear a lot of users have is that changes will be made to OH to support the newbees at the expense of the more technically able. This thread is to find a way where that doesn’t become the case and we continue to support both types of users.

This isn’t an either or discussion. It’s an and discussion.

I think part of what you are seeing is while you might imagine this vision with gladness, excitement, and positive expectation, for many many users both current and those who have never used OH before, that vision is full of fear, dread, and annoyance.

We all have different ways of interacting with computers and for a large portion of the population (perhaps an aging portion) GUIs are like putting weights on their ankles and telling them to run a race.

(Andrew Rowe) #164

Rich do you mean ‘you’ as in the metaphorical you or me directly? Reason I’m asking is because I myself am rather comfortable with text config having administered numerous postfix email servers in early 00s. CL only for security none gui

(YF) #165

I think everything that needs to be said has already been said. I am one of those stubborn command-line/text-based user, but I would rather prefer the term “power user” as those of us that’s been around long enough understand perfectly well how difficult it is to get a decent UI that can do everything correctly and effectively. This is why there is an inherent distrust to the UI generally among developers.

I do not think people are opposed to changes such as the format of the confile files as suggested in this thread. What I would like to suggest is to have a higher threshold for breaking changes. Now that we’ve seen the nightmare with the 2.4 upgrade (I wasted hours to restore mqtt items), perhaps there should be a serious discussion on change policies with regarding to existing features. OH has a very large user bases, and as other have said, we should be considerate of the effort people have to put in just to restore the previously working functionality.

Here’s an example of being backward compatible while still allowing changes. If a binding needs to replace an attribute with a new one, introduce the new one in 2.5, but continue to support the old attribute. The openhab.log file and other documentations would then make it clear that user needs to migrate. Then in 2.6 release the old attribute is dropped. This allows user times to migrate, and it also means everything continues to work after the upgrade to 2.5. If people didn’t migrate and things no longer work in 2.6, they have no one to blame as they’ve already given a window.

Another example is the Java APIs, many APIs were marked deprecated years ago, and only several of them got removed in the recent release (10 or 11).

The impact is obviously on the developer. There is more work to do whenever a change is introduced. However, I believe that knowing the breaking change policy will force a better initial design, and we end up with a more usable piece of software.

(Rich Koshak) #166

Yes, it is the metaphysical/rhetorical “you”.

(Andrew Rowe) #167

k… cause my openhab runs… just saying :laughing:

(David Graeff) #168

But you could just have enabled legacy bindings and install mqtt1 again. This was not seen as breaking change. It was more of a bug that “legacy bindings” got disabled on upgrading.

A big company can do that. Around 5 free time developers/maintainers and a small amount of company baked developers can’t handle those breaking changes, most of the time coming from binding authors. The community need to jump in here and provide guidance for migration etc.

But those few developers really need to care about fixing core bugs, redesign architectures for being more efficient, update dependencies periodically and refactor code again because dependencies change their interface. New feature development need to happen too. There are still a bunch of core services missing to allow me to really enable all features of the embedded mqtt broker for example. (Which also needs an update btw, a new version was released. And yes: The API changed -> bunch of work).

I don’t know if people don’t realize that we are running short on core developers. That is why I’m so pressing with getting rid of xtend and old cruft and having one way where there are two at the moment.

Cheers, David

(Markus Storm) #169

No, it’s just a matter of defining proper priorities.
Proper release management is a must. If that means to slow down development and spend more ressources on testing and writing migration docs also from the developer side, then so be it.
(yeah, understood with MQTT it wasn’t meant that way but an accident. But still it was bad release management).

Are you aware that most of the people in this thread are in fact those power users from the community to actually do that ?
Those that you now try to convince they shall give up on the core configuration mechanism that all their current knowdledge, documentation and support work on the forum is based on ?

Are you aware that most people, including newbies, are well content with OH and what it is offering today ?
There is no benefit to any new or old user in getting rid of any interface or feature. Overall it’s always inferior, a step backwards.
If that means those few developers have to slow down on new developments and spend more work on enhancing/fixing existing stuff instead, then so be it. Adding options is nice, but maintaining/improving on existing ones is more important and not dropping/not seriously changing any (widely used) options is a must.

I know you don’t see it that way but this thread should have at least made you aware that that’s how the vast majority of people sees it.

(YF) #170

@David_Graeff, I wish the 2.4 release doc was that clear on the mqtt change. I don’t think it initially even mentioned that v1 is removed and how to re-enable it. It’s the early adopters that were hurt badly. In my case I didn’t even expect that significant change, since it was all good in M5 build.

mstormi is really hitting all the points here. We all seen how understaffed the dev teams in the real world is. There is always a constant juggling on what to deliver with the given resources. And at the end of the day it is all about priorities. Sometimes we don’t appreciate how a small change (might even be considered insignificant from the core developer) can generate such a positive impact on the user bases. Likewise, a significant and necessary change from the core developer perspective might create gripes.

As has been stated multiple times, we need to understand what are the problems we are trying to solve, and analyze to see their priority based on the various impacts. It wasn’t entirely clear in the beginning. But as the discussion progresses further, I think it comes down to two points: 1) improve loading speed, and 2) ease of maintenance.

I can’t comment on #2. However, as a medium project size OH user, I can comment that #1 is not significant enough at this point to break backward compatibility. I am running a combination of xTend and jython rules on the Raspberry Pi. And yes, it is a bit slow in loading the rules. But I can live with that. Here’s another example problem that has more impact then doing big changes. How about we tackle the loading issues of JSR 223 rules that cause core objects to be not fully available for the jython rules. Currently jython users has to insert an artificial delay.

I hope that example can show you that from the user perspective, we have a very different expectation. We look forward to in-the-background changes that address nuances. New features are nice, as long as they provide a migration path forward, or sufficient times to migrate. In the example above, the Jython community is a small group of users, so the issue is certainly not on the top of the priority list, but just fixing it will make many users happy. Also, as you can see, if many dev can live with the current speed on the PI, I do not think speed is a priority.

(YvesHanoulle) #171

But you could just have enabled legacy bindings and install mqtt1 again. This was not seen as breaking change. It was more of a bug that “legacy bindings” got disabled on upgrading.

If the default action of openhab install, is to chose the new binding over an old OH1 binding, and the new binding is breaking (which is probably the case for all OH2 bindings)
I consider it a bug.
yes it could be an upgrade bug, yet the developer that creates the new binding could/should know that this will happen.

That is not to shoot you or anyone else (I have no idea who’s fault that is)

yet I do think if we as an OH community don’t learn from this bug that has bitten many people, then we do have a problem.

And especially because it was not documented, people did not know they could reenable the old binding.

On top, when I read that in 2.4M5 the binding worked and in 2.4 release it no longer worked, I do consider it a breaking change for the Milestones installers.

Power users that are quickly installing milestones and daily releases are the people who give feedback and help to get a good view of the quality of a release.
If we as an OH community don’t support them adequately, a lot of them will not be so quick on installing new versions, and then every developer will get less feedback.

(Rich Koshak) #172

So do Xtend users some times. There is a larger and moire fundamental issue here and there are open issues at ESH for this.

Here is my problem with this. I personally spend and I know Markus spends a good deal of time working with people on the forum on this very issue. It isn’t just that it’s slow but it is easy to get stuck with a queue of reloads that can bring just about everything to a halt for hours. You may be satisfied with this but it has become a minor time sink for us.

(YvesHanoulle) #174

It isn’t just that it’s slow but it is easy to get stuck with a queue of reloads that can bring just about everything to a halt for hours.

is that pure speed or is that more a memory problem?

(Rich Koshak) #175

It seems to be a CPU problem but I’m not really sure as I don’t run on an RPi and it appears to be mainly a problem on that platform. I believe at least part of the source of the slowdown comes from the Xtend libraries themselves. I do know that use of primitives in Rules will exacerbate the problem for some reason (I have theories but they are unsubstantiated) which would be from the Xtend libraries. But I think there are also some problems with how OH does file polling and reloads as well.

The way people run into trouble is:

  1. Edit file and save
  2. OH starts to load and parse all the files, this can take several minutes,.
  3. Edit and save a file while 2 is still ongoing
  4. OH queues a new load and parse of all the files to take place once 2 is done.

The edit of a few .rules files can put OH into a loading state for hours if one is not careful.

A work around is to do all your edits “offline” and then move them all over in bulk so there is only one load and parse.

(YvesHanoulle) #176

A work around is to do all your edits “offline” and then move them all over in bulk so there is only one load and parse.

By working on my machine and uploading them to a source control system to then download them on my openhab, I’m kind of working that way.

I have an PINE64, which is a little bigger, yet not that much.
indeed starting up takes some time, yet I consider that normal.

yet when I download a changed file, it does not seem that openhab starts to load all files.

Is that true for all files ?
or just rules?

(Rich Koshak) #177

I don’t know. I think just rules. And the more powerful machine may get you past it. I’m on a VM (two cores CPU, 8 gig ram) and my startup time is less than a minute. When I heard that RPi users routinely see load times in the tens of minutes I was very surprised. I would not be surprised is there is something about how the ARM processors on SBCs work that could be a problem. I should note that I only recently added a core to this VM because I plan on running IP Camera stuff on it (up to four cameras). I was seeing similar performance with only one CPU. And it has WAY more RAM than it needs, but I’ve plenty of ram to spare on the host.

(YF) #178

Rich, that speed statement is in the context of making backward-incompatible changes. I do see strange issues as well, and when the system acts up, the solution is to restart OH. It’s definitely not the proper way to fix it, but IMO this only affects a small set of people making changes on the production machine.

I also believe if we look deeply at the problem, the issue is not because of the slow loading of the xtend rules. It is likely caused by a lock or unsynchronized states. The slow hardware just makes the problem more likely to happen.

(marcel_erkel) #179

No need to apologize. Just because I’ve been around longer does not mean that I’m more important that you are. In my opinion we’re all equal sharing the same goal, we just have different opinions on some things. These different opinions can only make openHAB a better product.

So you don’t count, because you didn’t complain. That’s my point. The people that complain are usually not the ones stepping up and take action. And those that take action usually do not complain because they simply fix what is wrong.

If I read ‘imagine’ as in ‘dream’ then for me that would be a nightmare :wink:

I didn’t understand it that way, but maybe I was put on the wrong foot when David wrote that he wasn’t going to write a JavaScript parser for the current syntax. That to me seemed like the parsing was going to be part of PaperUI-NG, not a separate tool. But maybe he meant to use the parser for syntax highlighting and verification.

(David Graeff) #180

Exactly. If I want to support a textual mode in Paper UI with the current .thing / .item syntax, I would need to write a parser for that syntax. And that is a nightmare.

Because this thread is going out of hands a bit (a lot of repetitive comments), does it perhaps make sense to close this as resolved and open another Thread with the same name?

I would then summarise my proposal again, considering all comments. Unfortunately I can’t please Markus @mstormi in my proposal and let the current .thing / .item files stay as they are. XTend really needs to go. Starting up OH shouldn’t take half an hour for a decent setup.

(Markus Storm) #181

I don’t feel this to be that much of a repetition. No I don’t think that makes sense as it likely won’t change anyone’s position.

I do think that it makes the most sense to wait for @Kai to come up with his view on things as he announced.