Having the Architecture Council established, let’s go into some initial discussion on the overall directions for our beloved project!
A major step has been the re-integration of ESH into openHAB 2. From a technical perspective, this has only happened partially as although we have moved the code to our repos, the Java package names (and thus the API to develop against) is still under org.eclipse.smarthome namespace. Note that we are oblidged to get rid of this in the long term. We therefore have to start with discussions around an openHAB 3, which will have to break (developers) backward compatibility by changing the namespaces - no way around that.
My suggestion is to not make any bigger changes to openHAB 2 anymore and rather focus on merging as many of the pending add-on/binding PRs as possible - quite a long list has piled up (sorry to all contributors who didn’t hear back on those…).
openHAB 3 development should imho start in parallel already and we need to define the directions - there have been plenty of discussions in the community already, wrt setup, UIs, configuration, rules etc.
My personal opinion is that we should use the major version shift for simplifications. A few options worth to discuss:
Removing the compatibility layer, thus having all add-ons behaving the same way (in terms of setup & configuration)
The NG rule engine should be the only one - existing Xbase-rules should be supported by it as well and the Jython-rules should be made a first class citizen.
Simplify switching from UI to textual configuration and vice versa - maybe through a new database format, by export mechanisms or whatever.
It is important to note that we can only realise such “wishes”, if we also have contributors willing to work on such topics - so this will always have to be taken into account.
For the start, I’d like to start gathering the most important ideas/changes with you here. We should then open a dedicated topic for each where we think that the time is ripe for a decision (I did so for the “web UI” already).
Is there a deadline for this? I ask because do we need to fit all of these proposed changes into a specific calendar time period, or can we release 3.0 when it’s ready without needing to rush?
I’ve gathered a lot of lessons learned from the release of OH 2.4 so I’m very sensitive to how we roll out these breaking changes.
I completely agree. The backlog of new bindings awaiting approval is long.
Does this include deprecating/removing any 1.x bindings that have basically been abandoned? If yes (and I think it should) we need to be very vocal and communicate that these bindings are going away unless someone steps up to bring them up to at least running without the compatibility layer. And we need to give them adequate time for someone to step up or for users to make other arrangements.
I’d like to see all four languages be first class citizens in NGRE (Jython, JS, Groovy and Xbase). Though through influence on the forum I’d like to start de-emphasizing Xbase rules unless moving them to NGRE solves some of the structural problems with them (e.g. non-thread safeness, limited number of Rules threads). I spend a lot of time helping people deal with problems caused when they have long running Rules and weirdness that happens when they use global lambdas or have Rules that trigger really fast.
Yes, I’m strongly in favor of this. Users should not be locked in to just one approach. They should be able to move relatively freely between the two approaches.
OK, here are my current thoughts.
As you mentioned, Jython Rules have come a really really long way and with Scott et al’s library I’d venture to say that Jython Rules are now easier to write and understand than the current Rules DSL. If we can distribute the Jython jar and the helper library I’d start pushing people to use it by default starting now. Even if we can’t get the UI elements in place by the time it’s ready to release OH 3, I’d like to see Jython or JavaScript become the default recommended one and the one the demo and docs are based on.
For all of these proposed changes I’d like the developers to keep in the back of their mind “How will the folks on the forum provide support for this feature?” This is particularly important for the replacement to PaperUI. A bunch of screen grabs of web forms is not going to cut it. The users need easy access to some concise text based version of the configs they can paste into the forum that we can read and understand and provide help.
A demo config that matches the Beginner’s Tutorial. Of course, this means beefing up and completing the Beginner’s Tutorial. But if we will continue to develop and maintain a demo config, I feel that it should be documented somewhere and where better then through the Beginner’s Tutorial? And while we are on the subject, we need an easier way to move off of the Demo package. Right now I think the only ways are to edit addons.cfg or reinstall OH. My first idea for the Beginner’s Tutorial was that it would build up the demo config in steps so the users would get documentation for the demo config, gain an understanding about how all the parts work together, and be able to look ahead.
A well defined way to create, package, and distribute (through the IoT Marketplace?) Rules libraries. I think this is bigger than NGRE alone so I raise it as it’s own topic. One thing I’d like to see happen is the end to copy/paste/edit Design Pattern postings for common Rule approaches. Instead I’d like experts to build reusable Rules and let people download and just use them. But for that we need a distribution mechanism. My understanding is that technically everything is already in place, we just don’t have UIs, tools, or docs to aid in the process. It sure would be nice though if we could take the top 20 tutorials or so from the forum, implement them, package them up as Rule Templates or libraries, and have them available for use in OH 3.
A scheduler service of some sort. David’s PaperUI-NG Study has a mock-up of what that could look like. But next RBAC, a scheduler is the second most requested feature on the forum. Users have two use cases that come to mind and I’ve a third.
Alarm Clock that can be modified from the UIs (regular UI, not admin UI). Right now it is really painful to set up and configure a simple “send this event at a time configured on the sitemap”.
A repeating calendar of events akin to what one sees with HVAC interfaces. This is another case where I think users want to provide access to this from the regular UI, not just the admin UI.
A replacement for the Expire 1.x binding. Since it was introduced it has been adopted enthusiastically and it has replaced Rule Timers in many people’s code. But as a 1.x version binding it means many users who would like to move to UI only configs for Items are stuck with .items files. And there is a good argument to be made that the functionality would be better as part of the core, or perhaps as a Profile or something similar.
I may come up with more later and I’ll edit my post. But this is what I can think of right now.
If you ask for an “official” deadline, it would be December 2018. There simply should not be any use of that namespace in our repos. I’d hope that the Eclipse gods pardon us, if we manage to move all the active code development to openHAB namespace latest by the end of this year. I’d think that development of 2.x and 3.x will have to happen in parallel throughout this year.
I may come up with more later and I’ll edit my post. But this is what I can think of right now.
Note that I tried to think of things that removes burden from us, not about adding new features
Remember that for all good ideas, you will need to find somebody to implement it - and from what I have experienced in the past, there are usually enough ideas what could be done but too few people actually doing it. I’d actually suggest to enter many of your ideas into the issue trackers of the concerned component and have them discussed there. We should not try to draw all discussions and decisions here, but rather be the “last resort” for maintainers to go to, if there’s really something strategical/overarching to decide that they do not feel comfortable with or empowered for.
I agree. But as a helper on the forum, these are things that would remove burdens from us.
Some of them already are (scheduler for example).
I am, but short of hand editing JSONDB files (which isn’t terrible but it isn’t great either) I don’t know how to create them. Once created I don’t know where to put them so others can download them. Once downloaded I don’t know where to put them so they become part of my OH config.
I’m running 2.5 M1 and I don’t have a Rules Template tab so it’s all very opaque. I know these things are possible but I don’t know where to start.
I’ll figure out which of these has an open issue and open new ones for those that don’t.
The rule templates have never been rolled out to the openHAB users. My point is that they are the feature you are asking for. The format does not have to be JSON, other formats could be implemented (not sure how it would have to look in Jython). All this should be part of making the experimental RE the official RE - unfortunately, this is quite some work. Just to prove my point from before: The idea has been there for years, just the people who implement it are missing…
I’ve received a PM regarding the removal of the compatibility layer so I’m going to go ahead and open up a thread on Development for further discussion.
I don’t know if we should open a new thread now or not. But I have some real concerns about removing the compatibility layer without offering any way to continue to support 1.x bindings.
There are far too many 1.x bindings for which there isn’t a 2.x binding to replace them and sadly that will always be the case as there are some bindings that have lots of users that are basically abandoned and have no one to rewrite them as 2.x. The list of 1.x bindings that have a 2.x version in the works appears to have closed to a trickle and some very heavily used ones (e.g. Expire) do not have any replacement in the works as far as I know. But even obscure bindings like Insteon and Asterisk (which never actually got listed as officially supported in OH 2) have some number of users who will either be stuck on OH 2 forever or lose full support for their technologies.
For many 1.x bindings that have a maintainer those developers do not see the benefit or spending a lot of effort to rewrite the binding without adding any functionality. Apparently a consistent user experience is not sufficient enough benefit for these maintainers. But many users would rather live with the inconsistency then to lose access entirely.
The loss of support for the 1.x bindings at this time will result in a significant loss of users and potentially maintainers as well, and all the acrimony and bad feelings that would go with that. It could take a long time to recover from this as a community, if we ever do.
So if we do remove the compat layer, we need to provide a path for users that still depend upon 1.x version bindings to continue to do so. Some ideas have been raised on the thread I linked above.
Ultimately I think we either need to come up with a way to make the OH 1.x bindings work on the new core with all the necessary REST end points and everything, or continue to use something like the compatibility layer.
We’ll said @rlkoshak really echos my feelings. Following the if it isn’t broke philosophy I can say. I use insteon and expire. Really as a user I care that it works more than new features.
For expire not sure things and auto discovery apply.
For insteon yes I suppose auto discovery/channel links could help but not sure what other features could be added.
So, my main agreement is that we need a solution that encompasses many of these bindings that I am sure exhibit the same conditions. I feel some of the reason the bindings appear abandoned is because they function great and need few added features. So it would be more or less work for the sake of work.
I do understand we need to balance this thought process with whether or not things do just need a general review and update every few years.
For expire specifically, I think it should just be a setting in the Item definition window in PaperUI, much like Groups currently are. Expire directly acts on the “state” of the Item, so it should just be a configuration setting for each item, rather than a dedicated binding.
I absolutely agree and stated the same here a while ago. Imho, the expire1 binding should never have been introduced in the first place, I only learned about it when it was too late already.
So if they work, they should have been added to the distro. This is hardly any work and people were asked since FIVE years many times to do so. If nobody felt willing to take this little effort over 5 years, this speaks a lot about their engagement and help for the project.
The loss of support for the 1.x bindings at this time will result in a significant loss of users and potentially maintainers as well
Keeping up complexity will PREVENT maintainers from joining/staying with the project, so it’s hard to tell, which side is more relevant in the long run. Fact is, in 5 years time nobody ever took care of the compat layer but me. It’ll be interesting to see, which developers will jump to the rescue the next time it breaks and I don’t fix it. Let’s try that out. I am not very confident, though. A funny experience were the last 2-3 weeks: While hundreds of people had opinions and spent a massive time on discussions, nobody seemed to care about the fact that we had no working snapshot distro and I was more or less the only one chasing bugs and regressions… Pretty frustrating, really. So whatever you discuss, please always consider the situation where I might not be available and others have to keep the project running - that’s why I focus so much to make it easy for others, because otherwise it could be an instant death.
Ultimately I think we either need to come up with a way to make the OH 1.x bindings work on the new core
I don’t agree that it has to be “on the new core”. I think @David_Graeff made a great suggestion about it: A feature to connect multiple openHAB instances would be cool for distributed setups (and to allow this was a design goal of the openHAB architecture 10 years ago already) - so if that would be easily possible, you can run a 2.x instance with all 1.x legacy stuff and connect it to a lean and modern 3.x instance to have everything working seemlessly as one setup. Makes imho a lot of sense and I think it’d be great if @David_Graeff would work extending the MQTT support that way.
I just happened to stumble on this topic because I happen to be watching a topic which was referenced here. I can imagine that there are many users who hardly ever visit this forum if at all. Is feedback from the user base ever actively sought? One gap I encounter is that between what users are doing and what developers/maintainers are doing.
It is true that it is the maintainers that provide the functionality in the product. But the fact that they have not maintained their code for a long period of time does not mean that users aren’t using it… and are depending on it.
For the case where a OH2 binding has been made available, I can see making the case that users are given the option to migrate. But I go back to what I said at the front - that would have to be a more direct communication. Otherwise a user will likely face the situation where what is to be a simple update ends up removing some functionality they depend on. Even though the typical openHAB user is probably more technically savvy than the average Joe, my gut tells me that they aren’t strict subscribers to ITIL practices when it comes to running their home computing environments.
But what of the OH1 functionality that has never been brought forward? In some cases it’s because it just works. If it ain’t broke… What options will users have then? Telling them to follow up with the maintainer to plead with them to update the code to the latest architecture is going to be difficult to say the least. And probably not very efficient in getting the desired result. Yes, OH needs to make changes for continued improvement. But backward compatibility is an issue. One that will very likely result in losing users. Will it be offset by new users coming on board? Or will users just make the decision that if they have to completely change their installation, they might as well make the jump to a different home automation solution altogether.
Finally, I know that there is a code of behavior when it comes to touching someone else’s code.
If someone is working on a new binding and drops it for months, it stays there because they are the owners… even if someone else might want to see the work to the finish line. Trying to get permission to do so sometimes goes on deaf ears.
The same for a working binding. It was put out there and now let’s say it’s been orphaned. Is there no way to transfer that to someone willing to maintain it even w/o being given explicit permission? Maybe the terms to contribute to OH should include a “dormant” clause?
I’m sure that this post sounds disjointed perhaps. It’s just comes from my stream of consciousness dump when I saw the theme of the topic - dropping backward compatibility. Don’t get me wrong, I truly appreciate what the OH developer community has done in creating a great tool. I just want to make sure that the voice of the user is given the opportunity to be heard. How can that happen more effectively?
There is. That’s the sole reason why every binding is in one openHab controlled repository. Everybody can at any time suggest changes. And if they improve the binding (performance wise, usability wise) the original author will usually not veto such a change, even if it fundamentally changes his binding.
I have already presented my first option of having two independent OH run-times running, connected by some means like MQTT to sync the two event-bus events.
Threat every OH version as a different product
How about making that even more obvious.
Threat every major OH version as a different product. No migration paths at all (those cost developer hours as stated above). So there are two products currently available OH1 and OH2. Those products might be able to interact with each other, or might as well not. Depending on the core developers of each product.
Binding API versioned
At the podium discussion on the smarthome day 2018, there was also the proposal to version parts of the API independently. So there would be a binding API at version 2.x, a REST API version 2.x and maybe a service API version 2.x. That way the core could develop independently as well and be at version 6.x. As long as it provides a binding API at version 2, all those bindings will still work.
This also means, that we don’t need to auto-sync the binding versions in openhab2-addons anymore. The major version must match the binding API version, and the minor and revision version can be determined by the binding developers themselves. Breaking changes in bindings are more obvious as well. And if a binding developer really require advanced functionality of a newer binding version, he would increase the major version of his binding. Making it incompatible to older core versions. But very well communicated.
Threat every major OH version as a different product. No migration paths at all
That sounds really bad.
As a user I would feel completly f*cked.
When I was replying in all the discussions in the last month or so, my main concern was: make sure you have an upgrade path for your end-users.
I can understand we don’t want to force 2.4 users to upgrade the moment when we bring out 3.0 yet I do think we should offer upgrade paths.
those cost developer hours as stated above
Supporting 3 or more versions costs even more development hours.
Once there is a good upgrade path for a binding in 2.x, people no longer use the older 1.x binding.
That is true. But the concepts changed so radically that no upgrade path can be provided without a lot of development hours (migrations). You also need to differentiate what core developers we have. Personally I’m a pure OH2 developer. And as far as I know (please correct me if I’m wrong) Kai is the only OH1 developer left. And he stated something along the line of that he is not willing to do that in the future anymore.
So the “product” OH1 has no developers left. The question is actually: Can we find other solutions. For insteon there is the mqtt bridge for example. Anel would work with the Http+UDP binding. And so on.
This is where I actually disagree to some extent. There is benefit in consistency is setup and use of bindings. Even if OH 1.x bindings were to become first class citizens again, they will always be different with all the advantages and all the disadvantages of that.
So if someone were to take the Insteon binding and rewrite it as a 2.x version biding with exactly the same capabilities, it would still be an improvement from a usability perspective, particularly usability for new users. It’s not make work.
There is already an issue open on this. I don’t know where it ended up after the great merge.
It’s definitely an approach I can support. My concern is that there is some path for support, not that it necessarily has to remain on OH 3.
Yes but the forum is our only communication medium. Threads like this are actively seeking feedback from the user base.
No matter the outcome of this topic, this is a much requested feature so I hope it happens anyway.
I think going to that extreme would be disruptive, particularly if those older versions cease receiving bug fixes and security updates. If the answer is “well, when we stopped development of OH2 it still had this ugly memory leak that never got fixed, but if you need to use a binding no longer supported on OH3, your only choice is to continue to run that” it isn’t very satisfactory for the users. It means we are telling them that we don’t provide long term support for OH. Every two to three years you will lose functionality if you stick with OH.
I really like this idea for lots of reasons. But does this actually make the job of the OH maintainers easier? It certainly provides a way for users to know and understand what a binding is capable of and why it can’t work on a newer OH core (i.e. OH 3.1.23 supports only API versions 2.5 and above).
Ultimately, I think we are faced with a no-win situation. Any choice leads to harm to the community.
Option
Consequence
Remove support for 1.x bindings entirely
Loss of users and binding maintainers
Run multiple instances of OH
If the older instance is not longer maintained eventually this will break
Keep the compatibility layer
With only one developer who can or will maintain it this will eventually break
Make OH 1.x bindings first class citizens
There is no volunteer to develop this so this path is essentially not an option unless and until someone volunteers
All viable paths lead to the OH 1.x bindings becoming unusable on the new instances of OH. When that happens there will be a loss of users, perhaps binding maintainers as well.
If we break the currently viable options down further we are faced with the following choices:
Do we choose when the 1.x bindings finally break or let it happen naturally?
Do we take on effort for longer term support to the older versions of OH for those who need to keep these 1.x bindings running or just leave them and hope it continues to work?
There are some things that we can do I think to mitigate the user’s long term ability to continue to run older OH instances using Docker or VMs and the like (one of the problems will be continuing to be able to obtain Java 1.8) but even that will be a challenge. I don’t think it will be reasonable to expect in five years that a user will easily be able to continue to run a Java program that depends on a long abandoned JRE. So even the “continue to run older versions of OH and federate” approach only serves to buy us a little bit of time.
I stated this already in another discussion, but as this is on the table here as well: Two instances of OH (one OH2, one OH3) on a single Raspberry Pi is probably not a good idea (only 1GB RAM). As this is a very popular platform, this should be considered…
Please don’t propose to run two Raspberries to resolve this: Double the rules programming and maintenance effort, double the energy costs.
But many users would rather live with the inconsistency then to lose access entirely.