The Future of openHAB

Announcements
1

All, I would like to discuss a topic with you that I feel is vitally important for the future of openHAB.
This project has grown massively over the past 5 years since its first release and we have a great community with a steadily increasing number of contributions - this is really awesome and something I would have never thought of being possible!

The downside is though, that the current organisational setup does not keep up with the evolution of the project. You might have wondered when the openHAB 1.8 release will be available, since half a year has passed since the 1.7 release. The blatant truth is: We cannot tell. You might have noticed that there are currently about 70 open pull requests for openHAB 1.x, many unprocessed since many months. Most, if not all, of these pull requests should imho be merged for the 1.8 release.

Not finding the time to review and merge pull requests is distressing for us, but it must be awfully disappointing for any contributor who spent lots of efforts on creating the pull request, just to see it lingering there. From my point of view, this is an unacceptable situation that needs to be addressed asap.

The reason for this situation is pretty obvious: With mainly @teichsta and myself as maintainers on openHAB 1 and openHAB 2 respectively, we have simply reached a very unhealthy unbalance between contributors and maintainers. Although there are others helping a lot on the pull requests as well (like @chris on Z-Wave, @watou on Nest, Ecobee, etc., @theo_weiss on Debian, Tinkerforge etc.), I think that simply adding a few futher people won’t help us solving the problem - the only way to go is to change something about the processes.

I have a few ideas in mind that are not yet carved in stone, but which I would like to share with you in order to get feedback and have a discussion:

  1. New “Integration Rank” for Add-ons: Add-ons are optional parts and some are only interesting to a very small number of users. Not all of them might require a detailed code review and a place in the main repository. For openHAB 2, we already now have different places where add-ons sources are located: Some are in the Eclipse SmartHome repo, others in openHAB 2 and all the rest in openHAB 1. I would therefore like to completely split the (future) openHAB 2 runtime from the add-ons and with this introduce different repositories for the add-ons, like for “fully-reviewed and actively maintained” (rank 1) ones (like currently) and one for “not-in-depth-reviewed, but successfully tested” (rank 2) ones. All others (i.e. “not-reviewed and not tested” (rank 3)) could still be listed by referencing external repositories, so that at least they can be found.

  2. Transforming contributors to maintainers: Having separate repos makes it easier to deal with access rights. In any case, I would want to see every contributor of a rank 1 add-on to be an active maintainer of his code and thus to also have write access to the repo, so that PRs and issues can be dealt with directly. Having undergone a full code review themselves, I could also imagine a kind of round-robin assignment for the “not-to-be-reviewed-in-depth” PRs of rank 2; this should mainly cover checking the general guidelines and making sure that the formalities (like license compatibility and signed off and squashed commits) are fulfilled. All in all, the goal is to have any openHAB developer helping out a bit on the reviewing tasks as well.

  3. openHAB for the community: You might know that we currently have the “openHAB UG” company as a legal entity behind the openHAB project as a place to host and support the project - this company is run by myself, @teichsta and @belovictor. In order to reflect the fact that openHAB is effectively run by its community, I would like to give everybody the chance to better engage himself (officially) in this organization and thus scale it easily. As this is not possible with a regular company as the openHAB UG, I suggested to replace it by a non-profit (charity) organisation (for the Germans here: a “gemeinnütziger e.V.”) instead, where the board is chosen by the members and anybody is free to join the organisation. This should help us scale more easily and set a good legal frame for all operations (like having write access to repos, becoming a moderator or admin on the forum etc.)

Would you agree with those points? Do you have any other good ideas on measures that should be taken?

All those changes will mean quite some effort to get in place. But since we planned the openHAB 1.8 runtime to be the final version (and only continue with 1.9 for the add-ons) and openHAB 2 runtime becoming the mainstream version afterwards, this might be a good moment to go through these changes. It also will have a huge impact on the structure for documentation, the wiki, etc., so we very much depend on your support here - so please step up and help making these things happens!

Best regards,
Kai

22 Likes
3

I understand the problem you and Thomas are having and this sounds like some good steps in the right direction. Is there specific support you need immediately from the community?

1 Like
4

Hi @Kai, @teichsta and @belovictor,

first of all again MANY thanks for all efforts.

How about as well initiating a fund raising/ donating activities - as well to cover the costs for servers, warehousing, … that those costs are covered.

Cheers
Karsten

5

@Kai, I like thinking about changing how addons are incorporated and reviewed, the current model of reviewing everything that same way is not going to scale as more and more addon are created. I understand the consequence could be lower quality code, but your suggestions of “Ranks” would help mitigate that. I see this moving to more how communities like node handle plugins, but not as chaotic as those for the core/popular bindings. How do you see the structure for the repos for these ranks? Would there be a Rank 1, Rank 2 and Rank 3 repo, with contributors still merging into a shared repo? Or would addons exist in their own repos and somehow be linked into these? In any case I like moving to a more community oriented process only because you have to sleep now and again :wink:

6

I think the most pressing issue right now is to find a solution to get 1.8 out of the door. So if you find an unprocessed PR in this list that you think you are capable of reviewing wrt to the general architecture and the coding guidelines, please let us know and the we can assign the PR to you - if we find a few people to help on this (as a one time effort; you won’t be obliged for eternity ;-)) it would be gorgeous!

Thanks for bringing this up, Karsten. Although more donations would be nice, money is not our major concern (we are fine to cover the costs mostly privately for now). Once we have the charity organisation established, I will come back to you asking for donations - as it will then be much more attractive to donate (at least for German users), when it can be deducted from the taxes :smile:

I am not yet clear on the best structure. Having one repo per add-on is definitely too much, so combining them by rank is probably most pragmatic. Rank 3 should actually not even be a shared repository, but rather just a list pointing to other repos (e.g. forks), where people host and maintain code themselves.

7

I wouldn’t say that -You get a lot more flexibility by assigning a repo per add-on. That way you can assign write permissions to individual add-ons as required.

Example: https://github.com/puppetlabs

1 Like
8

I am mainly afraid of the maintenance overhead that would then mean again. Administrating them, setting up build plans, having different issue trackers, etc. Might not be that easy.

9

I’m really happy with these proposals. I felt a bit of beeing stuck since some monthes as some PR stood under review process for a while (and I know that it’s for a missing time reason).
Extending the current ESH/OH2 split in validation process to binding would really be great. I think that with a lighter review on Ring3, we’ll see new :
1°/ even more new binfings
2°/ some emerging and come to Ring2 and then one.

I’ll be more than happy to give a hand to the core team also.

10

I suspect this is a huge stressor to the team.

Is the thought to start over from scratch for OH 2’s documentation/wiki and leave in place the OH 1 wiki, replace OH 1’s wiki with OH 2’s, or something inbetween?

I see massive disruption and problems with any approach I can think of but if we can get a jump start on generating some docs (I can certainly help out).

Are there any parts of OH 2 that seem stable enough to start building out a skeleton of the documentation (I’m sure the answer is yes) so when OH 2 does reach Beta stage or is fully release the documentation (in some form) can be there? Even if we can identify some things that are not changing or changing minimally (e.g. the existing rules engine being the default so we can beef up and reuse a lot of what has already been written) we can at least get started porting the docs.

Will the general installation/getting started for OH 2 be roughly the same or will installation be significantly different?

If the OH 2 docs will be separate from OH 1’s, has a space to store and start work on them been identified?

Just a few questions that I think if we can get settled I and others can start trying to get some of the documentation started.

Thanks!

Rich

11

I like the idea of having different add-ons sources (ranks), but I would split out current list to 2 parts only:

  • most important add-ons for industry standards within OH repo (like your rank1), examples: http, ntp, tcp, serial, mqtt, zwave, knx, etc
  • a kind of marketplace for 3rd party add-ons where everybody can register its own repo or maybe even just a jar; this means no reviews, testing etc, just a feedback like comments, likes or whatever else to provide high quality of this bundle (something like your rank 3)
1 Like
12

It may be more difficult to separate out the addon ranks if the qualities of each rank aren’t first articulated very clearly up front. How many users use a particular binding is basically impossible to know currently, and after much effort it would likely still only be a rough estimate, differ by geography, and constantly change over time. I don’t have answers to my own questions at the moment, but I suggest the ranks are divided in such a way that it makes simple sense to users and developers alike, and is unaffected by time and geography, so that as little time as possible is spent deciding which repository it will forever be contained in.

1 Like
13

I think I prefer to keep the existing OH1 wiki in place as it is and start with OH2 more or less from scratch. I’d also like to split it into developer documentation and user documentation. The developer doc might be done as a wiki, while I would like to version the user doc together with the releases (so that you can still find the docs for your version, if you are not upgrading regularly). The docs should also directly incorporate stuff from ESH, so that this does not need to be duplicated - we generate HTML from Markdown there: https://www.eclipse.org/smarthome/documentation/index.html. I’d like to see something similar for the user docs of OH2.

Before starting with the docs, I want to wait for Use Apache Karaf as a runtime container · Issue #194 · openhab/openhab-addons · GitHub to be finalized as this will have some impact on the overall usage of openHAB (of course only to the better!). But once this is there (hopefully by the end of this month), we should jointly start going for the documentation.

I think having add-ons properly contributed to the project as source code under the EPL is an asset that you should not underestimate (and which you give up if you omit rank 2) - having the source code allows others to take over the maintenance of an add-on or to build their own stuff on top of it. Also it allows re-distribution, e.g. if you want to install openHAB for others (possibly even commercially). Having the source code also avoids getting malicious code - if you only have a jar, god knows what it is doing internally… That’s why I think the marketplace is only the “last resort”.

I agree that the rules should be clearly defined, but imho a decision does not have to be final. An add-on that gets much interest from the community can be properly reviewed and added to rank 1, while outdated add-ons that might have lost their maintainers could also be thrown out again there. We might lose the commit history in such cases, but this can be acceptable.

14

Hi,
I really would like to see the transition to “gemeinnütziger e.V.” for the reason you mentioned.

Also, clustering AddOn repository seems a very good idea. Maybe a fourth rank for ad dons which are currently still ins the developement-phase might be usefull. People could look there and some might be willing to do some testing and contribute to beta versions. This could be only a linklist or something like that but there would be the possibility to see which add-ons are getting developed.
So based on the discussion we might have

  • “fully-reviewed and actively maintained”
  • “fully-reviewed and not actively maintained” //Maybe this could be a tag to the repostory
  • “not-in-depth-reviewed, but successfully tested”
  • “not-reviewed and not tested”
  • and maybe “development phase”

A marketplace would be a concept I dislike very much. The source code has to be accessable and a simple browsable repository also eliminates the need for a marketplace.

15

I dont’t think that clustering is a good idea. I would create a repository for each addon. Why should not all addons have the same rank/priority. Who decides which plugin is used often and wich not. But nevertheless it’s a very good idea to seperate the addons and to let the owner of the addon all the work.

16

I really don’t like this kind of separation either, you will end up with all sort of group names to differentiate the add on by category of maintenance where as, as a user, I don’t really want to search in 5 different repo to find the addon I’m looking for.

It might be a little overhead, but one repo per addon makes it more clean, you can also give all management power to the owner of this addon in the repo and then users can just search at the base of the org to find all the available addons.

Now imagine a new addon is made and at first is not actively maintained but a new user comes in, has time and will to put effort in this module and now is becoming the number 1 maintained addon, will you move it of repo ? Same question for the other way around, a highly maintained addin is being less maintained after the original owner doesn’t have time or will to maintain it anymore, will you push it down to “lower priority” repo ? If you move them users will keep searching in the diff repo for the module they once knew was in a repo but is not there anymore …

17

A separation of repos could by overcome by the application layer. Puppet was already mentioned above. Those guys have created PuppetForge. Within the application you can search for modules and install them with a snap.
Every module developer can register its module on puppet forge. Once done its searchable & installable via the application.

The so called ranks seem to me like tagging which is just additional information for the enduser to know if the addon is used frequently or not. Tags can be changed easily, so “down ranking” an outdated module should be very easy.

Third, when doing installation of addons via the application it should be fairly easy to track at least the “number of downloads” to calculate a rank or popularity. Or if an enduser is willing to offer such information from his/her system also the status of an addon could be reported (numer of items/things created by an addon within the installation, etc).

I also like the idea of migrating into “gemeinnütziger Verein”. As a particiapant I went through that with another open source project and it went quite well.

2 Likes
18

I also see @Kai’s points and I would love to see openhab move to a “e.V.” That would make donations a lot easier and everybody can have an impact on the organizational structure.

Through the discussion I see a lot of “technology related” points. IMHO we should first decide on the core principles of the future organisation/architecture and then look after the technology that would solve the problem.
The areas I see:
"openHAB": Should have a stable core that is maintained by a few experienced people who manage release cycles etc. It should be extensible via “Add-ons” that can and should have different level of maturity depending on e.g. standardization in the industry, number of possible users, contribution possibilities of the author. In the end this will perhaps result in a kind of “attribution” or perhaps a ranking. [quote=“Spaceman_Spiff, post:14, topic:4048”]
“fully-reviewed and actively maintained”
“fully-reviewed and not actively maintained” //Maybe this could be a tag to the repostory
“not-in-depth-reviewed, but successfully tested”
“not-reviewed and not tested”
and maybe “development phase”
[/quote]
+1 @Spaceman_Spiff

Contributers/Maintainers: Everybody should have the possibility to contribute easily (as it is today). That means people who have developed a certain solution should be able to make it public and let others use it. The code should be open source of course for the reasons already being mentioned (security, enhancements,…)

Users: An end-user should be able to easily find and get that “add-on” and add it to his system. Call it a “marketplace” (why not?) And he should also see the utilization of that add-on to decide what he can expect from it.

Just my two cents,
Michael

19

Honestly,I beg to disagree on this point.

One repo per addon works excellently for jenkins and puppet (which was mentioned here many times). While setting up the build jobs might sound like lots of work, I would offload this to a robot / shell script. If you decide to go this way, I would offer my help in setting up the necessary infrastructure as this is what I am doing all the time in my day job.

My main argument in favour of seperated repos (which would then be part of the “openhab” organization on github) is that you can leave the resposibility for issue tracking and defining release schedules to the maintainer of each addon. And, if and when one maintainer loses interest, another can jump in and continue work.

Code quality could be defined via some kind of reputation metric. Again, this works just excellently for Jenkins and we should steal (oops, get inspiration from) the best :wink:

1 Like
20

Thanks for all your input. Ok, so I might have @hakan prove me that this many repos can be handled through clever automation - if this really works well, I agree that it might be a good option after all.

Nonetheless, I think we still will have different ranks/levels of integration of the bindings. If anybody is free to open a new repo under the openHAB Github org and just put any code in there, this simply won’t work out. So there has to be some governance and this usually means manual effort. And believe me: Without our code reviews, the openHAB code base would be a real mess today and I would never have had the chance to build something like the openHAB 2 compatibility layer, which relies on the fact that some principal architectural rules have been followed throughout the code. And this is something that you cannot check automatically, you really need a peer code review.

So this brings me back to why I originally started this topic: Every change we discuss now is merely a workaround for the fact that we have not enough maintainers, who are willing and capable of doing code reviews. Ideally, all code that ends up in one of the openHAB org git repositories should be clean and reviewed. Everything else is imho a compromise.

So to reach our most pressing goal, the 1.8 release: Are there any volunteers for reviewing open PRs that currently exist? I would prefer to only start with bigger changes once this is out of the door.

1 Like