Increase openness and competition on components developed outside the official source code repositories

Dear Developer,

today I want to discuss the topic of development of third party solutions outside the official openHAB repositories.

In my opinion a life and vivid platform should encourage the participation without strong restrictions and with the intention of competition of ideas in mind.

Regarding the first point, I see currently far to many restrictions for collaborating on or with openHAB. Let’s take a practical example on the addons. The main line is here that for each solution/technology there can be only one “official” addon in the openHAB repository. For taking other or competing solutions, this leaves you only with the option to develop an addon outside the openHAB source code.

To do this there is no easy way because the whole development is based on a complicated monolithic maven reactor build. If you go the suggested way of cloning the original repository you are left with a monstrous build just to develop a simple addon. Yes you can tweak the command line to only build your artifact but why should this be necessary?

The second solution is to build your own maven POM with lots of work and twists and tweaks, until you can manage to bring this to work.

Both solutions force you to subdue to many regulations like formatting rules, code rule checks, licenses, etc. In my opinion this should be not a requirement for a third party solution. You should be free to set your own set of quality rules. And it should be easy to declare your dependencies and just build your artifact.

So my intention here is to promote a way that ease the way of producing openHAB artifacts as third party components.

I for myself came across all of the mentioned problems when I decided to start an alternative addon for the well known KNX binding.

Kind regards,

Frank

If you want to have it listed as an openHAB addon under openHAB namespace, then there is no other way.
But you can check how other third party solutions did it. We already have a third party marketplace….

2 Likes

If you just want to have a single binding repository, you can remove most of the root-bundles (i.e. delete bom, features, itests and remove those from the POM) and all bundles in bundles. Keep the pom.xml, just remove the bundles from the modules section. Add your own bundle instead. If you want to change the groupId of your bundle (which I believe is necessary if you want to distribute a non-official binding), you might need to adjust the BND configuration in the root POM.

1 Like

I understand in part your irritation cause openHAB BOMs are useless for externalized development. They rely on maven dependencies and not dependency management. When I raised this issue in 2019 answer was that it is done this way cause it is easier to maintain 200 addons living in one repository.

With this small change it would be possible to have a baselines dependencies which could be easily imported into another pom. Yet, seems OH as a project is fine without external addon developers or project management is not interested in making their life easier for some reason.

My very subjective observation which I am not afraid to reflect - overall the free thought and fight of ideas doesn’t happen much either in github or forums as core project members/architecture court seems to be busy keeping status quo.
Why is that, that’s another topic.

Cheers,
Łukasz

2 Likes

As I already said, this is a cumbersome and error prone way. Not the best start if you want to promote third party components.

I’ve done this and it has cost me several days until I found out through the sparse information on this forum and other source code repositories on GitHub. And I guess that this can break any time because it is obviously not the intended way to do such builds.

I already did this and after lots of work on the POM, managing dependencies, configuring the OSGI parts and copy plugin configuration from three levels of aggregator inheritance, it finally produces an artifact that I can use in my openHAB installation.

It is just beginning. Now, given that you know the problem and you know what been missing you could in theory bring some changes through PR into OH core build by sacrificing even more time.
But what kind of confidence you have that it will be taken with care with by any core of maintainers who are busy with other things? Or that it will not be objected as a breaking change affecting addons build? OH had in the past people upset for their PRs stuck for a year or even years. I get that everybody’s time have a price, but holding others for such long is unjustified and reminds sometimes a “gardener’s dog” (blocks things but don’t make any use of them).
Clearly there is no guidance and policy for that in this project.

2 Likes

I feel rather bad answering my own answer, but according to multiple pointers - forum is not best place to discuss development issues. In this post, I am going over Governance | openHAB. Anyhow, maybe later this post could be moved to github and discussed by maintainers or AC itself if it will find it necessary.
I get that defining rules for open source project is a difficult task cause there is a little of people who can afford paying bureaucracy cost by sacrificing their own time.

Starting from its principles - Transparency, Openness and Meritocracy. One of first links which are mentioned is Sign in to GitHub · GitHub, a list of maintainers working on project which is invisible for anyone who is not a maintainer. On opposite side I can mention Apache Karaf project (used by openHAB) where all nominated committers, their names are publicly listed: Apache Karaf commiteers. I get that contributors are listed in each repository, but why not listing them in one place?
Another comparison to the transparency and oneness of Apache projects - in Apache projects there are usually three mailing lists (forums) - user, dev and private. First two are public while private is kept secret (for project members and apache members) to keep primarily two things - a committer elections/voting and security issues handling until its confirmed and fixed. In practice all major development decisions must be taken in a public discussion (at dev@myproject.apache.org). Agreement shall be recorded so anyone can confirm it later on. Not sure how the github teams translate to that and what is the role of core and adons maintainers teams.
For comparison, the maintainer teams rules which apply to all github teams (maintainers), hence contribution criteria, are invisible to external people.
The governance page states that:

Nominations must be done on the GitHub team discussion page (as a public post) and voting is done through reactions on that post.

Which seems to not be valid since these pages are invisible for external members unless github team is a private forum for project maintainers (just like private@apache). Its unclear to me. I suppose that documentation could be improved in that matter to answer why github teams for core/addon etc. repositories are invisible for outside collaborators. If it was answered in forums or github it still doesn’t count as governance policy shall be clear and available in one place.

Going to next point - the Architecture Council - as an external contributor I don’t really see if its acting whereas (for comparison) I can go to any Apache project and see what they do in their dev@ mailing lists, what they plan or discuss to improve their project. More over, even not being an apache committer I can bring there a topic to discuss without creating an issue in the tracker.
Again, I don’t think its feasible to expect anyone to sacrifice tremendous amounts of time to keep answering any single request coming to this body, but there is no way for external contributors to ask for AC attention. Usual way with github issues belongs to maintainers, hence problem reflected in this topic and brought by @franks are invisible for AC as an institution.

Going now to openHAB Architecture Council responsibilities:

The openHAB Architecture Council (AC) serves the community by identifying and tackling any issues that hinder openHAB’s continued technological success and innovation, widespread adoption, and future growth. This involves technical architecture as well as open source processes and social aspects. Comprising the finest technical leaders from all community stake holders, it is the council’s goal to keep the project successful and healthy, the processes simple and smooth, and the communities vibrant and cohesive.

How I, as an end user or supplier or external contributor can ensure that AC is tackling for a technological success and innovation? By going over github issues the AC members are taking care of? I could, but the AC members are not indexed.

Another point on AC, again comparing to Apache projects, is that this seems to be fairly limited body with no space for more people coming into it over the time. I think that current members were established somewhat when ESH migration been done and there was no single new member, whereas in Apache this is kind of metric if project is attractive for people to join effort and take responsibility about it. With just 5 people in the AC there is in practice less than 1 person for each main OH repository (core, addons, ui, ios, android, cloud). How governance page can talk about a meritocracy, if there is a fairly limited pool of people to decide, who can’t be removed (they have to remove themselves), and no way for other people who’s involvement can superseded existing members to join? To me this sounds more similar to oligarchy. Not sure how many project maintainers are there, someone who has access to all these pages can simply do math.
I did small comparison based on Apache and other projects and ohloh.net stats:

Project Started Commits Lines of code Contributors Committers Managers
openHAB 2010 13616 1634313 820 ? 5
Apache Arrow 2014 11317 926665 846 72 39
Apache Karaf 2010 8570 164800 208 31 17
Apache ActiveMQ 2005 11132 479796 161 63 28
ZeroMQ - C4 process 2009 8351 51485 604 ? ?
ioBroker - unknown process 2013 51632 1689350 559 ? ?
home assistant - unknown process 2001 156692 1372354 7128 ? ?
Eclipse SmartHome 2014 5198 295562 223 ? ?
Jersey 2007 4164 558174 112 ? ?
fhem 2007 25876 826347 250 ? ?

I don’t have any strict measures here, I just try to compare scale of openHAB project to others. I don’t have any data for other smarthome related projects other than lines of code and contributors, cause I don’t follow them and their process. What’s notable - ActiveMQ and Karaf are mostly written in Java, just like openHAB.
Despite of bigger codebase than ie. Karaf, openHAB have much smaller set of people who can vote on adding new members to “AC” who can decide on project future and drive its development. From scale perspective I put for comparison Eclispe SmartHome (RIP) and Jersey, where second one was utilized by OH < 3.0 to render all web services. Jersey is one of quite popular frameworks in Java lands and its still much smaller than openHAB. :slight_smile:
What is visible, current AC does not reflect the structure of contributions and does not give any weight to these who are committing most of the time and work to the project. More over, the form AC have been made does not leave space for that. I get that there are just 5 seats, but this limit is completely artificial.

Discussions and decisions are done on the Github team page. Decisions are taken unanimously. This is the case if no member of the AC vetos against a proposed resolution within a week of the call for votes on the topic.

Can we, as part of openess learn how many discussions AC had, how many of them got resolved unanimously, how many ended up with veto?

Overall I think that current way AC works is outdated. It does not reflect scale of the project, it does not reflect contributions to the project any more, just an earned flag. Looking at evolution of Architecture Council (or its lack) over the time, I think it might be a valid time to open a wider discussion if its form is still serving the project. AC is primary body to lead improvements of the project, so maybe its time for it to think how it can serve it better?
Looking at 12 month stats rendered by ohloh it seems that even 5 seat AC could be reshaped to reflect project activity. Yet, if we take openHAB core and each individual repository the picture will be completely different.

Cheers,
Łukasz

8 Likes

I don’t think it is out of place for me to answer that question.

There have been a grand total of 13 posts, one of which was a test.

5 had votes or were otherwise implement without a vote being necessary.

2 I would say are currently still open.

Of these 13, 8 were created by members of the AC itself. The rest were created by maintainers of various repos and, with only a couple of exceptions, the ones raised by members of the AC were raised in that member’s role as a maintainer of a repo with an issue that concerns that repo and others (e.g. a big change to the UI that also impacts core or the distro).

The rest never had a call for a vote since consensus was not reached (note that one of the open issues is how to manage these unvoted upon threads).

The first post was in February 2019. There was a flurry of opened issues in 2019 and since then we’ve had on average 2-3 posts per year. The types of topics that came up are things like removal of the 1.x compatibility layer, merging the UIs into MainUI as much as feasible, and stuff like that. The two posts from this year have to do with how to manage conflicts between maintainers and non-maintainers and the already mentioned how to manage issues that never get a vote.

Over the first year of the AC’s existence I think its role was refined slightly. I remember several posts from Kai, probably in the now no longer existing Architecture Council category on the forum before the GitHub Teams was set up but they were along the lines of

https://community.openhab.org/t/the-road-ahead-reintegrating-esh/64670/57

And later discussion that kind of rolled back the role of the AC to be more of a place to adjudicate conflicts and less of a directing force since, for the most part, the AC doesn’t have the power to direct anyone to do anything anyway. We can’t force anyone to implement something in a certain way. You all are not employees.

Based on the quote from above I don’t know that that governance document accurately reflects what the AC has been use for though given this limitation and refinement.

My understanding is that any member of the openHAB GitHub team can see, open, and contribute to the AC team’s thread. I also remember seeing a mention that anyone named a contributor (not just maintainers) will be added to the openHAB team. Has that not been happening? Maybe I’m wrong and only maintainers can see it? There are definitely non AC members who open threads and contribute to the discussion.

As for the rest I don’t have much of anything to say. While I’m on the AC, my role is to provide the user’s perspective for impacts caused by proposed changes. I’m not a maintainer nor am I an add-on developer so I don’t really have anything to say on these types of issues. But I can at least answer some of these questions.

6 Likes

I must say that until recently I had assumed that the AC discussions (as limited as they are) were visible to everyone, and not just the select few. I really think that the points @splatch makes about openness are good points - we should welcome everyones opinion - not just the select few.

IMHO it would be useful if the AC had a broader member base, and it was a majority rule. With a broader membership, and possibly removal of veto (or only having it in very special cases) we could have a more open, and more progressive development.

The description of the AC on Github states -:

The purpose of the AC is to be an authority to appeal to when there is an impasse among the developers and maintainers of the various parts of OH or to help maintainers from different parts of OH to come to a decision on how to address a cross cutting change.

Maybe AC should be renamed to the “Complaints Department” :slight_smile: . Seriously though, I’m not sure that this description is consistent with the system “Architecture”?

I think that maybe the reason that the AC forum is not getting much use is that, given that the any “impasse” between “different parts of OH” is almost certain to include the core maintainers, and the core maintainers sit on the AC, there is no point to bring issues to the AC if they are already rejected by core maintainers.

2 Likes

That is actually being discussed in one of the open issues. I’m very sensitive to conflicts of interest and it doesn’t feel right for someone to be a party to the issue and judge at the same time. There should be some sort of recusal and code of ethics in place.

3 Likes

Thanks Rich. I had another read through that discussion in the AC and see you raised almost exactly the same point :+1: . I guess my only comment here is maybe “being discussed” might be a little optimistic since this discussion seems to have finished at the beginning of February :wink: .

I guess maybe we’ve hijacked the original post a little here which was more focussed on binding development outside of the OH addons repository. However, I’ve personally been involved with OH for around 10 years now, and it does feel like things have plateaued in some areas, and there needs to be some change to facilitate further development and innovation. I think some open discussion on this is healthy :slight_smile:

6 Likes

I agree with most of what @splatch and @chris said. The “unanimous vote” rule essentially grants something similar to a BDFL status to the AC members as they can block every decision. This is even more true if two out of three core maintainers are members of the AC, because openhab-core is where the most important decisions are made. I would prefer the AC to make decisions based on a majority rule and have all openhab-maintainers as a member. To keep it manageable, all issues brought to the AC should have a two-week discussion period and a one or two-week vote period afterwards.

I also think that the group of maintainers (especially in core, but also in other repositories) needs to be larger. There are 26 open PR in core and 92 in addons of which some have not even received a single comment since several month (like Add osgi support for ThingHandlerService implementations. by cpmeister · Pull Request #1941 · openhab/openhab-core · GitHub or more recent [dbquery] JDBC database bridge by lujop · Pull Request #11428 · openhab/openhab-addons · GitHub). Obviously the amount of work cannot be handled by the current members of the respective groups (no offense intended, everyone’s time is limited and we are all volunteers) .

5 Likes

Thanks Chris for coming back to my original topic. And your are right. I mainly want to focus on the development and governance of external bindings. I only can repeat that IMHO there should be much less governance and more support for supporting external development.

Let’s take the example on the two best known digital assistants, Siri and Alexa, and how they address their openness and acceptance in terms of governance and support for development.

Whereas Alexa provides the cloud infrastructure and stable APIs and SDKs with nearly no restrictions on the addons (as known as skills), Siri is a closed system fully under control of Apple.

The differences are quite remarkable. Siri as a closed and governed system lacks innovation velocity and solution broadness especially if cooperation with non Apple development is needed. Alexa on the other hand is thriving on solutions and integrating areas of functionality in such way that even Amazon would not have guessed.

To take a strong position on my case: for me openHAB is much more like a Siri than an Alexa when it comes to supporting external functions to complement the openHAB plattform.

From my point of view there should be a distinction between Core and Addons. For the first part, I think that the current governance rules are needed to keep the openHAB platform sound and safe.

For the addons I would like to see a much more open approach. For the first steps, I would like to see the openHAB bill of materials (BOMs) brought to a state that you can actually use them as a addon developer outside the maven addon reactor project.

I really like the new market place. I think there should be less official openhab addons and utilize the marketplace more.

Regarding building an addon independent of openhab take a look at jrule GitHub - seaside1/jrule: OpenHAB Java Rule Engine
It can be built standalone or inside the big openhab-addons project.

Pr for the change: Fix Windows support by LumnitzF · Pull Request #24 · seaside1/jrule · GitHub

Thing is, you inherit a whole bunch of settings (bnd, compiler, header files, static analysis, null annotations) and plugins which might be (or are) irrelevant for your thing. I am referring to this point: jrule/pom.xml at main · seaside1/jrule · GitHub.

The directory plugin fix you mentioned is there to manage paths for some of the checks. So PR you mentioned really solves a problem which comes from how openhab addons pom is structured. It is really not suitable for external development and that’s what Frank is rising up. What he/we need is a BOM (bill of materials pom) which says that openHAB core 3.2 consists of this and that artifacts (config, discovery, thing, item etc), and 3.3 will ship this and maybe some more (think of ephemeris module which was added to core). Maybe some additional BOM with external dependencies would help too. After all these are pulled and managed in by core build itself. That’s all.

Instead of arguing, why not come up with a PR to suggest a solution without breaking existing addon builds.

1 Like

Not sure what you mean by arguing, so far most of above feedback indicates that there are gaps primarily in policies which hold on multiple areas - especially build, code and documentation.

If you ask me why not coming with PR:

  1. Because it is not clear if this change would be accepted in first place. It requires some touch in OH core build which I do not maintain nor have a badge. I don’t think anyone is willing to wait (especially me) 2 months until it will be reviewed and rejected because of typo or copyright date.
  2. I brought very similar topic in this forums earlier and it was left without resolution: Maven BOMs without dependencyManagement, not picked, guided or lead by any of core maintainers.
  3. There is split brain problem in handling external addons. As far I remember some of addons were pulled down from marketplace as cause they clashed with official ones, there is also no clear statement that forum is fine to discuss issues related to them.
  4. Investing time in preparing code changes without a clear acceptance policy, as it could always be vetoed by single maintainer or AC member for any reason, is bigger waste of time than complaining on how it works. Until there is a clear, objective and publicly written definition of acceptance criteria, what is welcome and what’s not for OH core, I don’t think there is a point in committing any time into OH core.
  5. I made this work already for my own addons, why I would waste time on this again given above points?
5 Likes

Neither do I. But nevertheless I have made PR’s to core, and have had them succesfully merged. So why not just try it. In my opinion, if you are honest, precise, and polite, in explaining your intentions, then maintainers are willing to support it. In my opinion also, making multiple smaller changes, is easier than making elephants.

4 Likes

Absolutely - and so have many others. The problem is that there are also many instances where PRs will sit for months or years without even a comment. I have personally had a number of such PRs and issues, and this means that people don’t want to waste their time submitting PRs that will not even be considered.

This is highlighted above by @J-N-K - I note that one of the PRs he linked was raised in December 2020 and has had no comment. I think this is the sort of issue that needs to be resolved - it’s not meant to criticise individuals who are volunteers, but more to motivate and empower people who want to contribute.

6 Likes