Restructuring of repositories / types of add-ons

Organisation Misc
9

How about using an OBR repository dedicated to openHAB? This could be used to store the different binary versions of add-ons.
We could add a market place layer on top of that with some publishing governance built in. Then all we would need is an add-on manager bundle which allows you to browse and install from the market place.

@Kai, if you have no objections, I’m happy to start hacking away at a solution for this over the next few months. It will be a good way for me to get started again with openHAB development.

For the repos, one option would be to split it into 3 types:

  • openhab core : the minimum runtime required
  • openhab add-ons : certified add-ons
  • standalone add-ons : addons under active development with their own lifecycle. These could move to openhab addons when their maintainer looses interest and wants to find a good home for the addon…
10

II really like the idea of @davy and @marcel_verpaalen.

I’m already dreaming of what would be the perfect solution from user point of view (german: Goldrandlösung ;-))…

As a user I could download and install addons from a central repository. I will also get notified if there is an update available for my addons, they are not directly connected to the SW-lifecycle of the OH-Core. I can select, per addon, which level of addon I want to run (released, beta, development, …) which can be changed at any time. With this I’m able to get an earlier version of certain addons when there are problems with the current version (can’t get worse with not finally tested new version) or the maintainer added cool new features I really want to have. Even as developer you can select “development” state for your addon and will always have your newest version via official distribution channel.

If then there is some kind of mechanism that prevents the addons to crash the whole system as they are running in a kind of sandbox…

… as I’ve said: just dreaming…

Dieter

1 Like
11

Guys, it is ok to start dreaming and I also have millions of ideas what would be cool for users and developers.
But please note that I started this thread for a specific reason: There are two few people doing the overall coordination vs. too many people doing contributions (which need to be coordinated to some extend).

For me as a maintainer of the runtime, it is VERY valuable to be able to have a workspace with all addons in it, so that on refactorings, things can be tested to some extend and “bulk” actions can be performed (like creating a release, applying code formatting or whatever).

From an individual contributor perspective, it is of course nicer and more flexible to have a single small repo - but this means that these are indeed 3rd party “addons” and not a part of the openHAB project itself anymore (since nobody will be track the governance, IP & license issues etc.

@davy, it is great to have you back on board!
If you are interested in improving the user experience here, I would suggest you join me and @maggu2810 on Use Apache Karaf as a runtime container · Issue #194 · openhab/openhab-addons · GitHub. We have actually implemented a lot already there; add-ons can be located in any Maven Repo and the new runtime comes with a UI where users can easily install/uninstall the add-ons.
Note that this is already the status quo of openHAB 2: The add-ons are already located in 3 different source repositories (ESH, OH1 and OH2), but for the openHAB 2 distribution they are all assembled, so that the user does not have to care. With the openHAB 1.8 release we also want to remove the runtime and designer code from the openHAB 1 repository, so that only the add-ons are left in there. For openHAB 2, I also plan to split the add-ons from the runtime. For add-on developers it will therefore in future be pretty easy to work on their code, because all they need in their workspace is their own bundle, nothing else. But again: To do all these changes is a lot of work that I am mainly doing alone - I would wish to have more people willing to help on such things!

12

So what is going to happen to the non-binding code changes that have been made since 1.7.1? In what ways will the 1.8 release differ in nature from prior releases? Your comment gives rise to worry of wasted effort, so please clarify and thank you.

13

Sorry if this was misleading: What I meant was that the 1.8 runtime will be the last one - and it will of course include all changes that have been done since 1.7.1! AFTER this release, the sources will be removed so that there isn’t any 1.9.0-SNAPSHOT runtime. Worries gone now?

14

Yes; thanks for clarifying. Makes perfect sense, except in the case where there would be strong arguments for a 1.8.1 maintenance release, but I suspect you’ve thought of that already. But knowing that there will be no 1.9 runtime is good!

Onward! :smile:

15

This would be branched off the 1.8 release branch, so the runtime would be in it and you can happily provide hotfixes for it :slight_smile:

1 Like
16

Except that it doesn’t scale. Imagine having 600 openHAB add-ons :wink:
I believe most of the things you mention can still be done by just importing different repos into one workspace. And some of the others, are probably things you shouldn’t spend time on. But I can understand the need for a good working environment and it makes sense to keep things simple.

Personally, I prefer an approach where a contributor would not even have to clone a repo. In my dreams, a contributor would be able download a single esh.binding.api.jar and implement a binding any way they want. Keeping all connections between the runtime and the bindings limited to a single api spec will make it easier to maintain both the binding and the runtime.
But there does need to be some governance on add-ons. I don’t think it needs to be a ‘hard’ governance, where bindings don’t get included in the release because they are not approved yet. I would much prefer a system of governance through ratings. Ratings like ‘official’, ‘certified’, ‘experimental’, etc should be enough. You could have a system that if a bindings gets too many defects, its rating is lowered automatically to ‘buggy’. That may motivate developers to resolve bugs. I also believe this ‘governance’ should be more limited than today. What is really important is that the code doesn’t pose a thread to application stability and that it’s reasonably documented. But does the code for the different bindings all need to use the same code formatter? For me, that would be a choice that can be left up to the developer of the binding as long as the binding is maintained outside of openHAB and with that, it’s one worry less for the reviewer… Of course, when bindings are donated back to openHAB, a more rigid review process could still be used to make sure the IP is OK.

Great news. I’ll check it out.

Count me in.

17

I did not contribute to the discussion because I am not a professional software developer, and I have zero experience in provisioning systems and so forth, and thus prefer to shut up. However, I do crank out a great deal of source code at various points in ESH and OH, so I might throw in my thoughts, albeit not very structured :wink:

I would prefer a system whereby

  • there is one overall “add-on” repo, community-owned by all “usual suspects”, eg. more that @kai and thomas
  • every author of an add-on takes on the responsibility of self-reviewing the code, and resolve bugs
  • every author has the possibility to push commits to the repo, do tagging, follow-up things. I know this implies a minimal level of trust, but if things are broken, we can always fix them afterwards
  • add-ons can be added immediately to a repo, so that at least everyone (from user to developer) finds its way to the add-on, and can use it in a test/evaluation/production environment. I have about 10 or so bindings that are still in the pipe as a pull-request. There are running fine > 1 year in my own production environment, I fix the bugs on them, but no one else is using them, so they do not evolve, code quality does not evolve, and so forth. I think stuff needs to be shared early on, even if it is not finished
  • I think we should stay pragmatic, and stick with a simple ranking system as suggested by @davy

Karel

18

Thanks Karel,

This is probably a good moment to give you an update of what I now did as a preparation for the new openHAB 2 distro (https://github.com/openhab/openhab-distro):

  • We have a repo that contains ALL reviewed/approved/clean openHAB 1 bindings (openhab/openhab)
  • We have a repo that contains ALL reviewed/approved/clean openHAB 2 bindings (openhab/openhab2-addons)
  • We have an IDE setup, which in the minimal setup does not require any of these repos to be checked out - so you can potentially work on code that is located somewhere completely different.
  • Targetplatform and core bundles are all published to Maven, so that it should not be to hard to create new github projects for individual add-ons (@hakan’s suggestion), but I am not sure we should go this way (just want to mention that my preparations make this possible as well).
  • We have the distro project, which assembles the runtime with the add-ons from these sources. It is fairly easy to add further locations, either directly to the official distro or by the user himself. The add-ons do not even have to be available as source code, but only need to be deployed to some Maven repo.
  • I think it would be great to have a build server setup that is building all PRs and publishes such add-ons to a Maven repo. We could then have an new “experimental” Karaf feature, which people can activate on their runtime and which would make all such add-ons directly available.
1 Like
19

I have about 10 or so bindings that are still in the pipe as a pull-request.

That is indeed a problem. It’s definitely not nice if you have put in all the effort to create a binding and then it just ends up stuck in limbo for a long time.

Targetplatform and core bundles are all published to Maven, so that it should not be to hard to create new github projects for individual add-ons (@hakan’s suggestion), but I am not sure we should go this way

It will be unavoidable that bindings move away from the main openHAB repo if the review/integration process takes too long. We should find a solution in [this thread] (Growing the number of active maintainers).

20

This is definitely my preferred solution. I think for the openHAB 1 add-ons, we made already good progress here as we have a bunch of maintainers for them meanwhile. For ESH and OH2 I am still completely alone, which definitely has to change when even more developers are now moving over.
A big part of the long binding review queue is also due to the fact that my priorities were more on getting the ESH framework and APIs stable and usable than to invite people to write new bindings for OH2. I still do not consider the APIs to be final, so here is still the risk that stuff gets broken. That’s why I prefer any effort of our @maintainer’s and developers to be spend on making existing bindings work smoothly on OH2, before having to many people actively porting stuff over.

21

That’s a catch22. If no bindings are written, then the API’s will never become final :wink:
But I don’t think anyone blames you for the long delays. You are already putting in so much effort and there is only so much a person can do.

I’ll continue in the other thread.

22

I agree with @davy. Better have in there, have it break down if necessary, and then fix it. (that too will be free time well spent btw). I mean, all the API changes we had so far did bring good stuff to the bindings, no? :wink:

K

23

Time to write or suggest a Maven tutorial for us mortals not doing code dev every day?

K

24

It is of today’s 2.0b1 announcement that I finally cached up, but why not adding a “contributions” distort for all the stuff that is not reviewed/approved/clean, with access for all the active add-on developers? Just a place to dump stuff, W.I.P and so forth? Something that can be extensively forked and so forth

1 Like
25

I assume you are not aware how much time I spend on not breaking the API when doing changes to it, because I would otherwise have to adapt all the bindings that are out there already… Having less bindings makes it easier to do and communicate breaking changes that would potentially simplify and enhance the API.

I don’t think it will help the project if everybody does the code in different ways and only provides the binary at an untrusted location.
So for official tutorials, I prefer the idea of automatically building and deploying PRs, even if they are not merged.

Well, this is what a DVCS like git is all about. This place IS the openHAB repo and everyone has access to it and can do changes by forking it. The link to the forks is nicely provided in the PRs.

26

I rather meant, the new stuff that got introduced over time. I for one did not mind rewriting my bindings to adapt to new stuff in the API’s. You are right, in fact, little broke over time, but new great things came along

Honestly, me too, but what I meant is that we get an proliferation of dev.tools to use, the learning curve will get steeper for all those that are not pro-developers. I also prefer homogenous code

I know, but what I meant was a contributions repo (“distort” → predictive spellchecker mistake) on GitHub with access to all of us (e.g. without bottleneck of maintainers’ time), e.g. read and write access

K

27

Hello Folks,

I read this post and feel guilty after using OH for so long to support actively in the dev of it. I am trying to push it in my company to become more known.

But Iam also an old OSGi developer and already created a telematics plattform on OSGi with embedded Java. So I feel you @Kai if you speak about dependency management. I think there is a big need for a build server, where all the jobs could be automated.

I will try to set up a local instance and IDE of ESH and as soon I can walk alone, I would like to support in the OH2 area. As much my wife doesnt get it :wink:

BR, Mehmet

1 Like
28

This thread is quite old but I didn’t find any newer one regarding this topic. I read your posts and understand the concerns of both sides.

First, I would like to add another major drawback of the current approach: As a user who is willing to experiment with addons that are under development and “living in the wild”, it’s just extremely hard to find them if they’re maintained in fork 645 of the openhab2-addons repository. In addition, there is no issue management available on GitHub for these kind of addons.

Coming back to the developer perspective, I think there should be a clear and fair path from how a developer should start with binding development to how he gets his addon into the officially maintained status. If the chances of the own addon of being taken over into the official repository are low (because of a small audience and limited maintainer resources), it’s not a good approach to mislead the developer to the (in my opinion awful) approach of using a fork (where there is no possibility of tracking issues etc.) and let him wait until the cows come home. In addition, if his addon maybe gets more popular later on, there should be be a good way of handing it over to the officially maintained repo.

I would like to bring two ideas up for discussion. I think they’re less invasive as the approaches previously discussed and take the requirements of all three stakeholder types (user, addon developer, maintainer) into consideration:

Idea #1

It would be good to have a best practice documentation for “not so important” addons that have low chances of ever being reviewed and taken over by a maintainer. Or also for addons that cannot be maintained in the official repository because of licensing concerns. Cherry on the cake would be to have clear criteria which addons are “important” and which not.

A good approach for addons “living in the wild” in my opionion: Seperate repository with nightly Travis builds to check whether they’re incompatible API changes. Example:

Idea #2

Regarding the repository structure I have the following idea: Carving every “officially maintained” addon to a seperate repository owned by the OpenHab project (like openhab/milight-binding). To keep them maintainable within the existing Maven structure and the current CI approach, the openhab2-addons repo could include them as Git submodules.

This approach would have several advantages in my opinion:

First, this would enable the project to nominate addon-specific maintainers which are known of having a good API understanding and being able to produce good-quality source code. I think there are quite talented maintainers out there who just do not have the time to take responsibility of the overall project.
Second, this would enable the possibility to manage addon-specific issues and PRs seperately (very important in my opinion, currently it’s just a mess!).
Third, it would make it possible for addon developers to start with a seperate repository which can be handed over to the OpenHab project later on (by using the “Transfer ownership” functionality and adding it as a submodule to openhab2-addons).
Last but not least I think the overhead for maintainers would be minimized (compared to the current approach on refactorings e.g. just the “git submodule foreach” would have to be prepended to the Git commands that are currently used and having a complete checkout is quite easy).

1 Like