Tutorial: Migrate your binding to the Maven + Bnd based build system

The build system has been migrated to a Maven + Bnd based build system. Bindings not yet merged into the openHAB2-addons repository or external bindings need to be migrated to the new build system in order to be accepted and to make them work with the current build system.

This tutorial will make use of git and maven command line tools.

Background

openHAB is a java project that is build around the OSGi framework. openHAB add-ons are OSGi bundles.

Over the years the bnd software (“bnd is the engine behind a number of popular software development tools that support OSGi”) has established itself as the defacto standard for creating OSGi bundles.

The openHAB maintainers have decided to move from the very Eclipse centric buildsystem to maven/bnd. A combination that not only allows to develop bindings seamlessly in other IDEs than Eclipse, but Java beginners and intermediates find a commonly known maven build system.

This tutorial describes in a step-by-step approach how to migrate a binding. The steps described here use git and maven on the command line.

Step 1: Backup your current work

If you don’t have a backup already of your binding, copy your binding from the your source tree to some other place. This will be your backup for the migration. We need these later in the migration step.
You can also make an additional backup by creating another branch in git with the following command:

git checkout -b <name your backup branch>

Step 2: Installing the IDE

Before you start, it’s best to freshly set up your development environment. For development ONLY USE JAVA 8. Either install Eclipse or if you prefer VisualCode or IntelliJ. See the documentation on how to install: Developer Guide | openHAB. This tutorial describes the steps using Eclipse.

When installing Eclipse ONLY select openHAB Development in the Eclipse installer.

Step 3: Getting your old code into the new installation

Copy your original clone of the openhab2-addons repo to the newly installed git directory.

Step 4: Getting your current branch up-to-date with the latest sources

Note: This step destroys the content of your current branch. So any changes on this branch will be lost. That’s why you should back it up

This step is only really relevant if you have a pr pending. If you don’t have a pr pending it’s easier to start with a new clean git branch and update that to the latest sources.

So if you have a pr pending first make sure your on the branch of your pr with your binding:

git checkout <my pr branch>

Now get the latest code from the openhab2-addons repo (this assumes the remote openhab2-addons is on origin. If you have a different name replace origin with your name. If you don’t know what I’m talking about here, you probably have origin):

git fetch origin

The following command will update your branch to the latest code that you just have fetched in the previous command. NOTE this will destroy any changes on your branch, do you have backups! (note this being git, it isn’t directly gone, but that is outside the scope here, hint: reflog)

git reset --hard origin/master

Cleanup the brute force way. You have likely untracked changes after this command.

You can now delete the addons folder.

Check if you still have any (untracked) changes with git status:

git status

If everything is clean it shows: nothing to commit, working tree clean. But if there are untracked files you can get ride of them with the following commands (I admit a bit brute force):

git stash -u
git stash drop

What it does. It pushes any (untracked) changes to the git stack. The second command simple deletes them from the stack.

Step 5: Create the new basis of your binding

With the create_openhab_binding_skeleton script in the bundles folder we’re going to create the basis for your migrated binding. The bindings have been moved from the addons folder to the `bundles folder. Run the following command to create the basis for your binding. The skeleton script also updates all files outside your binding folder that are needed to be adapted for new bindings.

On Linux:

create_openhab_binding_skeleton.sh <binding name> "<your full name>" <your github name>

On Windows:

create_openhab_binding_skeleton.cmd <binding name> "<your full name>" <your github name>

This will generate a new binding in the bundles/org.openhab.binding.<binding name> folder. The following steps are all relative to this folder. More on the skeleton script can be found here: Developer Guide | openHAB

Step 6: Clean up the generated binding

The skeleton script generates an example binding. Because we are going to migrate an existing binding we don’t need those files. So you can clean the following files/directories:

  • Remove README.md
  • Clean src/main/java
  • Clean scr/main/resources

Step 7: Copy your code back into the newly generated binding

To get your binding code into the newly generated binding copy the following files:

  • Copy your README.md and any related files (i.e. images) into your binding folder
  • Copy your java source files to src/main/java
  • Copy the ESH-INF folder into src/main/resources

If you have any other additional files they either go in the root of your binding or if the relevant for runtime. They go in the src/main/resources/

Additional Modifications:

  1. If you have an OSGI-INF folder you need to replace the xml files with annotations in JAVA. The OSGI-INF folder is related to how OSGI services are enabled. In the past we did this by xml files in OSGI-INF folder. But some time ago we switched to using annotations. And any pull request containing them is kindly asked to change it to annotations. If you already have annotations .gitignore files in the OSGI-INF folder can be removed. Because the xml configuration isn’t accepted anymore the documentation on the xml files has been removed. To migrate the xml configuration here is an old documentation page that shows both xml and annotations. openhab-docs/developers/prerequisites/osgids.md at c58e7b9b87bd69826cf90ce450cdfddba7b2d79f · openhab/openhab-docs · GitHub There is also more documentation on the annotations: OSGi Declarative Services | openHAB But probably the easiest way is to look at other bindings!
  2. The schema location in the ESH-INF xml files must be updated:
    OLD: http://eclipse.org/smarthome/schemas/..
    NEW: https://openhab.org/schemas/...
  3. The schema location xsi:schemaLocation in the pom.xml must be updated:
    OLD: http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd
    NEW: http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd

Step 8 (Optional): Handling additional libraries

If you have additional libraries in a lib folder. You need to migrate those. In the documentation this is described in detail: Build System | openHAB

Also this is might be a good time to check if you use libraries that contain functionality also supported by core libraries, and you actually don’t need those extra libraries.

Step 9: Importing the binding in Eclipse

In eclipse Import the binding via File import. Select as an existing project.

If you have a dependency on guava you need to replace those method calls, because we don’t support guava anymore in bindings. In most cases only a single class was used from this library. You will have compile errors in this case.

Step 10: Add your binding to the debug/demo app

With openHAB2 Development and demo app is available. To debug your binding, add the dependency of your binding to the pom.xml of the demo app. (The one next to app.bndrun). See also out Eclipse documentation for up-to-date details: Eclipse IDE | openHAB

Step 11: Check and Compile

To check if everything is ok. We can build the binding with maven. Use the following command from the openHAB2-addons root folder:

mvn clean install -pl :org.openhab.binding.<your binding>

This will also produce a html report file. It’s a good practice to fix those issues.

Step 12: Push your changes back to Github.

Now we’re completed the migration and can compile the new binding we need to push the changes back to the branch on Github. Use the following command:

git push --force-with-lease

Don’t forget to use the force, otherwise you will get a lot of merge commits.

Finished!!

You made it :tada:

If you have a pr pending ping us on the pr that you want to have your pr merged :smile:

And if add any more commits. Don’t forget to sign-off on them!

Bonus: Writing Unit Tests

One of the advantages of the new build system is that it allows you to add tests much easier. While with the old build system you needed to setup a whole project. You can now add your unit tests to src/test/java. This allows you to write unit and mocked tests. If you would have a test that needs a working OSGI container you still need to setup a separate project. These are located in the itests folder. But most cases can be handled by mocking and setting up your code architecture such it’s not all in a big class.

If you have any questions or improvements to this tutorial let me know in the comments.

This text was originally posted in the now closed issue 5005 https://github.com/openhab/openhab2-addons/issues/5005#issuecomment-467973902 To give it more visibility the text has been updated and placed here on the forum.

19 Likes

Excellent tutorial. :+1:

Can you put it into scope somewhere for java rookies like me. :smiley:
I am not really a pro with java annotations.

1 Like

Thanks for putting something together for this. I’m struggling through putting together a binding for openHAB, having never really used Java before. The move away from Eclipse has helped, but it’s still quite a lot of “now do X”, much like this comment here:

This is one of my minor annoyances with “tutorials” on open-source projects - for a beginner to contributing to the project (which, lets face it, is who is going to be reading these things!), they tend to throw a term around which relates to a really important part of the process, and then skip on to the next item.

In this case, you’re telling us to convert one existing format to another - why can’t you provide an example, or at the very least a link, so that we can see a before & after comparison…

I’m definitely not having a go, and I am truly thankful for these posts that help to explain some of the new concepts I, as someone who wants to add to openHAB, will need to follow… but please don’t assume that we all know the intricacies. Those who know how to do something can quite happily skip over something, but those of us who don’t, it becomes a massive stumbling block!

1 Like

The OSGI-INF folder is related to how OSGI services are enabled. In the past we did this by xml files in OSGI-INF folder. But already some time ago we switched to using annotations. And any pull request containing them is kindly asked to change it to annotations. What changed with the build system is that the OSGI-INF folder is gone from the bindings sources folder. With the old system it was still required to have that folder and put a .gitignore file in it to ignore the generated xml files. The reason I included the reference is because if someone still would have xml files and didn’t see the folder and than just removed the folder and wonders why it doesn’t work anymore (this actually happened). So I casually mentioned the OSGI-INF folder to trigger people, just in case. Apparently it hit a nerve. Because we don’t use the xml configuration anymore it’s also gone from the documentation. I found this old page that shows both xml and annotations. So maybe it can shine some light on it:

1 Like

I would like to get a simple OH2 system running so I can import other projects - namely the ZigBee and ZWave bindings that are in separate repositories.

I had hoped that I could simply install the IDE, and just select the “openHAB Development” project in the Eclipse installer, and I could get an environment that would work with OH2 and allow me to install other projects.

Fundamentally, this is step #2 of your tutorial. Step #3 I have not performed since this included the addons repository which I do not need, and on Github we have been talking about closing all projects in this working set, so I assume that this is the same as not including it? (maybe this is wrong?).

The rest of your tutorial relats to adding a new binding. Since I don’t want a new binding, I ignore this, however I assume that since this only adds the new binding to the runtime, that the runtime should work at this stage?

However, this is not the case. It does resolve, but it doesn’t really run. The JVM starts, and I get some debug messages in the IDE log, but only a few and there is no web server (opening localhost:8080 gives me a 403 from Jetty - so it’s at least running, but not enough to be of use).

In the log I get the following messages -:

____________________________
Welcome to Apache Felix Gogo

g! log4j:WARN No appenders could be found for logger (org.eclipse.xtext.parser.antlr.AbstractInternalAntlrParser).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
10:49:23.408 [main] INFO  o.e.s.m.c.i.ModelRepositoryImpl:123 - Loading model 'demo.items'
10:49:23.530 [main] INFO  o.e.s.m.c.i.ModelRepositoryImpl:123 - Loading model 'mapdb.persist'
10:49:23.659 [main] INFO  o.e.s.m.c.i.ModelRepositoryImpl:123 - Loading model 'demo.sitemap'
10:49:23.714 [main] INFO  o.e.s.m.c.i.ModelRepositoryImpl:123 - Loading model 'demo.things'
10:49:24.746 [main] INFO  o.e.s.m.c.i.ModelRepositoryImpl:123 - Loading model 'demo.rules'
10:49:25.494 [nfig-1-thread-1] WARN  o.o.p.w.u.ServletContainerInitializerScanner:136 - failed to parse and instantiate of javax.servlet.ServletContainerInitializer in classpath
10:49:25.494 [nfig-1-thread-1] WARN  o.o.p.w.u.ServletContainerInitializerScanner:136 - failed to parse and instantiate of javax.servlet.ServletContainerInitializer in classpath
10:49:25.589 [nfig-1-thread-1] WARN  o.o.p.w.u.ServletContainerInitializerScanner:136 - failed to parse and instantiate of javax.servlet.ServletContainerInitializer in classpath
10:49:25.589 [nfig-1-thread-1] WARN  o.o.p.w.u.ServletContainerInitializerScanner:136 - failed to parse and instantiate of javax.servlet.ServletContainerInitializer in classpath
10:49:25.608 [nfig-1-thread-1] WARN  o.o.p.w.u.ServletContainerInitializerScanner:136 - failed to parse and instantiate of javax.servlet.ServletContainerInitializer in classpath
10:49:25.608 [nfig-1-thread-1] WARN  o.o.p.w.u.ServletContainerInitializerScanner:136 - failed to parse and instantiate of javax.servlet.ServletContainerInitializer in classpath
10:49:25.629 [nfig-1-thread-1] WARN  o.o.p.w.u.ServletContainerInitializerScanner:136 - failed to parse and instantiate of javax.servlet.ServletContainerInitializer in classpath
10:49:25.629 [nfig-1-thread-1] WARN  o.o.p.w.u.ServletContainerInitializerScanner:136 - failed to parse and instantiate of javax.servlet.ServletContainerInitializer in classpath
10:49:25.648 [nfig-1-thread-1] WARN  o.o.p.w.u.ServletContainerInitializerScanner:136 - failed to parse and instantiate of javax.servlet.ServletContainerInitializer in classpath
10:49:25.648 [nfig-1-thread-1] WARN  o.o.p.w.u.ServletContainerInitializerScanner:136 - failed to parse and instantiate of javax.servlet.ServletContainerInitializer in classpath
10:49:25.666 [nfig-1-thread-1] WARN  o.o.p.w.u.ServletContainerInitializerScanner:136 - failed to parse and instantiate of javax.servlet.ServletContainerInitializer in classpath
10:49:25.667 [nfig-1-thread-1] WARN  o.o.p.w.u.ServletContainerInitializerScanner:136 - failed to parse and instantiate of javax.servlet.ServletContainerInitializer in classpath
10:49:25.687 [nfig-1-thread-1] WARN  o.o.p.w.u.ServletContainerInitializerScanner:136 - failed to parse and instantiate of javax.servlet.ServletContainerInitializer in classpath
10:49:25.687 [nfig-1-thread-1] WARN  o.o.p.w.u.ServletContainerInitializerScanner:136 - failed to parse and instantiate of javax.servlet.ServletContainerInitializer in classpath
10:49:25.765 [nfig-1-thread-1] WARN  o.e.j.util.thread.ThreadPoolBudget:156 - Low configured threads: (max=16 - required=9)=7 < warnAt=8 for QueuedThreadPool[ServletModel-20]@14c6e66f{STARTED,8<=8<=16,i=8,q=0}[ReservedThreadExecutor@11b2d4b0{s=0/1,p=0}]
10:49:25.776 [nfig-1-thread-1] WARN  o.o.p.w.u.ServletContainerInitializerScanner:136 - failed to parse and instantiate of javax.servlet.ServletContainerInitializer in classpath
10:49:25.776 [nfig-1-thread-1] WARN  o.o.p.w.u.ServletContainerInitializerScanner:136 - failed to parse and instantiate of javax.servlet.ServletContainerInitializer in classpath
10:49:25.794 [nfig-1-thread-1] WARN  o.o.p.w.u.ServletContainerInitializerScanner:136 - failed to parse and instantiate of javax.servlet.ServletContainerInitializer in classpath
10:49:25.795 [nfig-1-thread-1] WARN  o.o.p.w.u.ServletContainerInitializerScanner:136 - failed to parse and instantiate of javax.servlet.ServletContainerInitializer in classpath
10:49:25.815 [nfig-1-thread-1] WARN  o.o.p.w.u.ServletContainerInitializerScanner:136 - failed to parse and instantiate of javax.servlet.ServletContainerInitializer in classpath
10:49:25.815 [nfig-1-thread-1] WARN  o.o.p.w.u.ServletContainerInitializerScanner:136 - failed to parse and instantiate of javax.servlet.ServletContainerInitializer in classpath
10:49:25.832 [nfig-1-thread-1] WARN  o.o.p.w.u.ServletContainerInitializerScanner:136 - failed to parse and instantiate of javax.servlet.ServletContainerInitializer in classpath
10:49:25.832 [nfig-1-thread-1] WARN  o.o.p.w.u.ServletContainerInitializerScanner:136 - failed to parse and instantiate of javax.servlet.ServletContainerInitializer in classpath
10:49:25.862 [nfig-1-thread-1] WARN  o.o.p.w.u.ServletContainerInitializerScanner:136 - failed to parse and instantiate of javax.servlet.ServletContainerInitializer in classpath
10:49:25.863 [nfig-1-thread-1] WARN  o.o.p.w.u.ServletContainerInitializerScanner:136 - failed to parse and instantiate of javax.servlet.ServletContainerInitializer in classpath
10:49:35.179 [pool-4-thread-1] INFO  o.o.c.a.m.s.r.i.l.ScriptFileWatcher:171 - Loading script 'demo.js'

After this all activity stops.

Maybe I’ve missed some important steps, but it feels to me like this should work. If someone simply wants to develop a new binding, then a simple, minimal install such as this would really seem to me to be the way to go to try and minimise the potential for problems that we’ve all been having for the past 3 or 4 months.

(apologies for duplication - I posted some of this on Github yesterday and this morning before I saw your post)

It doesn’t stop. It’s just an empty system with no bindings installed and nothing configured. In one of the projects there is the configuration map where you can put in textual configuration (I’m not behind a computer so can’t checked exact names). It should however work with UI’ and http://localhost:8080/paperui/ should show the ui.

Ok, thanks. I think what I missed is there is no “landing page” on localhost:8080 in this configuration - paperUI is there though. I will try and add the bindings.

:+1:

After quite a bit more messing around today, I’m relatively happy that I can get a simple environment working if I only install the OpenHAB Development repo when in the Eclipse Installer. From here I don’t change anything in the POMs, but I import my bindings and mostly it works.

If anything changes with dependencies, bnd doesn’t seem to cope well and it needs a restart of the IDE to resolve it (or at least I’ve not found any other way to recover it as yet!).

Can I suggest that in step 2 above, you change it so as not to say what NOT to do, but instead say what SHOULD be done - it’s clearer this way. I think that for most users, if they only add the development repo, it should work and I think this is what you are saying in step 2, but I’m not 100% sure.

I fear that the more projects that are open, and possibly adding the core development which I have also always done in the past as often this is needed to understand the finer points of what’s happening, will likely lead to more problems. As above, bnd doesn’t seem super stable when resolving dependencies, and any changes in the system quickly seems to get it confused, so keeping a simple setup until this is resolved seems a surer way to success.

So far I’ve only tested out the ZigBee binding, but as that included half a dozen projects, it’s a good test I think. Let’s see if it survives overnight :slight_smile:

Thanks.
Chris

5 Likes

Thanks for jumping into this chris. Sounds really promising

2 Likes

Sorry to jump into this again, but…

I’m trying to work out how to resolve dependencies in the ZigBee binding again. I need to change some of the dependency versions, which is simple enough in the POM, and everything resolves just fine in the IDE. However, the demo app doesn’t resolve the new dependency versions - it seems to keep pulling in the ones defines in the reactor.

I know there’s a problem with bnd not pulling in the dependencies defined in the project POM, but I can’t get the demo app pom configured correctly to pull in the new versions…

I’ve locally updated the org.openhab.addons.zigbee.reactor pom to change dependency versions (eg from v 1.2.3 to 1.2.6 as this is defined as a property in the reactor for the multiple zigbee child bundles). This is picked up in the various child bundle poms (eg org.openhab.binding.zigbee) and Eclipse shows all the latest updates in the IDE.

However the demo app still wants to pull in 1.2.3 so I can’t get a working debug environment. @hilbrand or @wborn any thoughts on this? (thanks).

Have you been running Maven> Update project after changing the POMs and BND files? This has been helping me out, but every now and then ‘clean install’ will fail after doing this, but it works again after restarting Eclipse.

Yes, as I said the IDE resolves fine and of course this requires that maven is updated, it’s just the demo app that fails. It’s known that bnd has an issue - we discussed that a few months back but I can’t get it to resolve the dependency changes.

@chris I tried to see if I could reproduce your issue. If I don’t have the root reactor project not in Eclipse then it gives me the behavior you mentioned. When I have the reactor project in Eclipse than it does resolve to version 1.2.6.

Hmm - ok, thanks. I don’t think I have the reactror imported as it had caused other problems in the past. I’ll give this a try though - thanks.

Any idea why this is needed?

@hilbrand Eclipse won’t actually import the reactor -:

I’ve tried this a number of times. I assume when you said “the root reactor project in eclipse” you meant that the project was imported into the workspace - not just that the POM was in the editor? If so, it doesn’t seem that this wants to import.

I tried removing all projects from the workspace and reimporting, but I still don’t end up with a separate project for the reactor. How did you manage to get this to show as a separate project - or do I misunderstand what you did?

Yes I mean the root project that is checked in your screenshot. It shows selected, but you say it doesn’t want to import? What message does it give?

I don’t get any message. In the error log it says the org.openhab.binding.zigbee already exists, but of course that’s not what I’m importing. It also won’t import if I delete all the zigbee projects and select all projects to import…

I’m guessing you’re looking at Top Level Elements Working Sets and that the project is already imported, but not part of the Working Set your’re looking at. See if it changes if you use Projects:
image

When the project is not in Eclipse and you changed the pom and/or have maven installed it locally, the demo app still could override that file when it resolves. It just downloads the latest version that overrides your own (installed) version. This is unless you have the project in Eclipse. In that case it will take that version. Therefor all files that are changed need to be in a project in Eclipse.

1 Like

That’s already how I had it configured -: