Hello,
I have been exploring OpenHab integration tests for a few weeks, initially with the goal of getting the integration tests working for org.openhab.bindings.mqtt.homeassitant. Through this I have learned a few things, some may already be known to many developers, but were new to me, and hopefully can help with deciding how to proceed going forward.
Integration tests execute by launching via the itest.bndrun file in each integration test. This file configures launching via OSGI. OSGI is related to, but different from, the maven tools used for building add-on bundles. A primary function of OSGI is finding bundles that are needed to successfully launch a particular configuration. In order to determine which bundles are needed at runtime, OSGI uses a process called resolving to determine which versions of which bundles are required. This resolving step can be done at two points. It can be done on a development machine, in which case the resolving step will pick from bundles available in the maven repository on the developer’s machine. Alternatively the pom.xml file for either all itests, or an individual itest can be configured to do the resolving step just before the integration test is launched on the continuous integration machine, Jenkins or Travis.
By default, the resolving step, regardless of where it is run, finds a version of a matching bundle that is required from the dependencies in the pom.xml file for the project and all parent pom.xml files. It will then configure the itest.bndrun file to accept any version of the bundle, starting from the found version of the bundle, and less than the next minor version of the bundle. So for dependencies on other openhab bundles for example it will choose required versions in a range similar to the following [2.5.3,2.5.4) The square bracket means inclusive, the parenthesis means exclusive, so the notation means “accept versions starting with 2.5.3, but not including 2.5.4”, so importantly 2.5.3-SNAPSHOT is OK, 2.5.4-SNAPSHOT is not.
If the resolving step happens on the development machine, the developer can manually edit the itest.bndrun dependencies to accept a broader range of versions. But it is not clear how often it is done.
A problem that occurs because of this configuration is that each time the .x digit is increased, all itests that were configured on development machines fail, because now no matching openhab bundles can be found in the maven repository on the CI machine, since all new builds produce 2.5.4-SNAPSHOT version bundles, but the itest.bndrun files are configured to only accept versions less than 2.5.4. So every time a minor version is increased, owners of integration tests need to go back in and reconfigure their itest.bndrun file, clean their local maven repositories, build the latest dependencies, and run the OSGI resolve step to find the latest bundles.
Apologies for the wordy post, but my suggestion is to configure the default pom.xml file in the itests directory to automatically perform the OSGI resolve step each time before the integration test is run, this would avoid the need for developers to manually update the file after every minor version update.
Thoughts?