openHAB reliability and upgrade experience

[glen_m]

@Mherwege

Ideally there should be a new UI, proposing to clear the cache, run the upgrade tool, show breaking changes

I know there have been subsequent posts on technical limitations etc, of where/how/when things like the application cache must be cleared, but I thought I might throw this in the mix.

I also use an application called xwiki for my House & Automation documentation, and after every upgrade it presents me with a series of dialogue screens, to address remaining steps in the (post) upgrade process

https://www.xwiki.org/xwiki/bin/view/Documentation/UserGuide/Features/DistributionWizard

I’m not putting this here as some panacea/reference model that OpenHAB should adopt, but maybe just some ideas which could be taken from the approach.

I’m no xwiki expert, but at a HL, it sort of does this:

• First checks that I am logged in as an admin user (and allows me to log in if not)
• Then it checks that the versions of the extensions/flavours that I have are compatible with the new base version of the product, and presents me with a screen which warns that the current installed extensions etc, are not compatible
• It will then propose what changes it will make, and I get to accept (or deny) that
• It goes away, and downloads the updated extensions/flavour etc, and upgrades them
• At the end it lists the changes made (and in the case of the wiki, the affected pages)
• And from time to time (if my memory serves me correct), it also then asks if I want to remove deprecated items (I think pages/extensions)

So putting that in a OpenHAB context, perhaps if ‘extensions/flavours’ from the above was replaced with the words ‘Marketplace Add-ons/rules/widgets’, you could see how a similar experience could ‘theoretically’ feel for Openhab.

Reading the thread, I know there is potentially the issue that marketplace add-ons are ‘removed’ by the upgrade process, so there may be an issue of how you present a list of previously installed add-ons which need upgrading and/or re-installation (Perhaps the upgrade process can write this somewhere before it removes them?)

But I am also sure that the above could be adapted to show the release notes, etc, and for xwiki, once I finish the process, it is no longer presented to me, and there is also an option to suppress the dialogue from repeating on next startup, if you choose not perform the post-upgrade steps. (There was a process (but not in the UI) to make it run again on next startup, as well, if you had suppressed it initially)

There is also probably the dependency of the Marketplace being able to store multiple versions of the same add-on/rule/widget, so if the Author publishes a new version, say only compatible from a future 5.2 version onwards, somebody can still upgrade from say 4.x to 5.1, with having the prior compatible version still available via this process.

[florian-h05 (Florian H.)]

@Nadahar

That still leaves the question of where the script that does the cache clearing etc. lives, because in the update script that comes with the “manual install” archive that you can download, both things are done as part of the same operation:

The openhab-linuxpkg repo has its own package pre and post steps.

[gerrit (gerrit oehlmann)]

“So I am getting a cold sweat when I install a new version of OH“

A lot of what you wrote sounds quite familiar to me. If I remember correctly, I started using OH in 2016/17 and over the years I had some less-than-pleasant and labor-intensive experiences when upgrading. Over the years I also eliminated some pitfalls like the notorious RPi SD card. Now, ten years after I started, all my installations are running in containers with automated backups on a small PC, and I still only do upgrades when there is at least a weekend in sight with not much else to do.

I did consider moving to another software, but haven’t found anything more compelling in comparison. I also had some questionable interactions in this community over the years – but mostly it was helpful and nice. And yes – there is still space for improvement, but from my personal observations, it is happening.

Oh, and I integrated Gemini for support during my last upgrade. What can I say? It worked quite well and took some very tedious tasks off my hands.

Have a nice day and thanks to everyone for your support and work,
Gerrit

[Kim_Andersen]

@florian-h05

IMO we have to add a very real issue with the “wishlist” topics: Reading a hundreds of comments is quite time consuming … it would be better if such wishlists were organized as wiki topics where everyone could edit a single post and add his ideas if they aren’t already there, and where someone could organize the ideas.

Agree, comments on the comminuty is impossible to follow. There has been a few, and each time I feel, that its waste of time.
Having a wishlist would be a perfect idea.

[rlkoshak (Rich Koshak)]

@Nadahar

However, when I look inside the update script found on openHABian, it only has this content:

openHABian uses apt. apt will be the process that does the update for openHABian.

For the purposes of this discussion, openHABian is the same as apt.

@Nadahar

because in the update script that comes with the “manual install” archive

The update script is the same across all openHAB installs, even Docker. You can find it in github here. There is only the one version of the shell script.

It would seem that instead of invoking this script, the package managers implement the cache clearing independently. See openhab-linuxpkg/resources/control-runtime/postrm at 854aeb8fec77043b11a7257865fbb8db9cfdc5f9 · openhab/openhab-linuxpkg · GitHub

[Nadahar (Nadar)]

@florian-h05

The openhab-linuxpkg repo has its own package pre and post steps.

I understand that, I’m trying to see this from a user perspective. There has been talk about that you need to run the “upgrade script” after manually restoring/copying configuration from a different version - but it’s still very unclear to me how to run a script that I don’t even know how to identify.

@rlkoshak

openHABian uses apt. apt will be the process that does the update for openHABian.

I understand that, but how does people manually run the “upgrade script” then? You can’t expect them to dissect them using dpkg to figure out what scripts exist and where, so that they can run them..?

@rlkoshak

The update script is the same across all openHAB installs, even Docker. You can find it in github here. There is only the one version of the shell script.

Except that when installed using apt it’s not there - the content has been replaced with a message to use apt instead.

@rlkoshak

It would seem that instead of invoking this script, the package managers implement the cache clearing independently.

Yes, I see that they also do the “update tasks” from within these scripts:

resources/control-runtime/postinst

..which leads to what I’ve been suspecting for a while now: If you run openHABian or an apt (and probably also dnf) install, it doesn’t seem like you actually have any upgrade/update script to run after copying in files from a backup from a different version.

I think it would make sense if there was a separate script, what worked across all variants of OH, that showed the warnings and performed the tasks in update.lst that could be run manually in a predictable manner. All the other solutions that do this, could then just call this script, just like users could do manually.

[rlkoshak (Rich Koshak)]

@Nadahar

I think it would make sense if there was a separate script, what worked across all variants of OH, that showed the warnings and performed the tasks in update.lst that could be run manually in a predictable manner. All the other solutions that do this, could then just call this script, just like users could do manually.

Isn’t that basically what @florian-h05 recommended above? Then only addition is to make it so the user doesn’t have to invoke the script manually ever in the first place. It gets run automatically when it’s needed regardless of the installation method.

[Nadahar (Nadar)]

@rlkoshak

Isn’t that basically what @florian-h05 recommended above? Then only addition is to make it so the user doesn’t have to invoke the script manually ever in the first place. It gets run automatically when it’s needed regardless of the installation method.

Sure, doing it automatically using a check would be great. I’m just talking about the organization of the files here now, at a quite “low level” if you will. If a check is done during startup, it would also be easier if the script with this logic was isolated from the downloading/updating of versions, because then the “startup check” could merely call this script, regardless of installation method, if the check determined that running the process was needed.

[bjscheue (Bjoern)]

@rlkoshak

Correct. Think of it like this, the upgradeTool is a part of the upgrade script. The upgrade script will invoke the upgradeTool as one of the steps it takes to prepare a config to run with a new version of OH which includes clearing the cache and in some cases updating some files in userdata/etc.

Ok, I had gotten it right then. But while I can find the upgradeTool among the binaries in the .deb package, I can’t find the upgrade script. It seems it is in the packet metadata, but not (easily) accessible to run it manually, after configuration from an earlier version has been transferred. I guess that’s something that is very easy to improve - just (additionally) shipping it somewhere where it can be found and called without dissecting the .deb package?

@rlkoshak

Is this theoretical or something you’ve faced? On occasion, I’ve had to install old OH’s on newer OS (e.g. OH 2.5 on Debian Bookworm, and Java 8 and OH 2.5 installed and ran without problem). I’ve never had to do Java 11 or Java 17 but if 8 worked I would assume those would work too. Does it not?

That’s something I have faced.

The overall scenario that I described was a mashup of two distinct, but both very real incidents. In the course of getting things back on track, I was at the point where I though: “Well, I might try and install an old version, then upgrade from there.” And it proved really hard. Perhaps it would not even have been hard to do with a manual installation. But that wouldn’t have helped if the plan is to subsequently upgrade the package so that it can do its thing and run its scripts during the upgrade.

It’s been too long to accurately remember all the details. But I do recall that I didn’t manage to get closer to a solution along these lines before I finally decided to try other things first.

And mind you: we’re not talking about installing an ancient package because there’s any desire or need to actually run and use that old version. We’re discussing this only for one reason: as a workaround to make the new package run the upgrade script…

@rlkoshak

You don’t have to do this at all unless they were installed from the marketplace (i.,e. unofficial add-ons). And there, make sure there is a version compatible with this version of OH first.

Very good point, and yes of course that’s something to double-check first.

@rlkoshak

That might be something we can deal with, namely identify an OpenJDK implementation for these older Javas that work on newer OS.

Possibly, yes - but it sounds like a workaround to make the other workaround work. Calling the upgrade script of the newer version directly (i.e., putting it somewhere where it can be found and called) serves the same purpose with much less effort, both for developers and for users.

Of course, I am fully in line with you and several follow-up posters that fully automating the check and the necessary steps upon every start of OH would be even better. But that doesn’t rule out to provide the upgrade script in apt-based installations, as a first, very lightweight improvement and/or in addition to the automation.

[Nadahar (Nadar)]

@bjscheue

Of course, I am fully in line with you and several follow-up posters that fully automating the check and the necessary steps upon every start of OH would be even better. But that doesn’t rule out to provide the upgrade script in apt-based installations, as a first, very lightweight improvement and/or in addition to the automation.

This is essentially what I’ve been trying to say too. Separate the downloading of new versions from the part that does the necessary changes in update.lst, and leave the script that processes update.lst in place even for packages distributes with various distribution systems. The downloading part should only be part of the “manual install” archive.

[Benjy (Ben Clark)]

Just catching up on a few things mentioned here - a few recent life events mean that I’m not as active as I’d like to be at the moment.

@rlkoshak

[..on the Linux directory choices and the FHS]

I don’t know if that was the original reasoning. It’s been this way since OH 2.0. But that was what I understood to be the why.

Of course, it gets confusing with the whole $OH_USERDATA/etc folder. But there too karaf can and does update these files dynamically (e.g. log4j2.xml).

Yes, that was the original reasoning. In short anything we expect the user to directly edit by text should go in ${OPENHAB_CONF} (e.g. ~/openhab/conf or/etc/openhab) and anything which is edited by openHAB should go in${OPENHAB_USERDATA}(e.g. ~/openhab/userdata or /var/lib/openhab). I believe it’s because of limitations of Karaf that there are a few exceptions, the system files that must be replaced each upgrade should be tracked and listed here and found at ${OPENHAB_RUNTIME}/bin/userdata_sysfiles.lst

@bjscheue

[on using a package installer to upgrade]

I can’t find the upgrade script.

The package managers (apt (used by openHABian), yum and dnf) work by tracking a list of files that need to be replaced, vs a list of files that can be changed by the user in which it tries to leave alone unless the openHAB distribution contains newer versions. It also does the more openHAB specific part in a way that matches the ${OPENHAB_RUNTIME}/bin/update script. We overwrite the manual update script because any file it touches counts as a user change which affects how the Linux package manager treats the file on the next run. The consequences of running the manual script aren’t easily identifiable, so it was safer to remove the manual process if you were using apt, yum, or dnf. After all, these steps have already been completed and will be completed again on the next sudo apt install openhab.

In short, if you use apt,yum, or dnf, the functionality of the update script is done for you and upgradeTool.jar is also executed/

---

Then there’s the related, but seperate issue:

The ${OPENHAB_RUNTIME}/bin/backup and ${OPENHAB_RUNTIME}/bin/restore scripts were made shortly after the ${OPENHAB_RUNTIME}/bin/update scripts, The scripts are used in the apt/yum/dnf versions of openHAB.

At the time (9 years ago) I did not consider how the update process would evolve and did not add a shared function to do all those replacement steps. As a result, the restore script does not perform the steps that an updatedoes and this is probably not the correct behaviour which can be fixed by running the functions that update does after a restore. A PR is needed in GitHub openhab-distro for this.

---

To finish on a clear note - what I would personally recommend to anyone not using openHABian (which has menu items for such things) and is worried about upgrading:

1. Remember that anything that is 3rd party to openHAB needs its own backup/restore configuration.
2. Make a backup of openHAB using openHAB’s included backup script (${OPENHAB_RUNTIME}/bin/backup --help or openhab-cli backup --help)
3. Upgrade openHAB
4. If something goes wrong and you can’t figure out what, install the older version of openHAB first before using the included restore script.

I saw some points about implementing step 2 automatically when you try to upgrade openHAB. Whilst I agree that this is should be done, I would still recommend you make a backup manually, the included scripts have options for including everything such as the cache for those that feel they need it.

[Nadahar (Nadar)]

@Benjy

We overwrite the manual update script because any file it touches counts as a user change which affects how the Linux package manager treats the file on the next run. The consequences of running the manual script aren’t easily identifiable, so it was safer to remove the manual process if you were using apt, yum, or dnf. After all, these steps have already been completed and will be completed again on the next sudo apt install openhab.

In short, if you use apt,yum, or dnf, the functionality of the update script is done for you and upgradeTool.jar is also executed/

How does the package manager know what has been touched by “the user” and not? Does it save a copy after “it is done” and then compare that? If so, you would have the exact same problem if the update check is made during OH startup.

The problem that has been discussed here isn’t when upgrading OH using the package manager, but when restoring data manually, from a different version. When you do that, there needs to be a way to run the “conversion” manually, which is lacking on the package managed versions (and in reality, on the manual install version as well, as it doesn’t seem that the script allows running just the update part without installing a different version).

So, it’s more a case not having the flexibility to do anything but exactly following “the recommended path”, and it results in things like having to install a potentially very old version after a system have died, to allow it to “upgrade its way” to the target version.

Having seen the script, it doesn’t actually look like that is necessary, it seems to me that any version upgrade is enough to run “the full update script”, so you can probably just install the version immediately prior to the one you want, restore your data and then upgrade. But, this isn’t what you’re told to do, and frankly, it’s a “strange thing” to have to do to get around the “rules of the package managers”.

[ubeaut (Greg)]

When I was testing OH4 to OH5 I restored my OH4 backup and then deleted /var/lib/openhab/jsondb/org.openhab.core.tools.UpgradeTool and ran sudo dpkg-reconfigure openhab

This forced the upgrade tool to re-run.

Whether that was the correct way of doing things I don’t know but I didn’t have any issues. (That I know of)

I am on OH5.1 now and still have had any problems.

[rlkoshak (Rich Koshak)]

@ubeaut

and then deleted /var/lib/openhab/jsondb/org.openhab.core.tools.UpgradeTool and ran sudo dpkg-reconfigure openhab

That file keeps track of the upgradTool steps that have already been executed on your configs. If it’s already been run successfully, upgradeTool will skip running them again.

@ubeaut

Whether that was the correct way of doing things I don’t know but I didn’t have any issues. (That I know of)

That was an unnecessary step. Those steps were performed during previous upgrades and do not need to be done again. But if you ever do want to run them again, you can run upgradeTool manually with --force.

[mstormi (Markus Storm)]

Not fully sure what it already does but wouldn’t it be a good idea if the restore script (from openhab-distro), when called via openhab-cli, checks the installed oh version against the version the backup was taken on, and auto-runs the upgradetool?

Would help for example if some user installs a fresh openHABian (which installs latest OH) and restores his openHAB backup from any earlier version. Quite some people do and probably many expect this to work.
cc @Benjy

[Nadahar (Nadar)]

@mstormi

and auto-runs the upgradetool?

It would need to run the “update script” as well, the logic that apples the changes that is listed in update.lst.

[Benjy (Ben Clark)]

Absolutely!

As a result, the restore script does not perform the steps that an updatedoes and this is probably not the correct behaviour which can be fixed by running the functions that update does after a restore. A PR is needed in GitHub openhab-distro for this.

The most straightforward way of doing this is taking out the functions from update and placing them in a shared file so that they can be sourced from both update and restore. The backup zip file already has a record of what the old version is and this is read in as a variable by the restore script.

[Nadahar (Nadar)]

@Benjy

The most straightforward way of doing this is taking out the functions from update and placing them in a shared file so that they can be sourced from both update and restore. The backup zip file already has a record of what the old version is and this is read in as a variable by the restore script.

Yes, pretty much exactly what I suggested further up - that the processing of update.lst should be separated from other logic and then just called whenever it’s needed.