Update Docker container to 2.5 ends in Reboot loop

vlad1986 · December 16, 2019, 9:51am

this night watchtower updates my openhab docker container to 2.5 but this ends in a Reboot loop.

message before container crashes

openhab_1      | ++ cmp /openhab/userdata/etc/version.properties /openhab/dist/userdata/etc/version.properties
openhab_1      | + '[' '!' -z '/openhab/userdata/etc/version.properties /openhab/dist/userdata/etc/version.properties differ: byte 141, line 4' ']'
openhab_1      | + echo 'Image and userdata versions differ! Starting an upgrade.'
openhab_1      | + tee /openhab/userdata/logs/update.log
openhab_1      | Image and userdata versions differ! Starting an upgrade.
openhab_1      | ++ date +%FT%H-%M-%S
openhab_1      | + backup_file=userdata-2019-12-16T10-47-02.tar
openhab_1      | + '[' '!' -d /openhab/userdata/backup ']'
openhab_1      | + tar --exclude=/openhab/userdata/backup -c -f /openhab/userdata/backup/userdata-2019-12-16T10-47-02.tar /openhab/userdata
openhab_1      | tar: Removing leading `/' from member names
openhab_1      | + echo 'You can find backup of userdata in /openhab/userdata/backup/userdata-2019-12-16T10-47-02.tar'
openhab_1      | + tee -a /openhab/userdata/logs/update.log
openhab_1      | You can find backup of userdata in /openhab/userdata/backup/userdata-2019-12-16T10-47-02.tar
openhab_1      | + tee -a /openhab/userdata/logs/update.log
openhab_1      | + exec /openhab/runtime/bin/update
openhab_1      | 
openhab_1      | ################################################
openhab_1      |           openHAB Docker update script          
openhab_1      | ################################################
openhab_1      | 
openhab_1      | You are already on openHAB 2.5.0
openhab_openhab_1 exited with code 1

update.log

Image and userdata versions differ! Starting an upgrade.
You can find backup of userdata in /openhab/userdata/backup/userdata-2019-12-16T10-47-08.tar

################################################
          openHAB Docker update script          
################################################

You are already on openHAB 2.5.0

openhab.log the time stamp is after the update.

2019-12-15 22:38:07.900 [ERROR] [ternal.service.BootFeaturesInstaller] - Error installing boot features
org.apache.karaf.features.internal.util.MultiException: Error:
	Error downloading mvn:org.openhab.ui.bundles/org.openhab.ui.dashboard/2.5.0
	at org.apache.karaf.features.internal.download.impl.MavenDownloadManager$MavenDownloader.<init>(MavenDownloadManager.java:91) ~[?:?]
	at org.apache.karaf.features.internal.download.impl.MavenDownloadManager.createDownloader(MavenDownloadManager.java:72) ~[?:?]
	at org.apache.karaf.features.internal.region.Subsystem.downloadBundles(Subsystem.java:457) ~[?:?]
	at org.apache.karaf.features.internal.region.Subsystem.downloadBundles(Subsystem.java:452) ~[?:?]
	at org.apache.karaf.features.internal.region.SubsystemResolver.resolve(SubsystemResolver.java:224) ~[?:?]
	at org.apache.karaf.features.internal.service.Deployer.deploy(Deployer.java:393) ~[?:?]
	at org.apache.karaf.features.internal.service.FeaturesServiceImpl.doProvision(FeaturesServiceImpl.java:1062) ~[?:?]
	at org.apache.karaf.features.internal.service.FeaturesServiceImpl.lambda$doProvisionInThread$13(FeaturesServiceImpl.java:998) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_232]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_232]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_232]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
	Suppressed: java.io.IOException: Error downloading mvn:org.openhab.ui.bundles/org.openhab.ui.dashboard/2.5.0
		at org.apache.karaf.features.internal.download.impl.AbstractRetryableDownloadTask.run(AbstractRetryableDownloadTask.java:77) ~[?:?]
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_232]
		at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_232]
		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_232]
		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_232]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_232]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_232]
		at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
	Caused by: java.io.IOException: Error resolving artifact org.openhab.ui.bundles:org.openhab.ui.dashboard:jar:2.5.0: [Could not transfer artifact org.openhab.ui.bundles:org.openhab.ui.dashboard:jar:2.5.0 from/to openhab (https://api.bintray.com/maven/openhab/mvn/online-repo/2.5/): Not authorized]
		at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.configureIOException(AetherBasedResolver.java:803) ~[?:?]
		at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:774) ~[?:?]
		at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:657) ~[?:?]
		at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:598) ~[?:?]
		at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:565) ~[?:?]
		at org.apache.karaf.features.internal.download.impl.MavenDownloadTask.download(MavenDownloadTask.java:52) ~[?:?]
		at org.apache.karaf.features.internal.download.impl.AbstractRetryableDownloadTask.run(AbstractRetryableDownloadTask.java:60) ~[?:?]
		... 7 more
		Suppressed: shaded.org.eclipse.aether.transfer.ArtifactTransferException: Could not transfer artifact org.openhab.ui.bundles:org.openhab.ui.dashboard:jar:2.5.0 from/to openhab (https://api.bintray.com/maven/openhab/mvn/online-repo/2.5/): Not authorized
			at shaded.org.eclipse.aether.connector.basic.ArtifactTransportListener.transferFailed(ArtifactTransportListener.java:52) ~[?:?]
			at shaded.org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run(BasicRepositoryConnector.java:368) ~[?:?]
			at shaded.org.eclipse.aether.util.concurrency.RunnableErrorForwarder$1.run(RunnableErrorForwarder.java:75) ~[?:?]
			at shaded.org.eclipse.aether.connector.basic.BasicRepositoryConnector$DirectExecutor.execute(BasicRepositoryConnector.java:642) ~[?:?]
			at shaded.org.eclipse.aether.connector.basic.BasicRepositoryConnector.get(BasicRepositoryConnector.java:262) ~[?:?]
			at shaded.org.eclipse.aether.internal.impl.DefaultArtifactResolver.performDownloads(DefaultArtifactResolver.java:489) ~[?:?]
			at shaded.org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:390) ~[?:?]
			at shaded.org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:215) ~[?:?]
			at shaded.org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact(DefaultArtifactResolver.java:192) ~[?:?]
			at shaded.org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveArtifact(DefaultRepositorySystem.java:247) ~[?:?]
			at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:767) ~[?:?]
			at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:657) ~[?:?]
			at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:598) ~[?:?]
			at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:565) ~[?:?]
			at org.apache.karaf.features.internal.download.impl.MavenDownloadTask.download(MavenDownloadTask.java:52) ~[?:?]
			at org.apache.karaf.features.internal.download.impl.AbstractRetryableDownloadTask.run(AbstractRetryableDownloadTask.java:60) ~[?:?]
			at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_232]
			at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_232]
			at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_232]
			at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_232]
			at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_232]
			at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_232]
			at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
		Caused by: shaded.org.apache.maven.wagon.authorization.AuthorizationException: Not authorized
			at shaded.org.apache.maven.wagon.shared.http.AbstractHttpClientWagon.fillInputData(AbstractHttpClientWagon.java:1123) ~[?:?]
			at shaded.org.apache.maven.wagon.shared.http.AbstractHttpClientWagon.fillInputData(AbstractHttpClientWagon.java:1072) ~[?:?]
			at shaded.org.apache.maven.wagon.StreamWagon.getInputStream(StreamWagon.java:126) ~[?:?]
			at shaded.org.apache.maven.wagon.StreamWagon.getIfNewer(StreamWagon.java:88) ~[?:?]
			at shaded.org.apache.maven.wagon.StreamWagon.get(StreamWagon.java:61) ~[?:?]
			at shaded.org.eclipse.aether.transport.wagon.WagonTransporter$GetTaskRunner.run(WagonTransporter.java:567) ~[?:?]
			at shaded.org.eclipse.aether.transport.wagon.WagonTransporter.execute(WagonTransporter.java:435) ~[?:?]
			at shaded.org.eclipse.aether.transport.wagon.WagonTransporter.get(WagonTransporter.java:412) ~[?:?]
			at shaded.org.eclipse.aether.connector.basic.BasicRepositoryConnector$GetTaskRunner.runTask(BasicRepositoryConnector.java:456) ~[?:?]
			at shaded.org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run(BasicRepositoryConnector.java:363) ~[?:?]
			... 21 more
	Caused by: shaded.org.eclipse.aether.resolution.ArtifactResolutionException: Error resolving artifact org.openhab.ui.bundles:org.openhab.ui.dashboard:jar:2.5.0
		at shaded.org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:413) ~[?:?]
		at shaded.org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:215) ~[?:?]
		at shaded.org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact(DefaultArtifactResolver.java:192) ~[?:?]
		at shaded.org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveArtifact(DefaultRepositorySystem.java:247) ~[?:?]
		at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:767) ~[?:?]
		at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:657) ~[?:?]
		at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:598) ~[?:?]
		at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:565) ~[?:?]
		at org.apache.karaf.features.internal.download.impl.MavenDownloadTask.download(MavenDownloadTask.java:52) ~[?:?]
		at org.apache.karaf.features.internal.download.impl.AbstractRetryableDownloadTask.run(AbstractRetryableDownloadTask.java:60) ~[?:?]
		... 7 more

version.properties

openHAB Distribution Version Information
----------------------------------------
build-no        : Release Build
online-repo     : https://api.bintray.com/maven/openhab/mvn/online-repo/2.5

Repository        Version
----------------------------------------
openhab-distro  : 2.5.0
openhab-core    : 2.5.0
openhab1-addons : 1.14.0
openhab2-addons : 2.5.0
karaf           : 4.2.7

martn · December 16, 2019, 11:22am

Exact same here. After update of my Docker Container to openHAB 2.5 i have reboot loop with exact same log entries.

BrutalBirdie · December 16, 2019, 11:34am

Is this GitHub Issue related to this post?

EDIT:
No I think not. Sorry then.

mpuff · December 16, 2019, 11:34am

delete the userdata folder, i have done this, than it was working. (only do this if you have file based config)

BrutalBirdie · December 16, 2019, 11:41am

WARNING

You are suggesting just delete the whole userdata folder?

Please don’t just delete it.
Make sure you made a backup first!

With deleted the userdata folder completely remove every configuration done via PaperUI.
All Items, Things, Channels yada yada yada.
Unskilled users may just follow your advice and run into weeks of work destroyed.

mpuff · December 16, 2019, 11:47am

i have done these configuration in the config folder (items file, sitempa file …).
backups should be done regularly anyway!

BrutalBirdie · December 16, 2019, 11:50am

This means that you do textual configuration via *.items, *.things and so on files.

Many people do a lot of configuration via PaperUI, and PaperUI stores everything in userdata/jsondb/ and more places in userdata.

I have not one *.items file, all my items are created via PaperUI.

All I want to say with my Warning comment is that you need to be careful with what you suggest as solution.
Many users will follow your solution (since it worked for you) without thinking.
And YES I agree backups should be done regularly, but I would never expect this behavior as default.

I just want to protect some of my friends, I know to well who would/will follow such advice and destroy there work.

vlad1986 · December 16, 2019, 11:53am

I will test this is have done all my configuration the manual way so the the jsondb is empty for me as well, but thank you for this tip could be painfully

BrutalBirdie · December 16, 2019, 12:13pm

Oh btw, I could not recreate this behavior.
I do most of my config via. PaperUI.

I use docker-compose and just swapped out

- image: "openhab/openhab:2.4.0"
---
+ image: "openhab/openhab:2.5.0"

Run

docker-compose -f docker-compose.yml up --remove-orphans -d

And everything is working.

How does your docker setup look like?

vlad1986 · December 16, 2019, 2:13pm

It works now as well i also delete the complete userdata folder and restart and after i install all needed addons it works again…

version: '3'
volumes:
  openhab_addons:
    driver_opts:
      type: none
      device: /sharedfolders/Docker/openhab/addons
      o: bind
  openhab_conf:
    driver_opts:
      type: none
      device: /sharedfolders/Docker/openhab/conf
      o: bind
  openhab_userdata:
    driver_opts:
      type: none
      device: /sharedfolders/Docker/openhab/userdata
      o: bind
  zigbee2mqttData:
    driver_opts:
      type: none
      device: /sharedfolders/Docker/openhab/zigbee2mqtt
      o: bind
  mqttData:
    driver_opts:
      type: none
      device: /sharedfolders/Docker/openhab/mosquitto/data
      o: bind
  mqttConfig:
    driver_opts:
      type: none
      device: /sharedfolders/Docker/openhab/mosquitto/config/mosquitto.conf
      o: bind
  mqttLog:
    driver_opts:
      type: none
      device: /sharedfolders/Docker/openhab/mosquitto/log
      o: bind

services:
  mqtt:
    image: eclipse-mosquitto:latest
    restart: always
    network_mode: host
    ports:
      - "1883:1883"
      - "9001:9001"
    volumes:
      - /sharedfolders/Docker/openhab/mosquitto/config/mosquitto.conf:/mosquitto/config/mosquitto.conf
      - mqttData:/mosquitto/data
      - mqttLog:/mosquitto/log
  zigbee2mqtt:
    image: koenkk/zigbee2mqtt
    restart: always
    volumes:
      - zigbee2mqttData:/app/data
    devices:
      - /dev/ttyACM0:/dev/ttyACM0
    network_mode: host
    environment:
     - TZ=Europe/Amsterdam
  openhab:
    image: openhab/openhab:latest
    restart: always
    network_mode: host
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
      - openhab_addons:/openhab/addons
      - openhab_conf:/openhab/conf
      - openhab_userdata:/openhab/userdata
    devices:
      - /dev/ttyACM0:/dev/ttyACM0
    environment:
      OPENHAB_HTTP_PORT: 8080
      OPENHAB_HTTPS_PORT: 8443
      EXTRA_JAVA_OPTS: "-Duser.timezone=Europe/Berlin"

BrutalBirdie · December 16, 2019, 2:38pm

Oh yea the userdata folder also includes all addons.
Oh you do volumes first? I think that`s OK but I am used to see the services first before the volumes.

I will just quote @wborn

vlad1986 · December 16, 2019, 3:43pm

yeah maybe i should change this so far it worked fine.
I saw this post but iam blind? I saw Upgrade procedure for different installation methods except Docker so what steps are needed for docker?

rlkoshak · December 16, 2019, 4:37pm

When you start a container that differs from the version indicated by the files in userdata, the entrypoint.sh script will automatically perform the steps to upgrade/downgrade. It usually just involves backing up and then deleting the contents of the userdata/cache and userdata/tmp folders. This will force a reinstall of the bindings.

Keep in mind though that, as was mentioned on the other thread, perform any manual steps that may be required. Look in userdata/logs for a backup.log file which will contain the same list of warnings that you get when performing an apt update. Any breaking changes, known stuff that is not working, etc will be listed there.

tiborkovacs · December 21, 2019, 6:34pm

I have experienced the same issue and had to downgrade to 2.4. QNAP container station, docker container. “Error downloading mvn:org.openhab.ui.bundles/org.openhab.ui.dashboard/2.5.0”
Has this problem been resolved? Kai suggested to add the public key for the APT repos by adding “wget -qO - ‘https://bintray.com/user/downloadSubjectPublicKey?username=openhab’ | sudo apt-key add -”, however apt-key does not seem to work on QNAP.
Any suggestions?
Thanks Tibor

Falk · March 29, 2020, 1:09pm

By upgrading from 2.5 to 2.5.3 I deleted the userdata folder and … it was a (small) nightmare. Several hours of reconfiguring over 100 things, access to online services, openhab itself …
I really hope next upgrade attempts do not require this. Otherwise installing a non-docker version of OH, upgrading and then going back to docker may be my next approach.

hmerk · March 29, 2020, 4:18pm

You should not delete complete userdata folder, just temp and cache.

rlkoshak · March 30, 2020, 3:55pm

Why did you delete the userdata folder? If you saw instructions somewhere to do that on this forum than please let us know so we can correct it. If it was not on this forum those instructions are flat out wrong and should not be trusted.

You don’t need to manually delete anything when upgrading a Docker image. When the new container starts up it will see that it’s internal version number is different from the version number in your userdata folder and delete the contents of cache and tmp for you. You shouldn’t be deleting anything yourself.

Falk · March 30, 2020, 8:24pm

Well, i did follow this conversation for quite a while. One of my first attempts was to delete just temp and cache. This did not change the error message or the loop issue.
And yes, I interpreted this thread in a way that some did delete the whole folder (with all problems that come with this). At least that worked and installed/configured the new version but created pain on the other side which is also referred to in this thread.

My userdata folder has been copied from a pure “linux” installation to the docker based version. I assumed problems with this and finally decided for the plain vanilla approach.

Glad to read this will not happen again if just temp and cache folders are being deleted. Happy to report accordingly in case this happens again with the next update.

openls · April 9, 2020, 6:41pm

I am experiencing the same issue. I run openHAB inside a Docker container on my Synology NAS. I use Portainer to create containers, as the Synology Docker app does not allow advanced configurations like forwarding USB devices.

What I noticed is the following: When the openHAB container gets created, a lot of environment variables like OPENHAB_VERSION are created and somehow persisted in the container. These variables do not change anymore after I recreate the container with a new image, even if I manually delete them before deployment. I guess the same problem applies if I use watchtower. So after deployment openHAB makes the upgrade but does NOT update the environment variables and runs the same update again and again…

The only way for me to circumvent that is to create a completely new container and set all volumes, variables and resources again after every openHAB update. This seems rather odd, openHAB is the only Docker container I use which adds its own environment variables after deployment and has this problem.

I am no Docker or openHAB expert, so this might not be the real problem but I hope someone with more knowledge about this can comment here and at least tell me if this is a bug or not the intended way of updating the openHAB Docker container.

rlkoshak · April 9, 2020, 10:39pm

One thing to understand about Docker is a container is ephemeral and it’s intended to be immutable. When you upgrade the image, you will need to create a brand new container when you start it. In order to persist data you therefore have volumes which contain the data you want to keep from one container to the next.

Those environment variables like OPENHAB_VERSION are created when the Docker image was created. They exist even before the container is created. If those variables are not changing after you recreate the container, you are either not actually recreating the container, or you are not recreating the container with the new Docker image.

It’s impossible for it to make the upgrade and not update the environment variables. Both are done before you ever download the image. What upgrading there is that occurs is if the version of OH that is in the volume you mounted to userdata is different from the OPENHAB_VERSION, the entrypoint.sh script will backup your userdata folder and clear the cache.

This is literally what is required to update any container. When you want to move to a new Docker image, you must create a new container based on that image. I would imagine that Portainer or Watchtower or whatever you use has a way to say “create a new container using this already existing cofiguration” and that is what you need to do to upgrade.