[SOLVED] Migration to docker failed

1

Hi,

i finally took the plunge in an attempt to move away from the old rpi3 and oh 4.3.9 to a newer platform.

so, i did openhab-cli backup --full (while OH service is stopped)

backup was 360mb

moved that file to a docker host

spun up temp container for restore purposes (manipulated so it doesnt start java service so i can actually restore backup

services:
 openhab-restore:
    image: openhab/openhab:4.3.9-debian
    container_name: openhab-restore

    entrypoint: /bin/bash
    stdin_open: true
    tty: true

    volumes:
      - ./conf:/openhab/conf
      - ./userdata:/openhab/userdata
      - ./addons:/openhab/addons
      - ./backups:/backups

i run the restore

########################################## openHAB restore script ########################################## Using '/openhab/conf' as conf folder... Using '/openhab/userdata' as userdata folder... awk: cannot open /openhab/userdata/etc/version.properties (No such file or directory) Making Temporary Directory Extracting zip file to temporary folder. Backup Information: ------------------- Backup Version | 4.3.9 (You are on ) Backup Timestamp | 26_08_18-22_06_03 Config belongs to user | openhab from group | openhab Your current configuration will become owned by 999:986. Any existing files with the same name will be replaced. Any file without a replacement will be deleted. Okay to Continue? [y/N

for some weird reason FULL backup didnt backup this file: version.properties so i copied it manually.

after that started the normal docker container with the 4.3.9 version and it craps out.

any clue? corrupted backup? run backup again?

 âś” Container openhab Started                                                                               0.2s
+ IFS='
        '
++ find /usr/lib/jvm -mindepth 1 -maxdepth 1 -type d
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
+ '[' unlimited = unlimited ']'
Configuring Java unlimited strength cryptography policy...
+ echo 'Configuring Java unlimited strength cryptography policy...'
+ sed -i 's/^crypto.policy=limited/crypto.policy=unlimited/' /usr/lib/jvm/java-17-openjdk-amd64/conf/security/java.security
+ capsh --print
+ grep -E Current:.+,cap_net_admin,cap_net_raw,.+
+ rm -f '/var/lock/LCK..*'
+ rm -f /openhab/userdata/tmp/instances/instance.properties
+ NEW_USER_ID=999
+ NEW_GROUP_ID=986
+ echo 'Starting with openhab user id: 999 and group id: 986'
+ id -u openhab
Starting with openhab user id: 999 and group id: 986
++ getent group 986
Create group openhab with id 986
+ '[' -z '' ']'
+ echo 'Create group openhab with id 986'
+ groupadd -g 986 openhab
Create user openhab with id 999
+ echo 'Create user openhab with id 999'
+ adduser -u 999 --disabled-password --gecos '' --home /openhab --gid 986 openhab
adduser: Warning: The home dir /openhab you specified already exists.
Adding user `openhab' ...
Adding new user `openhab' (999) with group `openhab (986)' ...
useradd warning: openhab's uid 999 outside of the UID_MIN 1000 and UID_MAX 60000 range.
adduser: The home directory `/openhab' already exists.  Not touching this directory.
adduser: Warning: The home directory `/openhab' does not belong to the user you are currently creating.
Adding new user `openhab' to supplemental / extra groups `users' ...
Adding user `openhab' to group `users' ...
+ groupadd -g 11 audio2
+ groupadd -g 14 uucp2
+ groupadd -g 16 dialout2
+ groupadd -g 17 audio3
+ groupadd -g 18 dialout3
+ groupadd -g 32 uucp3
+ groupadd -g 63 audio4
+ groupadd -g 490 dialout4
+ groupadd -g 492 audio5
+ groupadd -g 997 gpio
+ adduser openhab audio
Adding user `openhab' to group `audio' ...
Done.
+ adduser openhab audio2
Adding user `openhab' to group `audio2' ...
Done.
+ adduser openhab audio3
Adding user `openhab' to group `audio3' ...
Done.
+ adduser openhab audio4
Adding user `openhab' to group `audio4' ...
Done.
+ adduser openhab audio5
Adding user `openhab' to group `audio5' ...
Done.
+ adduser openhab dialout
Adding user `openhab' to group `dialout' ...
Done.
+ adduser openhab dialout2
Adding user `openhab' to group `dialout2' ...
Done.
+ adduser openhab dialout3
Adding user `openhab' to group `dialout3' ...
Done.
+ adduser openhab dialout4
Adding user `openhab' to group `dialout4' ...
Done.
+ adduser openhab gpio
Adding user `openhab' to group `gpio' ...
Done.
+ adduser openhab uucp
Adding user `openhab' to group `uucp' ...
Done.
+ adduser openhab uucp2
Adding user `openhab' to group `uucp2' ...
Done.
+ adduser openhab uucp3
Adding user `openhab' to group `uucp3' ...
Done.
+ initialize_volume /openhab/conf /openhab/dist/conf
+ volume=/openhab/conf
+ source=/openhab/dist/conf
++ ls -A /openhab/conf
+ '[' -z 'automation
html
icons
items
misc
persistence
rules
scripts
services
sitemaps
sounds
tags
things
transform' ']'
+ initialize_volume /openhab/userdata /openhab/dist/userdata
+ volume=/openhab/userdata
+ source=/openhab/dist/userdata
++ ls -A /openhab/userdata
+ '[' -z '192.168.4.119_lgtvallapps.xml
192.168.4.119_lgtvallchannels.xml
192.168.4.5_lgtvallapps.xml
192.168.4.5_lgtvallchannels.xml
backups
cache
Californium.properties
config
etc
hs_err_pid469.log
jsondb
logs
marketplace
miio
mqttembedded.bin
openhab.id_rsa
openhab.id_rsa.pub
persistence
secrets
smarthome:send
sony
tail
tmp
zigbee
zwave' ']'
++ cmp /openhab/userdata/etc/version.properties /openhab/dist/userdata/etc/version.properties
+ '[' '!' -z ']'
+ chown -R openhab:openhab /openhab
+ sync
+ '[' -d /etc/cont-init.d ']'
+ sync
+ '[' false == false ']'
++ IFS=' '
++ echo su-exec openhab tini -s ./start.sh
+ '[' 'su-exec openhab tini -s ./start.sh' == 'su-exec openhab tini -s ./start.sh' ']'
+ command=($@ server)
+ exec su-exec openhab tini -s ./start.sh server
Launching the openHAB runtime...
Cannot parse null string
Error occurred shutting down framework: java.lang.NumberFormatException: Cannot parse null string
java.lang.NumberFormatException: Cannot parse null string
        at java.base/java.lang.Integer.parseInt(Integer.java:630)
        at java.base/java.lang.Integer.parseInt(Integer.java:786)
        at org.apache.karaf.main.ConfigProperties.<init>(ConfigProperties.java:251)
        at org.apache.karaf.main.Main.updateInstancePidAfterShutdown(Main.java:232)
        at org.apache.karaf.main.Main.main(Main.java:197)
2

It wouldn’t. That file is how OH identifies to itself what version it’s running. You don’t want to replace that file when restoring a backup or else OH will think it’s running a different version that it really is.

The error is coming from parsing an integer from a config file. Since you replaced a config file you should not have, that’s going to be your first suspect.

Restore that file from the default set of configs in the container.

docker exec -it openhab cp dist/userdata/etc/version.properties userdata/etc/version.properties

Then restart OH.

Or wipe out the current volumes, restore the backup again but this time don’t mess with versions.properties.

3

i didnt replace the file, the initial OH startup error was due to not having version.properties file in ./etc. I guess it failed to copy it from ./dist if that was his plan.

so then i gave him the file (which should be the same file specifying the version i’m running (which is the same version that is on the container. I’m migrating 4.3.9 → 4.3.9

4

this is the failure without version.properties

 âś” Container openhab Started                                                                               0.3s
+ IFS='
        '
++ find /usr/lib/jvm -mindepth 1 -maxdepth 1 -type d
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
+ '[' unlimited = unlimited ']'
+ echo 'Configuring Java unlimited strength cryptography policy...'
+ sed -i 's/^crypto.policy=limited/crypto.policy=unlimited/' /usr/lib/jvm/java-17-openjdk-amd64/conf/security/java.security
Configuring Java unlimited strength cryptography policy...
+ capsh --print
+ grep -E Current:.+,cap_net_admin,cap_net_raw,.+
+ rm -f '/var/lock/LCK..*'
+ rm -f /openhab/userdata/tmp/instances/instance.properties
Starting with openhab user id: 999 and group id: 986
+ NEW_USER_ID=999
+ NEW_GROUP_ID=986
+ echo 'Starting with openhab user id: 999 and group id: 986'
+ id -u openhab
++ getent group 986
+ '[' -z '' ']'
+ echo 'Create group openhab with id 986'
+ groupadd -g 986 openhab
Create group openhab with id 986
+ echo 'Create user openhab with id 999'
+ adduser -u 999 --disabled-password --gecos '' --home /openhab --gid 986 openhab
Create user openhab with id 999
adduser: Warning: The home dir /openhab you specified already exists.
Adding user `openhab' ...
Adding new user `openhab' (999) with group `openhab (986)' ...
useradd warning: openhab's uid 999 outside of the UID_MIN 1000 and UID_MAX 60000 range.
adduser: The home directory `/openhab' already exists.  Not touching this directory.
adduser: Warning: The home directory `/openhab' does not belong to the user you are currently creating.
Adding new user `openhab' to supplemental / extra groups `users' ...
Adding user `openhab' to group `users' ...
+ groupadd -g 11 audio2
+ groupadd -g 14 uucp2
+ groupadd -g 16 dialout2
+ groupadd -g 17 audio3
+ groupadd -g 18 dialout3
+ groupadd -g 32 uucp3
+ groupadd -g 63 audio4
+ groupadd -g 490 dialout4
+ groupadd -g 492 audio5
+ groupadd -g 997 gpio
+ adduser openhab audio
Adding user `openhab' to group `audio' ...
Done.
+ adduser openhab audio2
Adding user `openhab' to group `audio2' ...
Done.
+ adduser openhab audio3
Adding user `openhab' to group `audio3' ...
Done.
+ adduser openhab audio4
Adding user `openhab' to group `audio4' ...
Done.
+ adduser openhab audio5
Adding user `openhab' to group `audio5' ...
Done.
+ adduser openhab dialout
Adding user `openhab' to group `dialout' ...
Done.
+ adduser openhab dialout2
Adding user `openhab' to group `dialout2' ...
Done.
+ adduser openhab dialout3
Adding user `openhab' to group `dialout3' ...
Done.
+ adduser openhab dialout4
Adding user `openhab' to group `dialout4' ...
Done.
+ adduser openhab gpio
Adding user `openhab' to group `gpio' ...
Done.
+ adduser openhab uucp
Adding user `openhab' to group `uucp' ...
Done.
+ adduser openhab uucp2
Adding user `openhab' to group `uucp2' ...
Done.
+ adduser openhab uucp3
Adding user `openhab' to group `uucp3' ...
Done.
+ initialize_volume /openhab/conf /openhab/dist/conf
+ volume=/openhab/conf
+ source=/openhab/dist/conf
++ ls -A /openhab/conf
+ '[' -z 'automation
html
icons
items
misc
persistence
rules
scripts
services
sitemaps
sounds
tags
things
transform' ']'
+ initialize_volume /openhab/userdata /openhab/dist/userdata
+ volume=/openhab/userdata
+ source=/openhab/dist/userdata
++ ls -A /openhab/userdata
+ '[' -z '192.168.4.119_lgtvallapps.xml
192.168.4.119_lgtvallchannels.xml
192.168.4.5_lgtvallapps.xml
192.168.4.5_lgtvallchannels.xml
backups
cache
Californium.properties
config
etc
hs_err_pid469.log
jsondb
logs
marketplace
miio
mqttembedded.bin
openhab.id_rsa
openhab.id_rsa.pub
persistence
secrets
smarthome:send
sony
tail
tmp
zigbee
zwave' ']'
++ cmp /openhab/userdata/etc/version.properties /openhab/dist/userdata/etc/version.properties
cmp: /openhab/userdata/etc/version.properties: No such file or directory
+ '[' '!' -z ']'
+ chown -R openhab:openhab /openhab
+ sync
+ '[' -d /etc/cont-init.d ']'
+ sync
+ '[' false == false ']'
++ IFS=' '
++ echo su-exec openhab tini -s ./start.sh
+ '[' 'su-exec openhab tini -s ./start.sh' == 'su-exec openhab tini -s ./start.sh' ']'
+ command=($@ server)
+ exec su-exec openhab tini -s ./start.sh server
Launching the openHAB runtime...
Cannot parse null string
Error occurred shutting down framework: java.lang.NumberFormatException: Cannot parse null string
java.lang.NumberFormatException: Cannot parse null string
        at java.base/java.lang.Integer.parseInt(Integer.java:630)
        at java.base/java.lang.Integer.parseInt(Integer.java:786)
        at org.apache.karaf.main.ConfigProperties.<init>(ConfigProperties.java:251)
        at org.apache.karaf.main.Main.updateInstancePidAfterShutdown(Main.java:232)
        at org.apache.karaf.main.Main.main(Main.java:197)
5

and now i did what you suggested, copied file from /dist because obviously OH fails to copy it, but of course, behaviour is identical as the first failure I posted, because the file is identical to the file from my backup.

root@47d676a01afe:/openhab# cat userdata/etc/version.properties
openHAB Distribution Version Information
----------------------------------------
build-no        : Release Build
online-repo     : https://openhab.jfrog.io/openhab/libs-release

Repository        Version
----------------------------------------
openhab-distro  : 4.3.9
openhab-core    : 4.3.9
openhab-addons  : 4.3.9
karaf           : 4.4.6
root@47d676a01afe:/openhab# exit
6

Not an expert like @rlkoshak , but if it was me I might be tempted to spin up a new OH4.3.9 and let Docker create all the volumes then just overwrite the userdata & conf folders after OH is up and running. When I create a test OH, I do that. Also probably possible to restore the backup with a docker exec -it, but I haven’t done that

7

bizzaro world…this time I injected version.properties in my backup.zip

(Because last time i got complaint from restore script its missing

########################################## 

openHAB restore script 

########################################## 

Using '/openhab/conf' as conf folder... 

Using '/openhab/userdata' as userdata folder... 
awk: cannot open /openhab/userdata/etc/version.properties 
(No such file or directory)

so, this time, restore script did not complain about missing version.properties, nor did the startup complain about it,

great success, it works :slight_smile:

8

@rlkoshak wouldn’t it be logical that version.properties is INCLUDED in the backup, so that restore script can compare your version from which you brought in the backup with the actual running version to which you are trying to restore the data into.

This way, i came with my backup, the new vanilla system was like WTF is this, who are you, backup of OH2? OH5? I’m failing you, goodbye.

But when I injected the version.properties file, suddenly all is good.

9

Did you start the container first and then restore? Or did you restore and then start the container for the first time?

Again, it doesn’t fail to copy it. It deliberately doesn’t copy it. It’s not just forgotten or missed. There is logic to deliberately exclude it.

Which is the exact same error with the version.properties which is interesting.

That’s definitely going to be the more tested work flow as most of the time when restoring the backup it will be over a config that is already there.

That would work for Docker and break every other way to install openHAB. versions.properties is deliberately excluded from the backup for this reason.

The general workflow to do what you did in a way that would work would have been to:

  1. start the docker container with empty volumes
  2. copy your backup file to userdata/backup
  3. docker exec -it runtime/bin/restore userdata/backup/<nameOfBackup.zip
  4. docker restart openhab

Alternatively, following @apella12’s suggestion

  1. start the docker container with the empty volumes
  2. unzip and distribute your backup as appropriate after the container prepopulates the volumes
  3. docker restart openhab

The first workflow will be the most reliable as if running upgradeTool ever becomes part of the restore process, that’s where it will be run. The second approach you’d have to remember to run upgradeTool manually.

Unzipping a backup and just pointing a new container to that from the start is an unexpected, or at least a rarely tried work flow. It should probably fail with a nicer error than that though (you can open an issue).

The presence and contents of version.properties is how the entrypoint.sh knows what to do when the container is a different version from the configs. But entryPoint.sh is only a thing for Docker. What it does is compare the version in dist with the version in userdata/etc and if they are different it goes through the upgrade process (taking a backup, clearing the cache, running upgradeTool, etc). It appears entryPoint.sh always expects the file to be there and since the normal backup deliberately never has this file it fails if you restore first before initializing the mounted volumes.

10

I started the container, but manipulated entrypoint so OH cannot start. so, to answer your question, I guess no, OH did not run to initialize the folders before restore procedure. Honestly, I can’t remember if i tried to run proper (not manipulated) OH container, maybe I run it for few seconds just to see compose is working.

So if I understood correctly, what I did (implant version.properties to the backup.zip) would crap out normal bare metal installations, but docker likes it.

OK, all in all, I understand this backup restore process is not really optimized, or even compatible with docker.

I mean, you cannot even use restore on docker because it asks you to stop the service.

so once you start manipulating docker image into tricking it to do things, you’re in unchartered waters, and on your own.

I really don’t mind tinkering and getting it to work one way or another.

But we all have to accept that these kinds of things are precisely the reason we are not widely adopted like HomeAssistant, but remain a niche product for a handful of power users, engineers and tinkerers.

Regular people run docker containers on home NAS devices nowadays all by clicking some gui like portainer, gone are the days when docker was geeky. But there is no procedure which normal people would be able to follow to dockerize their RPI OH installation..

11

I didn’t run it so missed that. But in that case you could do a one-off run of the container to do the restore. Or just do it manually.

Once you’ve started the container once and let it initialize the volumes you could stop the container and pass a different command as one of the options:

docker run --rm -it \
  -v openhab_addons:/openhab/addons \
  -v openhab_conf:/openhab/conf \
  -v openhab_userdata:/openhab/userdata \
  --entrypoint runtime/bin/restore userdata/backup/nameOfYourBackup.zip \
  openhab/openhab:latest

And then run it normally when that completes.

The container also has a feature where you can mount a folder of scripts over /etc/cont-init.d and the entryPoint.sh script will run those scripts before openHAB is started. I use this approach to install ffmpeg into the container so the IP Camera add-on works. You could even keep this mechanism in place and the container will auto restore when there is a zip file in the folder the script uses.

#!/bin/bash
# If a backup zip exists, run the restore script and then remove it
if [ -f "/openhab/userdata/toRestore/backup.zip" ]; then
    echo "Found backup file. Starting restore..."
    /openhab/runtime/bin/restore /openhab/userdata/toRestore/backup.zip
    rm /openhab/userdata/toRestore/backup.zip // or to be safe move it to userdata/backup
fi

As you can see, there are ways to do this without treading into uncharted waters. Both of these mechanisms are standard parts of Docker and/or the openHAB Image. The cont-init-d approach with the permanent restore script would be particularly well suited for Portainer/NAS type setups.

If you’ve got a better way PRs are always welcome. My focus is always on what is now, not on what could/should be.

But I’ll also say that HomeAssistant address this sort of thing by taking away options and choices from their users. It’s simpler because you have to do it their way or you’re gonna have a bad time.

I just outlined two. There are more.

What won’t work is just unzipping a backup and pointing the container to those folders. However,

  • letting the container start and then unzipping the folders over those prepopulated volumes
  • running the container with a different entrypoint to run the restore
  • mounting a folder of pre-run scripts to the container to do the restore when a file is present in a watch folder

will all work, even on a Portainer or NAS deployment of openHAB regardless of what type of machine/install the backup was taken from in the first place. It will even work if the backup was taken from a different version of OH.

But if you can come up with a way to make it also work when you just unzip the backup and use those as the volumes I’m sure that would be a most welcome improvement. I looked into it once upon a time and couldn’t come up with a good approach, but that doesn’t mean there isn’t one.

12

well, i dont know about other suggested methods, but this one didnt work for me, thats what I’m trying to explain.

  1. run the container with different entry point
  2. run the restore - restore complains it cannot find version.properties
  3. after restore, OH fails to start.

Now when I think about it, was the restore script complaining about no version. properties in the backup or in /etc folder? hmm…maybe really the process should be - let OH container run without obstructing the entrypoint for few minutes, to create this file.

then kill it. then run the modified entrypoint container, then run the restore.

Probably restore was confused because OH never run and never created anything, so it was not clear what version is running…

13

Right, that’s why I said

Once you’ve started the container once and let it initialize the volumes you could stop the container and pass a different command as one of the options:

In all but one of the approaches you must start the container and let it initialize the mounted volumes. Then stop the container and proceed with the restore.

The one exception is the cont-init.d approach. That will get called after the regular entryPoint.sh initializes the volumes but before openHAB is started. So restore won’t have any problem with openHAB being already running.

But I’ll also note you are not modifying the entry point script nor the official Image in any of these approaches. When you use docker run or docker compose you can configure the container to run some other command instead of the entrypoint.sh.

docker run --rm -it \
  -v openhab_addons:/openhab/addons \
  -v openhab_conf:/openhab/conf \
  -v openhab_userdata:/openhab/userdata \
  --entrypoint runtime/bin/restore userdata/backup/nameOfYourBackup.zip \ # <- Restore instead of starting OH
  openhab/openhab:latest

Also note the -rm will remove this temporary container once it’s done its job. That’s why we don’t give it a name. It’s a temporary container, so the name doesn’t matter.

The image is configured to run the cont-init.d scripts as a deliberate feature.You are not stepping outside the normal and expected path with any of these.

I recommend the cont-init.d approach over all but any of them will work.