Testing auto backup

narf27 · September 18, 2020, 12:01pm

like this?

#!/usr/bin/env bash -x

BASEDIR=/opt/openhabian
CONFIGFILE=/etc/openhabian.conf

source $CONFIGFILE

source ${BASEDIR}/functions/helpers.bash
source ${BASEDIR}/functions/backup.bash

mirror_SD "$@"

with “-x”:

[13:59:19] root@ohab2:/media/influxdbstick/data# /usr/local/sbin/mirror_SD "raw" /dev/sda
/usr/bin/env: ‘bash -x’: No such file or directory
/usr/bin/env: use -[v]S to pass options in shebang lines

withoud “-x”

[13:32:08] root@ohab2:/media/influxdbstick/data# /usr/local/sbin/mirror_SD "raw" /dev/sda
blockdev: cannot open : No such file or directory
FAILED (bad destination)

mstormi · September 18, 2020, 12:02pm

then use “set -x” as the 2nd line

narf27 · September 18, 2020, 12:43pm

okay:

[14:42:46] root@ohab2:/media/influxdbstick/data# /usr/local/sbin/mirror_SD "raw" /dev/sda
+ BASEDIR=/opt/openhabian
+ CONFIGFILE=/etc/openhabian.conf
+ source /etc/openhabian.conf
++ hostname=openhab
++ username=openhabian
++ timeserver=0.pool.ntp.org
++ locales='en_US.UTF-8 de_DE.UTF-8'
++ system_default_locale=en_US.UTF-8
++ wifi_ssid=
++ wifi_psk=
++ wifi_country=
++ ipv6=enable
++ repositoryurl=https://github.com/openhab/openhabian
++ clonebranch=master
++ debugmode=maximum
++ java_opt=Zulu8-32
+ source /opt/openhabian/functions/helpers.bash
++ ESC='\033['
++ COL_DEF='\033[39;49;00m'
++ COL_RED='\033[31;01m'
++ COL_GREEN='\033[32;01m'
++ COL_YELLOW='\033[33;01m'
++ COL_BLUE='\033[34;01m'
++ COL_MAGENTA='\033[35;01m'
++ COL_CYAN='\033[36;01m'
++ COL_LGRAY='\033[37;01m'
++ COL_DGRAY='\033[90;01m'
++ export COL_DEF COL_RED COL_GREEN COL_YELLOW COL_BLUE COL_MAGENTA COL_CYAN COL_LGRAY COL_DGRAY
+ source /opt/openhabian/functions/backup.bash
+ mirror_SD raw /dev/sda
+ local src=/dev/mmcblk0
+ local dest
+ local start
+ local syncMount=/storage/syncmount
+ [[ -n '' ]]
+ dest=
+ [[ /dev/mmcblk0 == '' ]]
++ blockdev --getsize64 ''
blockdev: cannot open : No such file or directory
+ [[ -n '' ]]
+ echo 'FAILED (bad destination)'
FAILED (bad destination)
+ return 1

mstormi · September 18, 2020, 2:38pm

Fix L361 in backup.bash to read

 dest="${2:-${backupdrive}}"

and in L393 change 2 to 3

narf27 · September 18, 2020, 7:04pm

new error:

[20:26:41] root@ohab2:/media/influxdbstick/data# /usr/local/sbin/mirror_SD "raw" /dev/sda
+ BASEDIR=/opt/openhabian
+ CONFIGFILE=/etc/openhabian.conf
+ source /etc/openhabian.conf
++ hostname=openhab
++ username=openhabian
++ timeserver=0.pool.ntp.org
++ locales='en_US.UTF-8 de_DE.UTF-8'
++ system_default_locale=en_US.UTF-8
++ wifi_ssid=
++ wifi_psk=
++ wifi_country=
++ ipv6=enable
++ repositoryurl=https://github.com/openhab/openhabian
++ clonebranch=master
++ debugmode=maximum
++ java_opt=Zulu8-32
+ source /opt/openhabian/functions/helpers.bash
++ ESC='\033['
++ COL_DEF='\033[39;49;00m'
++ COL_RED='\033[31;01m'
++ COL_GREEN='\033[32;01m'
++ COL_YELLOW='\033[33;01m'
++ COL_BLUE='\033[34;01m'
++ COL_MAGENTA='\033[35;01m'
++ COL_CYAN='\033[36;01m'
++ COL_LGRAY='\033[37;01m'
++ COL_DGRAY='\033[90;01m'
++ export COL_DEF COL_RED COL_GREEN COL_YELLOW COL_BLUE COL_MAGENTA COL_CYAN COL_LGRAY COL_DGRAY
+ source /opt/openhabian/functions/backup.bash
+ mirror_SD raw /dev/sda
+ local src=/dev/mmcblk0
+ local dest
+ local start
+ local syncMount=/storage/syncmount
+ [[ -n '' ]]
+ dest=/dev/sda
+ [[ /dev/mmcblk0 == \/\d\e\v\/\s\d\a ]]
++ blockdev --getsize64 /dev/sda
+ [[ -n 31914983424 ]]
++ mount
++ grep /dev/sda
+ [[ -n '' ]]
+ [[ raw == \r\a\w ]]
+ echo 'Creating a raw partition copy, be prepared this may take long such as 20-30 minutes for a 16 GB SD card'
Creating a raw partition copy, be prepared this may take long such as 20-30 minutes for a 16 GB SD card
+ cond_redirect dd if=/dev/mmcblk0 bs=1M of=/dev/sda
+ [[ -n '' ]]
+ echo -e '\n\033[90;01m$ dd if=/dev/mmcblk0 bs=1M of=/dev/sda \033[39;49;00m'

$ dd if=/dev/mmcblk0 bs=1M of=/dev/sda
+ dd if=/dev/mmcblk0 bs=1M of=/dev/sda
15193+1 records in
15193+1 records out
15931539456 bytes (16 GB, 15 GiB) copied, 980.984 s, 16.2 MB/s
+ return 0
+ cond_redirect fsck -y -t vfat /dev/sda1
+ [[ -n '' ]]
+ echo -e '\n\033[90;01m$ fsck -y -t vfat /dev/sda1 \033[39;49;00m'

$ fsck -y -t vfat /dev/sda1
+ fsck -y -t vfat /dev/sda1
fsck from util-linux 2.33.1
fsck.fat 4.1 (2017-01-24)
0x41: Dirty bit is set. Fs was not properly unmounted and some data may be corrupt.
 Automatically removing dirty bit.
Performing changes.
/dev/sda1: 247 files, 110143/516190 clusters
+ return 1
+ echo 'FAILED (fsck /boot)'
FAILED (fsck /boot)
+ return 1

mstormi · September 18, 2020, 7:18pm

well not exactly an error as it corrected your filesystem.
Will change that nonetheless.

get & replace latest backup.bash

narf27 · September 18, 2020, 8:32pm

same:

[20:43:13] root@ohab2:/media/influxdbstick/data# wget https://raw.githubusercontent.com/openhab/openhabian/master/functions/backup.bash
...
backup.bash                                                 100%[========================================================================================================================================>]  26.62K  --.-KB/s    in 0.02s

2020-09-18 21:21:24 (1.62 MB/s) - ‘backup.bash’ saved [27264/27264]

[21:21:24] root@ohab2:/media/influxdbstick/data# cp backup.bash /opt/openhabian/functions/backup.bash
[21:21:41] root@ohab2:/media/influxdbstick/data# /usr/local/sbin/mirror_SD "raw" /dev/sda
+ BASEDIR=/opt/openhabian
+ CONFIGFILE=/etc/openhabian.conf
+ source /etc/openhabian.conf
++ hostname=openhab
++ username=openhabian
++ timeserver=0.pool.ntp.org
++ locales='en_US.UTF-8 de_DE.UTF-8'
++ system_default_locale=en_US.UTF-8
++ wifi_ssid=
++ wifi_psk=
++ wifi_country=
++ ipv6=enable
++ repositoryurl=https://github.com/openhab/openhabian
++ clonebranch=master
++ debugmode=maximum
++ java_opt=Zulu8-32
+ source /opt/openhabian/functions/helpers.bash
++ ESC='\033['
++ COL_DEF='\033[39;49;00m'
++ COL_RED='\033[31;01m'
++ COL_GREEN='\033[32;01m'
++ COL_YELLOW='\033[33;01m'
++ COL_BLUE='\033[34;01m'
++ COL_MAGENTA='\033[35;01m'
++ COL_CYAN='\033[36;01m'
++ COL_LGRAY='\033[37;01m'
++ COL_DGRAY='\033[90;01m'
++ export COL_DEF COL_RED COL_GREEN COL_YELLOW COL_BLUE COL_MAGENTA COL_CYAN COL_LGRAY COL_DGRAY
+ source /opt/openhabian/functions/backup.bash
+ mirror_SD raw /dev/sda
+ local src=/dev/mmcblk0
+ local dest
+ local start
+ local syncMount=/storage/syncmount
+ [[ -n '' ]]
+ dest=/dev/sda
+ [[ /dev/mmcblk0 == \/\d\e\v\/\s\d\a ]]
++ blockdev --getsize64 /dev/sda
+ [[ -n 31914983424 ]]
++ mount
++ grep /dev/sda
+ [[ -n '' ]]
+ [[ raw == \r\a\w ]]
+ echo 'Creating a raw partition copy, be prepared this may take long such as 20-30 minutes for a 16 GB SD card'
Creating a raw partition copy, be prepared this may take long such as 20-30 minutes for a 16 GB SD card
+ cond_redirect dd if=/dev/mmcblk0 bs=1M of=/dev/sda
+ [[ -n '' ]]
+ echo -e '\n\033[90;01m$ dd if=/dev/mmcblk0 bs=1M of=/dev/sda \033[39;49;00m'

$ dd if=/dev/mmcblk0 bs=1M of=/dev/sda
+ dd if=/dev/mmcblk0 bs=1M of=/dev/sda
15193+1 records in
15193+1 records out
15931539456 bytes (16 GB, 15 GiB) copied, 989.669 s, 16.1 MB/s
+ return 0
+ cond_redirect fsck -y -t vfat /dev/sda1
+ [[ -n '' ]]
+ echo -e '\n\033[90;01m$ fsck -y -t vfat /dev/sda1 \033[39;49;00m'

$ fsck -y -t vfat /dev/sda1
+ fsck -y -t vfat /dev/sda1
fsck from util-linux 2.33.1
fsck.fat 4.1 (2017-01-24)
0x41: Dirty bit is set. Fs was not properly unmounted and some data may be corrupt.
 Automatically removing dirty bit.
Performing changes.
/dev/sda1: 247 files, 110143/516190 clusters
+ return 1
+ echo 'FAILED (dirty fsck /dev/sda)'
FAILED (dirty fsck /dev/sda)
+ return 1

mstormi · September 18, 2020, 9:35pm

download & try again please

narf27 · September 18, 2020, 10:20pm

i still get the error regarding the “Dirty bit”:

[00:00:40] root@ohab2:/home/openhabian# /usr/local/sbin/mirror_SD "raw" /dev/sda
+ BASEDIR=/opt/openhabian
+ CONFIGFILE=/etc/openhabian.conf
+ source /etc/openhabian.conf
++ hostname=openhab
++ username=openhabian
++ timeserver=0.pool.ntp.org
++ locales='en_US.UTF-8 de_DE.UTF-8'
++ system_default_locale=en_US.UTF-8
++ wifi_ssid=
++ wifi_psk=
++ wifi_country=
++ ipv6=enable
++ repositoryurl=https://github.com/openhab/openhabian
++ clonebranch=master
++ debugmode=maximum
++ java_opt=Zulu8-32
+ source /opt/openhabian/functions/helpers.bash
++ ESC='\033['
++ COL_DEF='\033[39;49;00m'
++ COL_RED='\033[31;01m'
++ COL_GREEN='\033[32;01m'
++ COL_YELLOW='\033[33;01m'
++ COL_BLUE='\033[34;01m'
++ COL_MAGENTA='\033[35;01m'
++ COL_CYAN='\033[36;01m'
++ COL_LGRAY='\033[37;01m'
++ COL_DGRAY='\033[90;01m'
++ export COL_DEF COL_RED COL_GREEN COL_YELLOW COL_BLUE COL_MAGENTA COL_CYAN COL_LGRAY COL_DGRAY
+ source /opt/openhabian/functions/backup.bash
+ mirror_SD raw /dev/sda
+ local src=/dev/mmcblk0
+ local dest
+ local start
+ local syncMount=/storage/syncmount
+ local failed=no
+ [[ -n '' ]]
+ dest=/dev/sda
+ [[ /dev/mmcblk0 == \/\d\e\v\/\s\d\a ]]
++ blockdev --getsize64 /dev/sda
+ [[ -n 31914983424 ]]
++ mount
++ grep /dev/sda
+ [[ -n '' ]]
+ [[ raw == \r\a\w ]]
+ echo 'Creating a raw partition copy, be prepared this may take long such as 20-30 minutes for a 16 GB SD card'
Creating a raw partition copy, be prepared this may take long such as 20-30 minutes for a 16 GB SD card
+ cond_redirect dd if=/dev/mmcblk0 bs=1M of=/dev/sda
+ [[ -n '' ]]
+ echo -e '\n\033[90;01m$ dd if=/dev/mmcblk0 bs=1M of=/dev/sda \033[39;49;00m'

$ dd if=/dev/mmcblk0 bs=1M of=/dev/sda
+ dd if=/dev/mmcblk0 bs=1M of=/dev/sda
15193+1 records in
15193+1 records out
15931539456 bytes (16 GB, 15 GiB) copied, 1001.76 s, 15.9 MB/s
+ return 0
+ cond_redirect fsck -y -t vfat /dev/sda1
+ [[ -n '' ]]
+ echo -e '\n\033[90;01m$ fsck -y -t vfat /dev/sda1 \033[39;49;00m'

$ fsck -y -t vfat /dev/sda1
+ fsck -y -t vfat /dev/sda1
fsck from util-linux 2.33.1
fsck.fat 4.1 (2017-01-24)
0x41: Dirty bit is set. Fs was not properly unmounted and some data may be corrupt.
 Automatically removing dirty bit.
Performing changes.
/dev/sda1: 247 files, 110143/516190 clusters
+ return 1
+ echo 'FAILED (dirty fsck /dev/sda1)   '
FAILED (dirty fsck /dev/sda1)
+ failed=yes
+ cond_redirect fsck -y -t ext4 /dev/sda2
+ [[ -n '' ]]
+ echo -e '\n\033[90;01m$ fsck -y -t ext4 /dev/sda2 \033[39;49;00m'

$ fsck -y -t ext4 /dev/sda2
+ fsck -y -t ext4 /dev/sda2
fsck from util-linux 2.33.1
e2fsck 1.44.5 (15-Dec-2018)
rootfs: recovering journal
Clearing orphaned inode 24293 (uid=0, gid=0, mode=0100600, size=0)
Setting free inodes count to 864023 (was 864057)
Setting free blocks count to 2375959 (was 2376094)
rootfs: clean, 71977/936000 files, 1447017/3822976 blocks
+ return 0
+ [[ yes == \n\o ]]
+ return 1

mstormi · September 18, 2020, 10:26pm

that is sort of correct as the filesystem indeed was not unmounted.
Maybe I should not output that as FAILED.

I made one last change but it’s only the output that changed not the logic
You can test again but don’t have to.

narf27 · September 19, 2020, 7:08am

ok, dirty bit is “okay” now

[08:37:26] root@ohab2:/home/openhabian# /usr/local/sbin/mirror_SD "raw" /dev/sda
+ BASEDIR=/opt/openhabian
+ CONFIGFILE=/etc/openhabian.conf
+ source /etc/openhabian.conf
++ hostname=openhab
++ username=openhabian
++ timeserver=0.pool.ntp.org
++ locales='en_US.UTF-8 de_DE.UTF-8'
++ system_default_locale=en_US.UTF-8
++ wifi_ssid=
++ wifi_psk=
++ wifi_country=
++ ipv6=enable
++ repositoryurl=https://github.com/openhab/openhabian
++ clonebranch=master
++ debugmode=maximum
++ java_opt=Zulu8-32
+ source /opt/openhabian/functions/helpers.bash
++ ESC='\033['
++ COL_DEF='\033[39;49;00m'
++ COL_RED='\033[31;01m'
++ COL_GREEN='\033[32;01m'
++ COL_YELLOW='\033[33;01m'
++ COL_BLUE='\033[34;01m'
++ COL_MAGENTA='\033[35;01m'
++ COL_CYAN='\033[36;01m'
++ COL_LGRAY='\033[37;01m'
++ COL_DGRAY='\033[90;01m'
++ export COL_DEF COL_RED COL_GREEN COL_YELLOW COL_BLUE COL_MAGENTA COL_CYAN COL_LGRAY COL_DGRAY
+ source /opt/openhabian/functions/backup.bash
+ mirror_SD raw /dev/sda
+ local src=/dev/mmcblk0
+ local dest
+ local start
+ local syncMount=/storage/syncmount
+ local dirty=no
+ [[ -n '' ]]
+ dest=/dev/sda
+ [[ /dev/mmcblk0 == \/\d\e\v\/\s\d\a ]]
++ blockdev --getsize64 /dev/sda
+ [[ -n 31914983424 ]]
++ mount
++ grep /dev/sda
+ [[ -n '' ]]
+ [[ raw == \r\a\w ]]
+ echo 'Creating a raw partition copy, be prepared this may take long such as 20-30 minutes for a 16 GB SD card'
Creating a raw partition copy, be prepared this may take long such as 20-30 minutes for a 16 GB SD card
+ cond_redirect dd if=/dev/mmcblk0 bs=1M of=/dev/sda
+ [[ -n '' ]]
+ echo -e '\n\033[90;01m$ dd if=/dev/mmcblk0 bs=1M of=/dev/sda \033[39;49;00m'

$ dd if=/dev/mmcblk0 bs=1M of=/dev/sda
+ dd if=/dev/mmcblk0 bs=1M of=/dev/sda
15193+1 records in
15193+1 records out
15931539456 bytes (16 GB, 15 GiB) copied, 988.415 s, 16.1 MB/s
+ return 0
+ cond_redirect fsck -y -t vfat /dev/sda1
+ [[ -n '' ]]
+ echo -e '\n\033[90;01m$ fsck -y -t vfat /dev/sda1 \033[39;49;00m'

$ fsck -y -t vfat /dev/sda1
+ fsck -y -t vfat /dev/sda1
fsck from util-linux 2.33.1
fsck.fat 4.1 (2017-01-24)
0x41: Dirty bit is set. Fs was not properly unmounted and some data may be corrupt.
 Automatically removing dirty bit.
Performing changes.
/dev/sda1: 247 files, 110143/516190 clusters
+ return 1
+ echo 'OK (dirty bit on fsck /dev/sda1 is normal)   '
OK (dirty bit on fsck /dev/sda1 is normal)
+ dirty=yes
+ cond_redirect fsck -y -t ext4 /dev/sda2
+ [[ -n '' ]]
+ echo -e '\n\033[90;01m$ fsck -y -t ext4 /dev/sda2 \033[39;49;00m'

$ fsck -y -t ext4 /dev/sda2
+ fsck -y -t ext4 /dev/sda2
fsck from util-linux 2.33.1
e2fsck 1.44.5 (15-Dec-2018)
rootfs: recovering journal
Setting free inodes count to 864023 (was 864056)
Setting free blocks count to 2375956 (was 2376094)
rootfs: clean, 71977/936000 files, 1447020/3822976 blocks
+ return 0
+ [[ yes == \n\o ]]
+ return 1

just to be sure: my sd card is ok? or does this (previous) error indicate a problem?

mstormi · September 19, 2020, 7:28am

yes it should be fine. All errors were either due to the code or because the filesystem was not unmounted when we checked it. Because the source (the internal SD) was not unmounted when we copied it.

I think it works fine now. Thanks for your testing help.

narf27 · September 19, 2020, 8:34am

atm there’s nothing that would inform a “normal” would if something went wrong, is there?

thinking about alle the things i had trouble with (obviously also because of my own fault): zram-config, amanda, auto-backup… well, there enough things that could go wrong.

(all this things will still be relevant to oh3, right?)

have a nice day!

mstormi · September 19, 2020, 9:33am

There is, you should see a FAILED line in unattended install. It’s logged in /boot/first-boot.log.

narf27 · September 19, 2020, 10:09am

i’m not sure if unexperienced users would set already set this options in unattended install.
some people might need to get a bigger card for backup before setting up auto-backup f.e…
but as this i running now it shouldn’t be a problem anymore!

mstormi · September 19, 2020, 10:11am

these can use interactive setup. I don’t get your point?

narf27 · September 21, 2020, 11:17am

my auto-backup was not working after the (successful) setup, but i only discovered this because of your help.
a “normal” user might not look into logs to check if the script worked as expected.

another point: i ran openhabian-config and this message pops up:

## August 29, 2020
## Auto-backup (BETA)
openHABian can automatically take daily syncs of your internal SD card to
another card in an external port. This allows for fast swapping of cards
to reduce impact of a failed SD card.
The remaining space on the external device will also be used to setup
openHABian's Amanda backup system.

i’ve i’m not mistaken this message would only pop up only after a successful openhabian-config, right?
if that were the case the unattended installation already happened. if a user hadn’t previously opted for the auto-backup installation, the following installation via openhabian-config wouldn’t setup amanda. so the last sentence might not be 100% true.
obviously it becomes clearer when the user checks the openhabian-docs

mstormi · September 21, 2020, 11:34am

“bc”? what are you trying to tell ?
What sort of problem did you hit ?

well you cannot expect to get the full story in an announcement summary, can you

narf27 · September 21, 2020, 11:43am

“bc” > “because” - corrected it in post.

the problems we discussed in this thread (until your modifications solved the problem!).

absolutely
but maybe

The remaining space on the external device can also be used to setup openHABian’s Amanda backup system.

but that’s really just a minor detail and it’s all the same for me. just some input

mstormi · September 21, 2020, 8:10pm

Run blkid and compare UUIDs of internal and external SD.
If they’re equal, update openhabian to master branch and run dd sync via menu 54.