Daily Backup Solution

lukics · September 26, 2018, 6:32pm

I looked through the forum about a automatic backup solution and setup myself something I want to share and get your feedback.

I decided to backup my configuration to google drive. So I created a google account just for backup which includes 15GB, which should be enough for a while.

I looked for a utility which can connect to the google drive and found rclone: https://rclone.org/install/. THe installation is straight forward.

I created a script:

#!/bin/bash

listOfDirs=(/var/lib/openhab2 /etc/openhab2 /var/log/openhab2 /opt/backup )
localServer=raspiohab-pro
rcloneConfig=/home/openhabian/.config/rclone/rclone.conf
rcloneOptions="-v --ignore-checksum --ignore-size --local-no-check-updated"

echo  $(date +"%Y-%m-%d %H:%M:%S") : backup started

for dir in "${listOfDirs[@]}"
do
   :
   echo  $(date +"%Y-%m-%d %H:%M:%S") : backup of  $dir
   rclone  $rcloneOptions --config=$rcloneConfig sync $dir raspiohab:/backup/$localServer$dir
done

echo  $(date +"%Y-%m-%d %H:%M:%S")  : backup complete

which I added to cron on root level:

0 0 * * * /opt/backup/backup.sh >> /opt/backup/backup.log 2>&1

As you can see I am backing up /var/lib/openhab2 /etc/openhab2.
Which additional folders should be backup-ed.

What is your feedback

rlkoshak · September 26, 2018, 6:49pm

How would restore work?

Were I to go with this approach, I would use sudo openhab-cli backup to generate the backups then only backup the contents of /var/lib/openhab2/backups to Google Drive. That way you know you will always have everything needed backed up (OH config wise) and you have a built in way to restore.

lukics · September 26, 2018, 7:16pm

This seems to be more reasonable to run the built in backup and then copy the zip to gdrive.
Additionally the script can be used to backup of influxdb and grafana . Influxdb and grafana I run on differnt server whe I first start infludb own backup routine and then backup the output.

#!/bin/bash

listOfDirs=(/tmp/ifxbackup /var/lib/grafana /etc/grafana \
            /opt/landroid-bridge /etc/init.d/landroid-bridge \
            /etc/default/ebusd /etc/ebusd /opt/backup)

localServer=raspiohab-utl
rcloneConfig=/home/lukics/.config/rclone/rclone.conf
rcloneOptions="-v --ignore-checksum --ignore-size --local-no-check-updated"

echo  $(date +"%Y-%m-%d %H:%M:%S") : backup started

echo  $(date +"%Y-%m-%d %H:%M:%S") :  create influxdb in /tmp/ifxbackup
# Create influxdb backup
influxd backup -portable /tmp/ifxbackup


for dir in "${listOfDirs[@]}"
do
   :
   echo  $(date +"%Y-%m-%d %H:%M:%S") :  backup of $dir
   rclone $rcloneOptions --config=$rcloneConfig sync $dir raspiohab:/backup/$localServer$dir
done

echo  $(date +"%Y-%m-%d %H:%M:%S") :  backup complete

dastrix80 · September 27, 2018, 12:00am

@lukics I keep getting errors about file open when backing up grafana

How have you managed to do this?



2018-09-27 09:58:17 : backup of /var/lib/grafana
2018/09/27 09:58:31 ERROR : sessions: error reading source directory: failed to open directory "sessions": open /var/lib/grafana/sessions: permission denied
2018/09/27 09:58:31 ERROR : png: error reading source directory: failed to open directory "png": open /var/lib/grafana/png: permission denied
2018/09/27 09:58:31 ERROR : grafana.db: Failed to copy: failed to open source object: open /var/lib/grafana/grafana.db: permission denied
2018/09/27 09:58:31 ERROR : plugins: error reading source directory: failed to open directory "plugins": open /var/lib/grafana/plugins: permission denied
2018/09/27 09:58:31 INFO  : Google drive root 'backup/openhab2/var/lib/grafana': Waiting for checks to finish
2018/09/27 09:58:31 INFO  : Google drive root 'backup/openhab2/var/lib/grafana': Waiting for transfers to finish
2018/09/27 09:58:49 ERROR : Google drive root 'backup/openhab2/var/lib/grafana': not deleting files as there were IO errors
2018/09/27 09:58:49 ERROR : Google drive root 'backup/openhab2/var/lib/grafana': not deleting directories as there were IO errors
2018/09/27 09:58:49 ERROR : Attempt 1/3 failed with 4 errors and: failed to open source object: open /var/lib/grafana/grafana.db: permission denied
2018/09/27 09:58:59 ERROR : grafana.db: Failed to copy: failed to open source object: open /var/lib/grafana/grafana.db: permission denied
2018/09/27 09:59:00 ERROR : sessions: error reading source directory: failed to open directory "sessions": open /var/lib/grafana/sessions: permission denied
2018/09/27 09:59:00 ERROR : plugins: error reading source directory: failed to open directory "plugins": open /var/lib/grafana/plugins: permission denied
2018/09/27 09:59:00 ERROR : png: error reading source directory: failed to open directory "png": open /var/lib/grafana/png: permission denied
2018/09/27 09:59:00 INFO  : Google drive root 'backup/openhab2/var/lib/grafana': Waiting for checks to finish
2018/09/27 09:59:00 INFO  : Google drive root 'backup/openhab2/var/lib/grafana': Waiting for transfers to finish
2018/09/27 09:59:00 ERROR : Google drive root 'backup/openhab2/var/lib/grafana': not deleting files as there were IO errors
2018/09/27 09:59:00 ERROR : Google drive root 'backup/openhab2/var/lib/grafana': not deleting directories as there were IO errors
2018/09/27 09:59:00 ERROR : Attempt 2/3 failed with 4 errors and: failed to open source object: open /var/lib/grafana/grafana.db: permission denied
2018/09/27 09:59:00 ERROR : grafana.db: Failed to copy: failed to open source object: open /var/lib/grafana/grafana.db: permission denied
2018/09/27 09:59:01 ERROR : png: error reading source directory: failed to open directory "png": open /var/lib/grafana/png: permission denied
2018/09/27 09:59:01 ERROR : sessions: error reading source directory: failed to open directory "sessions": open /var/lib/grafana/sessions: permission denied
2018/09/27 09:59:01 ERROR : plugins: error reading source directory: failed to open directory "plugins": open /var/lib/grafana/plugins: permission denied
2018/09/27 09:59:01 INFO  : Google drive root 'backup/openhab2/var/lib/grafana': Waiting for checks to finish
2018/09/27 09:59:01 INFO  : Google drive root 'backup/openhab2/var/lib/grafana': Waiting for transfers to finish
2018/09/27 09:59:01 ERROR : Google drive root 'backup/openhab2/var/lib/grafana': not deleting files as there were IO errors
2018/09/27 09:59:01 ERROR : Google drive root 'backup/openhab2/var/lib/grafana': not deleting directories as there were IO errors
2018/09/27 09:59:01 ERROR : Attempt 3/3 failed with 4 errors and: failed to open source object: open /var/lib/grafana/grafana.db: permission denied
2018/09/27 09:59:01 Failed to sync: failed to open source object: open /var/lib/grafana/grafana.db: permission denied

lukics · September 27, 2018, 5:11am

Are you running the script with root permission, e.g. sudo backup.sh , sudo crontab -e ?

--ignore-checksum --ignore-size --local-no-check-updated

did some magic for me!

dastrix80 · September 27, 2018, 5:16am

It backs up the Oh2 zip fine along with influx, just not the Grafana files… yes, its run with sudo

SCRIPT


#!/bin/bash

listOfDirs=(/var/lib/openhab2/backups/ /home/kris/.config/rclone/ )

localServer=openhab2
rcloneConfig=/home/kris/.config/rclone/rclone.conf
rcloneOptions="-v --ignore-checksum --ignore-size --local-no-check-updated"

echo  $(date +"%Y-%m-%d %H:%M:%S") : backup started

echo  $(date +"%Y-%m-%d %H:%M:%S") :  create influxdb in /home/kris/influxbk
# Create influxdb backup
influxd backup -portable /home/kris/influxbk


for dir in "${listOfDirs[@]}"
do
   :
   echo  $(date +"%Y-%m-%d %H:%M:%S") :  backup of $dir
   rclone $rcloneOptions --config=$rcloneConfig sync $dir Google:backup/$localServer$dir
done

echo  $(date +"%Y-%m-%d %H:%M:%S") :  backup complete

RULE


rule "Backup OH2/Influx/Grafana"
when
 Time cron "0 0 1 ? * * *" or
 Item backupserver received command ON
then
 logInfo("Server Backup", "Backing up the Server")
 var String RCloneOutput = executeCommandLine("sudo /etc/openhab2/scripts/backup.sh", 60000)
 logInfo("Server Backup", "Backup Executed + RCloneOutput")
end

lukics · September 27, 2018, 5:35am

I do not see that you have grafana directory in the list of directories, ok, see your comment
No need to backup rclone config, can be recreated at any time
I would suggest to run the backup.sh from the system cron, e.g. sudo contab -e an redirect to backup.log

dastrix80 · September 27, 2018, 5:37am

1)I removed it out, it was there before but created errors
2) OK
3) Any reason over the cron time in rules and or a switch? sometimes i want to manually initiate backups from habPanel, this works

lukics · September 27, 2018, 5:38am

did you try to run the command in the console and you could stop and start the grafana service in the backup.sh if needed

dastrix80 · September 27, 2018, 5:39am

I readded the grafana directories

Yes, running it in the console now. Yes, I could stop/start Grafana - how would I add that to the script? i use systemctl

Here is the result after adding them:

2018-09-27 15:39:16 : backup of /var/lib/grafana
2018/09/27 15:39:19 ERROR : grafana.db: Failed to copy: failed to open source object: open /var/lib/grafana/grafana.db: permission denied
2018/09/27 15:39:19 ERROR : sessions: error reading source directory: failed to open directory "sessions": open /var/lib/grafana/sessions: permission denied
2018/09/27 15:39:19 ERROR : png: error reading source directory: failed to open directory "png": open /var/lib/grafana/png: permission denied
2018/09/27 15:39:19 ERROR : plugins: error reading source directory: failed to open directory "plugins": open /var/lib/grafana/plugins: permission denied
2018/09/27 15:39:19 INFO  : Google drive root 'backup/openhab2/var/lib/grafana': Waiting for checks to finish
2018/09/27 15:39:19 INFO  : Google drive root 'backup/openhab2/var/lib/grafana': Waiting for transfers to finish
2018/09/27 15:39:19 ERROR : Google drive root 'backup/openhab2/var/lib/grafana': not deleting files as there were IO errors
2018/09/27 15:39:19 ERROR : Google drive root 'backup/openhab2/var/lib/grafana': not deleting directories as there were IO errors
2018/09/27 15:39:19 ERROR : Attempt 1/3 failed with 4 errors and: failed to open source object: open /var/lib/grafana/grafana.db: permission denied
2018/09/27 15:39:20 ERROR : grafana.db: Failed to copy: failed to open source object: open /var/lib/grafana/grafana.db: permission denied
2018/09/27 15:39:21 ERROR : png: error reading source directory: failed to open directory "png": open /var/lib/grafana/png: permission denied
2018/09/27 15:39:21 ERROR : plugins: error reading source directory: failed to open directory "plugins": open /var/lib/grafana/plugins: permission denied
2018/09/27 15:39:21 ERROR : sessions: error reading source directory: failed to open directory "sessions": open /var/lib/grafana/sessions: permission denied
2018/09/27 15:39:21 INFO  : Google drive root 'backup/openhab2/var/lib/grafana': Waiting for checks to finish
2018/09/27 15:39:21 INFO  : Google drive root 'backup/openhab2/var/lib/grafana': Waiting for transfers to finish
2018/09/27 15:39:21 ERROR : Google drive root 'backup/openhab2/var/lib/grafana': not deleting files as there were IO errors
2018/09/27 15:39:21 ERROR : Google drive root 'backup/openhab2/var/lib/grafana': not deleting directories as there were IO errors
2018/09/27 15:39:21 ERROR : Attempt 2/3 failed with 4 errors and: failed to open source object: open /var/lib/grafana/grafana.db: permission denied

lukics · September 27, 2018, 5:41am

sudo systemctl stop grafana-server

lukics · September 27, 2018, 5:43am

but:
did you check the directory permissions for grafana, just adapting these could give the solution

dastrix80 · September 27, 2018, 5:43am

I more mean, in the script. I know how to start it from the console.

lukics · September 27, 2018, 5:47am

try

sudo chmod -R a+r /var/lib/grafana

lukics · September 27, 2018, 5:47am

should be no difference as on the command line

dastrix80 · September 27, 2018, 5:49am

This didnt work…



listOfDirs=(/home/kris/OH2Backup.zip /var/lib/grafana /etc/grafana )

echo  $(date +"%Y-%m-%d %H:%M:%S") : Stopping Grafana
sudo systemctl stop grafana-server


localServer=openhab2
rcloneConfig=/home/kris/.config/rclone/rclone.conf
rcloneOptions="-v --ignore-checksum --ignore-size --local-no-check-updated"

echo  $(date +"%Y-%m-%d %H:%M:%S") : backup started

echo  $(date +"%Y-%m-%d %H:%M:%S") :  create influxdb in /home/kris/influxbk
# Create influxdb backup
influxd backup -portable /home/kris/influxbk


for dir in "${listOfDirs[@]}"
do
   :
   echo  $(date +"%Y-%m-%d %H:%M:%S") :  backup of $dir
   rclone $rcloneOptions --config=$rcloneConfig sync $dir Google:backup/$localServer$dir
done


echo  $(date +"%Y-%m-%d %H:%M:%S") : Starting Grafana
sudo systemctl start grafana-server

echo  $(date +"%Y-%m-%d %H:%M:%S") :  backup complete

lukics · September 27, 2018, 5:52am

you do not sudo inside script, run the script with sudo permission
try also to adapt the permissions of whole grafana as alternative

dastrix80 · September 27, 2018, 5:53am

by removing sudo from the command, the console wants me to enter the password

I dont know what that means ‘adapt permissions’

Im learning unix

lukics · September 27, 2018, 6:01am

sudo  bash -c '/opt/backup/backup.sh >> /opt/backup/backup.log 2>&1'

lukics · September 27, 2018, 6:08am

I meant that it might be enough to change the permission on the directory level by calling the obove command

sudo chmod -R a+r /var/lib/grafana

There you give read permission to grafa dir and subdirs to all