Recommended way to backup/restore OH2 configurations and things?


(Markus Storm) #61

Amanda is CLI only. There’s a brief usage guide in docs/openhabian-amanda.md.


(John) #62

thanks, I will take a look

but what I was saying would be nice is just a simple web gui that easily backed up to a file and restored from same file the control and the zwave, ect… :0)


(Andreas Kuhn) #63

Hi, i just found this thread and wanted to share my backup solution.
I have an USB Stick attached to my PI and run this script every day.
It covers:

  • OpenHab
  • Influxdb
  • Grafana
#!/bin/bash
# stop openhab instance (here: systemd service)
echo "+-+-+-+-+-+-+ Stopping service...+-+-+-+-+-+-+-+"
sudo systemctl stop openhab2.service

# backup current installation with settings
echo "+-+-+-+-+-+-+ timestamp and folder setup +-+-+-+-+-+-+-+-+";
TIMESTAMP="`date +%Y%m%d_%H%M%S`";
sudo mkdir  "/media/usb/openhab2-backup-$TIMESTAMP";
sudo mkdir  "/media/usb/openhab2-backup-$TIMESTAMP/influxdb";
sudo mkdir  "/media/usb/openhab2-backup-$TIMESTAMP/grafana";
#bakdir="$HOME/openhab2-backup/$TIMESTAMP";
#mkdir -p "${bakdir}"

echo "+-+-+-+-+-+-+ OH backup +-+-+-+-+-+-+-+-+-+-+"
#cp -arv /etc/openhab2 "${bakdir}/conf"
#cp -arv /var/lib/openhab2 "${bakdir}/userdata"
#find /var/lib/openhab2 \( -path /var/lib/openhab2/tmp -prune -o -path /var/lib/openhab2/cache -prune \) -o -name '*' -exec cp -arv {} "${bakdir}/userdata"  \;
sudo cp -arv "/etc/openhab2" "/media/usb/openhab2-backup-$TIMESTAMP/conf";
sudo cp -arv "/var/lib/openhab2" "/media/usb/openhab2-backup-$TIMESTAMP/userdata";

echo "+-+-+-+-+-+-+ Clear cache and tmp folder +-+-+-+-+-+-+-+-+"
sudo rm -rf "/media/usb/openhab2-backup-$TIMESTAMP/userdata/cache"
sudo rm -rf "/media/usb/openhab2-backup-$TIMESTAMP/userdata/tmp"

echo "+-+-+-+-+-+-+ Grafana backup +-+-+-+-+-+-+-+-+"
sudo systemctl stop grafana-server
sudo cp -arv "/etc/grafana/grafana.ini" "/media/usb/openhab2-backup-$TIMESTAMP/grafana/grafana.ini";
sudo cp -arv "/var/lib/grafana/grafana.db" "/media/usb/openhab2-backup-$TIMESTAMP/grafana/grafana.db";

echo "+-+-+-+-+-+-+ Influxdb backup +-+-+-+-+-+-+-+"
sudo cp -arv "/etc/influxdb/influxdb.conf" "/media/usb/openhab2-backup-$TIMESTAMP/influxdb/influxdb.conf"
sudo influxd backup "/media/usb/openhab2-backup-$TIMESTAMP/influxdb/metastore/"
sudo influxd backup -database openhab_db "/media/usb/openhab2-backup-$TIMESTAMP/influxdb/db/"

# restart openhab instance
echo "+-+-+-+-+-+-+ Starting service...+-+-+-+-+-+-+-+-+"
sudo systemctl start openhab2.service
sudo systemctl start grafana-server

# Folder size
# sudo ls -1d */ | sudo xargs -I{} du {} -sh && sudo du -sh
sudo df -h /media/usb/; sudo du -sh -- /media/usb/*


(Rich Koshak) #64

Keep an eye on that thumb drive. While not as bad as SD cards, thumb drives will wear it with writes. And when it starts to go bad it isn’t always obvious.

For these reasons I do not recommend thumb drives as a reliable backup solution. But you’re scrips will work with any storage type so should work just fine with USB hdd or sdd.


(Andreas Kuhn) #65

Hi Rich,

you are absolute right, these tiny thumb drives are not a very good reliable backup medium, but they are not worse than the sd card :slight_smile:

So the usb stick has the advantage that i easily can exchange file between my 2 PIs running OH for testing.

As final Backup Medium i put the Backup files on my Synology NAS, but this is still yet all manual, so i need to find a way to save, lets say, always the last 2 backup files on the usb stick automatically to my syno nas. It is on my ToDo list :wink:


(SiHui) #66

Very nice script, I like it.
If you want to save some time and work, take a look for a fully automated backup to any backup medium:

https://www.linux-tips-and-tricks.de/en/backup


(Rich Koshak) #67

Are your Pis not on the same network as your NAS? If they are you can just NFS/CIFS or whatever mount a backup folder in your Synology NAS to the Pis and copy straight to that and bypass the thumb drive.

My biggest concern with the thumb drive is not how quickly it wears out or that you are using it as the final destination for the backups but the fact that you won’t necessarily know that it has failed unless you are somehow validating the files you copied off of the drive with the source files every time (in which case that means you can see your Pis from you NAS and have no need for the thumb drive in the first place).

You can happily be making backups for months only to discover one day when you need to restore that your thumb drive failed and all of your backups are corrupted or stuck back at a distant point in time, or more likely a combination of the two. It isn’t always obvious when these drives fail. Just look around at some of the strange behaviors people have seen when their SD cards fail. I myself discovered my SD card failed only because logrotate stopped working. It would delete the old files and rotate them only to have the old files reappear the next day. You don’t always get errors or flashing red lights. Sometimes it acts like everything is working just fine but that file you thought you wrote just disappears without a warning and without a trace.

tl;dr - even using the thumb drive to copy the backups from one machine to another is a bad idea


(Andreas Kuhn) #68

Hi Rich,

my PIs and NAS share the same Network. I just had not the passion yet to learn how to map my NAS drive :). I guess Linux will not become my best friend, so every new thing i have to look it up how to do it and sometimes things are different by the distros.
But the Forum here already helped a lot and your thoughts made change my mind and i will see how to map the drive manually and perform backups on the NAS then.
I think MOUNT will be the one command i need to look closer at, cause my nas is not running 24/7 and will be WOLed whenever i need it i have to map and unmap the share for backups.

Thanks
Andi


(Ben Clark) #69

Hi all, I wanted to direct people that were interested in a PR for openHAB that contains a backup and restore script for Linux and MacOS to be included in future versions of openHAB. It hasn’t been merged yet as it needs people to test it.

The scripts should work with the snapshot versions.


(Austris V) #70

@aksnet, I wonder, do u also have the other side of script - for restore? Thank You!


(SiHui) #71

(Andreas Kuhn) #72

Hi @Austris_V, i have not yet needed the restore, but let me see what i can come up with during the day…
I already have some in preparation, but need to check if this should work.
I will follow up later on this.

Andreas


(Andreas Kuhn) #73

@Austris_V, so this is my actual backup script, fully working so far:

#!/bin/bash
# stop openhab instance (here: systemd service)
echo "+-+-+-+-+-+-+ Stopping service...+-+-+-+-+-+-+-+"
sudo systemctl stop openhab2.service

# backup current installation with settings
echo "+-+-+-+-+-+-+ timestamp and folder setup +-+-+-+-+-+-+-+-+";
TIMESTAMP="`date +%Y%m%d_%H%M%S`";
sudo mkdir  "/media/usb/openhab2-backup-$TIMESTAMP";
sudo mkdir  "/media/usb/openhab2-backup-$TIMESTAMP/influxdb";
sudo mkdir  "/media/usb/openhab2-backup-$TIMESTAMP/grafana";
#bakdir="$HOME/openhab2-backup/$TIMESTAMP";
#mkdir -p "${bakdir}"

echo "+-+-+-+-+-+-+ OH backup +-+-+-+-+-+-+-+-+-+-+"
#cp -arv /etc/openhab2 "${bakdir}/conf"
#cp -arv /var/lib/openhab2 "${bakdir}/userdata"
#find /var/lib/openhab2 \( -path /var/lib/openhab2/tmp -prune -o -path /var/lib/openhab2/cache -prune \) -o -name '*' -exec cp -arv {} "${bakdir}/userdata"  \;
sudo cp -arv "/etc/openhab2" "/media/usb/openhab2-backup-$TIMESTAMP/conf";
sudo cp -arv "/var/lib/openhab2" "/media/usb/openhab2-backup-$TIMESTAMP/userdata";

echo "+-+-+-+-+-+-+ Grafana backup +-+-+-+-+-+-+-+-+"
sudo systemctl stop grafana-server
sudo cp -arv "/etc/grafana/grafana.ini" "/media/usb/openhab2-backup-$TIMESTAMP/grafana/grafana.ini";
sudo cp -arv "/var/lib/grafana/grafana.db" "/media/usb/openhab2-backup-$TIMESTAMP/grafana/grafana.db";

echo "+-+-+-+-+-+-+ Influxdb backup +-+-+-+-+-+-+-+"
sudo cp -arv "/etc/influxdb/influxdb.conf" "/media/usb/openhab2-backup-$TIMESTAMP/influxdb/influxdb.conf"
sudo influxd backup "/media/usb/openhab2-backup-$TIMESTAMP/influxdb/metastore/"
sudo influxd backup -database openhab_db "/media/usb/openhab2-backup-$TIMESTAMP/influxdb/db/"

# restart openhab instance
echo "+-+-+-+-+-+-+ Starting service...+-+-+-+-+-+-+-+-+"
sudo systemctl start openhab2.service
sudo systemctl start grafana-server

echo "+-+-+-+-+-+-+ Clear cache and tmp folder +-+-+-+-+-+-+-+-+"
sudo rm -rf "/media/usb/openhab2-backup-$TIMESTAMP/userdata/cache"
sudo rm -rf "/media/usb/openhab2-backup-$TIMESTAMP/userdata/tmp"

# Packen
echo "+-+-+-+-+-+-+ Pack Backup Folder into tar.gz +-+-+-+-+-+-+-+-+"
tar cfvz /media/usb/openhab2-backup-$TIMESTAMP.tar.gz /media/usb/openhab2-backup-$TIMESTAMP

# Entpacken-> tar xfvz archiv.tar.gz

# Folder size
# sudo ls -1d */ | sudo xargs -I{} du {} -sh && sudo du -sh
sudo df -h /media/usb/; sudo du -sh -- /media/usb/*



For restore this SHOULD work, i have not tested yet and unfortunately don’t have the time to do so in the next couple days. So if you can test it and let us know would be perfect.
I did care also of the influxdb restore which requires a few more steps than the backup.

The $TIMESTAMP has to be replaced yet manually in the script before execution. This could be done better in a future version of the script. If you have any ideas, let me know.

#!/bin/bash
# stop openhab instance (here: systemd service)
echo "+-+-+-+-+-+-+ Stopping service...+-+-+-+-+-+-+-+"
sudo systemctl stop openhab2.service

# restore current installation with settings
#echo "+-+-+-+-+-+-+ timestamp and folder setup +-+-+-+-+-+-+-+-+";
#TIMESTAMP="`date +%Y%m%d_%H%M%S`";
#sudo mkdir  "/media/usb/openhab2-backup-$TIMESTAMP";
#sudo mkdir  "/media/usb/openhab2-backup-$TIMESTAMP/influxdb";
#sudo mkdir  "/media/usb/openhab2-backup-$TIMESTAMP/grafana";
#bakdir="$HOME/openhab2-backup/$TIMESTAMP";
#mkdir -p "${bakdir}"

# Sample
# $TIMESTAMP -> 20171130_054320
# search and replace it via editor

echo "+-+-+-+-+-+-+ OH restore +-+-+-+-+-+-+-+-+-+-+"
#cp -arv /etc/openhab2 "${bakdir}/conf"
#cp -arv /var/lib/openhab2 "${bakdir}/userdata"
#find /var/lib/openhab2 \( -path /var/lib/openhab2/tmp -prune -o -path /var/lib/openhab2/cache -prune \) -o -name '*' -exec cp -arv {} "${bakdir}/userdata"  \;
sudo cp -arv "/media/usb/openhab2-backup-$TIMESTAMP/conf" "/etc/openhab2";
sudo cp -arv "/media/usb/openhab2-backup-$TIMESTAMP/userdata" "/var/lib/openhab2";

echo "+-+-+-+-+-+-+ Grafana backup +-+-+-+-+-+-+-+-+"
sudo systemctl stop grafana-server
sudo cp -arv "/media/usb/openhab2-backup-$TIMESTAMP/grafana/grafana.ini" "/etc/grafana/grafana.ini";
sudo cp -arv "/media/usb/openhab2-backup-$TIMESTAMP/grafana/grafana.db" "/var/lib/grafana/grafana.db";

# https://docs.influxdata.com/influxdb/v1.3/administration/backup_and_restore/
echo "+-+-+-+-+-+-+ Influxdb backup +-+-+-+-+-+-+-+"

service influxdb stop

sudo cp -arv "/media/usb/openhab2-backup-$TIMESTAMP/influxdb/influxdb.conf" "/etc/influxdb/influxdb.conf"
sudo influxd restore -metadir /var/lib/influxdb/meta "/media/usb/openhab2-backup-$TIMESTAMP/influxdb/metastore/"
sudo influxd restore -database openhab_db -datadir /var/lib/influxdb/data "/media/usb/openhab2-backup-$TIMESTAMP/influxdb/db/"

service influxdb start

# restart openhab instance
echo "+-+-+-+-+-+-+ Starting service...+-+-+-+-+-+-+-+-+"
sudo systemctl start openhab2.service
sudo systemctl start grafana-server

#echo "+-+-+-+-+-+-+ Clear cache and tmp folder +-+-+-+-+-+-+-+-+"
#sudo rm -rf "/media/usb/openhab2-backup-$TIMESTAMP/userdata/cache"
#sudo rm -rf "/media/usb/openhab2-backup-$TIMESTAMP/userdata/tmp"

# Packen
#echo "+-+-+-+-+-+-+ Pack Backup Folder into tar.gz +-+-+-+-+-+-+-+-+"
#tar cfvz /media/usb/openhab2-backup-$TIMESTAMP.tar.gz /media/usb/openhab2-backup-$TIMESTAMP

# Entpacken-> tar xfvz archiv.tar.gz

# Folder size
# sudo ls -1d */ | sudo xargs -I{} du {} -sh && sudo du -sh
#sudo df -h /media/usb/; sudo du -sh -- /media/usb/*




(Austris V) #74

Just to verify - it will backup also bindings and additions (e.g. zwave, expire, transformations; influx, graffana) and it’s config?
e.g. if I have it backed-up and then run restore on top of fresh openhab from openhabian - it shall recreate it all?

or I rather shall have a image ready with all the sw staff installed and the can use restore for all the configuration?


(Austris V) #75

will give it a try tonight. also, just to verify - your restore shall work on fresh openhabian instance? or I shall, as a minimum, install the influxdb and grafana first and the restore takes care of the content (config and data)? Thanks!


(Andreas Kuhn) #76

Hi,
it should backup everything.
And yes at a fresh install you have to install grafana and influxdb first and then restore.

BR
Andreas


(Ben Clark) #77

The two scripts backup and restore the configuration of everything inside openHAB. If you restored the backup on top of a fresh installation then it will reload the configuration exactly as it was. This also means that the bindings you had previously would be reinstalled on the next launch.

In practical terms this means that if you backed up a configuration from openHAB 2.1.0, then restored it on top of a new openHAB 2.2.0 instance. It would load the configuration and install the 2.2.0 version of the bindings.

It will also restore cross platform. The same scripts work for manual or automatic installs on any MacOS or Linux OS. Of course if you do this you’d need to reconfigure which port is used by the zwave dongles etc and the default permissions of the openHAB user.

My recommendation would be to call these openHAB scripts with the scripts for the other programs that @aksnet has posted.


Unable to boot on RPI3 with latest update
(Austris V) #78

I guess I was little lucky:) Openhab was half running, but could not logon through putty (connections refused; even after few restarts). And my previously made backup, which I thought is on flash, apparently was not there. ha :smiley:

Found ext2explore and used to access the sd from pc; Following your backup script, manually copied all the mentioned files to safe location (did not do proper influxdb backup though). And then repeated the same - manual copy following your restore script. had to add few sudo in front of service starts. had to fix permissions at the end (because was copying through win?). had to remember to copy jars from addons. but other than that all up and running. such a good feeling. and no more enhancements before I try full backup-restore process and become sure I have it fixed :smiley:
Thank you!

And will eventually also uprgade to snapshot (again) so I have the built in scripts, too.


(Jochen Breh) #79

Hi all,
I’m very new to OpenHab and I just want to give you my opinion on this discussion.
My background is technical and in a former life I was even a software engineer :slight_smile:

First, the PaperUI helped me to get a jump start and I made first successful experiences. Therefore, I think it is needed for a first introduction.
Second, to understand the entire concept of openhab is really hard. In particular, making the step from PaperUI to the configuration via the eclipse designer was very hard for me. It seemed to me as two different concepts with now relation to each other
Btw, I love text file configuration, but only if I understand it :wink:

Third, backup us a real issue for me, because in my trail phase I sometime crashes my Pi or I want to go back to a previous state.
Now, as I understand the config stuff, I can backup this, but what about the PaperUI detected stuff? Best for me would be a twofold approach. 1. an export/import function for the PaperUI DB (its not important to get it human readable/editable but it must be restore-able) and 2. saving the text files (which is available today).

And last but not least, I really like and appreciate what you guys have done here. Now I need to go back to my system to learn more about it :slight_smile:

Thanks for reading.


(SiHui) #80

It’s there, fully automatic :grinning:. You just need to copy it to another drive to be more safe: