OS and OpenHab rebuild

[idkpmiller (Paul Miller)]

Hi All,
I have placed this thread in ‘off-topic’ as I was not sure where it belonged.
I feel with the holiday season almost upon us, the need to change OS to a new major releases and the introduction of OH3, A need to do some serious maintenance preparation.

I have Amanda scheduling backups everynight and I get the emails to say whe has done so and what slot has been used etc… But to be honest it mans very little to me.

My Setup
I have just upgraded to the very latest OH 2.5.11-1 version and so far my process for that has worked quite well, so I am happy with that and my starting position

As my Openhab instance runs in a virtual machine, which is shutdown once a week, a full image copy is performed and then the VM is started back up all this is automated while I sleep. This has been running for a year really well and I keep two full image backups before overwriting, so if anything ever goes wrong I go and start up the last image.

I am currently on Debian and have no driver to change to another distro. But as it is using Jessie I need to upgrade to Buster. I have previously tried an upgrade direct between versions and it did not go well, mainly because of my-sql. Since then I have removed my dependency from my-sql and I am considering another attempt at a in place upgrade rather than a new build. Any views on this?

I would also like to prepare myself for the new build approach and would like to make sure I have the process as clear as possible in my head.

Objective
I would like to rebuild the OS while still using OH2 and then tackle the move from OH2 to OH3 as a different event.

Doing it this way I am thinking for an OS new build, I need to

1. Build Debian Buster Virtual Machine
2. Install OH2 (openhabian on linux)
3. Install all the extra packages that I need for extra scripts, tools, etc
4. Amanda Recovery from latest backup
5. Confirm working . solve minor issues

Concerns
The two main areas that scare me are #3 and #4.
Over the years, I will have introduced many different tools that I have loaded and installed on the OH server and now I have to find which ones are still needed and which can be left behind, scary!
And I have NEVER tried to recover a backup from Amanda, I am thinking I should try this as a separate task using a copy of one of my clones to make sure that they recover with the same machine before I try doing the procedure on a new OS machine.

Where do I look at the way to recover from Amanda, so far when I look on line my eyes glase over, I think I set up amanda using the openhabian config tool so I am hoping there may be a simple way to test that is true, and also the process to recover is something that will have been thought about with the configuration applied and possible some steps outlined to perform the task. Summary, Is there a simple command I run to recover using latest backup?

Next Steps
I am looking for discussion, a little hand holding / assurance, that I am doing things sensibly and not running into a burning building.

Even writing this down is preparing me for the tasks I need to perform

Hoping this thread may assist others that are looking at this task also.

Regards

Paul

[rlkoshak (Rich Koshak)]

@idkpmiller

Any views on this?

For the most part OH isn’t going to care. But as you’ve seen, some of the stuff that you will use with openHAB can have problems. It’s not quite the same but on my RPis I’ve a couple that I did direct upgrades from wheezy to jessy to buster. But I am only running some Python scripts on them so there isn’t much that could go wrong.

Since it’s a VM, I’d take a snapshot and give it a try. It might work and you’ll probably save yourself a good deal of time. If it doesn’t work, well it was worth a shot and you are not out much time.

@idkpmiller

Doing it this way I am thinking for an OS new build, I need to

You probably don’t want to do 4. At least if Amanda is configured to back up your whole machine than at step 4 you will undo everything you did in steps 1-3. However, if you are just recovering your openHAB configs than that’s ok.

@idkpmiller

Over the years, I will have introduced many different tools that I have loaded and installed on the OH server and now I have to find which ones are still needed and which can be left behind, scary!

It’s too late now but this is one of the reasons why I use Ansible for all my configuration. It is more complicated than just going out and making configuration changes and such, but when I need to rebuild (e.g. because of an OS upgrade) I just need to run the scripts. I don’t even need to remember how to do it.

But even keeping a notebook of some sort with a change log so you remember everything you’ve done. So my suggestion now is to start that notebook (or consider learning a configuration as code tool like Ansible) now. It’ll be a lot of work but you’ll be happy you did in a year or so from now when you need to move stuff to another VM or rebuild.

I don’t use Amanda so I can’t offer any advice on that point. But a common saying among us security engineers is “An untested backup is no backup at all.”

Anyway, what I would do is the following:

1. Take a snapshot of your VM
2. Try an in place upgrade of the OS to Buster
3. If and only if that fails, create a brand new VM and install Buster.
4. Go through the configuration steps necessary to bring that new VM up to where you need it to be. Take LOTS of notes, and take the notes in a way that you will be able to continue adding to them as time passes and you add more to the new VM.
5. For openHAB configs use openhab-cli backup on the old machine, copy the zip file over to the new machine and use openhab-cli restore to move over the OH configs. Configs for other apps you can probably move over manually.

Since this is a VM and you can take snapshots, I’m not sure what value Amanda provides here, especially since you don’t know how to use it. A backup isn’t much good if you don’t know what’s backed up and how to restore from the backup.

[idkpmiller (Paul Miller)]

@rlkoshak

openhab-cli backup

Some great points here, I love the one about amanda, I came to the same conclusion myself. Incidentally it is only copying the Openhab data, however I need to learn how to restore from it or stop backing up as its a waste of resources

I love the backup and restore option from cli, I will give that a go, as my restore option.

I have so far upgraded 4 VMs fine to Buster, however the simpler there are the easier it is. An early attempt with OH failed as I said mainly around my-SQL being removed and replaced with mariedb. As I no longer have a dependency it is certainly worth another shot, as you said a lot to gain and little to lose.

I think I will start putting together a process for myself to check the different package managers (apt, pip, /opt and the file locations I have used for scripts, which I have changed over the years, I really need to start being more disciplined on where I keep these scripts. I think allowing some time to audit and reshape the current setup is worth doing no matter what outcome I eventually go with.

Thanks
Paul

[idkpmiller (Paul Miller)]

As preparation I have gone through the current server checking and collecting information taking notes as you suggested about the following:

• Apt package list
• Python pip package list
• /opt folders
• Cron (all users)
• /etc/rc.local
• /etc/fstab
• Running processes
• /usr/local/sbin
• /usr/local/bin
• /usr/sbin
• /usr/bin
• /root
• /home directory’s
I already know my networking setup so no need to check that.

I will be using the ‘openhab-cli backup’ to grab the configuration.
As I will try an in-place upgrade first (next weekend after another complete week of running with the latest 2.5.11 release.
If that fails then I will start the process to create a new buster version of openhabian and setting it up as per my notes.
In this way I can simply shutdown the existing system and run this one up locally to test from VMWare workstation on my desktop before committing to moving across. So by the end of 2020 I should be using the latest 2.5.11 on Buster.

Then I can see what is going on with OH3

I did find that most of my extra tools I had installed into /opt folders and were no longer required, and also many of them used python2.7 so no use to me now anyway as I really only want to use python>3.7 going forward.

Do not get me wrong there is a lot of work needed to rebuild the openhabian system, so I am hoping that I can simply upgrade. Even if that does work I need to start archiving the add-on’s and eventually deleting them as part of a clean up.

One of my many good projects planned for this break

Regards
Paul

[idkpmiller (Paul Miller)]

Hi all,
So today, I took my last clone backup of my VMWare based Openhab VM and after changing its MAC address in the .vmx file I configured the network interfaces as ‘host only’, then after booting the clone VM up in this configuration I modified the IP address, then powered it back off. changed the network interfaces on the VM guest to be bridged and I powered it back on. I brought it online, checked it had internet connectivity and then stopped the openhab service from running and then proceeded to perform some last minute prep.

At this point I will say I had already set up a few days earlier the backup to occur every night and for the last 30 days to be retained. Thanks again Rich for the tip.
As I had a few days under my belt I now uninstalled the Amanda package, and for the time being left the data slots still in place, I will need to go back and remove these in a couple of weeks, as a clean up activity.

apt-get autoremove amanda*

I performed an apt update and upgrade from Debian Jessie to Stretch, please note prior to starting this I had got rid completely of my dependency for MySQL, as this from my experience is where every upgrade from Jessie to Stretch fails. I performed a dist-upgrade and then rebooted.

apt-get clean
apt-get update
apt-get upgrade
apt-get dist-upgrade 
apt-get autoremove
reboot

I removed all no longer required packages using autoremove and let OH2 start up. I gave it some time and went back and checked it and it seemed reasonable ok, so I stopped it once again.
I considered the upgrade from Jessie to stretch to be successful.

I once again cleaned the apt cache and changed the apt sources to now point to buster and began the update, to buster. once again doing a dist-upgrade after the standard upgrade. This time before I rebooted

apt-get clean
service openhab2 stop
apt-get update
apt-get upgrade
apt-get dist-upgrade 
apt autoremove
reboot

I took a backup on my still operating OH2 system and copied this to my new OH2 installation, I performed a restore on the buster system to get the latest changes into the new system and changed the IP address back to the normal OH system IP address and then rebooted the system.

service openhab2 stop
ip addr
/usr/bin/openhab-cli restore /root/Openhab-Backup_20201228.zip 
nano /etc/network/interfaces
poweroff

I allowed the system to bootup gave it around 10 minutes to get as much done as it could and then stopped the openhan service, waited a minute or so and then started it again, this time it went more smoothly and some basic items and calls were all found. After around 10 minutes things were settling and after around 20 minutes it all looked pretty good.
I tested a few things and it all looks good.

I will keep my jessie version on the VM host server until next 2 weeks when I will delete this and the Amanda backups as they will be of no use by then.

This has gone far smoother than I thought and I think it was because of some great advice and solid preparation work after trying the double upgrades on a few other systems first
I really hope this thread helps someone going forward

Thanks
Paul