Hi, seems like my OH 3.4 on Pi4 died. Had some file corruption apparently in the main OS files and I’m unable to get it to boot again.
So… looks like I’ll need to reinstall from scratch and bring back the backups which I was doing with Amanda. I didn’t backup all folders, but the ones below:
/etc/openhab
/home/openhabian
/use/lib/node/modules (I use nodered extensively)
/usr/share/openhab
/var/lib/openhab
Question: if, after a fresh install of the latest 3.4.x with Openhab processes off/shutdown, I simply go and recover those folders overwriting what comes in the standard install, should everything go back to normal or should I expect side effects, and which?
Didn’t work. Seems like I have a problem with influxdb. I didn’t have a backup of it, but, /var/lib/influxdb was still readable in the corrupted disk, so I could recover it. But that didn’t work as well.
This is what I see in the logs:
2023-08-29 02:53:17.904 [INFO ] [el.core.internal.ModelRepositoryImpl] - Loading model 'influxdb.persist'
2023-08-29 02:53:18.179 [WARN ] [el.core.internal.ModelRepositoryImpl] - Configuration model 'influxdb.persist' has errors, therefore ignoring it: [1,1]: mismatched input '<EOF>' expecting 'Strategies'
All my Things have this state and error, except from 1 binding (Wiz):
Status: UNINITIALIZED
HANDLER_MISSING_ERROR
Handler factory not found
And I think that’s because I used a self-installed file for Wiz binding.
But the bindings are there, at least according to the UI… so must be something related to the influxdb.
Any hints will be appreciated. Would love to avoid having to recreate everything, and lose my historical data.
Thanks @rlkoshak, I cleared the cache with openhab-cli and those errors are gone. Most of my Things were recognized then. Just the MQTT ones had a Bridge error.
Then removed the InfluxDB binding, since I noticed all my data was in OpenHab despite the Influxdb errors, so seems like I was really not storing anything in InfluxDB anymore.
Then installed Mosquitto, my MQTT Things returned to life.
Last but not least, installed Node-red. My flows were gone. Had to Import them from the backup of /home/openhabian/.node-red/flows_openHABianDevice.json
Seems like everything is normal now.
A note of caution to other out there: this whole experience left me uneasy with Amanda. It’s not a backup that easily gets you back online by bringing the files back. I’m going to try the SD card mirroring this time, seems safer. Maybe doing it in parallel with Amanda to a networked drive/folder to have extra copies of files just in case.
That’s the recommendation anyway as is to validate your backups after setup.
If your SD to contain backups breaks you’ll be in trouble.
It’s all in the Amanda docs.
I think the documentation for Amanda is fine, it is clear and helps to configure and to recover.
The concern isn’t really Amanda, but what you need to have backed up and how to pull everything back in. I’m thinking now one probably needs to backup everything, from / down, to be able to use it to recover easily. Or maybe backup the two partitions in the SD card to a networked drive.
IMHO validating backups is out of reach for most people, timewise. What would I do, make a new installation of OH 3.4 in another SD card, then use Amanda to bring back all the files, showdown my Pi, boot the new card, and see if it works? How frequently? It’s just not practical, even though I understand it is the only way to be sure it is reliable.
Bringing back a whole partition would be simpler, assuming that works. I’ll give it a try at some point soon.
An unvalidated backup is little better than no backup at all. So you are basically saying that backing up at all is out of reach for most people.
Yes, if for no other reason to than to learn and document the process as well as to verify all the technical parts are working as expected and everything necessary is actually backed up.
At least once. I’d recommend every time there is a major change (e.g. installation or removal of a new major version of a service, major OS upgrades, etc) or maybe once a quarter but a minimum of at least once.
Then backing up at all is just going to be an arcane ritual one does to ward off the evil eye. If you don’t know that the backing up process is working and it’s backing up everything you need it’s worthless. If you don’t know how to restore from the backup it’s worthless. And you can’t know either of those without testing it. If you can’t test it, why bother with backing up at all?
That’s a different discussion about backup strategy. What to back up and how can be greatly informed by your experience testing a restoration.
Well this obviously cannot work as long as you only have one single medium for both, content and meta data about the content you backup. It’s like having a replica of a valuable painting and keeping both in the same house that can be robbed or burnt down, isn’t it.
You need to copy the Amanda database itself somewhere off-site at times. IIRC there’s already a cron job to do that you just need to copy the result somewhere off the box at intervals.
As said, the recommendation is to have both, mirroring and Amanda .
He he he this is an interesting conversation. Look, I fully agree with you in a professional setting. Unacceptable to not test backups and be sure they work, regularly.
Problem is that for personal use I’m wanting to have a consumer-like device, where I don’t need to go through all those pains. Because, you know, in the end the risk of things going wrong is relatively low. Risk being likelihood x impact, with low likelihood (because the backup solution is considered good by others) and low impact of total loss (in my case, YMMV).
In other words, my dream scenario is that OpenHAB offers some way to backup configs and stored data, which I can trust without having to check it because I know others have used it and it worked for them, especially the developers or maintainers of that part who take pride in what they’re doing (I’ve been there), and so end up testing and retesting that part a lot of times. If it works for them and for other users, it should work for my setup as well. I run vanilla Openhabian on a vanilla RP4… okay, maybe node-red isn’t vanilla, more like raspberry jelly mixed in… anyway…
So completing the dream scenario: if a big crash happens I just install the same version of OH I had into a new card then bring back the backed up configs and data.
Okay, I might try that once, after adding a Thing. If it works, I’ll trust it will always work. In the very unlikely event it doesn’t, well, I’ll be disappointed, but whatever. It’s not like I’m running a life support system on OpenHAB, you know?
Again, same here, fully agree in a professional setting. For my personal stuff, with a consumer-product-mindset, and especially knowing how zealous and competent most open source developers of established projects like OpenHAB are, I can trust it should all work and be fine. And if not, okay, c’est la vie…
Yes, I will have Amanda backing up to a networked drive. Not off site (won’t survive a fire) just off-device, but would survive the cat deciding to pee on the RPi, remove and chew the SD card, etc…
In fact my Amanda backup was to another device, an RPi400 not running OH, but with an attached SSD accessible via network.
Even in a personal setting this applies. The only difference is how often and the degree of the checks the person needs to perform. But even a personal consumer must test the restore process to make sure: 1. it works 2. everything needed is indeed backed up.
But OH is not a consumer product. For it to become a consumer like product it would need to be way more locked down than it is and your options would have to be way more limited. How many consumer products allow you to log into the device, have root, install arbitrary new software, install on non-standard hardware and/or install in non-standard ways?
Consumer products are able to provide simpler processes for backup and restore (you should still test it because even there they can be surprisingly incomplete) but they do so because they limit the end users’ options.
Limiting user’s options is not what OH is all about.
He he he this is an interesting conversation. Look, I fully agree with you in a professional setting. Unacceptable to not test backups and be sure they work, regularly.