OH3, SD Card failed ... next steps?

Running Openhabian on a Pi4 with OH3.2 …

Started getting “Error writing JsonDB” in the log, system was still running … attempted to SSH in to attempt a reboot , could not … ended up cutting the power. System wont boot.

I inserted the SD card into my Mac reader and it appears that the OH system is gone, only the boot partition appears.

I have a backup from several days ago, but would like to see what, if anything, can be recovered.

What are the next steps I should try before giving up and getting a new SD card and restoring from backup?

Not gone, it’s formatted with an ext4 file system which the Mac can’t understand. I believe Paragon makes an EXT4 driver for Mac (they definitely do for Windows) or you can run a Linux VM and you’ll be able to read that partition without trouble.

What you find there might not be in all that bad of shape. Though you’d really need to see why it’s not booting by hooking the RPi up to a monitor to see what errors it’s generating that is preventing it from booting.

Ultimately you’ll probably need to restore from backup anyway, but you might be able to recover the OH settings you’ve changed since your last backup.

2 Likes

Thanks Rich - Ok, I’ve downloaded the Paragon ExtFS for Mac and can see a rootfs partition and see my various files in the /etc/openhab directory … Is there a way to make a backup now that will be readable by a freshly installed OH system? It appears that my previous backup saved the conf and userdata directories but I’m not sure how those map to the rootfs files … i see the zigbee and zwave folders in /var/lib/openhab so the data seems to be there.

So, basic question, what’s the best way to restore a fresh system install with all the openhab files I need without an actual backup zip file?

conf is /etc/openhab and userdata is /var/lib/openhab.

The paths will be all wrong for the backup script to work so your best bet is probably to just grab both folders and copy them to the new machine.

1 Like

Great thanks. So just to be clear, once I get a fresh system going, I can stop openhab, replace those two directories with the previous ones (that I’ve saved) and restart the system. (i.e. there no other directories needed).

Correct, for openHAB. If you have other stuff running on that machine though, like Mosquitto or InfluxDB those will have to be backed up and restored separately.

1 Like

Yes, understood. Thanks!

Ok … have the system 95% backup … thank you Rich for your quick help.

This error appeared in the log after restarting the system after copying the previous SD cards openhab folders. It is the same or very similar to the errors I was seeing just before the SD card went belly up …

2021-09-09 14:13:54.633 [ERROR] [re.storage.json.internal.JsonStorage] - Error reading JsonDB from /var/lib/openhab/jsondb/org.openhab.core.thing.Thing.json. Cause com.google.gson.stream.MalformedJsonException: Unterminated string at line 28793 column 26 path $.zwave:device:0d28f614c7:node4.value.properties.dbReference.

Perhaps that was where corruption occurred and that file was unexpectedly terminated?

Yep, OH was in the middle of writing to the JSONDB when the power was pulled. Before writing though, it will copy the file to $OH_USERDATA/jsondb/backup so you haven’t lost anything. Look for the version of the file with the biggest number in front (that’s the timestamp) and copy that over instead.

1 Like

FYI, the Pi didn’t lose power, (i.e. there was no reboot), but there could have been a power glitch or a fatal memory failure. I was seeing dozens of JSONDB Errors before I tried to log into the Pi to attempt a graceful shutdown. I couldn’t ssh in, so I pulled the power.

openhabian running openHAB 3.2.0 Build #2469 on a Pi4 4GB with a 32GB SD Card

So, I have my system back, wasn’t difficult, but I’m thinking about how to fool proof it more … without adding more complexity and therefore more potential failure points, I’m looking at the following:

  1. Battery backup / UPS / surge protection
    a) Graceful power-down after power failure, w/restart when power returns
  2. Keep a cloned SD Card with same Openhab system
    a) needs updating everytime Openhab system is updated to keep it identical.
  3. USB stick for off-card storage of openhab userdata and conf folders (is this recommended?)
    a) What about event and openhab log files?
  4. Doing 2&3, doesn’t necessarily improve reliability, but makes it easy to quickly swap SD card with new system.
  5. Nightly Rebooting of OpenHab
    a) Reduce likelyhood of SW failures

I’ve read the Amanda openhab docs and in terms of protection from just SD card failures, I’m not sure I need it. I realize there are many other parts of the system that can (and will fail), but for now, I’m only concerning myself with SD card failures (from power failures or other memory/sw glitches).

In terms of SD card failure protection, what else should I be considering?

  1. Absolutely. This is by far the best thing you can do to protect your file systems.

  2. Lots of people do this but there are alternatives. You mention Amanda already which does incremental full machine backups. The advantage there is the backuos happen automatically with no down time. But restores take a little longer than just dealing the sd card.

  3. USB thumb drives are flash too and have all the same problems as sd cards. In openHABian, logs and such get logged to a zram partition and but to the sd card, saving wear and tear.

  4. This is as likely to cause problems as it is to prevent prevalent and is not recommended. It’s always better to find and stove the root cause of a problem and solve that. OH and systems like this are designed to run for long periods of time. Rebooting is quite disruptive over all.

1 Like

That is already available in openHABian. Get another card reader and setup SD mirroring.

It DOES improve reliability. Correct it does not increase MTBF but “reliability” is not just MTBF but essentially the quotient of MTBF and MTTR, and mirroring greatly lowers MTTR.
Read: The faster you’re back up when things go wrong, the higher your reliability (or “availability”) is.

You should use it, it has various advantages like to store multiple generations of a file over time.
It’ll also take care of your #3. You can store to essentially any medium, including an USB stick, a NAS and AWS.

Use “Endurance” type SD cards.
I’d get a 16GB one, install to that from scratch, and use your existing card as the mirror, plus storage for Amanda. You can have openHABian setup everything automatically, see these docs.

1 Like