Openhab lost 3 weeks rrd data after power down

Setup, Configuration and Use Beginners
1
  • Platform information:
    • Hardware: Pi4
    • OS: openhabian openhabian-pi-raspios64-202308121313-gitc6a2fa1-crcaf8d881d.img.xz
    • Java Runtime Environment: 17.0.10+7-1~deb11u1
    • openHAB version: 4.1.1-1
  • Issue of the topic:
    I’ve been running open habian with a working 1-wire setup for about 3 weeks, collecting data from 10 temp sensors, and it’s been working very nicely - I even got a nice home screen set up with various graphs and temps.
    I went away for a day and had to take the USB-C cable that was powering the Pi with me so I pulled the power. I plugged it back in after 2 days and when I looked at my graphs they just started from then. The previous 3 weeks RRD data was all gone!
    That’s not expected is it? The whole point about RRD is that it’s hard to lose your data.

I also seem to have lost my ‘overview’ screen setup after another couple of days (It was working after the powerdown but maybe only because it was cached in a browser window?)

So can someone tell me where the RRD files live - maybe my old data is still there and can be rescued?
Why might this happen? OK, I didn’t do a proper shutdown, but I don’t see a way in the UI to do that anyway, and it shouldn’t lose weeks of data from an rrd database.

Similarly where does the overview page config live. Might I be able to rescue that from somewhere?

The ‘things’ configuration is still there.

Looking in /var/log there are some trivial files from feb 7th and then bothing until mar 6th 09:17 (when I plugged the power back in). I see that /var/log seems to be in-memory, which would explain why all the log files are gone: overlay2 on /var/log type overlay (rw,relatime,lowerdir=/opt/zram/log.bind,upperdir=/opt/zram/zram2/upper,workdir=/opt/zram/zram2/workdir,redirect_dir=on)

Looking in var/lib/openhab/persistence/rrd4j there are rrd4j files for the 10 temp sensors. Can I interrugat them directly to find out how far back they go (I bet it’s just to the 6th)?

I saw a /var/lib/openhab/backups directory which was encouraging, but it’s empty.

Anyone got any idea what’s gpoing on here? That data was important - it was for my heat-load analysis. It’s spring now - I may not get another 0C night before about November now. I really wasn’t expecting it to just evaporate!

Oh and is there a way to backup the data I have now to avoid the same problem again? I just wandered round the interface and didn’t see a ‘backup’ option.

Clues welcome.

2

It is always a really bad idea to just yank the power from a computer. It’s doublely a bad idea to yank power from a computer running off of flash media (I.e. an SD card). That is likely to lead to file system corruption given how flash memory works.

openHABian uses a service called zram to prevent wear on SD cards by saving most changes, including rrd4j databases, in RAM, only writing that data out to the SD card on a normal shutdown.

If you just yank the power, everything that has been saved to rrd4j since the last proper shutdown is lost because you never gave it a chance to flush the contents of zram to the SD card.

They live in /var/lib/openHAB/persistence/rrd4j but the old data is gone. It was lost when you yanked the power without a proper shutdown.

See above. Furthermore, the way flash memory works, if there happens to be a write to the SD card taking place when you yank the power, you not only lose the data being written but everything else that is stored in that sector on the flash memory. You could lose a part of the kernel, file system tables, anything.

It’s really bad to just pull the power.

openHAB is just a program running on an operating system. You don’t see an option to shutdown the whole computer in Chrome or Word either. You need to interact with the OS to initiate a shutdown. For Linux that’s sudo shutdown -h now from a command prompt. There might be an option from openhabian-config too.

The Locations, Equipment, and Properties tabs are generated dynamically based on the semantic model. Everything else is stored in /var/lib/openHAB/jsondb. But when you yanked the power, you lost all the configs you’ve made since the last normal shutdown or reboot.

I think there are tools out there but I can tell you right now that everything saved between the last normal shutdown/reboot and when you yanked the power is gone.

You can also get the data through Developer Tools → API Explorer on MainUI.

That’s where the backups created when you run opengab-cli backup go.

openhab-cli backup and don’t let the machine just suddenly lose power. You can turn off zram through openhabian-config but you’ll have to watch the machine like a hawk for SD card wear out failures. When an SD card wears out, it just stops saving stuff and weird things start to happen. Zram is configured by default for a good reason.

openHABian has *several" backup options including SD card mirroring and Amanda backup. There’s openhab-cli which just captures OH stuff. See the openHABian docs for details.

1 Like
3

To add another detail :slight_smile:
The current openHABian version will create a cron job which then will flush zram on a regular basis (err… once a day?).
I don’t know, though, if it’s sufficient to update openhabian-config or if you would need to reinstall zram or to do something else.
mstormi had mentioned this feature in another thread, I’m just to lazy to search for it… :wink:

As for power-down command, I’m not aware of such an option in openhabian-config.

sudo poweroff

should also do the job.

1 Like
4

OK, thanks for the comprehensive, if chastening, answer. It did a sufficiently good job of behaving like a gadget such as a router, that I assumed it would be configured to be more tolerant of powercuts. I am well aware that you ought to shut machines down properly, but these days if your laptop or phone goes flat, or the router or TVbox, or solar-thermal controller (all running linux-based distros) are powercycled, actual problems and significant data loss are quite rare. 3 weeks of stuff in a tmpfs with no flash syncing is not what I expected (obviously). Still full marks to someone for avoiding SD wear, which is otherwise a big problem.

So if you get a powercut, and openhab has been running for (maybe months), that could be pretty serious from a data-loss POV. Still,now I know. It could have been worse.

openHAB is just a program running on an operating system. You don’t see an option to shutdown the whole computer in Chrome or Word either

Well, yes but openhabian-on-a-pi is a whole distro-configured-as-gadget, and my laptop and TV box both show ‘reboot/power-off’ options. If it’s important to shut-down properly, I’m surprised that I’m expected to ssh in and shutdown, like it was a buildd or server. I hope you can see why I made the wrong assumption. 20 years ago I’d agree: anyone just pulling the power on a machine got what was coming to them (never do that to an XFS machine, for example), but these days it’s a surprise.

Oh well. I learned something this week.

5

Please be aware that this is a Raspberry Pi. It’s not a simple router (which, by the way, will not save lots of data, in most cases the os is read only anyway and configuration is sort of static)

And of course you shall never ever cut power to a “real” server.
If critical, they tend to have redundant power, one of them with UPS, the other without (or with another independent UPS), and that is not only because the machine should never fail, but also to avoid data loss through a power loss.

6

I get why you would think that given that many devices can just have their power cut without consequences. As pointed out, it’s entirely about the SD card. It’s common knowledge in the RPi community, but many new RPi users unfortunately don’t realize this until it’s too late (as in this case).

I’m currently working on a rewrite of the openHABian documentation, and am planning to mention this for the benefit of new users.

Correct, unless you’re on the version of openHABian that schedules a daily cron job (as Udo mentioned). But more than that, I recommend a small UPS that can keep your RPi and modem/router going for a short period of time.

Note that Network UPS Tools can now be installed from openHABian-config. I haven’t tried this myself, so I haven’t updated the tutorial yet.

This depends on your point of view. I’d agree that openHABian presents this way to the new user, but it’s actually just Raspberry OS Lite with scripts to install openHAB and some other software. So, we kind of need to separate the three main elements that you get from the image:

  1. Raspberry OS Lite.
  2. openHABian-config, a program for simple configuration of services and software, accessed through the Linux command line.
  3. openHAB, a program that is accessed through a web interface.

That’s why it doesn’t make sense for Main UI to have built-in shutdown/reboot commands–for all it knows, it could be running on Windows (where sudo poweroff does nothing).

If we were to make comparisons with a TV box, openHAB is equivalent to the YouTube app. We wouldn’t expect YouTube to be able to reboot/poweroff a TV.

But as you say, it’s nice not having to SSH into the RPi to shut it down. So…

7

Just a thought ,
You could install the exec binding and set it up with a shutdown command at least then you would be able to shutdown the system from GUi. There are a few different discussions that do just that. You should be able to write a script perhaps even get elegant and have that would trigger a graceful shutdown of Openhab a full backup and then finally power off the device.

8

The link @rpwong posted at the end of the post just above yours explains exactly how to do that.

9

Not sure why him saying go to SSD is same as installing exec binding but thx pointing it out to me that he also provided a link to do a exec binding install as well.I did not see (click on that) link.

10

No where in the reply I replied to does it say anything about going to SSD. I was only referring to that post.

And I wasn’t pointing it out to you specifically but to future readers of this thread who may have skipped over Russ’s post without realizing that he posted such a tutorial given the way your reply was worded it sounds like it doesn’t.

11

Sorry I read that part of his post when I replied as not having a SSD after rereading it was a typo I guess and he meant not having to SSH into host.

12

I totally missed that typo. I know what he posted and mentally just changed it to “ssh” in my head. This is why I’m not an editor.

2 Likes
13

Yep, typo. Of course, the title of the link directly referred to a reboot/shutdown switch.

Fixed now, thanks!