HELP! Big trouble with ZRAM and Read Only File System

I am running on a RPi 4. I am running OpenHAB 3.0.2. The system has been running for quite a while and /var/log filled up, probably because I had the logging set too high. I changed this, then attempted to clean up the excess logs to free space. In this process, I clearly upset things and the overlay file system went into Read Only. Now, I am afraid to reboot it and really don’t know how to proceed.

The underlying problem seems to be with Overlay 1. The RPi forum was asking me why I chose to use ZRAM…I didn’t realize I was(!!!) so I knew it had to come with a package I had installed. Since OpenHAB is the only application of consequence running on it, I searched and found this: ZRAM status so it would appear this is indeed and OpenHAB issue.

PLEASE HELP! I really like my installation and don’t want to need to rebuild it…

If you started from an openhabian image, zram is installed by default. If this is the case you can simply reboot: you will only loose the logs and the openhab persisted data.

You have to realize that the openhabian image has a slightly modified version of raspberry OS, explicitly optimized to work with openHAB and some other accompanyng application (as mosquitto or nodered or similar). If you install something else you are on your own and people from RPi forum may not know the details of openhabian tweaks.

Thank you SO much. I didn’t realize and therefore I think that RPi advice was probably inconsistent with the implementation. So, a reboot should be fine? I am SO paranoid that I am making a bootable USB right now because I’m afraid to reboot it…

Okay, I bravely rebooted and all is indeed fine. Thank you SO very much. I appreciate the fact that you spent your valuable time to help me.

Just to answer that question in particular. An openHABian system is designed to run non-stop without booting for long periods of time (years) and for a number of reasons, it does a lot of writing out to the file system. Each and every write slowly wears out the underlying SD card (each cell in an SD card can only support a certain number of writes before it’s done and can’t be written to any more). In order to extend the life of the SD card, ZRAM was implemented so that those many hundreds of writes per minute instead are made in RAM, preserving the SD card.

Before openHABian implemented ZRAM, posts about corrupted SD cards were pretty common here on the forum.

Now that you are back up and running now is a good time to consider a backup strategy so when this happens again (and it will) you are not faced with rebuilding from scratch. In fact, good backups are kind of freeing because you can go mucking about like a bull in a china shop and know you can get back to where you started if you break something.

I’m really glad you got it working.

The feedback about backups is right on. I realize I haven’t been properly managing the toolset and I need to do a better job. I am anxious to hear suggestions as to the best methods. I have a hard drive installed on my RPi, so I am not clear how to best manage this environment. Suggestions…please.

With appreciation for all the help.

It depends on a lot of things. At a minimum openhab-cli has a backup option that will capture all your openHAB configs. You can run that and transfer the zip file it generates somewhere safe. I don’t know what else you might be running so you’d have to look into what needs to be backed up from those. In some cases it could be as simple as backing up a couple of files or folders (e.g. Mosquitto) and in others it might be quite involved (e.g. InfluxDB).

However, if the RPi is running off of a hard drive, then ZRAM really isn’t needed. A hard drive isn’t an SD card and doesn’t wear out in the same ways. However, if you are running openHABian, running off of an external drive is also no longer officially supported anyway so we are in uncharted territory. If the hard drive is just plugged in though as additional storage, it might be worth while setting up Amanda (one of the openHABian options) which will take incremental backups of the machine and put those on the hard drive.

All-in-all, it gets difficult to provide any concrete advice without a lot more details about the environment (e.g. is there a shared network folder available somewhere?) and what all you need to back up.

Oh my! I added the hard drive because I thought I would need it. If I am in uncharted territory, maybe I should think about unwinding it…can this be done? The only other thing running on my RPi is email, but I don’t use it (because of other issues with it). So, in the end, OpenHAB is the only think of any value running. Where I would like to get to: 1) a stable, “vanilla” install; 2) something that I don’t need to babysit; 3) a set up where back ups are easy to do (and easy to recover). Please advise. I really have become quite dependent on the automation OpenHAB provides and any bump-in-the-night sends me into high anxiety.

I am not unhappy with the hard drive. If I can use it and stay ‘vanilla’, that would be the smoothest path for me. I am a newbie to Linux (not to systems but all my experience was on much bigger systems).

Again, thank you for all the help. It is most certainly appreciated.

It all depends on what you’ve done. If you are booting off of the hard drive you are in an unsupported configuration (not necessarily uncharted, just not directly supported). But if it’s just a drive connected to the machine, no problems. All you’ve done is added some extra storage somewhere under /mnt or /srv or /media (I don’t know where you’ve mounted it).

I’m not trying to scare you or anything like that. I just don’t know what you’ve done so I try to cover all the bases.

Given your 1-3 I would recommend the following, which very well may get you to the same place you are now:

  1. Create a backup of anything important on that machine. Make sure the backup is a good one by opening it and making sure everything looks reasonable. I think you’ll see a conf and userdata folder and under that you should see some familiar folders. But the biggest thing is that you can open the zip file. That alone is a good indication the backup is probably good. For openHAB you can create the backup using openhab-cli or menu option 50 in openhabian-config. It will tell you where the file was created.

  2. Copy that backup off of the RPi (thumb drive, Samba shared folder, FTP, email it to yourself, whatever).

  3. On a new SD card burn openHABian. Boot that and let it set up OH for the first time.

  4. Copy that backup file back to the machine and use openhab-cli restore or openhabian-config. That will put your OH config right back the way it was.

  5. Test it out to make sure all is working as expected.

If disaster happens and nothing works, you’ve still the old SD card you can boot to while we figure out what went wrong.

Once you know everything is working well you’ve some options.

Under that backup and restore menu in openhabian-config you can set up:

  1. Amanda: takes incremental backups of changes to your system
  2. SD mirroring: keeps an external SD card in sync with the running SD card as a backup
  3. Raw SD: makes a byte for byte copy of the running SD card
  4. Sync SD: copies the contents of the running SD card to another machine using the rsync command

In all of these cases you’ll need some other storage. Maybe that’s what your hard drive could be used for. Maybe that’s what you’re already doing with that hard drive. 1, 3 and 4 can all use that hard drive.

This may sound like a silly question, but would I be just as far ahead to get a NEW RPi and start there, using backup (I did backup OpenHAB and copied to a thumb drive).?

I do not remember all the steps I went through to put in the hard drive ( I followed the manufacturer instruction so I can locate that). I do boot from it, For the minimal cost of another RPi, maybe I could (to a large extent) parallel my current environment. I used to boot from an SD card, and originally set up OpenHAB that way. I believe I changed the boot sequence in RPi config tool, so maybe I could back to booting from the card.

Forgive all the questions, I am just trying to come up with a good plan to get me to where I want to be.

Update: I have confirmed I have a good backup on a thumbdrive.
The hard drive is: Crucial BX500 240GB 3D NAND SATA 2.5-Inch Internal SSD, up to 540MB/s The SSD required this second part to interface with RPi: Geekworm NASPi 2.5 inch SATA HDD/SSD NAS Storage Kit with Safe Shutdown Function for Raspberry Pi 4 Model B(Not Include Raspberry Pi 4)

The case/interface did use one…oh maybe two USB ports to connect. I put it all together in Jan and I am not finding the instructions. I contacted the seller to get a copy.

If I can change my boot sequence and use my SD card with the drive just being extra storage, does that make my situation better?

(head spinning)

Not quite. It only (r)syncs the internal SD with the mirror device you’ve already setup.

I would not use a hard drive as the mirror unit of the internal SD.
That’s not the way it’s designed to work. It might work or not but either way it’s untested and not guaranteed to keep working with future openHABian changes.
Use at least the same type of medium, i.e. SD.

To advise, I’d get a second RPi, an USB card reader with another SD and install that from scratch using openHABian defaults, then import your existing config there using menu 50/51 or openhab-cli on the command line.
You can also change your existing Pi’s boot order but we cannot advise on modified systems like yours so since you should be having spare parts on site anyway and you say you’re dependent on your system I’d play it safe like this.

Why did you do this without even reading the official docs ?
Oh my! No offense there’s several who do it. But I just cannot understand why people do, even more so when they’re reliant on their home automation.

It’s all in there:

Points well taken. I started in as an experiment, never dreaming that I’d get so “into” it. OpenHAB is one heck of a great product (I recognize this having spent a career in software dev), but when I started, I didn’t do much reading because…well…it was an experiment and I didn’t have high expectations (WRONG!!). NOW, I see the errors of my ways and want to make it right. Hopefully, others starting out will find this thread and will my mistakes.

I will start again, then bring in the work I have already done…FOLLOWING THE DOCUMENTATION! However, as I start reading, I immediately see that OpenHAB can run on many platforms, not just the RPi. That leads me to more questions — what is the OPTIMAL platform? I do have a Linux laptop that has nothing live on it so I can use it, I can buy another RPi, I have windows machines. What is the BEST, given my situation?

With appreciation,

What’s your definition of “optimal”? What do you want to optimize?

What’s optimal for me is likely not going to be optimal for you. For me I wanted to optimize hardware given the dozen or so services I run (in addition to openHAB). So I’ve a server class machine running ESXi with now two VMs and all my services run in containers (everything is managed by Ansible). That’s optimal for me. It’s almost certainly not optimal for most.

We like to encourage those who are a little less technical and experienced to use a “stock” configuration: openHABian on an RPi 3 or RPi 4. That will optimize your ability to quickly and efficiently get help from the forum since it’s a known configuration we don’t have to ask 20 questions to figure out what’s what even before we even get to your actual problem. So that maximizes commonality which maximizes our ability to help you on the forum.

But we also have said in the past (I don’t think it’s in the docs anymore) that the best hardware to run openHAB on is hardware you already have. But that’s often going to drive how you install OH and such. That will optimize your hardware investments.

I could go on.

And your optimal solution will change over time and you may want to migrate to new approaches.

Good points…Optimal for me (at this point) is 1) reliable; 2) supportable; 3) does not require a lot of babysitting. Yes, I can handle more complex environments, but at this point in my life, I don’t want to. I like the RPi because it is small. reliable and relatively straightforward. That said, I will order a new RPi 4 and do my best to bring it up.

I plan to follow these instructions: openHABian | openHAB to bring up my new box. Am I in the right place? Once up, I am sure I will need guidance to properly import my backup. One thing that strikes me is the IP situation. Since I have a live production system, I don’t think I want the new RPi using the same IP. Or does it matter? Perhaps I am overthinking…

Yes, that will get you to the generally recommended solution. openHABian handles a lot of the config for you.

It does matter but not because of OH. You can’t have two machines on the same network with the same IP address, period. If you are using DHCP, the new machine will be given a different IP as a matter of course. That’s what DHCP does. It’s a good idea to configure in your router (where available) that your machine gets served the same IP address. That almost always is easier and works better than trying to do static IP on the machines themselves. I’m not certain the openHABian supports that anyway.

But as long as you are running both machines, each will need their own IP address.

Sorry, I think I worded the question incorrectly…let me try again.

I have ordered a new RPi. When it comes to life, it will have a different IP address (whether static or DHCP). [Since I am now using OpenHABCloud, I believe I can now go DHCP with the new RPi]. At any rate, it will have a different IP address. If I import the existing OpenHAB config from my production system, that will include the IP address currently used (it is a static IP at the present time). When I import the configuration, will that not cause a conflict?

Sorry for all the questions, but I am trying to make sure I don’t take another wrong turn!

With appreciation,

Ah, no, that won’t be a problem. OH doesn’t really have anything in it’s config that is directly tied to its IP address. The same config will work no problem even if the new machine has a different IP.

  • There might be one thing that could cause problems. If you’ve modified any of the config files in /var/lib/openhab/etc by hand to change which IP address OH listens on you might need to change that. It’s almost certain you haven’t so there should be no problems.

Under OpenHAB Settings, Network Settings, there is an IP address. While I do not recall changing anything in /var/lib/openhab, but I do see that static IP is stored in the Network Settings. I suspect I may have set it when I went to static IP. This is the setting that I was wondering if it would cause things to go a bit crazy if I import the backup…

New RPi won’t be here for a day or so…I am just trying to think through the work ahead of me and make a list of the things I will need to adjust after I do the basics.