Well, I recently migrated from OH2 to OH3…
A new rPi4, new OH3 install eight days ago.
One rule change 2 days ago; since then it has been pottering along
Today OH3 simply stopped working!
The terminal session was still responsive.
Checking the service:
openhab.service - openHAB - empowering the smart home
Loaded: loaded (/lib/systemd/system/openhab.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/openhab.service.d
└─override.conf
Active: active (running) since Sun 2022-11-27 22:44:18 AEST; 1 weeks 2 days ago
Docs: https://www.openhab.org/docs/
https://community.openhab.org
Main PID: 774 (java)
Tasks: 270 (limit: 4915)
CPU: 1d 9h 30min 2.896s
CGroup: /system.slice/openhab.service
└─774 /usr/bin/java -XX:-UsePerfData -Dopenhab.home=/usr/share/openhab -Dopenhab.conf=/etc/openhab -Dopenhab.runtime=/usr/share/openhab/runtime -Dopenhab.userdata=/var/lib/openhab -Dopenhab.logdir=/var/log/openhab -Dfelix.cm.dir=/var/lib/openhab/config -Djava.library.path=/var/lib/openhab/tmp/lib -Djett>
Dec 07 01:05:32 openhabian karaf[774]: org.ops4j.pax.logging.pax-logging-api [log4j2] ERROR : Unable to copy file /var/log/openhab/events.log.1.gz to /var/log/openhab/events.log.0.gz: java.nio.file.FileSystemException /var/log/openhab/events.log.0.gz: Read-only file system
Dec 07 01:05:33 openhabian karaf[774]: org.ops4j.pax.logging.pax-logging-api [log4j2] ERROR : Unable to delete 1, /var/log/openhab/events.log.1.gz: Read-only file system
Dec 07 01:05:33 openhabian karaf[774]: org.ops4j.pax.logging.pax-logging-api [log4j2] ERROR : Unable to copy file /var/log/openhab/events.log.1.gz to /var/log/openhab/events.log.0.gz: java.nio.file.FileSystemException /var/log/openhab/events.log.0.gz: Read-only file system
Dec 07 01:05:33 openhabian karaf[774]: org.ops4j.pax.logging.pax-logging-api [log4j2] ERROR : Unable to delete 1, /var/log/openhab/events.log.1.gz: Read-only file system
Dec 07 01:05:33 openhabian karaf[774]: org.ops4j.pax.logging.pax-logging-api [log4j2] ERROR : Unable to copy file /var/log/openhab/events.log.1.gz to /var/log/openhab/events.log.0.gz: java.nio.file.FileSystemException /var/log/openhab/events.log.0.gz: Read-only file system
Dec 07 01:05:34 openhabian karaf[774]: org.ops4j.pax.logging.pax-logging-api [log4j2] ERROR : Unable to delete 1, /var/log/openhab/events.log.1.gz: Read-only file system
Dec 07 01:05:34 openhabian karaf[774]: org.ops4j.pax.logging.pax-logging-api [log4j2] ERROR : Unable to copy file /var/log/openhab/events.log.1.gz to /var/log/openhab/events.log.0.gz: java.nio.file.FileSystemException /var/log/openhab/events.log.0.gz: Read-only file system
Dec 07 01:05:35 openhabian karaf[774]: org.ops4j.pax.logging.pax-logging-api [log4j2] ERROR : Unable to delete 1, /var/log/openhab/events.log.1.gz: Read-only file system
Dec 07 01:05:35 openhabian karaf[774]: org.ops4j.pax.logging.pax-logging-api [log4j2] ERROR : Unable to copy file /var/log/openhab/events.log.1.gz to /var/log/openhab/events.log.0.gz: java.nio.file.FileSystemException /var/log/openhab/events.log.0.gz: Read-only file system
Dec 07 01:05:35 openhabian karaf[774]: org.ops4j.pax.logging.pax-logging-api [log4j2] ERROR : Unable to delete 1, /var/log/openhab/events.log.1.gz: Read-only file system
… showed read-only file system.
Checking it:
# [2022-12-07 14:35] openhabian@openhabian /var/log/openhab $
la
total 38M
drwxrwxr-x 1 openhab openhabian 4.0K Dec 6 14:10 ./
drwxr-xr-x 1 root root 4.0K Dec 4 00:00 ../
-rw-rw-r-- 1 openhab openhab 0 Nov 18 16:03 audit.log
-rw-r--r-- 1 openhab openhab 28M Dec 7 08:28 events.log
-rw-r--r-- 1 openhab openhab 1.3M Dec 4 03:55 events.log.1.gz
-rw-r--r-- 1 openhab openhab 1.5M Dec 4 13:01 events.log.2.gz
-rw-r--r-- 1 openhab openhab 1.4M Dec 4 23:03 events.log.3.gz
-rw-r--r-- 1 openhab openhab 1.3M Dec 5 09:34 events.log.4.gz
-rw-r--r-- 1 openhab openhab 1.5M Dec 5 18:03 events.log.5.gz
-rw-r--r-- 1 openhab openhab 1.3M Dec 6 05:00 events.log.6.gz
-rw-r--r-- 1 openhab openhab 1.4M Dec 6 14:10 events.log.7.gz
-rw-r--r-- 1 openhab openhab 780K Dec 7 09:50 openhab.log
-rw-r--r-- 1 openhab openhab 317 Nov 18 16:07 openhab.log.1.gz
-rw-r--r-- 1 openhab openhab 522 Nov 27 22:44 openhab.log.2.gz
-rwxrwxr-x 1 openhab openhab 0 Jun 27 16:19 Readme.txt*
While this is a SanDisc brand-new card from the shop, as blister-ware, to make sure I got an orginal (actually four of those); I also ran a full write test on them to ensure all is good.
I never liked SD cards in SBCs, and prefer SSDs instead.
What is then the tenor in how to move forward?
Should I:
a) just burn the SD card onto another SD card?
b) burn the SD to a SSD?
c) rebuild the machine, but this time with apt, rather than the openHABian image
d) any other ideas?
There’s no need for SSD or leaving openHABian for that in my experience. I always went with option a). There’s a menu in openHABian for that - if you’re able to access the SD card that is. otherwise make sure to export your config (backup option in openHABian or CLI) and at best you have amanda-backup in place, to be sure not to lose anything, if the SD card suddenly goes blank.
on the other hand: if the SD is really brand-new (I even had openHAB instances running on cheap nonames for years), it also could have been some OS-related mishap, which made the filesystem RO. Can’t rule out openHABian/openHAB related issues in your specific use case, but seems very unlikely.
is it openHABian only? or do you have anything other installed/configured on the same raspberry?
openHABian with influx/grafana installed via the config tool, but not being used yet.
This new rPi4 is supposed to be the new OH3 machine.
It runs in parallel to the current OH2 system…
But has since I installed it:
a) stopped logging and needed a rebuild as I could not fix it
b) shows random script errors
c) and today stopped working
… none of that I experienced in OH2 on an SSD in over 4 years.
My experience is completely the opposite. It’s not a question of “if” but “when” the sd card will die. Based on that, do you honestly want to carry that in the back of your mind?
Whenever you have any kind of problem with openHAB, you’ll always be wondering “if the sd card working correctly?” And just think of all of the weird errors you start getting when the file system starts dying. How many hours will you put into it before you decide “nope, gotta be the sd card?”.
Like… for openHAB you don’t even need an ssd. Just get a cheapo 2.5” laptop hdd. It works fine, and you can check that future issue out of the list.
Edit: power draw?? I’m pretty sure we’re discussing peanuts here right? An ssd might use 50% less energy, but 50% less of 5 watts is still as insignificant as 5 watts anyway…
perhaps I’m lucky?
My main openHAB2 ran approx 4 years with the same SD Card (>1000 items, most of them changing at least every minute - some every 10 or 1-2 seconds). Logfiles, RRD4j persistence, etc. on that SD Card.
my main openHAB3 now runs almost two years on the same SD Card - same.
What did run into problems was my openHAB2 instance, which only had my smart meter and Zehnder comfoair (those with “physical” RS232-to-USB connections). Those had only a bunch of items, not much to do and every now and then a OH-configuration change or OS updates…
and my Pi for my magicmirror.builders.
My remote OH2 (updated to OH3) instance ran off a laptop SSD, but I reverted back to SD Card since two years. And this Pi is in an extreme situation: in winter it runs with down to -20° in the room and if someone is there the place heats up to 30° and more.
You also can use ZRAM for logging and RRD4J persistence - it’s in an openHABian menu also.
And is it really that big of a problem? Amanda runs daily, so I don’t lose anything at all. And if it should happen, then I just plug in another SD Card, which I either copy within that same Pi (you could setup “clone SD” every week or so?) or on another machine. 10mins tops.? plus like 10bucks for a new SD Card. OH (on openHABian) doesn’t need tons of GB to run.
But:
I never let grafana or any other I/O-heavy application run on the same Pi as openHAB. That’s running on my Synology.
If @Max_G is using a default install of openHABian, and there is nothing to indicate otherwise, the file system in question (where OH logs) is in zram. That means this has nothing to do with the SD card itself. None of the zram stuff is written to the SD card except on a restart of the zram service or a reboot of the whole machine.
Typically, a zram file system will only become read only under these three conditions:
the amount of space allocated to that zram file system ran out
the amount of RAM on the machine ran out
and of course there could be something weird in the OS or the zram service to mess things up
Moving to another SD card is unlikely to change the root cause. Moving to an SSD or HDD might solve this only insofar as upon doing so there’s no really good reason to run zram so zram should be disabled, thereby disabling the part that is causing the problem.
We’ve seen that the file system didn’t run out of space (assuming that df command was run while the file system was set to read only).
However, the free command reports memory usage in KB. 464 KB free is really low. So I’m going to guess the root cause of the problem here is you’ve run out of RAM. That’s surprising given what you are running on this machine so that’s worth exploration. What’s consuming 4 GB of RAM?
Well, yes and no.
The most often encountered reason for readonly is that either of the limits defined in /etc/ztab are hit. It doesn’t necessarily mean the disk is full. There’s also a limit on compressed data size that might be inadequate especially if your data doesn’t compress well or at all. zramctl --output-all will tell you. Given the ‘maximum’ use case with InfluxDB active, that’s not all that unlikely. I vaguely remember talks about an excessive number of items, too.
You can increase the figures in ztab if they’re too small. But pay attention to a slow, controlled increase in multiple steps.
InfluxDB isn’t really a great idea to run. rrd4j is the default for good reasons. One not so well known of these reasons is it uses a fixed amount of storage per item, no matter how long you run it. On a resource limited SBC, that’s definitely more clever than to run a full scale database.
I have taken a openhab-cli --full backup (just in case).
Based on this thread, I have put the SD card back in.
I have then uninstalled grafana and influx. However, openhab complains vigorously and endlessly:
2022-12-08 06:55:02.736 [ERROR] [org.influxdb.impl.BatchProcessor ] - Batch could not be sent. Data will be lost
org.influxdb.InfluxDBIOException: java.net.ConnectException: Failed to connect to localhost/0:0:0:0:0:0:0:1:8086
