Most statements on SD card wearout and filesystem corruption you can read about on this forum and the internet are missing the important points so I try to de-mystify and correct misconceptions with this post. It is constantly being updated and is meant to serve as a user reference. You probably got directed here by me or some other forum responder in response to a question or post of yours.
If you find any of the information contained herein to be incorrect please let me know.
Note I’m assuming you’re using a Raspberry Pi to boot off the internal SD reader.
Information may or may not apply to other computers or modified RPi setups.
Power
File storage corruption can happen when your server loses power while writing to disk - SD cards in particular - because every flash controller provides some caching memory so not every finished write command really means that the data was successfully written to the medium. Note this isn’t about file system level handling.
Power losses happen a lot in home automation setups, particularly if you’re in a build phase and there’s you or others to work on the electrical system of the house.
Fortunately there’s a simple solution: get a UPS. There’s RPi addon HATs such as the one from Waveshare, but you should also consider getting an external unit. Most of those add surge protection, too, and allow to run the OH server, Internet router and other critical systems on battery for at least a couple of minutes.
If you’re using a RPi, you might be tempted to use a simple powerbank as a UPS but make sure to get one that allows for charging and powering at the same time - most do not provide this capability and none of the vendors tells you upfront.
Any power supply (uninterruptible or not) must be able to provide the full amount of power the regular power adapter for your RPi provides. Raspberry foundation recommends 2.5A for a RPi3 with power hungry USB peripherals, and for the RPi4 they even had to move to USB-C and supplies to provide 3A. Common supplies are 1A or 2.1A at most.
It’s true that you usually get away with 1A, but you must not forget to factor in all your RPi HATs (HArdware-on-Top) and USB-attached devices as well as to remember that you need to size your system for peak power consumption such as at boot or backup time and not for the average value.
Note that if underpowered, a RPi3 or older will power down components, the USB chipset being among the first. On RPi 1-3, Ethernet is connected via USB so a first symptom of this to happen usually is network problems.
You’ll usually get to see ‘under-voltage’ messages in syslog, too, as well as the lightning symbol on the screen if you connect HDMI. The red power LED on newer RPIs (3, 4) will also flicker.
OFF means input current is insufficient.
SD and other media
Second, with maybe one exception (see #1 below), there’s no way to increase reliability of a SD card. They suffer from wearout leading to corruption, and you can do little about it. Reality is even worse because this is not a SD thing but a memory chip thing: the very same technology is used in USB sticks, eMMC cards and even SSDs, so the following applies even if you use one of those.
There’s variances w.r.t. error-free runtime, some cards or models or brands are better than others, but all but the most cheapest SD controller/card combos do wear levelling to some extent already. Read on if you’re interested in details.
Unfortunately, that isn’t enough. All of this is ignoring the fact that once setup, openHAB keeps writing to SD again and again in rapid succession (logs, data recording and paging) - wear levelling may simply not be enough in this case.
For what’s it’s worth, unlike many believe, SD size is not a good indicator for buffer size - twice the size does not imply all the extra capacity is available as extra buffer and it’s only doubling buffer at best. And even those with a large buffer fail at some point in time.
You should be getting a card that the vendor tagged to be suited for video recording. They have larger buffers and often also use more resilient electronics (storing less bits per cell) and can thus stand more writes. The major vendors (SanDisk, Samsung, Kingston) are selling these with an ‘Endurance’ name extension.
Don’t confuse with cards tagged as ‘Industrial’. While those may have a larger buffer, too, they’re not guaranteed to. ‘Industrial’ merely refers to use environment i.e. this is about temperature range, mechanical stability etc and not about buffers and wearout.
Also don’t think eMMC gives you any advantage in terms of reliability, it’s rather the opposite.
eMMC is essentially the same flash chips as are used in SDs but soldered to the controller.
So it’s usually a daughterboard rather than a card hence it’s much more cumbersome and expensive to exchange it (like it is when your Tesla is hit by this very same wearout problem).
It’s sometimes faster but then again, disk speed is not important to openHAB - it’s all about reliability. However as we’re just touching this, get an “A1” or even “A2” rated SD card. It’s not more reliable but faster than traditional ones rated UHS-1 or less under random access conditions as we have them in openHAB(ian).
On SSDs, be aware they also use the same flash memory. But all except the cheapest of them have a DRAM cache, too. That effectively results in relatively few writes to flash memory which is why flash wearout affects them way less than it affects SD and eMMC.
SSDs have drawbacks including some that pose yet more threats to overall system availability, such as power requirements and boot issues, in addition to cost and packaging and that it just isn’t there in a RPi by default. All of these are reasons why I will not recommend to use SSDs over SDs. Your mileage may vary if you feel this is of relevance to your own situation or not.
Advertising break
If you shop on Amazon, use smile.amazon.de and select openHAB foundation to donate to. Thanks !
Either way, to select a ‘better’ card or ‘proper’ medium is no solution to the corruption problem.
You need to take a complementary measure (#2 below).
There’s two real useful things you can do to fight corruption:
- reduce write operations (to SD or flash memory in general)
-
Ideally, put persistence, logs and swap into RAM and sync them to a permanent medium.
You can use any permanent medium (USB stick, SSD or NFS mount on NAS) to put these on.
Losing RAM (on reboot) or the medium with these files is not critical. openHAB usually keeps working, and you can restore them from backup.
Corruption of the system and data you need to keep on the other hand side is critical.
-
in a nutshell: use ZRAM.
That’s a RAM disk with compression for swap and the most active directories.
See this thread.
I recommend to keep existing swap as a fallback solution. Note the ZRAM swap is created with a higher priority so it’s used first.
-
adding an option like commit=60 to /etc/fstab will result in files being written to the medium only every 60 seconds, greatly reducing the number of writes, but note it doesn’t apply to swap or NFS.
Backup
Reducing writes and moving write-intensive files off the boot medium is a small one-time effort and will greatly reduce the risk of a SD card corruption caused crash, but it won’t fully mitigate it, the takeaway point is that to offload logging and persistence all by themselves is not sufficient. So either way, you also need to
- make daily backups
This will not increase runtime, but it will mitigate the impact of a SD (or USB stick or USB attached SSD or other disk) crash or accidential admin failure.
openHABian now comes with Amanda, a professional backup system.
You might be unaware that openHABian is not just a RPi disk image - it is a set of scripts that can be installed on top of any Debian like UNIX as well. Amanda is available on x86, too.
Some features, however, are only available on RPi hardware.
If on RPi, make sure to use the new SD mirroring feature in openHABian to clone your SD right at installation time or via menu option. In case of crash, you just need to exchange SD cards and you are good to go.
Here’s the link to the official docs.
)
