Sd-card corruption?

Hardware Server
20

QUOTE: I have a NAS but it’s usually running during the day only, so a usb stick might be the better choice.
I am confused about SSD though.
Some people say that USB sticks are not better than sd-cards and some even say, that this is the same for ssd.
So does it help to use a ssd instead of a USB stick of the same size or is the reliability on the same level?

NCO: There is USB, SD and SSD, they all run flash of different qualities BUT SSD will do far better active wear levelling AND they most likely have power supplies that keep the power running long enough to dump the caches back onto the flash before the power fails, or they may use the wear levelling to write the changes to a new area before erasing the old area. Better ecc and CRC error checks and things like that make SSD’s far better than platter drives on everything except price. USB and SD do NOT have those very important things so they are essentially the SAME as each other just with different plugs but they are both vastly different to SSD. Sure you can get better flash controllers and industrial grade flash but they both still don’t have the space for what is needed to make a rock solid SSD and this is reflected in the price. Having said this, SD cards can work very well if they are understood and they are very cheap.

Here is why flash causes issues: They need to prepare a BLOCK of flash before they can write to even a small area, so if the flash needs 1 file updated they have to read a BLOCK of the flash into ram and this block may contain 5 other file pieces that do not need to be changed ! They then prepare the BLOCK of flash that wipes it clean and then they write the contents of ram back on to the newly prepared flash. If the power fails at the wrong time before it can write the ram back to flash you loose parts of 5 files that were never going to change on the drive ! Since it was only pieces of files the result is a file that exists but is corrupt.

This is the reason for a number of points:

  1. Putting log files onto a USB drive or ram drive means if the power fails during a log write, it can not then effect other files as the USB only contains log files. Same idea to directing log files to a NAS, if you remove the need to write to the flash you don’t have the flash changing blocks that contain other file pieces as well. I opt to use ram disks as I don’t care about the logs and it is easy to redirect them if needed. USB sticks are cheap so some people prefer to use them.

  2. Linux by default when it READS a file, it will then WRITE what time that file was last accessed, so every read of a file results in a write. Imagine how many times that creates the opportunity for corruption as every block of flash may contain parts of other files as well. Years ago when I started playing with linux I was stunned at how quickly linux would corrupt flash when windows never/rarely did. Many people use the “noatime” switch and most of the time it causes no issues to how a linux machine runs. From doing some quick checks the openhabian image for the PIx by default appears to not update the atime so the noatime switch appears to do nothing as it must be achieved another way. Edit: latest builds of raspbian now use noatime by default when not specified which is great news, if u roll your own u need to check this still.

  3. The PI has a stupid tiny USB port for power and when plugging in it may cut the power in and out, many usb cables and power supplies cant supply enough current to properly power the PI’s. I have been using OSMC for years on a PI2 and they have a great feature that draws a rainbow icon on the screen to show when the power is not enough, does openhabian do this in any way, maybe to a log file? Raspbian last time I used it also used a rainbow icon so if you want to check your power supply and usb cable out it is quick to load a SD card with raspbian and have a play.

This is why I love the PI range so much. Cheap SD cards you can pull out, backup, change the entire system in seconds to another OS and purpose. If you buy a NUC and you get a hardware failure you either have downtime whilst you source another identical NUC that costs heaps and you have downtime for days, or you need to have a spare NUC sitting in your house costing you way more upfront. Spare raspberry PIx is cheap and you can even use your spare PI’s for kodi or other uses around the house and all it takes is a SD card swap out and you have replaced a failed device in seconds. Why change to SSD as then you have to pay more for a spare SSD in case it fails. What happens when your NAS is not reachable with a network issue? Be careful not to make things overly complex as you could make doing backups far harder and swap 1 cause of issues for another that will occur way more often.

To check what files get written use these commands:

sudo apt-get update
sudo apt-get install inotify-tools
sudo nano /proc/sys/fs/inotify/max_user_watches
inotifywait -mr --exclude 0 -e modify,attrib,close_write,move,create,delete /

You need to increase the number in the file /proc/sys/fs/inotify/max_user_watches if you want to do a global watch, otherwise you can watch a specified folder. Add two ‘0’ to the end of the number and save and exit nano. Press Ctrl+C to stop watching with the last command.

21

I also used Raspi 2 and 3 in the past for openhab and had lots of problems. Almost a year ago I switched to Odroid C2 which uses emmc instead of sd cards. The linux system runs very stable since then. I still have openhab crashes sometimes but it is a big difference between a java application crashing and the whole OS being broken.

22

Thanks for your detailed explanation. Now I understand the SD, USB, SSD differences - That totally makes sense.

I see, that it’s not that simple to make the right decision to go away from RPi.

To summarize there are some very valid points in this thread and in some other threads adressing the same issue (SD-card problems).

Pro Pi:

  • Cheap
  • Big community and experience (OH and raspi itself)
  • tools to simplify things such as openhabian (thanks @ThomDietrich - that helped a lot)
  • easy to backup (I really like the sd-card image backup option, because I use to play around a lot and sometimes mess up the system :wink:
  • low power consumption / footprint
  • flexible for further expansions.
  • (other hw will have it’s own issues)
  • barebone setups do not provide such easy backup options like a complete SD-card backup

Contra Pi:

  • SD-card corruption (which is really annoying when not at home and you don’t know which SD-card backup from the past is already corrupt and which is not).

It’s actually that easy - I don’t see any other contras, but I was so annoyed that I did not see the wood for the trees.

So my (preliminary?) approach to solve / address my issue will look like:

  • Stay with RPi (start with an older backup image of my runtime system to reduce efforts against starting from scratch) - close observation required) - otherwise start from scratch with brand new SD-card
  • use my NAS for the persistence (Yes, @mstormi, @rlkoshak, I will keep it running 24/7 from now :wink: )
  • redirect my /var/log to the USB drive
  • will check out various backup functions (Amanda seems to be a good option).

So thanks a lot for all your input.

23

Persistence on the nas sounds like an interesting solution. Let us know how it goes for you.

24

Put persistence db AND logs on NAS, you won’t need the USB drive any longer.

PS: and since I believe you’re running ZWave, too, get your replacement procedure ready in case your controller breaks down. See also the introductory section of the Amanda readme.

25

cschneider111: The odriod range are great as they support h265 where the rpix does not in Kodi. Been looking for a while but the ability to use ready made images for the rpi just keeps me from making the change.

NCO: I’m choosing to send logs to a ramdrive and have not choosen if I’ll use a NAS or a USB drive directly in the pi for the persistence yet.

You can see how I am doing it here…

26

Thanks, I will consider migrating the logs as well.

I have read the Amanda introduction about HW redundancy and actually never considered to prepare a second set.
This will be the next project though :wink:

27

Well, of course the type of interface doesn’t affect the reliability, but there are more differences between the cards than just the interface. For example many of them seem to have wear leveling, which sd cards seldom (or never?) has. I have read multiple report from people having big problems with trashed sd cards and who have then changed to eMMC cards and after that having no corruption problems at all.

28

Thanks! However just to be clear: This is not really a “Pro Pi” as you can use openHABian elsewhere as well :slight_smile:

29

Touché
That’s true, but I did not try it on any other hardware.
On my RPi it works flawlessly - so that’s good :wink:

30

One more (possibly stupid) question.
If the SD-card is corrupt, does this mean the card itself is ready for trash or just the data?

When I setup the system from scratch I wonder if formatting would re-animate SD-card after corruption.

31

Well it means you have hit a (eventually intermittent) HW problem. Reformatting might or might not spot that as a bad sector and might or might not exclude it from future use.

Bottom line is, you can’t be sure. I would not reuse that card in a Pi, but you can still make use of it in less write intensive applications like say a camera.

32

99% of the time it is only the data that is corrupt and flashing a new/backup image will work fine. I have personally worn out multiple USB sticks, but I was working full time doing beta testing for an embedded system and hammered them big time in ways they were not designed for. Of course it may have been from another fault as they were low quality no name brands. Never seen issues with brand name industrial grade flash.

Flash the SD and if it works then great, if not try flashing another SD card with the same image and if that one works then you know the other needs to be trashed after first checking that the image is not larger than the space on the SD card. Two ‘identical’ cards even same brand, size and model can have slightly different amounts of space so using a linux program like gparted / parted to reduce the partitions down helps to create smaller images and helps when you want to write the image to another card. Take an image before playing with parted if new to it :slight_smile: By reducing the partition size your backup images also take up less HDD space to store them.

33

I run an Odroid U3 for three years now with an emmc. And now my emmc is broken. At the moment I run it temporary with an sd card and it is much slower than before. :frowning:
But now I will switch to an Odroid HC1 - a cheap ARM computer with a SATA connector. I hope this helps to prevent corrupt cards.

34

Another hint is to move your swapfile to USB or NAS. Swapping (or paging, to be more precise) happens all the time and goes to /var/swap in default Raspian. Configure a new location in /etc/dphys-swapfile.
Use dphys-swapfile swapon to activate and swapon -s to validate the outcome.

35

Damn it - I feared that someone would say somthing like this :wink:
Thanks for the confirmation.

36

Hmmm… might be an option as well.
I will check out which suggestion will work for me best. :slight_smile:

37

All,

to get of frequently appearing issues with openhab (cron not working properly, persistence hangups) I decided to eliminate sd-card corruption issues (maybe causing the issues above?!)

To do so I just purchased an element14 desktop housing for the pi including a ssd:
https://www.element14.com/community/docs/DOC-83477/l/desktop-computer-kit-with-expansion-board-that-can-turn-a-raspberry-pi-into-a-real-desktop-pc

and installed openhabian 1.3 from scratch.

Just a short (maybe stupid) question:

@ThomDietrich and others
If I did a backup using openhabian-config on my NAS and assuming that one or the other file might be corrupt already - can I import the corruption back into my brand new setup?

Does this mean, I need to configure everything again?

38

Everything in conf can be verified by simply opening the file with a text editor, or even using less on the command line on the Pi itself and scan through them for anything that looks out of place. Even better would be to use VSCode or ESH Designer.

You can do the same for the contents of the /var/lib/openhab2/jsondb files. I don’t know if there is an online JSON checker but I would be surprised if there isn’t.

I believe those are the vital files. If those are all OK then you only need make settings and install bindings again to start from scratch which isn’t that big of an effort. I have to do the same pretty much every time I update my Docker image and it rarely takes more than a few minutes.

39

Thanks, Rich,

I will check the files you mentioned manually.
I am really keen on seeing my system run for months without any issues with the new ssd approach…