SD Card write cycle mitigation - my experience so far!

Setup, Configuration and Use Installation
, , , ,
1

Dear community,

I have OpenHABian / OH 4.3 with, well, a number of bindings and rules running on a Raspberry Pi 4 and therefore am very interested in the longevity of my SD card.

Over time (almost 18 months of operation and improvements) I found quite some tricks and hints of how to further and further reduce the number of write actions on the SD and would like to share these experiences with you.

First, there are some cool tools I found for analyzing what is going on:

inotifywait

this is part of the inotify-tools package (sudo apt install inotify-tools).This sets up watches (even recursively) on directories and tells you which file or which file attributes were changed, causing writes onto the device it’s mounted on.

I used this command for finding when any file on the filesystem was changed:
sudo inotifywait -m -r -e modify,create,delete,attrib / | grep -v -E ‘tty|/dev/pts/|/var/lib/openhab/persistence|/opt/zram|/var/log/’
with the grep part filtering out any directories mounted on a tempfs or zram. Let this run for a while, and if you find recurrent writes, you quickly get the candidates for files that could write the SD to death. This tool runs on the filesystem level, and potentially displays stuff which is in reality in a virtual buffer or cache, but is very detailed and file specific.

iotop

this is a standalone tool (sudo apt install iotop).

you can run it standalone, and it will show you any I/O operations on the storage devices, but it’s a bit hard to watch what is going on because the entries change quickly and the commandlines are often too wide for the screen.
But with sudo iotop -o -b -n 100 | grep mmc it runs for 100 seconds and geves you filtered operations to the SD card. The fundamental difference to inotifywait is that it does not run on filesystem level but almost on hardware level; if you see a write there, this really means that there was a data flow onto the SD card. Lower level operations like journaling can be seen here as well.

Now to my findings and solutions so far:

General jsondb

  • org.openhab.core.config.discovery.DiscoveryResult.json
  • StorageHandler.For.OAuthClientService.json
  • thing_status_storage.json
  • users.json
  • the whole backup folder

are files I found quite dynamically modified (partially several times a minute, permanently).

As my impression was that potentially losing the most recent status of these files (autodiscovery results, user logins, online/offline status of Things, backups) would not kill the system, but they nevertheless should stay there, I thought the best solution would be moving these files into the persistence zram and to symlink them to their original location.
So far, for about one year of operation, I never had any negative side effects from this, but a huge reduction of SD writes!

Hue emulation

  • hueEmulationUsers.json

I am using this service, and I found the file above updated on every access to the hue bridge - often multiply in one second (!!), and it was just a “last access timestamp”.
I did the same as in General jsondb - move it into the persistence zram. Works fine!

Samba

I found

  • /var/lib/samba/wins.dat
  • /var/lib/samba/wins.tdb

modified frequently, about once or twice a minute. Checking out what wins is doing, I saw that this is a discovery service for Microsoft OSes from Windows XP downwards which is required in a network environment with many subnets. Well, I don’t have any of them, and I guess most of as won’t have as well. So it’s easy to turn this off completely:
In /etc/samba/smb.conf,set the following entries:

  • wins support = no (it’s “yes” by default)
  • wins server = (just leave empty, it’s commented out by default and has “w.x.y.z” example)
  • disable the wins service:
    sudo systemctl stop samba-ad-dc.service
    sudo systemctl disable samba-ad-dc.service
  • delete the wins files named above

The writes will disappear, the files won’t be recreated. My smb / cifs shares continued working perfectly on my linux machine.

EDIT / Add-on:
********
There also is a directory /var/cache/sambawhich contains browse.dat and smbprofile.tdb.

These files are used by nmbd, and also are frequenty written. Especially for stopping these writes, it is important to disable the samba-ad-dc.service; only setting wins support = no does not stop that! When this is done, browse.dat is no longer written at all, and smbprofile.tdbis only updated and written on any reboot, but no longer while the system is up and running.
********

Timesync service

In /var/lib/systemd/timesync, i found a file timer, which has no content, but has its creation timestamp updated every few seconds, causing masses of journaling writes to the SD card. To my knowledge, this file is used to keep an approximate time for reboot time when NTP sync is not yet up. This functionality is already covered by the fake-hwclock service, which does the same, but with a 30 or 60 minute cronjob. Also writes, but orders of magnitude less per day.
Good thing is, this behavior of systemd-timesync can be switched off without disabling NTP:

  • create an override file for the service config:
    sudo mkdir -p /etc/systemd/system/systemd-timesyncd.service.d
    sudo nano /etc/systemd/system/systemd-timesyncd.service.d/override.conf

  • Insert the lines (yes, just empty):
    [Service]

    StateDirectory=

  • Reload the system config and check the override is working (StateDirectory should be empty and Runtime Directory should be somewhere in /run, which is a tmpfs):
    sudo systemctl daemon-reload
    systemctl show systemd-timesyncd --no-pager | grep -E ‘StateDirectory|RuntimeDirectory'

  • Delete the dir / file: sudo rm -rf /var/lib/systemd/timesync

  • Restart the service: sudo systemctl restart systemd-timesyncd

With this, the permanent writing is gone permanently, and NTP timesync as such stays fully functional.

Signal
If you use the Signal binding, you can find a folder /signal inside the JSONdb.
Here, the <uid>.d file is updated whenever the Signal server is contacted, which also sums up to several times a minute even when the account is idle - and every transaction triggers that as well.
This file I found to be better symlinked like described in the “General jsondb” section.

I hope this helps anyone, and I would be curious for any of your ideas! :slight_smile:

4 Likes
2

The pretty much best advice is to use a standard openHABian setup, it has ZRAM and many smallish mitigations built in.
Enable SD mirroring or even the auto backup feature combo.
Then you can boot from the 2ndary card should your primary one fail.

3

Yup, actually, doing these kind of backups should (at least in my opinion) be independent from such tweaks, no matter how deep they go. Just for reliability reasons. In my case I have a nightly Amanda backup to a NAS configured and regularly do manual raw dumps of the SD card.

Talking about deeper details, for sure these tweaks I did are some deeper above-beginner digs into Linux (e.g. execution and mount orders at boot, etc.), but especially as they are actually functional for quite some time, I would really be curious if someone knows if I was just lucky so far and that something could cause major trouble at some point, or if it has a good chance to work on properly.

4

Thanks for sharing the results of your work :+1:t2:

5

Thanks for some good tips, but I just had to comment on this: WINS isn’t “for XP and down”, it’s supported by all Windows versions since
 3.11 or something like that, including Windows 10 and Windows 11.

WINS (Windows Internet Name Service) is a name resolution service for (local) networks. It fills the same role as DNS does on the Internet. Having local name resolution isn’t necessary if you only have a few devices, you can just learn their IP addresses, use hosts files or mDNS where applicable. But, if you have a few more computers/units, I find WINS to be very handy. Sadly, not everything supports it these days, as some think it’s “old-fashioned”. As far as I know, no other name resolution service provides quite the same:

  • DNS has been attempted used for local name resolution, but you need clients that can dynamically register with the DNS server every time their IP changes, and a DNS server that support this. Then there’s authentication
 In short, it quickly gets complicated.
  • mDNS/Bonjour is “zero config”, so even simpler to use, but it’s unreliable and don’t traverse subnets. Not everything supports that either.

But, to have any use of WINS, you need to run a WINS server somewhere on your network. So, for all those that don’t, WINS can be turned off with no consequences. Samba can act as a WINS server as well as Microsoft’s WINS server.

WINS needs no configuration other than that you tell the clients where to find the server, usually by configuring DHCP to automatically set the WINS server address for clients. The WINS server will accept registrations without any kind of authentication, and will serve its records to anyone who asks. Configured clients will notify the WINS server when they start up, their IP address changes and when they shut down. That’s it, and it “just works”.

2 Likes
6

One thing: All those writes as a result of reads is easy to get rid of. Mount with noatime.

This is a “crazy feature” that shouldn’t exist, and certainly not be the default, in the first place. What is the possible reason for even keeping this, it’s not a record of who/what accessed the file, only when it was “last accessed”. It’s very hard to see how that information is useful. It’s a great waste also for SSDs to do this, so it shouldn’t be enabled generally IMO.

Since you found that those reads resulted in writes, I had to check my own openHABian installation, and it has the root filesystem mounted with noatime by default. So, since you write that you use openHABian, how come you see these “last accessed” writes?

Just disabling writing the “last access” should dramatically reduce the number of writes.

7

Thanks for the tips!

Do you guys perform preemptive SD card replacements? I have a pi4 that I’m using as a web browser to display a mainui page on a wall mounted monitor. I use endurance sd card and installed debian on it. It has been running for 4+ years on the same card.

Since there’s no SMART-like predictive failure, I often wonder. At the very least I should probably clone the card and sticky-tape it so it’s easy to replace should it ever happen.

8

I have replaced my openHABian card once, and that was after a lot of errors from rrd4j, so the answer in my case is “no”. I do have the “mirror backup” running though, so I’ll let it run till it drops.

9

See also Corrupt FileSystems every 2-3 month? - #20 by mstormi

Most of the stuff already is there inside openHABian.

Please create a PR if you think there’s an important one missing but mind complexity/error probability vs. benefit and that it must deploy non-interactively.

10

FYI: I updated the original post as I had forgotten to mention my observations with the Signal binding.

11

I’m still not sure if the file is actually modified or only read several times a minute. It sounds strange if it is modified repeatedly. Have you checked that your file system is actually mounted with noatime in /etc/fstab?

12

Hey all,

Just some more explanation. :grinning_face:

My approach towards these tries and tests I described here is, at first, curiosity with a grain of perfectionism :sweat_smile:.

It’s not being afraid of premature SD wear, and definitely not mistrust in the measures described in the OH forums. In contrary, I applied more or less all of the them - use OpenHABian, set up Amanda Backup, do regular raw copies, and use high grade Endurance / Long-time video recording type SD cards.

This should cover 95% of wear mechanisms and ensure years of stable operation. Great job, especially such reliable setup! :+1:

Which does not stop me from being interested in the last 5% :grin: and that’s why.

So, well, I did achieve quite some additional reduction in SD writes and did not see any malfunction over some time.

Applying these measures may look simple, but before testing my measures, I thoroughly learned what they might cause and if they might have glitches. For example, the systemd-tinesync service is started extremely early in the boot process, and the only virtual filesystem available at that time is /run, but that’s a tmpfs and therefore lost at boot. The zram service is started way down later in the boot process. In that sense, and especially due to the fake hwclock doing the same, I found it best to deactivate the boot time setting of the last known time of systemd-tinesync.

That makes these measures some good candidates for improvements.

But they are working fine for me, just me. Which says not much about what may happen on other systems.

And I might be wrong with an assumption. Or just was lucky not to encounter a severe consequence yet. Or I did, but didn’t notice yet.

And that’s why I decided to rather share these topics with you in the forum rather than just dumping it on the as a Git issue or a merge request. That’s something to come when it’s more certain that they do not accidentially break something.

My ideal hope is to give you some useful hints, to get some good feedback if I really did reasonable things, and in the very best case to achieve some improvement for OH in general. :slight_smile: So I am looking forward to your feedback!

@Nadahar : Thanks for your detailed description of the WINS functionality!
Especially for this topic I have the impression that the majority of users is not using it, while it’s heavily write-dumping files in the background. One might think of disabling it by default, and include the activation in the config menu for giving those actually in need a good way to enable?

I do have the filesystem mounted with noatime. It’s OpenHABian, and it’s inside /etc/fstab, so everything okay.
Concerning the “last accessed” timestamps, it’s something specific to the services instead:

  • systemd-timesync explicitely writes an empty file with the current timestamp, and for recovering the last known time at next boot, it looks for that timestamp. The write time is the actual content / payload for this function!
  • Hueemulation and Signal as well: Here, it’s no filesystem timestamp. It’s an UTC timestamp (unix time) as JSON content within the file’s content which is rewritten over and over again.

Not really; besides the measures in @mstormi ‘s link, I have two “Endurance” SD cards which I regularly exchange - I take a raw dump from the currently active card 1, write it onto card 2, and make card 2 the new active card. And vice versa. Once every 3
6 months, mostly after an upgrade to the OH packages. This has the advantage of both distributing the load as well as implicite verification of the backups.

1 Like
13

I haven’t studied how this is set up in openHABian, but I find it strange if a WINS client would need to write to disk at all. If what is configured is a WINS server, it will probably write the “database” now and then.

Looking at my own smb.conf, I see:

# WINS Support - Tells the NMBD component of Samba to enable its WINS Server
  wins support = no

I don’t know what the default was, but if it is true, I absolutely agree that is should be set to false. I don’t think a configuration option is needed for this either, the chance that you want your OH device to act as a WINS server is so slim, that those that would need that should configure that manually.

The next setting in the file is where you configure the WINS client, which is configured in my case, but that has to be done manually as it must point to the actual address of the WINS server on your network:

# WINS Server - Tells the NMBD components of Samba to be a WINS Client
# Note: Samba can be either a WINS Server, or a WINS Client, but NOT both
  wins server = 10.99.xx.xx

So, I’d say that both WINS Server and WINS Client should be disabled in the default configuration. It’s very unlikely that a server is desired and a client will need configuration anyway.

Hmm, bad design I’d say, but I don’t know if there are better options they could use. But, isn’t it potentially problematic to disable this?

From what I understand, the RPi 4 has no RTC, while the RPi 5 seems to have one..? Without a RTC (and a battery backup), the device will have no idea what date/time it is when booting, which could lead to some very strange timestamps in logs etc. Other things that depend on a time source like journalling and encryption, might fail too. So, there might be that writing this timestamp frequently is required to have a reasonable idea what time it is if the device is shut down suddenly/lose power.

On the RPi 5 on the other hand, given that the battery is intact, the RTC should keep track of the date/time during startup and avoid these problems. Once the system is up and running, NTP can be used to retrieve a precise time, but you need it to be “approximate” also before NTP can act.

I don’t know these services intimately, but you refer to fake-hwclock also being there. But, 30–60 minutes is way too rare in case of power failure, as I see it. I can only imagine all the problems that can arise if the time is almost an hour off. All HTTPS/SSL requests will probably fail, logs will be a mess, etc.

Not having an RTC is stupid, really. For RPi 5 users, one would think that both these services could be dropped.

This sounds like potentially bad design, depending on. There’s certainly no reason to write the same, unmodified, information over and over again. I should be easy for the binding to keep track of whether the information has actually changes before writing it to disk.

edit: It seems that while the RPi 5 has an RTC, it doesn’t do much unless you connect a battery to some connector, which I assume that most people won’t do. So, the problem might in reality be much the same for the RPi 5 and for the RPi 4.

14

Not really. I have five or so RPis running 24/7 with SD cards that are five/six years now. These old systems are pretty light weight and are not doing much and way oversized for the job which probably accounts for their longevity. But I’ve done almost nothing to ensure their longevity.

The one instance of OH I have running on an RPi uses openHABian and it’s been going for a number of years now (one RPi Z w has been running, reporting the door sensors to OH over MQTT for about seven years now).

I’m just prepared to replace the SD card when the strangeness starts happening on these machines (e.g. log files filling up because they can’t be deleted, which is what happened on the one SD card I had fail, which was running a RaspiCAM at the time) but I’ve done nothing special to back them up. But all my machines are configured through Ansible so when the time comes it won’t take any time to set them back up again. Backups are automated and the Ansible scripts can restore from backup when necessary. So I’m not completely running without a plan.

But If I needed a family member to fix what went wrong, I’d probably do just as you mention and have a preconfigured spare SD card they can just swap out for the broken one.

I think it all depends on your recovery plan as to what the best approach is.

15

Fascinating
 I use OpenHABian (don’t know which version exactly, it was one of the first OH 4.3.x versions, 64bit, still with Frontail viewer), never messed around with Samba, and for me it was set to wins support = yes by default (and with no override file existing). Maybe a distro thing? Maybe it came with an apt update at some point?

I agree on things may be going miserably wrong if datetime falls back into 1970 / “zero” unix time. Being off by “only” weeks may as well be not any better. But being off by 30
60 minutes might mess up logfiles, in a bad case the persistence may have a stretch of inconsistent data, but it’s hard to imagine an overall failure in that case. Question is, what may happen even with systemd-timesync enabled
 if power is off for hours, similar effects, maybe slightly better for the logs. If these very small writings to the file’s date are first cached by OS, later cached by the SD card’s firmware (which “endurance” type cards often do), and what happens at a sudden power failure then, it might be not so far off the 30-minute period covered by fake-hwclock. And, with both services enabled, it might happen that one service overwrites the other at bootup, causing even more time jumps. And who knows what NTP does in an actual case - if there’s internet available, everything’s fine. If there is no internet, NTP might decide to sync to the next available timesource, often this is the router - which might be great, or not if no one guarantees that the router has the correct time.

Long story short, I agree on what effects timestamp failures might cause, but even with systemd-timesync and fake-hwclock enabled, there is no certainty to mitigate this. Unless you provide a real hardware clock. Even for RPi 4, there are choices to retrofit - I would be curious to try this device at some time (with a supercap instead of a battery, including a setup guide).

But I strongly doubt that under all these circumstances it makes sense to do several writes to the SD card per minute. In my opinion, and unless someone proves me wrong :upside_down_face: , the fake-hwclock 30 minute interval is sufficient.

Concerning the hueemulation and Signal binding json files being continuously written with a timestamp, maybe should we contact the maintainers? I see the sense of keeping a timestamp there for communication reasons, but that should not reside directly on the SD card. What do you think?

16

FWIW, I did a fresh install of OpenHABian on a new RPi 5 and I have it set to yes as well. So it is probably worth creating a PR to change this default on installation.

The reason to have it enabled may be: A simple fix for accessing raspberry pi by hostname from windows with samba - Raspberry Pi Forums

17

When thinking about it, I kind of agree that it’s won’t help much to write it frequently. I assume that both services write a timestamp during shudown if the shutdown is graceful. And, whatever time elapses from shutdown to startup will be a problem regardless. During a normal restart, it error is probably just a few seconds anyway.

All things considered, the only good solution is a RTC. Everything else will result in the system starting up with a clock that is quite a bit off if it has been shut down for a while.

It’s hard to know the exact situation that has led to this behavior, but chances are that the developer never thought about this consequence, and that it could easily be avoided. Holding the timestamp in memory should be enough for most circumstances that I can think of. What is the use of, when you start the binding, to know when the last communication took place?

18

I created a Github ticket for the hueemulation observations, just checked again. It really is only a timestamp, and it’s written once per API access, meaning once per second in my case.

EDIT:
I also wrote into the Signal binding forum thread (it’s not an official binding yet, pity
), as I found accounts.db also being written on a secondly base - but no deeper hints yet what is updated so frequently; accounts.db is binary
 :stuck_out_tongue:

EDIT 2: Created ticket for WINS in GIT :slight_smile:

EDIT 3: The WINS issue was solved in GIT already :tada:
I created another ticket for the systemd-timesync issue now; looks like it’s really worth it as systemd-timesync and fake-hwclock are redundant in the timestamp-saving functionality and both enabled by default, and some docs of systemd-timesync actually include warnings for SD based systems due to the high number of write cycles

1 Like
19

While the prerequisites and actions for most of the write-heavy files were clear, one topic with some more ambiguity remains: the JSON database.

As mentioned earlier, I found the following files being modified frequently:

  • org.openhab.core.config.discovery.DiscoveryResult.json
    The autodiscovery works a lot in the background, especially the Chromecast binding. Each time, no matter if a change is found or not, it writes to that file.

  • StorageHandler.For.OAuthClientService.json
    Looks like each time a certificate is queried or renewed, write actions here.

  • thing_status_storage.json
    Each time some WiFi, Zigbee or cloud Thing encounters just a short connection drop, writes going here. Happens quite frequently for me; over 300 physical devices, 188 Things partially with dozens of channels


  • users.json
    LastRefreshTime looks like to update its timestamp each time a user logs in.

  • /backup/*
    The backup folder contents are, of course as no surprise, changed and partially rewritten on each of the above changes as well.

As already said, I found peace for my setup by symlinking each of those into ZRAM. I just tried and did not encounter any negative side effects yet.

As for my understanding, the idea of ZRAM is very well suited for such kind of files - Stuff you definitely need after a reboot, but changed so often that you don’t want to put it on Flash storage everytime. So putting it into ZRAM by default, and normally on main storage if ZRAM is disabled, just exactly like the persistence, could be an option to consider.

That being said, I do not have sufficient insights which consequences could come if, e.g. on unexpected power loss, these files are either lost or drop back to a previous state.

While at first sight it would not be overly problematic for user logins and OAuth tokens (just requires manual renewal) and Autodiscovery results, the Thing status might even become asynchronous from other parts of the JSON database - e.g. a Thing was recently added, unexpected reboot happens, and then the Things database has the newly added entry while the thing_status_storage drops back to a state before it was added. Would OH be able to recover from such status?

And what’s it worth? Heavily writing the SD card for having some better feeling, but still being at risk of data loss if the outage happens within a write cycle?
Because: If you (really :wink: ) do regular backups as recommended at many places, even an asynchronous JSON database won’t break you. If you want to be extra sure, do a graceful system reboot after changes, that will trigger the ZRAM to be written consistently to the SD card (once, not permanently!).

So from my personal point of view, writing these files to SD card on every change only gives you a false feeling of safety, you’ll need backups anyway, and if you do backups, there is nothing to be afraid of putting these files to ZRAM and heavily relieve the SD card from probably unnecessary writes.

How do you feel about this?

20

This is the already the default openHABian configuration for a reason.

You lose all the changes made since the last write out to the SD card. OH only reads these files on startup.

This almost couldn’t happen. You added the Thing and lost power. When power is restored, OH returns to a state from before the Thing was added because all those changes were lost in the power outage. The Thing will no longer exist.If it’s not in the JSONDB it doesn’t exist.

With zram, changes to files are only written to disk on a sync which only happens during a normal showdown or when explicitly called for from the command line. So if your machine boots, runs for a week and then loses power, every change that happened during that week which is in zram will be lost. The next time OH starts, it will be as if your machine has been off the whole week.

The only time a problem remotely like this could occur is if you are actively doing a zram sync back to the SD card and you lose power before that sync finishes. In that case you’ll have at least one incomplete file somewhere unless the timing were so perfect that the power was lost immediately after a file was closed but before the next file was opened. So in almost every case what will happen is you’ll immediately know because OH will fail to load properly because of a corrupted config file.

It’s really great that you are putting a lot of thought into all this, but don’t forget the openHABian folks have already done a lot of this already.