Reducing SD card writes with openhabian

Looking for feedback on how I have reduced the SD writes on my raspberry Pi2 and openhabian and if they are likely to cause issues in the future as I build the server up to have more abilities. I should stress the main reason for doing this is not wear and tear of the SD card, but instead for reasons to help prevent corruption when dirty shutdowns are made. ie pulling the power out when the kernel locks up or the power fails. I have a good UPS which auto records power events and I see around 12 a month on average where I live, so they are very common.

I found the main writes were caused by:
/var/log/openhab/events.log
/var/log/openhab/openhab.log
/var/lib/samba/wins.dat
/var/cache/samba/

Less often writes caused by:
/var/log/syslog
/var/log/daemon.log
/var/log/auth.log
/var/log/mosquitto/mosquitto.log (if using mosquitto for MQTT)

After making some basic changes, my green LED for SD access on the rpi will go for minutes with no flashes. I tried to keep the changes minimal and easy to make and reverse.

sudo mkdir /sambaBUP/
sudo cp -af /var/lib/samba/. /sambaBUP/
sudo nano /etc/rc.local

add the following to rc.local file which the last command above will open the file for editing:

chown openhab:openhab /tmp/
cp -af /sambaBUP/. /var/lib/samba/
ln -sf /tmp/events.log /var/log/openhab2/events.log
ln -sf /tmp/openhab.log /var/log/openhab2/openhab.log
ln -sf /tmp/syslog /var/log/syslog
ln -sf /tmp/daemon.log /var/log/daemon.log
ln -sf /tmp/auth.log /var/log/auth.log
ln -sf /tmp/mosquitto.log /var/log/mosquitto/mosquitto.log

Run this command:

sudo nano /etc/fstab

inside the fstab file add this and also add noatime to every mount. If you are using the latest raspbian build this is the default but it wont hurt to add it.

tmpfs                        /tmp                       tmpfs defaults,nosuid,nodev,noatime,size=50m 0 0
tmpfs                        /var/cache/samba           tmpfs defaults,nosuid,nodev,noatime,size=5m 0 0
tmpfs                        /var/lib/samba             tmpfs defaults,nosuid,nodev,noatime,size=5m 0 0

Type ‘sudo reboot’ and that should be everything done. Any issues you can disable lines by using a ‘#’ at the start of the line should you want the log files to once again be sent to the SD card. Still testing to see if 100mB is enough for the log files without lowering the logs from Info to Error. So far all seems good.

To check what files get written use these commands:

sudo apt-get update
sudo apt-get install inotify-tools
sudo nano /proc/sys/fs/inotify/max_user_watches
inotifywait -mr --exclude 0 -e modify,attrib,close_write,move,create,delete /

You need to increase the number in the file /proc/sys/fs/inotify/max_user_watches if you want to do a global watch, otherwise you can watch a specified folder. Add two ‘0’ to the end of the number and save and exit nano. Press Ctrl+C to stop watching with the last command. This will list all file writes even if they are not on the SD card.

EDIT: As others have pointed out your persistence will write and can not be stopped, so if setting this up you need to point it to a place like a NAS or USB. Because flash uses blocks to store data, any write that is interrupted due to power loss will cause corruption in other files. By shifting the persistence writes to another location you will keep a stable Openhab system for far longer.

3 Likes

I had started a generic discussion on this a while back (here).

As of today, I would refrain from using tmpfs. It’s built to increase performance, but it doesn’t do that in the mostly write-only scenario that we do have.
And it’s eating RAM which is very limited on a Pi. If you set its size too large, you put openHAB/java at risk. If you set it too low, you easily run out of log disk space (can lead to Pi lockups, too). I’d use a USB stick “drive” instead. A NAS is even better.

I cannot validate if your list of ‘most active’ files is correct (I didn’t know the inotify-tools, so I don’t know what they’re actually measuring … and to quickly set them up for a comparison didn’t work right away for me), but it’s surprising that you have samba writes but no system log and no persistence writes.
Now that might be valid for your very specific installation or not (don’t know what you’re using samba for and don’t know if you use persistence), but it’s for sure different in an average Pi/openHAB(ian) installation.
I would expect system logs, openHAB logs and persistence to make up for the vast majority of writes.

Thanks for the info Markus and yes I will be keeping an eye on the logs and free space. I should have plenty of ram for my needs and it is simple to run a cron to dump log files back to the SD every 24 hours or less if needed.

It is interesting as this is a brand new openhabian image downloaded last week completely setup by the first boot of the system only with mosquitto installed and some manually added site, rule and item files for some testing before I leap into openhab instead of HASS. By default the installation of openhabian setup samba shares and there is no persistence setup as that is the way a fresh openhabian was setup by the first boot script.

I don’t see anything at first glance that gives me concern with your setup.

However, I’ll second Markus’s caution about balancing having enough space for logs and enough ram for your programs. You have already said you want to add more to this server so you might be setting yourself up for problems.

I’m of the opinion that if your logs are not important enough to preserve past a system reboot (i.e. putting them in a tmpfs) they are probably not important enough to be logged in the first place. Before I put these logs in a tmpfs I would just turn them off, or at least dial them way down to only log errors. If they are important enough to preserve, set up a NAS or a centralized logging server to push those writes off of the RPi.

The same logic will apply to persistence. If it is important enough to set up a tmpfs for it it is probably worth putting it on something that is less likely to become corrupted when you lose power.

I don’t know much about samba’s cache so can’t really say whether that will cause problems in a tmpfs.

1 Like

@Matthew: As I have also serious problems with sdcard corruption: at which position of rc.local your edits should be added? So they are to be executed only at first boot?

We need to work out WHY you have serious problems with it. First work out why and then create a solution that fixes the root of the problem as my changes will not address the problem if you are writting to a persistance file 100 times a second or you have a fake SD card, are overclocking, or the power supply and usb cable you use is not good enough for some examples. Run the inotifywait commands in my first post and work out what files are getting written as a first step.

To answer the question, the rc.local file does have some lines that should not be disturbed as they have IF, FI and THEN statements. Place the code either at the top after the last of the lines starting with a hash #, or directly above the exit 0 line at the bottom. To make it easier to read it is fine to press ENTER a few times to create space from what your adding to what was there before. The hash lines are comments and you can add your own to help you work out what you have changed. Make a IMG of the SD card before you start, or backup the files you change so you can reverse it should something go wrong.

In a terminal type the command ‘free -m’ and it will tell you how much free memory in MB you have to use as a ram drive aka tmpfs (temporary file system). NOTE: It is my understanding that linux works by never clearing out the memory until you run out of memory. Linux keeps things in ram in case they are needed again and does not clean up so don’t be surprised if the amount of free mem is dropping the longer your setup runs. Look at the swap lines and if 0 is used then all is good. Actual Free Memory = Free + Used for Buffers + Used for Cached.

My background was doing hours of daily testing on a device similar to a PI long before they were invented. For years I would pull the power on this device 100’s of times per day that was running flash storage and linux. Not once did it ever have an issue as the entire flash was setup as read only. Because I have first hand experience with this for years, I know how well flash can work and also how bad it can be if it is written to excessively and made worse by dirty shutdowns. I also have a rpi2 running kodi (osmc) that never gets powered down and is still fine after 2 years since I made changes similar to the above to it.

1 Like

@matt1: Thank you for your detailed answer. I am using a Raspi 3 with no overclocking, nothing in its USB plugs, 16 GB Sandisk SDCard with 81% free (root), RAM 66% free, Swap 100% free, CPU load 1-2% typ.
My persistence is rrd4j, once a minute, two values.
Here is a log of accesses in about one minute.

Watches established.
/srv/openhab2-logs/ MODIFY events.log
/srv/openhab2-logs/ MODIFY events.log
/srv/openhab2-logs/ MODIFY events.log
/srv/openhab2-logs/ MODIFY events.log
/srv/openhab2-logs/ MODIFY events.log
/var/lib/samba/ CREATE wins.dat.2113
/var/lib/samba/ MODIFY wins.dat.2113
/var/lib/samba/ CLOSE_WRITE,CLOSE wins.dat.2113
/var/lib/samba/ ATTRIB wins.dat.2113
/var/lib/samba/ DELETE wins.dat
/var/lib/samba/ MOVED_FROM wins.dat.2113
/var/lib/samba/ MOVED_TO wins.dat
/srv/openhab2-logs/ MODIFY events.log
/srv/openhab2-logs/ MODIFY events.log
/var/lib/samba/ CREATE wins.dat.2136
/var/lib/samba/ MODIFY wins.dat.2136
/var/lib/samba/ CLOSE_WRITE,CLOSE wins.dat.2136
/var/lib/samba/ ATTRIB wins.dat.2136
/var/lib/samba/ DELETE wins.dat
/var/lib/samba/ MOVED_FROM wins.dat.2136
/var/lib/samba/ MOVED_TO wins.dat
/srv/openhab2-logs/ MODIFY events.log
/var/cache/samba/lck/ CREATE 2143
/var/cache/samba/lck/ MODIFY 2143
/var/cache/samba/lck/ DELETE 2143
/var/cache/samba/lck/ CLOSE_WRITE,CLOSE 2143
/srv/openhab2-userdata/persistence/rrd4j/ CLOSE_WRITE,CLOSE Robo_Battery_Batterieladung.rrd
/var/lib/samba/ CREATE wins.dat.2167
/var/lib/samba/ MODIFY wins.dat.2167
/var/lib/samba/ CLOSE_WRITE,CLOSE wins.dat.2167
/var/lib/samba/ ATTRIB wins.dat.2167
/var/lib/samba/ DELETE wins.dat
/var/lib/samba/ MOVED_FROM wins.dat.2167
/var/lib/samba/ MOVED_TO wins.dat
/srv/openhab2-logs/ MODIFY events.log 

My problem is as follows, as detailed here. Once in two or more months I cannot access openhabian by ssh port 22 any longer. Openhab is still running, but no more access to openhabian. Thus, I cannot do a soft shutdown. Disconnecting and connecting the power plug does not help, and installation soon fails completely and needs reflashing. Now, I use the systeminfo binding to monitor system ressources in my sitemap.

For my use case, lost logs or persistence data are not as critical as failed Openhab operation. Therefore, your approach possibly could help provided it is in fact a disk corrruption issue.

Give it a go and off load the persistance to a USB stick or a NAS as a trial. Are you using a UPS at all? if you get power cutting in and out that is very bad as linux may/will be writting to files during the boot process when the power is cutting in and out. When I get some spare time I will look into what files change on boot.

I have no UPS. I will offload persistence to a USB stick. Thanx!

I’d suggest also adjusting the mount options for the SD card file system. I’ve been using commit=360 parameter with ext4 to reduce write access, yet still have everything written to card (eventually). The maximum delay in this case would be 6 minutes and the file system should be safe & consistent no matter power outages. I’ve been running with this setup for several months now.
https://www.kernel.org/doc/Documentation/filesystems/ext4.txt

Good hint, thanks. I’ve added this to my list of recommendations (this post).

Thanks for suggesting will take a look when I get some time. So far my tweaks are working well and the only way I see them causing issues is if Samba gets upgraded and on a reboot the files are downgraded, so I keep an eye on the updates for any mention of samba which has not happened yet. Backing up is how I handle this which before doing updates it is a good idea to backup first anyway.

Logs appear to reach a certain size and then they get renamed which moves them to the sd card so never seen my RAM disk go anywhere near filling up. I could probably drop it’s size even lower but I have not had the need to.

EDIT: When openhab.log reaches approx. 17mb in size, it gets renamed and a new file is created. This breaks the symlink so the new file then writes to the SD card again, no crashes or issues it just keeps working fine besides the SD card starts to get the writes happening again until I reboot. So by rebooting before the log hits 17mb or by turning off or reducing the log outputs it is easy to keep the file from reaching this size.

Thanks, great stuff.

I have been following your advice and minimized the storage load by samba and /tmp.

I am also seeing a lot of writes for the /run/resolvconf/lock folder on Raspbian Stretch (2018-06 download):

  • creating the lock folder
  • creating a PID file
  • removing PID file
  • removing lock folder

Any idea about this? Can it be safely memfs-d?

I wouldn’t care to reduce processes to do more or less one-time writing which is what most demon processes do w.r.t their PID… quite some risk to break things and no real benefit.
If you’re really seeing “lots of” writes to that dir, something’s wrong (permanent startup loop ?) and you should find out what it is instead.

Great let me know what you find as I have been doing this for a long time on multiple projects not just Openhab with great results.

Sorry no idea, best asked on the Raspbian forum perhaps… But yes it is safe in the sense you can not damage your hardware in any way and if you have a backup of the SD cards contents it is very easy to reverse, so go for it and try it, that is the best way to learn. Linux has for years allowed itself to be loaded into RAM, non persistant USB loads, LIVECD’s, the list can go on not to mention every professional project that can not accept flash corruption…

This link may interest you as it mentions the lock files with a solution, some of what is written in that article I dont agree on for example data corruption does not damage the drive requiring you to throw it away…
http://hallard.me/raspberry-pi-read-only/

It is hard to predict what a programmer does in a binding or other type of addon so your behaviour or any other type of extra writes could be from what is being done in an addon. Worth checking into to see if it is an issue as Markus is indicating or perhaps a binding is doing it. Have fun playing.

I was reading the whole discussion.

By testing my openhabian. I found only the logs to be really active in Writting, most of other files use read.

I did Move the logs on a remote location. I covered this on another thread as HOW TO. But only for openhab logs. not system.

By testing my system, the most active location after logs was my persistence db, everything else was more or less read only.

What I did for these was:

  • Mount on /mnt an external USB flash drive
  • I created folders inside the mnt/ (persistence, logs)
  • I stopped openhab2 and rsynced the /var/log to /mnt/log (the same for /var/lib/openhab2/persistence to /mnt/peristence).
  • Instead of Symlinks! I used the mount command and bound some folders to relocate writing

My /etc/FSTAB:

proc /proc proc defaults 0 0
PARTUUID=578b7f5f-01 /boot vfat defaults 0 2
PARTUUID=578b7f5f-02 / ext4 defaults,nodiratime,noatime 0 1
UUID=“039c7f8a-b8d6-4428-8962-36555f669175” /mnt ext4 defaults,nodiratime,noatime 0 0

/mnt/log /var/log none defaults,bind 0 0
/mnt/persistence /var/lib/openhab2/persistence none defaults,bind 0 0

/usr/share/openhab2 /srv/openhab2-sys none bind 0 0
/etc/openhab2 /srv/openhab2-conf none bind 0 0
/var/lib/openhab2 /srv/openhab2-userdata none bind 0 0
/var/log/openhab2 /srv/openhab2-logs none bind 0 0
/usr/share/openhab2/addons /srv/openhab2-addons none bind 0 0

This way 99,9% of my writes has moved to /mnt (external flash thumb)

Next step is to use overlayfs to force all changes to move to USB thumb drive. And not on
This way I can mount the sdcard as readonly.

I also do keep all my /etc/openhab2 on a remote git server (therefore some of my config is there. Use Borgbackup (opensource) to backup specific folders on remote location.

Also this is my Inotify collection for 30 minutes at whole system: https://pastebin.com/6XFAYK43

1 Like

Be aware that USB flash drives use the exact same technology as SD cards so if your persistence is storing important data, it too is at risk of wearing out and corruption.

I have no important data, but also Nightly backups if no io issues exist.
I also plan to test the new USB SSD from sandisk for important stuff

Newer writes that have started in my Ubuntu based setup since this guide was first written:
/run/samba/msg.lock/ (multiple files)
/var/lib/mosquitto/ (mosquitto.db and mosquitto.db.new)
/var/lib/ntp/ (ntp.drift.TEMP and ntp.drift)
/var/lib/openhab2/jsondb/org.eclipse.smarthome.config.discovery.DiscoveryResult.json
/var/lib/openhab2/jsondb/backup/ (multiple files)

Since I am no longer using Openhabian I can’t confirm or test any ‘fixes’ for them. I now use my install script for my Odroid C2 found here to shut down the writes and this is done in a number of ways. You can open the install script in a text editor and read how I handle these files as I test and implement a work around to stop the writes.

Thanks for this post. By following this I was able to transfer my logging and persistance to an external flash drive. However initially mosquitto was not working. But after creating a “mosquitto” folder in log folder it started working.