I have a new install of OH2 running on a Pi3 and wonder what can be moved to tmpfs to save on SD Card writes. I don’t want to move it to a hard disk, as that defeats the object of using a small, silent, low power usage device, with a card I can remove regularly and image to disk. I have a 32gb SD card with 75% unpartitioned space that has worked well on OH1.8 for many months with just the logging moved to tmpfs, and the access log turned off.
I have moved /var/log and /tmp to tmpfs. What else can I move to tmpfs that can be recreated after every reboot?
Can the auto database backup be tamed, as that is creating around 1mb per day, and can it be moved to a memory stick as insurance for when the memory card eventually dies?
Looking at /var/lib/openhab2/tmp and /var/lib/openhab2/cache, can any of that be moved to tmpfs?
Any other suggestions?
Not directly connected to “saving SD Card Writes”… but, if you have a Pi3, why not boot it from a USB stick? USB boot mode is now supported on Pi3.
Reliability and speed will be definitely improved over using a SD card…
I’m actually using a NAS and moved /var/log and my persistence storage there. Losing logs is no big deal, to lose data is. In your case I’d move it to a USB stick. You can and should copy that to offline storage at regular intervals.
[What do you mean by ‘auto database’ ? The OH persistence storage or some Linux mechanism ? ]
Overall, trying to save on SD card writes is a good idea, BUT it will NOT safeguard you from SD corruption and HW crashes.
You still MUST apply additional safety measures here. A backup+restore concept, essentially.
I’m right now working on adding such a system to openHABian, but it’s a quite challenging task because of the large number of options in HW and SW.
It’ll of course be applicable to non-openHABian installs as well. You can get an early version if you’re willing to do beta testing.
@Mihai_Badea : it’s a dangerous misconception that USB sticks are more reliable than SD cards are. They contain the very same technology so once you start writing intensively (logs etc.) and not just occasionally (as in the most common use as a temporary storage medium to exchange data between systems), they start corrupting/crashing much easier .
To move the system to USB will NOT increase your Pi’s reliability! (and will not be faster either if you use the same cards).
Take a look at the activity in folder /var/lib/openhab2/jsondb/backup/
Ideally I’d like to plug a USB stick in and let something run automatically, and then rotate the stick after every major change, so I have a set of sticks that I can roll back to when the SD Card eventually wears out. Kind of like DDS Tapes, back in the olden days 
I’d also like to postpone the day of reckoning by minimising my SD Card writes
Thanks for the tip, didn’t know the Pi3 could do that
That’s the OH things/item database. Wondering why there’s so many writes, but yes, you can move it, too.
Hehe, exactly my approach. Amanda actually is a system I used to operate on DDS (and -whew!! - DLT tapes) 20 years ago when SD and USB weren’t even invented yet (I believe).
Just rediscovered it when I started thinking how to best backup my openHAB server.
Regarding SD card reliability: You also can use high reliability SD cards made for camera recording applications (Car Dashcams, supervision cams,…). They cost about 60% more than the standard ones, but have a much higher lifetime.
I tried to find some solid data on the Samsung website, but they are not very specific there 
But this will only extend the time until the next crash maybe to some years,… but it will come 
So a solid backup concept is still needed.
Not even that is guaranteed. A quality card or USB stick might increase mean time between failures (you never know anyway whether what you bought is ‘quality’). Either way, that will not increase the minimum TBF, and we all know Murphy’s law and many reports of corrupted cards.
From the technology point of view, you are, of course, right. However, statistically speaking, a premium USB flash drive proves more reliable than a premium SD card and I noticed that using a USB stick lowers the possibility of corruption in case of power failure.
A backup/restore strategy is needed even if you run your system with HDDs in a RAID1/5/6/10/etc configuration - nobody denies that.
From this backup/restore point of view, taking the USB approach is even better… Let me describe my setup: I run my Pi3 from a USB3 flash disk. As the Pi only has USB2, the fact that the disk is USB3 makes no difference. However, it makes a great difference when I restore the backup stored on my NAS, which has USB3 ports.
I have a backup script that runs every night and backs up the whole system on my NAS and retains the backups for 30 days (I have a lot of space to sparE).
So, in the event of a misconfiguration, a failed update or even a failed USB flash, I’m able to restore the system in something like 10 minutes.
Here is my script, for example, and the backup is run at 5 o’clock in the morning and takes around 10 minutes also:
#!/bin/sh
NOW=`date +%Y%m%d_%H%M%S`
apt-get clean
mount /remote/Backup
find /remote/Backup/raspi-smartclim/* -mtime +30 -delete
/usr/bin/curl http://192.168.200.202:8083/ZWaveAPI/Backup > /home/pi/razberry.zbk
systemctl stop smbd nmbd openhab2 nginx mongoose z-way-server cron mysql mosquitto pulseaudio
cmd="/bin/tar \
c \
--one-file-system \
--warning=no-xdev \
--warning=no-file-ignored \
--numeric-owner \
--exclude=/proc/* \
--exclude=/lost+found/* \
--exclude=/sys/* \
--exclude=/dev/* \
--exclude=/tmp/* \
--exclude=/run/* \
-C / . \
| /usr/bin/pigz > /remote/Backup/raspi-smartclim/raspi-smartclim_$NOW.tgz"
eval "$cmd"
echo "Backup done, rebooting..."
reboot
A restore, in case is needed, means no more than untar the archive wanted onto the USB stick in a USB3 port…
was thinking as its a backup that there should be an option to have it directly back up to a memory stick. having another copy on a corrupted card is about as much use as a chocolate fireguard
I wonder whether there is a bug making it backup too often? A timed option with the target being to a separate location would be better
@mstormi: SD cards normaly don’t get a hard crash (except maybe some overvoltage events).
They just wear out due to the excessive usage. This is a continous process. The included NAND flash is managed by the memory controller chip and weak cells get marked as bad and a new unused cell takes its place.
So the SD crashes in the end, when all unused cells have been consumed and the spare gets negative 
The high endurance cards have much more of these reserve cells, and I think they get differently tested on the NAND level to select the higher endurance chips.
Another important factor is temperature, which excellerates the effect very much. So keep your Raspi cool!
Cards and even more so sticks are in danger of kids, cats, wives (“oh, you didn’t tell me you needed that one”),coffee, …
Seriously. You can do a number of things to reduce the probability of your card or stick failing, and people spend a lot of time trying out and discussing on the details.
But the bottom line is that time spent on this is basically wasted time because whatever you do, it cannot prevent a failure from happen right the next day.
The only reasonable countermeasure is a (tested and proven) backup and recovery concept. And this is way more complex than to copy a couple of files to some offline storage. Don’t get me wrong, continue doing so, so do I, but that isn’t sufficient either. Think how long it will take you to rebuild your system from scratch, including all your modifications that you will have forgotten about in your backup file list. Or think what will happen if your ZWave controller or other gateway hardware breaks. That’s a way better investment of your time than to discuss wear levelling etc.
Now that caught my attention.
Do you have Z-Way running in parallel to openHAB ? No problems with concurrent accesses to the serial /dev/ACM0 device?
We are being off-topic here, but maybe some other users find the info
useful.
Actually, I believe my backup strategy to be near perfect (NEAR because
nothing is perfect 
)… Not only I backup to a NAS with RAID5, but
he NAS itself backs up the backups off-site, to anothe server (not only the
backups from my openHAB Pi, but all other backups I make of other
devices/PC). My backup script also makes a backup of the z-wave controller
(in my case a RaZberry daughter board) so I got this covered as well…
I missed that…
I’m not using z-wave binding, but z-way one. So no concurrent access…
Yes…
As a pure backup strategy, that’s a fine one (looks very similar to mine), but it’s not well suited for long term storage with different versions of a file like say a SVN or the Git stuff does.
But you only backup files that you remembered to add to your list. Files that you forgot to enter there will be missing after a restore. All that stuff you added to and changed in the system over time …
The other point isn’t about the backup strategy, it’s rather about the restore process:
How do you setup your server when the card breaks? How long will that take you before you can start restoring your backup ?
Long story short, you should (also) do raw dumps of your system SD card and prepare clone cards at intervals.
I’m trying to accomplish for all of that in a single backup solution and make that one the default for openHABian, see Github issue link above.
PS: you’re the first to have thought of the ZWave network config data - kudos.
Ever gone through the hassle of re-including all your devices to a replacement controller ?
Actually, I’m backing up th WHOLE disk, not some files. Everything gets backed up! A restore gets EVERYTHING back: packages, config files, scripts, etc. The whole disk is back in 10 minutes - remember, I’m running of a USB3 flash disk. Take a look at tar command: /bin/tar \ (...) – I’m backing up the root directory…
Well, not yet 
and I pray it won’t come to that.
I did however, a while back, a small test: I backed up z-way config, turned off a device (a z-wave wall plug), forced removed it from z-way (without exclusion I mean, just like when the device is damaged or missing) and restored the backup… Everything worked perfecty! The system didn’t even noticed that I messed with it
Two weeks ago, it looked as if my RaZberry card was broken. I went through replacing the Pi, the RaZberry card, and various SD cards, and all combinations thereof.
It was 3AM when I finally found the issue, which turned out to be related to file corruption, but the good news is now I know to use Z-Way for backing up and restoring the ZWave network to a replacement hardware DOES work without the need for re-inclusion of devices (and I do have a lot of them, some of which are hard to access).
So what happens if anything interesting occurs on the Z-wave network while the backup is running? openhab is off. What if the sun rises at about that time and you had something scheduled? Or you woke up early and wandered down the hallway expecting the motion detector to turn on the lights? Seems like there needs to be a way to back up while the automation continues to run?
But what if the LED or lighting bulb fails?
Or if you have a power outage and home automation is off? You could buy a generator, of course. But what if the power generator malfunctions? Well, let’s have a power generator to back up the first generator… And so on and so forth… 
As I said before: nothing is ever perfect! You cannot forsee everything and be prepared for everything. You have to take some chances.
You are of course perfectly right about what you said, but my setup, for me, covers pretty much what I needed: fast backup and, more than that, fast restore - I tend to get myself into trouble a lot and I need often to go back to a previously working config
If the system would’ve been designed to allow backups without stopping services… but…
Right now, stopping the daemons assures me that a restored backup would work out-of-the box.