Corrupt FileSystems every 2-3 month?

I would add moving /var/lib/openhab/persistence in addition to /var/logs to another medium. In a mid to large sized system with persistence configured would result in a whole ton of writes as well, perhaps as much if not more than logging depending on the number of Items and persistence approach.

Though the risk of losing that medium produces more of an impact than just losing the logs, particularly if you are using persistence to track long term data.

Also, moving ‘everything that writes’ isn’t really all that hard. I just had a catastrophic failure of one of my Pis (not SD card related) and just rebuilt it as read only. I’ve written an Ansible playbook (posted below for those interested) to do it for me. I’m not arguing against whether it provides that much benefit (I did it more for security reasons and to learn how to do it) but it isn’t that difficult. NOTE: this Pi is not running openHAB.

My Read Only Jesse Ansible Playbook:
NOTES:

  • Treat as Alpha, it is only partially tested.
  • With a read only runtime Tripwire probably doesn’t add anything so much of that stuff will probably go away
  • The command to run the relink.sh script relinks a bunch of var folders to the new tempfs /tmp. Sometimes ansible gets stuck after that as the networking gets interrupted. I had to put this into a script because sshd refuses new logins after messing with /var/run and /var/lib/dhcp and each line in an Ansible script is a separate ssh into the target machine
  • I’ve configured my Pis to remotely rsyslog to my main server so I don’t bother with any of the techniques to periodically write the logs to the SD card
  • Some of the steps are unnecessary for Jesse Lite but they were part of the posting I based this off of and I’ve not scrubbed it: Protect your Raspberry PI SD card, use Read-Only filesystem – Charles's Blog
  • I plan on posting all of my Ansible scripts which include setting up a bunch of other security related changes (UFW, Tripwire, config changes) to github at some point when I have a chance to more fully test it.
---
- name: Add aliases and fancy prompt to show status of FS
  blockinfile:
    state: present
    dest: /etc/bash.bashrc
    block: |
      # set variable identifying the filesystem you work in (used in the prompt below)
      set_bash_prompt(){
          fs_mode=$(mount | sed -n -e "s/^\/dev\/.* on \/ .*(\(r[w|o]\).*/\1/p")
          PS1='\[\033[01;32m\]\u@\h${fs_mode:+($fs_mode)}\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '
      }

      alias ro='sudo mount -o remount,ro / ; sudo mount -o remount,ro /boot'
      alias rw='sudo mount -o remount,rw / ; sudo mount -o remount,rw /boot'

      # setup fancy prompt"
      PROMPT_COMMAND=set_bash_prompt

- name: Set up /tmp, /var/log, and /var/tmp as tempfs
  blockinfile:
    state: present
    dest: /etc/fstab
    insertafter: "#   use  dphys-swapfile swap[on|off]  for that"
    block: |
      tmpfs           /tmp            tmpfs   nosuid,nodev         0       0
      tmpfs           /var/log        tmpfs   nosuid,nodev         0       0
      tmpfs           /var/tmp        tmpfs   nosuid,nodev         0       0

- name: Set permissions on /tmp
  file:
    mode: a+rwx
    path: /tmp
    state: directory

- name:  Mount /tmp
  mount:
    name: /tmp
    src: /tmp
    fstype: tmpfs
    state: mounted

- name: Mount /var/log
  mount:
    name: /var/log
    src: /var/log
    fstype: tmpfs
    state: mounted

- name: Mount /var/tmp
  mount:
    name: /var/tmp
    src: /var/tmp
    fstype: tmpfs
    state: mounted

- name: Remap folders to /tmp
  script: relink.sh

- name: Waiting for {{ inventory_hostname }} to come back from reboot
  local_action: wait_for host={{ inventory_hostname }} state=started delay=30 timeout=300
  become: false

- name: Configure boot command line
  replace:
    dest: /boot/cmdline.txt
    regexp: 'otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait'
    replace: 'otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait fastboot noswap ro'

- name: Move dhcpd lock file to temp
  replace:
    dest: /etc/systemd/system/dhcpcd5
    regexp: '\=/run/dhcpcd.pid'
    replace: '\=/var/run/dhcpcd.pid'

- name: update fake-hwclock cron job to remount / rw and then mount it back
  copy:
    dest: /etc/cron.hourly/fake-hwclock
    src: fake-hwclock

- name: Check for presence of Tripwire cron job
  stat: path=/etc/cron.daily/tripwire
  register: tripwire_cron

- name: Update cron job to remount rw before running check
  copy:
    dest: /etc/cron.daily/tripwire
    src: tripwire-cron
  when: tripwire_cron.stat.exists == True

- name: Remove some start scripts
  shell: /sbin/insserv -r bootlogs; /sbin/insserv -r console-setup

- name: Set boot FS as readonly
  replace:
    dest: /etc/fstab
    regexp: '/dev/mmcblk0p1  /boot           vfat    defaults          0       2'
    replace: '/dev/mmcblk0p1  /boot           vfat    defaults,ro          0       2'

- name: Set root as readonly
  replace:
    dest: /etc/fstab
    regexp: '/dev/mmcblk0p2  /               ext4    defaults,noatime  0       1'
    replace: '/dev/mmcblk0p2  /               ext4    defaults,noatime,ro  0       1'

- name: Reboot as read only
  include: tasks/reboot.yml

relink.sh

#!/bin/bash

rm -rf /var/lock
ln -s  /tmp /var/lock

rm -rf /var/spool
ln -s /tmp /var/spool

rm -rf /var/run
ln -s /tmp /var/run

cp -r /var/lib/dhcp/* /tmp
rm -rf /var/lib/dhcp/*
ln -s /tmp /var/lib/dhcp

fake-hwclock: the main addition is a remount of / as rw so it can write then mounting it back to ro

#!/bin/sh
#
# Simple cron script - save the current clock periodically in case of
# a power failure or other crash

if (command -v fake-hwclock >/dev/null 2>&1) ; then
  mount -o remount,rw /
  fake-hwclock save
  mount -o remount,ro /
fi

tripwire-cron, same as above, mount rw then as ro when done

#!/bin/sh -e

mount -o remount,rw /

tripwire=/usr/sbin/tripwire

[ -x $tripwire ] || exit 0

umask 027

$tripwire --check --quiet --email-report

mount -o remount,ro /
1 Like