RPI 2 B+ crashes frequently

rlkoshak · November 18, 2016, 6:27pm

Not /etc/ Nothing really changes much in /etc unless you physically do it yourself. On a Pi there are a couple of exceptions (e.g. fakehardwareclock gets written to once an hour by a cron job, resolv.conf gets written to once a day, etc) but for the most part etc is pretty static by design. It is intended to be configuration data, not runtime data.

I’ve posted my Ansible playbook I use to set up my Pis as readonly here.

If you use Ansible you should be able to use this as is. If not you should be able to figure out the steps to make it work.

Spaceman_Spiff · November 20, 2016, 8:21pm

This works fine, too (and is how I did it)!
Because I use rrd4j and mapdb so there are multiple folders in /etc.
With mounting the complete folder I do not need to mount the sub folders individually.

Tweedleburger · May 11, 2017, 12:20am

Sooo I know it’s been long time but I FINALLY got around to implementing this and realized I didn’t 100% understand how to initially set these folders in Openhab. Is it a change to openhab.in.sh in /usr/share/openhab/bin/ that I need?

I think i can figure out the cron job that copies it to the SD card once an hour…

Edit: found this https://www.howtoforge.com/storing-files-directories-in-memory-with-tmpfs and used this:
sudo mount -t tmpfs -o size=100M,mode=0755 tmpfs /home/seth/openhab/configurations/persistence/
and
sudo mount -t tmpfs -o size=100M,mode=0755 tmpfs /var/log/openhab
to mount the files (I think this is right?) and added the following lines to fstab:
tmpfs /home/seth/openhab/configurations/persistence/ tmpfs size=100M,mode=0755 0 0
tmpfs /var/log/openhab tmpfs size=100M,mode=0755 0 0

So now I just need to cron/copy the folders back once an hour…I thought this would be the easy part…but I’m not so sure how the mount command works. It feels like it creates like a symlink from that folder location to the tmpfs on ram. But if that’s true, do I just copy it once an over over the same folder path as it was before? or…?

Tweedleburger · May 11, 2017, 1:58am

Well I’m not so sure that worked the way I wanted it too. After those commands, my /var/log/openhab and ~/configurations/persistence are now empty as well as /dev/shm/ sooo I don’t think I quite have it yet…

Spaceman_Spiff · May 11, 2017, 4:44am

On shutdown you have to copy the files to the sd card and on startup you have to restore them from the sd-card to tmpfs. There is a script in the pi-forums which does exactly that (google “varlog”) or I can post the script that I am using.

Tweedleburger · May 11, 2017, 5:00pm

@Spaceman_Spiff first off, thanks so much for responding despite my long delay. I would love to see your example script, but I tried hard to find this script in the pi-forums but to no avail (sorry I really didn’t want to have to come back here for something as easy as googling ). I tried both the official RPi forums and the /hardware/server/ sections of this forum but nothing was immediately obvious.

Tweedleburger · May 11, 2017, 5:02pm

also, conceptionally, wouldn’t I want some script run at startup and shutdown AND once an hour (or some regular interval)?

Spaceman_Spiff · May 11, 2017, 7:58pm

etc/systemd/system/varlog.service:

DefaultDependencies=no
Before=local-fs.target shutdown.target
Conflicts=umount.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/local/bin/varlog start
ExecStop=/usr/local/bin/varlog stop

[Install]
RequiredBy=systemd-journal-flush.service

/usr/local/bin/varlog:

#!/bin/bash

case $1 in
    start)
        /bin/mount -t tmpfs tmpfs /var/log -o defaults,size=70M
        /bin/chmod 755 /var/log
        /bin/cp -Rpu /var/log_save/* /var/log
    ;;
    stop)
        /bin/rm -fr /var/log_save
        /bin/cp -Rp /var/log /var/log_save
    ;;
esac

exit 0

/etc/systemd/system/openhabmount.service:

DefaultDependencies=no
Before=local-fs.target shutdown.target
Conflicts=umount.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/local/bin/openhab start
ExecStop=/usr/local/bin/openhab stop

[Install]
RequiredBy=systemd-journal-flush.service

/usr/local/bin/openhab

#!/bin/bash

case $1 in
    start)
        /bin/mount -t tmpfs tmpfs /opt/Openhab/logs -o defaults,size=200M
        /bin/chmod 777 /opt/Openhab/logs

        /bin/mount -t tmpfs tmpfs /opt/Openhab/etc -o defaults,size=200M
        /bin/chmod 777 /opt/Openhab/etc
        /bin/cp -Rpu /opt/Openhab/etc_save/* /opt/Openhab/etc

        /bin/mount -t tmpfs tmpfs /opt/Openhab/webapps -o defaults,size=200M
        /bin/chmod 777 /opt/Openhab/webapps
        /bin/cp -Rpu /opt/Openhab/webapps_save/* /opt/Openhab/webapps
    ;;
    stop)
        /bin/rm -fr /opt/Openhab/etc_save
        /bin/cp -Rp /opt/Openhab/etc /opt/Openhab/etc_save

        /bin/rm -fr /opt/Openhab/webapps_save
        /bin/cp -Rp /opt/Openhab/webapps /opt/Openhab/webapps_save
    ;;
esac

exit 0

Tweedleburger · May 11, 2017, 8:24pm

Oh wow, thanks so much! That is more involved than I was anticipating. I’ll comb through this to try to understand it before implementation, then report back. Thanks again

Spaceman_Spiff · May 12, 2017, 4:43am

Basicly it just does this file copy thing as I have described earlier.
It mounts the folders as tmpfs without the fstab and then copies the saved files to the new mount.
On shutdown it saves all files on the sd card.
Additionally (I think) I run a cron job once a day with “/usr/local/bin/openhab stop” to persist my data on my sdcard.

Tweedleburger · May 14, 2017, 8:55pm

I’m very close…

Edit6: I apologize. My post had degenerated into babbling nonsense. Short story: been hacking at this all day learning (and relearning) lots about a bunch of stuff that I’m sure a more experienced user just breezes through, but its all very useful.

I still can’t quite get it to work though. The services start w systemctl but varlog gives an error /bin/cp: cannot stat '/var/log_save/*': No such file or directory even though I do have that folder. One question is that it contains very different content from /var/log/. Do I need to do an initial copy over? Or is this maybe an issue with the asterix

I think i discovered that you probably did a manual install and I an apt-get one, so our folder structure is slightly different (I gather this from the /opt/Openhab folder locations). Let me know if that’s incorrect, but if not I assume I replace many of the references with my specific openhab installation’s folders. If I have an apt-get installation and my logs are located in /var/log/, that would already be included in the varlog script, right? So is it correct that I wouldn’t need the first 2 lines of this script?

Spaceman_Spiff · May 15, 2017, 9:51am

You have to call /usr/local/bin/openhab stop the first time you want to use the script.
I am running a manual installation, so additionally you have to replace the folder names in /usr/local/bin/openhab.
Please leave the varlog untouched.

I have mounted the openhab-log folder via the fstab-file.

Tweedleburger · May 16, 2017, 5:56pm

@Spaceman_Spiff can you elaborate on what exactly you added to the fstab file? is it tmpfs /var/log/openhab tmpfs size=70M,mode=0755 0 0 or something like that to /etc/fstab?

That last line confuses me because it was my understanding that the varlog script moved ALL of /var/log (in which openhab resides on my installation) to tmpfs with copies back and forth at boot, shutdown (and eventually regular intervals w cron).

Which actually leads me to another clarification that may be related. My current /var/log_save/ folder contains the following:

auth.log
daemon.log
debug
kern.log
messages
syslog

And obviously the contents are vastly different in /var/log/ (not least of which, /var/log includes a openhab folder). If I enable varlog.service and reboot my system, I am able to SSH in, but openHAB doesn’t open on my computer or the android app. And the /var/log/ folder will contain the above hierarchy that used to be in /var/log_save/ (and still is, both log/ and log_save/ contain that)

On a more positive note. I have a /home/openhab folder, which isn’t the real home of user openhab, /var/lib/openhab still is, /home/openhab just symlinks to the 4 main folders of openhab (in /var/log, /etc, /var/lib and usr/share/). In the openhabmount script (that’s what I renamed it) I have 3 “entries” for /var/lib/workspace/, /var/lib/persistance/, and /usr/share/webapps/ that all save to a respective folder in /home/openhab/openhab_save. When enabling openhabmount.service those folders in ~/openhab_save seem to be populated by the contents of their respective folders. So I’m obviously very close…I think there is maybe something small but pertinent about tmpfs shares that I don’t quite grasp…

edit: had a thought: do I need to call /usr/local/bin/varlog stop to initialize the process as well as the openhab one? I’m a little hesitant to try without confirm, because of the contents already in /var/log_save/…

Spaceman_Spiff · May 17, 2017, 4:12pm

Yes - of course. You have to call stop for both scripts. Sorry that I forgot to mention that. :S
I have to mount the openhab/logs/ because I do not use the apt-get installation.

Tweedleburger · May 18, 2017, 4:41pm

ahhh, no my fault, I should’ve figured that out. Just was a little nervous trying because there was already stuff in /var/log_save, and you specifically didn’t mention it (I clearly hold your opinion in the highest esteem )

I SEEM to be up and running. I have the 4 shares the scripts set up show up when I type mount (for /var/log/ /var/lib/openhab/{persistence,workspace} and /usr/share/openhab/webapps), and the corresponding save folder seems to be populated by the hierarchy I’d expect… So thank you so much for all your help @Spaceman_Spiff!!!

2 final questions if you don’t mind?

@SS i know your answer to this, but just curious if anyone else has any other folders to recommend adding to tmpfs, both inside openhab and just linux in general
You mentioned putting /usr/local/bin/openhab stop into cron… which I will do, but I’m curious if I’d also have to have a cron entry to restart it? Or will the service continue to run via systemd (and the stop command, just copies the tmpfs folder over to the save location)?

Spaceman_Spiff · May 19, 2017, 6:31am

exactly.

Tweedleburger · May 20, 2017, 3:42am

Cool cool cool. In case its useful for anyone else here’s my /usr/local/bin/openmount:
#!/bin/bash

case $1 in
    start)
        /bin/mount -t tmpfs tmpfs /var/lib/openhab/persistence -o defaults,size=70M
        /bin/chmod 777 /var/lib/openhab/persistence
        /bin/cp -Rpu /home/openhab/openhab_save/persistence_save/* /var/lib/openhab/persistence

        /bin/mount -t tmpfs tmpfs /var/lib/openhab/workspace -o defaults,size=70M
        /bin/chmod 777 /var/lib/openhab/workspace
        /bin/cp -Rpu /home/openhab/openhab_save/workspace_save/* /var/lib/openhab/workspace

        /bin/mount -t tmpfs tmpfs /usr/share/openhab/webapps -o defaults,size=70M
        /bin/chmod 777 /usr/share/openhab/webapps
        /bin/cp -Rpu /home/openhab/openhab_save/webapps_save/* /usr/share/openhab/webapps
    ;;
    stop)
        /bin/rm -fr /home/openhab/openhab_save/persistence_save
        /bin/cp -Rp /var/lib/openhab/persistence /home/openhab/openhab_save/persistence_save

        /bin/rm -fr /home/openhab/openhab_save/workspace_save
        /bin/cp -Rp /var/lib/openhab/workspace /home/openhab/openhab_save/workspace_save

        /bin/rm -fr /home/openhab/openhab_save/webapps_save
        /bin/cp -Rp /usr/share/openhab/webapps /home/openhab/openhab_save/webapps_save
    ;;
esac

exit 0

I pretty much used @Spaceman_Spiff’s varlog and both service files and they work great. I used ‘crontab -e’ while logged into the root user (sudo su) and added:
0 * * * * /usr/local/bin/varlog stop
1 * * * * /usr/local/bin/openhabmount stop

and the crontab is firing at the correct times i.e. /var/log/syslog and everything seems to be saving correctly. Thanks again!

Now on to moving everything to a USB stick!

Tweedleburger · May 22, 2018, 7:43pm

Sorry to bump an old thread, but does anyone have suggestions for which folders to put into tmpfs in OH2?

edit: after looking at the openhabmount it seems I could reduce these 3 folders down to using just /var/lib/openhab2 inclusive. Any foreseeable issues with this? And, if anyone is kind enough to respond (@Spaceman_Spiff are you still out there:)?) Do i need to still mount the folders in fstab? or will the scripts take care of them. (I noticed i had commented out the tmpfs mount commands in my fstab, but I don’t remember if I did that when I upgraded to OH2 or before) All help is greatly appreciated.

Spaceman_Spiff · May 24, 2018, 4:39am

I think you only need to put OH2/userdata into tmpfs.
Make sure to persist at least the folders “etc”, “openhabcloud”, “zwave” and the uuid -file (if they exist)

No - the scripts have a “mount” command in them.

Tweedleburger · May 24, 2018, 7:51pm

to clarify the OH2/userdata folder on an apt installation would be the /var/lib/openhab2/ folder, right? the one with cache, config, etc, jsondb, kar, openhabcloud, persistence, tmp and uuid …?

And I’m sorry for what is almost certainly a dumb question, but what does the verb ‘persist’ mean in this context? (e.g. Make sure you persist at least these folders)

Gotcha, thanks for the clarification.