ZRAM nightly sync, openhab service restart

Hello,

I’m using OH4 with rpi4.
For my shutters and lights I’m using Bticino OpenWebNet binding, and it worked great for last 3 years on OH3. Since clean reinstall to OH4 I’m having difficulties with shutters position.
I’ve noticed that position of all shutters restarts to “UNDEF” during the night, so today I’ve decided to investigate.
I’ve noticed that it happens every day around 1am, and few seconds before reset this is in my syslog:

Nov 28 01:04:56 openhabian systemd[1]: Starting Perform nightly zram sync to persistent storage...
Nov 28 01:04:56 openhabian zram-config[12913]: zram-config sync 2023-11-28-01:04:56-CET
Nov 28 01:04:56 openhabian systemd[1]: Stopping Mosquitto MQTT Broker...
Nov 28 01:04:56 openhabian systemd[1]: mosquitto.service: Succeeded.
Nov 28 01:04:56 openhabian systemd[1]: Stopped Mosquitto MQTT Broker.
Nov 28 01:04:56 openhabian systemd[1]: mosquitto.service: Consumed 1min 5.997s CPU time.
Nov 28 01:04:57 openhabian systemd[1]: Stopping Samba SMB Daemon...
Nov 28 01:04:57 openhabian systemd[1]: smbd.service: Succeeded.
Nov 28 01:04:57 openhabian systemd[1]: Stopped Samba SMB Daemon.
Nov 28 01:04:57 openhabian systemd[1]: smbd.service: Consumed 3.989s CPU time.
Nov 28 01:04:57 openhabian systemd[1]: Stopping Samba NMB Daemon...
Nov 28 01:04:57 openhabian systemd[1]: nmbd.service: Succeeded.
Nov 28 01:04:57 openhabian systemd[1]: Stopped Samba NMB Daemon.
Nov 28 01:04:57 openhabian systemd[1]: nmbd.service: Consumed 1min 14.036s CPU time.
Nov 28 01:04:57 openhabian systemd[1]: Stopping Frontail openHAB instance, reachable at http://openhabian.local:9001...
Nov 28 01:04:57 openhabian systemd[1]: frontail.service: Succeeded.
Nov 28 01:04:57 openhabian systemd[1]: Stopped Frontail openHAB instance, reachable at http://openhabian.local:9001.
Nov 28 01:04:57 openhabian systemd[1]: frontail.service: Consumed 25.465s CPU time.
Nov 28 01:04:57 openhabian systemd[1]: Stopping openHAB - empowering the smart home...

So I went into investigation, why would this happen, since this only stores zram into persistant storage and restarts openhab, but since state is in persistant store it shouldnt matter… or should it?
Well, after checking binding documentation I’ve found this:

  • if openHAB is restarted the binding does not know if a shutter position has changed in the meantime, so its position will be UNDEF. Move the shutter all UP/DOWN to synchronise again its position with the binding

So it all makes sense now, openhab service restarts and resets all positions to UNDEF.

Now to my question, where is zram sync called from and does it need to restart openhab service to work? “crontab -e” is empty. And also times are differing by 5-10min everyday, between 0:55am and 1:05am.

Any ideas?

ty,
Tomi

Welcome to openHAB!
As you are on RPi 4 I recommend to replace your SD card by an USB-SSD and uninstall ZRAM.

I can’t answer the first part but the second part is yes. Otherwise OH might be actively writing while the contents of zram are being dumped and you might end up with corrupted files in the dump.

Unless something has changed in openHABian I’m unaware of, the developer of openHABian actively discourages automatically dumping the zram like this so I wonder if this comes from openHABian in the first place.

The syslog is saying that it’s a systemd cron job. Try looking for a script in /etc/cron.daily or cat /etc/crontab /etc/cron.d/* to see all the jobs scheduled through systemd.

Given that the time isn’t consistent I’m going to guess it’s a script in the cron.daily folder.

Is there a simple migration from SD to USB, or does it require a clean install?

Note, running openHABian on ssd is not supported. It will work for sure but the support you will get from the developers and here on the forum will be limited.

There are two topics which have been discussed in this community: Installing Pi OS on an SSD boot disk and using Pi OS 64bit. Some have still the opinion that it is still advisable to install PiOS 32bit on SD card and activating ZRAM.

As 32bit is the right option for Raspberries with 1GB of RAM, I recommend to install PiOS/bullseye on a SSD and deactivate ZRAM.
The world moved on and more and more users here go for a Pi4 with at least 2 or 4 GB. Then it makes sense to install 64bit PiOS as the first run of a rule takes up to 20 seconds on 32bit and just 2 seconds on 64bit.

Migration is fairly easy. Simply backup oh conf and userdata folder and restore it after a fresh installation (does not apply to apps like mosquitto, persistency databases, etc).
With openhabian it is even easier as you throw the backup zip file into the boot directory and during first-boot it gets automatically restored. Having said this, in the context of openhabian I am not allowed to recommend usage of SSD as boot disk, otherwise this post gets deleted.

1 Like

It’s unclear what you did to your system. Did you reinstall the openhab package only ? Or the OS ?

Syncing ZRAM no longer requires to restart OH so openHABian no longer does this in quite some time. As it obviously does on your system I believe you’re still running an ancient OS setup.

It’s /etc/systemd/system/zsync.service usually calling /usr/local/sbin/zram-config.
You can replace the latter with the latest version from https://github.com/ecdye/zram-config.

Well or you can install everything from scratch on SSD as recommended by some lonely shouters here. But then you will no longer receive help from myself or others on this forum as it’s no longer a supported openHABian setup so if you do and when you’re in need for help then for support please turn to those that recommended doing so.

1 Like

I did a clean reinstall of openhabian OS via RPI imager. From what I see image is from 2023-08-12, so seems recent to me. I did restore openhab configuration which was used on earlier openhabian OS version, but OS was clean.

When I SSH into openhabian, it says:
openHABian
openHAB 4.0.4 - Release Build

I lost power yesterday, and subsequently lost all historic data since I booted the Raspberry Pi.

Do I understand the statement I quoted correctly, that zram is syncing nightly to disk?

I just ran sudo /usr/local/sbin/zram-config sync which caused OH to restart. This implies to me running the command is not advisable when OH is running.

So, is there a way to sync the zram data to disk? Ideally automatically.


[edit1] Reading on Git: When running zram on a directory that has services accessing it, they will need to be stopped before starting or stopping zram. For example, in the log zram device zram-config stops the services that run by default in the /var/log directory before starting or stopping. If your system has other services that write to /var/log that are not stopped zram may fail to properly sync files and remove the zram device when stopping, and will probably outright fail to start when initializing a zram device. This issue is not limited to logs, if you are running zram on another directory that is written to by a service you will run into the same issue.

It seems zram cannot be synced to disk while OH is running.

It can but zram-config doesn’t do by default.
I don’t know if you have the most recent version (of zram) installed, but if so, try this:

SERVICE=1 /usr/local/sbin/zram-config sync

Put it into a cron job or service timer.

1 Like

I’ve just added that to openHABian, please test.
Only in main branch for now.

I didn’t find it after updating openhabian to 1.9(main).
Is it in crontab or a service timer? Or do I have to select something in openhabian-config?

A service timer, it only gets installed on zram install during fresh openHABian installs.
You could uninstall reinstall zram via menu but don’t do on production systems.

1 Like

Hello fellow Openhab users and maintainers.

I have a (fairly) fresh openhabian install:

Version:     4.1.2 (Build)

User:        openhab (Active Process 739)
User Groups: openhab tty dialout audio bluetooth gpio

Directories: Folder Name      | Path                        | User:Group
             -----------      | ----                        | ----------
             OPENHAB_HOME     | /usr/share/openhab          | openhab:openhab
             OPENHAB_RUNTIME  | /usr/share/openhab/runtime  | openhab:openhab
             OPENHAB_USERDATA | /var/lib/openhab            | openhab:openhab
             OPENHAB_CONF     | /etc/openhab                | openhab:openhab
             OPENHAB_LOGDIR   | /var/log/openhab            | openhab:openhabian
             OPENHAB_BACKUPS  | /var/lib/openhab/backups    | openhab:openhab

ZRAM is running its nigthly sync.

I’ve found that openhab.log and syslog freezes at the time the sync is done.
for openhab.log is helps to change some logging setting i.e. set log level from info to warn in the web UI.
Syslog can be restarted by a

killall -HUP rsyslogd

(there is no PID file to use with kill)

killall -HUP openhab

Doesn’t work as openhab is not a know process, the process is Java.

I could find out the PID number via PS (or openhab-cli info), and use that for

kill -HUP [PID]

Can we add some post deploy scripts to /usr/local/sbin/zram-sync
to have syslog and openhab.log reopened after the ZRAM sync?

If any more info is needed please let me know.
Thanks in advance for any help.

Regards André

That is not the best way to restart a service, assuming you are on an Debian based OS.

systemctl restart rsyslog
systemctl restart openhab

You can add these two commands to the same cron job that has the zram sync command.

Hi Rich,
Thanks for your rapid reply.
I’ve mistyped the part that you quoted.
I don’t actualy want to restart the services (with all the included logrotation and start-up actions in openhab).
I just want to get the logging up and running again after the sync.
(I have the latest version of openhabian, the sync runs via the systemd timer).
The rest of openhab keeps on running nice at/after the sync, including rules and persistence.

so I don’t think there is a need to restart the services.

Am I missing some thing here?

Just for reference, seems to be the same as reported here:

I found the same issue with openhab.log, but did not notice syslog. As @mstormi pointed out in the other thread, the openhab.log issue might need a fix in OH core. It’s a bit strange that events.log doesn’t suffer from the same issue, so maybe differences between the handling of the two could be identified and a fix could be created. But - if the same issue exists for syslog, it won’t be enough to “fix” OH.

It doesn’t surprise me that the sync causes problems. A zram sync is a pretty disruptive operation from a file system perspective. I would not be surprised if the sync causes OH to lose the file handle.

Probably just a change to log4j2xml in the openhab-distro repo. openhab.log uses a RollingFile appender but events.log uses a RollingRandomAccessFile appender. The difference is consistent with the observed behavior. A RollingFile appender likely opens the file once and just keeps appending to that open file. However on a sync the open file gets swapped out from under the logger and the logger never knows and continues blindly writing to the old file that is no longer there.

A RollingRandomAccessFile appender, since it allows the file IO to read and write to any part of the file instead of just blindly appending to the end, needs to be a bit more active in monitoring and following the file as it gets swapped out and back during the ZRAM sync.

That’s a theory not based on actually looking at the code. I could be way off base. But the reason matters less than whether changing the appender type for openhab.log to a RollingRandomAccesssFile works or not. And that’s an easy test to perform.

As for syslog/rsyslog, that’s probably going to be a problem best left for the zram community. It’s probably a similar root cause but I don’t think we have the knowledge to address that here.

3 Likes

In an other thread a user posted this screenshot ( see first four rows )

Are similar messages also shown in your cases of problems ?