/srv in ZRAM

mstormi · February 21, 2020, 12:08am

I’ve tried figuring that out and have created a preliminary version.
Could you please help with testing?
Please download https://raw.githubusercontent.com/mstormi/openhabian/systemd-zram-mount-order/includes/mount_template to /opt/openhabian/includes/ and replace /opt/openhabian/functions/system.bash by https://raw.githubusercontent.com/mstormi/openhabian/systemd-zram-mount-order/functions/system.bash.

Do not let openhabian-config update itself on start (as that would overwrite your changes).
Take a copy of /etc/fstab (just in case) and run openhabian-config option 13, then boot to see if that makes a difference. Thanks.
Please continue over at Github.

Lionello_Marrelli · February 23, 2020, 9:17am

I followed your procedure on my production openhabian.

The zram-service log shows some errors but then it says that the zram service is started.

● zram-config.service - zram-config
   Loaded: loaded (/etc/systemd/system/zram-config.service; enabled; vendor preset: enabled)
   Active: active (exited) since Sun 2020-02-23 09:21:30 CET; 7min ago
  Process: 255 ExecStart=/usr/local/bin/zram-config start (code=exited, status=0/SUCCESS)
 Main PID: 255 (code=exited, status=0/SUCCESS)

Feb 23 09:21:28 openhab zram-config[255]: mv: cannot stat '/usr/local/share/zram-config/zram-device-list.new': No such file or directory
Feb 23 09:21:28 openhab zram-config[255]: mv: cannot stat '/usr/local/share/zram-config/zram-device-list.new': No such file or directory
Feb 23 09:21:28 openhab zram-config[255]: stat: cannot stat 'stat': No such file or directory
Feb 23 09:21:28 openhab zram-config[255]: stat: cannot stat 'stat': No such file or directory
Feb 23 09:21:28 openhab zram-config[255]: stat: cannot stat 'stat': No such file or directory
Feb 23 09:21:29 openhab zram-config[255]: mv: cannot stat '/usr/local/share/zram-config/zram-device-list.new': No such file or directory
Feb 23 09:21:29 openhab zram-config[255]: stat: cannot stat 'stat': No such file or directory
Feb 23 09:21:29 openhab zram-config[255]: stat: cannot stat 'stat': No such file or directory
Feb 23 09:21:29 openhab zram-config[255]: stat: cannot stat 'stat': No such file or directory
Feb 23 09:21:30 openhab systemd[1]: Started zram-config.

The /srv/openhab2-log/ contents are significantly older and not updated compared to /var/log/openhab2. I checked the fstab file and I noticed that the srv mounts are not in the fstab file anymore as they used to be.

When applying option 13 of openhabian-config It does not show errors apparently

[09:58:10] openhabian@openhab:~$ sudo openhabian-config
2020-02-23_09:58:18_CET [openHABian] Checking for root privileges... OK
Hit:1 http://archive.raspberrypi.org/debian buster InRelease
Hit:2 http://raspbian.raspberrypi.org/raspbian buster InRelease
Ign:3 https://dl.bintray.com/openhab/apt-repo2 stable InRelease
Get:4 https://dl.bintray.com/openhab/apt-repo2 stable Release [6,051 B]
Fetched 6,051 B in 1s (4,069 B/s)
Reading package lists... Done
2020-02-23_09:58:25_CET [openHABian] Loading configuration file '/etc/openhabian.conf'... OK
2020-02-23_09:58:25_CET [openHABian] openHABian configuration tool version: [master]v1.5-541(5158a5f)
2020-02-23_09:58:26_CET [openHABian] Checking for changes in origin... OK
2020-02-23_09:58:40_CET [openHABian] Preparing openHAB folder mounts under /srv/...
$ rm -f /etc/systemd/system/srv-openhab2-addons.mount /etc/systemd/system/srv-openhab2-conf.mount /etc/systemd/system/srv-openhab2-logs.mount /etc/systemd/system/srv-openhab2-sys.mount /etc/systemd/system/srv-openhab2-userdata.mount

$ mkdir -p /srv/openhab2-sys /srv/openhab2-conf /srv/openhab2-userdata /srv/openhab2-logs /srv/openhab2-addons

$ cp /opt/openhabian/includes/srv_readme.txt /srv/README.txt

$ chmod ugo+w /srv /srv/README.txt

$ create_mount /usr/share/openhab2 sys

$ create_mount /etc/openhab2 conf

$ create_mount /var/lib/openhab2 userdata

$ create_mount /var/log/openhab2 logs

$ create_mount /usr/share/openhab2/addons addons
2020-02-23_09:58:45_CET [openHABian] Applying miscellaneous system settings...
$ setcap cap_net_raw,cap_net_admin=+eip cap_net_bind_service=+ep /opt/jdk/zulu8.40.0.178-ca-jdk1.8.0_222-linux_aarch32hf/bin/java

$ setcap cap_net_raw,cap_net_admin=+eip cap_net_bind_service=+ep /usr/sbin/arping
mkdir: cannot create directory ‘/home/openhabian/.ssh’: File exists
Creating persistent systemd journal folder location: /var/log/journal
Keeping at most 30 days of systemd journal entries
Vacuuming done, freed 0B of archived journals from /var/log/journal/0f2d65b363c04e06b7da2cca0b4d3ee0.
OK
2020-02-23_09:58:49_CET [openHABian] Checking for default openHABian username:password combination... FAILED
2020-02-23_09:58:51_CET [openHABian] We hope you got what you came for! See you again soon ;)

I then manually removed /srv/openhab2-logs

$ sudo umount /srv/openhab2-logs
$ sudo rm -rf /srv/openhab2-logs

applied option 13 and rebooted. After that, the /srv/openhab2-logs and /var/log/openhab2/ show the same files.
Still zram-service status gives the same errors and fstab file is empty.

mstormi · February 23, 2020, 10:53am

Insert this into [Service] section of /etc/systemd/system/zram-config.service and let me know if restarts still spit log errors:

ExecStartPre=-/bin/rm -f /usr/local/share/zram-config/zram-device-list /usr/local/share/zram-config/zram-device-list.rev

Just to make sure: this is before you applied option 13, right ?

Do you mean /srv/... now also shows the up to date files, so it’s working as expected now ?

fstab empty (no /srv in there) is correct.

Lionello_Marrelli · February 23, 2020, 11:18am

To be precise: I checked only after application of option 13 and reboot. I applied option 13 and rebooted three times but behavior was always the same. It did not happen any more only when I finally removed the /srv/openhab2-logs directory, applied option 13 and rebooted.

Yes, this is exactly what I meant. In this way, I can access to the log file directly from my windows PC through the samba mount.

Now I cannot take offline my system. I’ll modify zram-config in the afternoon and let you know.

Thanks for your help,
LionHe

Lionello_Marrelli · February 23, 2020, 1:54pm

I inserted it but the status of the zram-config.service contains exactly the same errors I reported in my previous post.

UPDATE: actually I just realized that /var/log/openhab2 is not updated anymore, while /srv/openhab2-logs/ contains the updated log files

mstormi · February 23, 2020, 2:30pm

Could you please retrieve the latest version of system.bash from GitHub and try again?
Also append smbd.service to the Before= line of the [Unit] section of /etc/systemd/system/zram-config.service and do systemctl daemon-reload.
Before rebooting, please try going back to that status, i.e. stop zram, create /srv/... dirs with some test contents, or eventually manually loop mount them they were before (mount -t none -o bind /var/log/openhab2 /srv/openhab2-logs/)
Then run openhabian-config and selectop option 13 again.
If that works, finally try to reboot.
Query status of mounts (mount | grep srv) and Samba (systemctl status smbd) while you test.

Did you reboot or systemctl daemon-reload ? That’s required to make that addition take effect.
Should be just cosmetical anyway.

And don’t let openhabian-config update itself and don’t (un)install ZRAM again as doing so would overwrite the changed code.

Lionello_Marrelli · February 23, 2020, 6:12pm

I tried to follow your instructions but I got inconsistent results , that were not reproducible after reboots. I probably need to start from my last backup, before the modifications taken from git. I leave in the following the log of my last attempt.

After modifing system.bash and zram-service.config I disabled zram, restored bind mount as per your instructions, applied option 13 and finally rebooted with zram disabled.

The mount are as follows (zram is disabled)

[18:29:03] openhabian@openhab:~$ mount | grep srv
/dev/mmcblk0p2 on /srv/openhab2-conf type ext4 (rw,noatime)
/dev/mmcblk0p2 on /srv/openhab2-addons type ext4 (rw,noatime)
/dev/mmcblk0p2 on /srv/openhab2-logs type ext4 (rw,noatime)
/dev/mmcblk0p2 on /srv/openhab2-sys type ext4 (rw,noatime)
/dev/mmcblk0p2 on /srv/openhab2-userdata type ext4 (rw,noatime)

I then started zram-config and the mounts are mixed

/dev/mmcblk0p2 on /srv/openhab2-conf type ext4 (rw,noatime)
/dev/mmcblk0p2 on /srv/openhab2-addons type ext4 (rw,noatime)
/dev/mmcblk0p2 on /srv/openhab2-logs type ext4 (rw,noatime)
/dev/mmcblk0p2 on /srv/openhab2-sys type ext4 (rw,noatime)
/dev/mmcblk0p2 on /srv/openhab2-userdata type ext4 (rw,noatime)
overlay1 on /srv/openhab2-userdata type overlay (rw,relatime,lowerdir=/opt/zram/openhab2.bind,upperdir=/opt/zram/zram1/upper,workdir=/opt/zram/zram1/workdir,redirect_dir=on)

when openhab starts /var/log/openhab2/ contains updated files while /srv/openhab2-logs/ are older files. On the other hand /var/lib/userdata and /srv/openhab2-userdata are showing the same files.
I tried rebooting but all srv mounts disappeared! I had to select option 13 again.
I probably need to start over from a backup

Lionello_Marrelli · January 19, 2020, 8:32am

I am using zram on a freshly installed openhabian buster image on a Raspberry pi v3b.
Everything works as expected, I only noticed that the links

/srv/openhab-log

refer to the sd (I.e. contain files whose date correspond to when the zram started) while

/var/log/openhab

contains the actual files in zram, which are continuously updated. The same apply for user data.

Is this behavior intentional or may be it will be removed in further versions of openhabian set of scripts?

Thanks for the very useful zram script,
Lionello

mstormi · January 19, 2020, 10:36am

No. I guess it’s because /srv/openhab-logs is mounted first in the boot process. It’s actually a loopback mount of what /var/log/openhab is at that time in the boot process, i.e. the “original”.

Not sure how to easily change that in a way that keeps /srv mounts working for non-ZRAM users, so I think it will stay like that - unless I get an idea how to properly work around this.

mstormi · February 23, 2020, 6:50pm

You probably don’t have to. Option 13 redoes all startup script things.

You must not artificially stop or disable ZRAM. Double check /etc/systemd/system/zram-config.service is as below. Edit, issue systemctl daemon-reload, then reboot.
You’re probably lacking the Before= and/or PartOf= dependency of ZRAM upon openhab2 so ZRAM has not necessarily fully initialized when OH and Samba start.

[19:49:21] root@devpi:/home/openhabian# cat /etc/systemd/system/zram-config.service
[Unit]
Description=zram-config
After=local-fs.target
Before=reboot.target halt.target smbd.service openhab2.service
PartOf=openhab2.service

[Service]
Type=oneshot
ExecStartPre=-/bin/rm -f /usr/local/share/zram-config/zram-device-list /usr/local/share/zram-config/zram-device-list.rev
ExecStart=/usr/local/bin/zram-config start
ExecStop=/usr/local/bin/zram-config stop
TimeoutSec=180
RemainAfterExit=yes

[Install]
WantedBy=basic.target

I get consistent results on my box, see below how I expect it to be.

[19:49:38] root@devpi:/home/openhabian# mount|grep srv
/dev/mmcblk0p2 on /srv/openhab2-sys type ext4 (rw,noatime)
/dev/mmcblk0p2 on /srv/openhab2-addons type ext4 (rw,noatime)
overlay2 on /srv/openhab2-logs type overlay (rw,relatime,lowerdir=/opt/zram/log.bind,upperdir=/opt/zram/zram2/upper,workdir=/opt/zram/zram2/workdir,redirect_dir=on)
overlay1 on /srv/openhab2-userdata type overlay (rw,relatime,lowerdir=/opt/zram/openhab2.bind,upperdir=/opt/zram/zram1/upper,workdir=/opt/zram/zram1/workdir,redirect_dir=on)
/dev/mmcblk0p2 on /srv/openhab2-conf type ext4 (rw,noatime)

Lionello_Marrelli · February 23, 2020, 9:22pm

I edited /etc/systemd/system/zram-config.service, issued systemctl daemon-reload and rebooted. Then I set option 13 and rebooted. It took three reboot to get everything working. Mounts are the same as yours and /srv/openhab2-logs is equivalent to /var/log/openhab2.
The zram-config status still give the same errors as before

[22:20:00] openhabian@openhab:~$ sudo systemctl status zram-config.service
● zram-config.service - zram-config
   Loaded: loaded (/etc/systemd/system/zram-config.service; enabled; vendor preset: enabled)
   Active: active (exited) since Sun 2020-02-23 22:12:53 CET; 7min ago
  Process: 161 ExecStart=/usr/local/bin/zram-config start (code=exited, status=0/SUCCESS)
 Main PID: 161 (code=exited, status=0/SUCCESS)

Feb 23 22:12:52 openhab zram-config[161]: mv: cannot stat '/usr/local/share/zram-config/zram-device-list.new': No such file or directory
Feb 23 22:12:52 openhab zram-config[161]: mv: cannot stat '/usr/local/share/zram-config/zram-device-list.new': No such file or directory
Feb 23 22:12:52 openhab zram-config[161]: stat: cannot stat 'stat': No such file or directory
Feb 23 22:12:52 openhab zram-config[161]: stat: cannot stat 'stat': No such file or directory
Feb 23 22:12:52 openhab zram-config[161]: stat: cannot stat 'stat': No such file or directory
Feb 23 22:12:53 openhab zram-config[161]: mv: cannot stat '/usr/local/share/zram-config/zram-device-list.new': No such file or directory
Feb 23 22:12:53 openhab zram-config[161]: stat: cannot stat 'stat': No such file or directory
Feb 23 22:12:53 openhab zram-config[161]: stat: cannot stat 'stat': No such file or directory
Feb 23 22:12:53 openhab zram-config[161]: stat: cannot stat 'stat': No such file or directory
Feb 23 22:12:53 openhab systemd[1]: Started zram-config.

mstormi · February 23, 2020, 10:09pm

Good.

I had forgotten the ExecStartPre= line. I’ve edited my post above. Add that, daemon-reload, that should make a difference w.r.t. those error messages.

mstormi · March 1, 2020, 11:31am

Any news, have you tried ExecStartPre= ?

Lionello_Marrelli · March 1, 2020, 11:44am

Sorry for not reporting earlier. I modified the service script, issued the daemon-reload and rebooted but the error is still the same.

The service file is the following:

[12:39:23] openhabian@openhab:~$ cat /etc/systemd/system/zram-config.service
[Unit]
Description=zram-config
After=local-fs.target
Before=reboot.target halt.target smbd.service openhab2.service
PartOf=openhab2.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/zram-config start
ExecStop=/usr/local/bin/zram-config stop
TimeoutSec=180
RemainAfterExit=yes
ExecStartPre=-/bin/rm -f /usr/local/share/zram-config/zram-device-list /usr/local/share/zram-config/zram-device-list.rev

[Install]
WantedBy=basic.target

The status of the service is

[12:42:41] openhabian@openhab:~$ sudo systemctl status zram-config
● zram-config.service - zram-config
   Loaded: loaded (/etc/systemd/system/zram-config.service; enabled; vendor preset: enabled)
   Active: active (exited) since Sun 2020-03-01 12:38:39 CET; 4min 29s ago
  Process: 253 ExecStartPre=/bin/rm -f /usr/local/share/zram-config/zram-device-list /usr/local/share/zram-c
  Process: 302 ExecStart=/usr/local/bin/zram-config start (code=exited, status=0/SUCCESS)
 Main PID: 302 (code=exited, status=0/SUCCESS)

Mar 01 12:38:37 openhab zram-config[302]: mv: cannot stat '/usr/local/share/zram-config/zram-device-list.new
Mar 01 12:38:38 openhab zram-config[302]: mv: cannot stat '/usr/local/share/zram-config/zram-device-list.new
Mar 01 12:38:38 openhab zram-config[302]: stat: cannot stat 'stat': No such file or directory
Mar 01 12:38:38 openhab zram-config[302]: stat: cannot stat 'stat': No such file or directory
Mar 01 12:38:38 openhab zram-config[302]: stat: cannot stat 'stat': No such file or directory
Mar 01 12:38:38 openhab zram-config[302]: mv: cannot stat '/usr/local/share/zram-config/zram-device-list.new
Mar 01 12:38:38 openhab zram-config[302]: stat: cannot stat 'stat': No such file or directory
Mar 01 12:38:38 openhab zram-config[302]: stat: cannot stat 'stat': No such file or directory
Mar 01 12:38:38 openhab zram-config[302]: stat: cannot stat 'stat': No such file or directory
Mar 01 12:38:39 openhab systemd[1]: Started zram-config.

PS. In order to reboot I simply issued the command “sudo reboot now”. After reboot openhab charts were not showing any data, implying that persistence files were lost, I do not know if it is related.

mstormi · March 1, 2020, 12:07pm

EDIT: First of all, get the latest version of zram-config from the openhabian-zram repo (https://github.com/mstormi/openhabian-zram) and install it as /usr/local/bin/zram-config.

If that still shows the error, can you share your /usr/local/share/zram-config/log/zram-config.log ?
Eventually also append “-x” to the first line of /usr/local/bin/zram-config, then in journalctl -u zram-config.service you should see in log what’s happening, i.e. which commands fail and cause the output.

You can also add the -x and simply (as root !) run /usr/local/bin/zram-config with stop or start as the argument.

Lionello_Marrelli · March 1, 2020, 2:50pm

I installed the latest version of zram-config and previous errors no longer appears.
A different error claiming that /boot/cmdline.txt does not exist appears (but the file actually is there).
Persistence are always losts after reboots, possibly due to wrong srv mounts (even rebooting multiple times does not fix this). The mounts are now as follow

/dev/mmcblk0p2 on /srv/openhab2-addons type ext4 (rw,noatime)
/dev/mmcblk0p2 on /srv/openhab2-conf type ext4 (rw,noatime)
/dev/mmcblk0p2 on /srv/openhab2-userdata type ext4 (rw,noatime)
/dev/mmcblk0p2 on /srv/openhab2-sys type ext4 (rw,noatime)
overlay1 on /srv/openhab2-logs type overlay (rw,relatime,lowerdir=/opt/zram/log.bind,upperdir=/opt/zram/zram1/upper,workdir=/opt/zram/zram1/workdir,redirect_dir=on)

The log is very long so that it is at the following pastebin link

mstormi · March 1, 2020, 5:18pm

But something’s weird with your box. The error keeps reappearing not every time you boot, but often. zram-config does not change that file so it’s related to your OS setup.
Are you sure you always booted from the same device/partition ?

Persistence is stored under /var/lib/openhab2/persistence, but I don’t see /var/lib/openhab2 ever be processed by zram that’s why it’s not mounted
Is it in /etc/ztab ? Try option13 again if it’s missing.

Lionello_Marrelli · March 1, 2020, 5:57pm

I tried applying option13 and rebooted.
The /etc/ztab still does not contain all the mounts.


# swap  alg     mem_limit       disk_size       swap_priority   page-cluster    swappiness
swap    lz4     250M            750M            75              0               80

# dir   alg     mem_limit       disk_size       target_dir              bind_dir
#dir    lz4     50M             150M            /home/pi                /pi.bind

# log   alg     mem_limit       disk_size       target_dir              bind_dir                oldlog_dir
log     lz4     50M             150M            /var/log                /log.bind               /opt/zram/oldlog

Maybe the sd is failing : I’ll try to burn a new SD with a backup and try again.

mstormi · March 1, 2020, 6:01pm

Unlikely. Save your time.

Sorry, it isn’t option 13 to create that. It’s the ZRAM option (38 I think).
You can also manually edit ztab instead, here’s mine:

# swap  alg     mem_limit       disk_size       swap_priority   page-cluster    swappiness
swap    lz4     300M            600M            75              0               1

# dir   alg     mem_limit       disk_size       target_dir              bind_dir
dir     lz4     500M            700M            /var/lib/openhab2       /openhab2.bind

# log   alg     mem_limit       disk_size       target_dir              bind_dir                oldlog_dir
log     lzo     100M            300M            /var/log                /log.bind               /volatile/oldlogs

Lionello_Marrelli · March 1, 2020, 6:50pm

I applied option 38 for zram, the mounts are ok now, but i think your modifications got overwritten.
The log of openhabian-config is

Note: checking out 'cb4423bb4df1c4bc85448ed2117c8dfda6700db4'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

Reading package lists... Done
Building dependency tree
Reading state information... Done
libattr1-dev is already the newest version (1:2.4.48-4).
0 upgraded, 0 newly installed, 0 to remove and 66 not upgraded.
Cloning into 'overlayfs-tools'...
remote: Enumerating objects: 14, done.
remote: Counting objects: 100% (14/14), done.
remote: Compressing objects: 100% (14/14), done.
remote: Total 110 (delta 5), reused 0 (delta 0), pack-reused 96
Receiving objects: 100% (110/110), 40.44 KiB | 476.00 KiB/s, done.
Resolving deltas: 100% (61/61), done.
gcc -Wall -std=c99 -c main.c
gcc -Wall -std=c99 -c logic.c
gcc -Wall -std=c99 -c sh.c
gcc -lm main.o logic.o sh.o -o overlay
#####          Reboot to activate zram-config         #####
#####       edit /etc/ztab to configure options       #####

The error in zram-config status is the same as this morning, i.e. before the installation of the github version of ram.