ZRAM issues

[michael_p (Michael)]

Sorry for not being clearer and I hope this is not considered high jacking because the problem was mentioned here before…
Today it happened again.
I’m trying to give as much useful information as possible but because all logging in the system essentially stops it is difficult. Please let me know if I missed something!

openhabian@openHAB:~ $ zramctl
NAME ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram1 lzo-rle 250M 240.6M 46.1M 120.9M 4 /opt/zram/zram1
/dev/zram0 lzo-rle 200M 12K 1.4K 8K 4 [SWAP]

openhabian@openHAB:~ $ cat /etc/ztab
# swap	alg		mem_limit	disk_size	swap_priority	page-cluster	swappiness
swap	lzo-rle		200M		600M		75		0		80

# dir	alg		mem_limit	disk_size	target_dir			bind_dir
dir	lzo-rle		150M		1000M		/var/lib/openhab2/persistence	/persistence.bind

# log	alg		mem_limit	disk_size	target_dir		bind_dir		oldlog_dir
log	lzo-rle		250M		500M		/var/log		/log.bind

openhabian@openHAB:~ $ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 29G 6.7G 22G 25% /
devtmpfs 1.8G 0 1.8G 0% /dev
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 1.9G 179M 1.8G 10% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/sda3 30G 45M 28G 1% /storage
/dev/mmcblk0p1 253M 45K 253M 1% /boot
/dev/zram1 227M 222M 0 100% /opt/zram/zram1
overlay1 227M 222M 0 100% /var/log
tmpfs 388M 0 388M 0% /run/user/1000

events.log & openhab.log just stop mid line and will not continue to be filled.
It seems like the 250mb memory limit is used as a maximum size for zram device and not the 500mb that it should if I understood the config correctly.
Also shouldn’t old logs not gotten rotated out of that folder? I have a few log.1 log.2 and log.gz files there that fill up the device.

[mstormi (Markus Storm)]

@michael_p

Sorry for not being clearer

Read more carefully.
You still fail to give the basicmost information. OS, OH versions ? openHABian branch ?

@michael_p

It seems like the 250mb memory limit is used as a maximum size for zram device and not the 500mb that it should if I understood the config correctly

Right. That should not happen. Have you restarted ZRAM or rebooted ? Reinstalled ?

@michael_p

Also shouldn’t old logs not gotten rotated out of that folder? I have a few log.1 log.2 and log.gz files there that fill up the device.

no
OH handles that

[ndye (Ethan Dye)]

There was a bug in the creation of zram devices that was causing the mem limit to be used as both the max disk size and mem limit for all devices. If you reinstall it should be fixed and resolve your issue.

[michael_p (Michael)]

@mstormi

You still fail to give the basicmost information. OS, OH versions ? openHABian branch ?

This is what openhabian-config gives me:

openHABian Configuration Tool [main]v1.6.3-1192(9b398a9)

I’m on the main branch with OH3 Milestone 1
Not sure where I can find the exact Openhabian OS Version.

@mstormi

Have you restarted ZRAM or rebooted ?

If I restart the issue goes away for a bit till the zram device is full again.

@mstormi

Reinstalled ?

I just reinstalled zram (uninstall, reboot, install) via openhabian-cli.

@ndye

There was a bug in the creation of zram devices that was causing the mem limit to be used as both the max disk size and mem limit for all devices. If you reinstall it should be fixed and resolve your issue.

How can I verify i have the correct version?
It seems to still limit the disk size by the mem_limit.

openhabian@openHAB:/var/log/openhab $ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 29G 6.8G 22G 25% /
devtmpfs 1.8G 0 1.8G 0% /dev
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 1.9G 9.4M 1.9G 1% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/sda3 30G 45M 28G 1% /storage
/dev/mmcblk0p1 253M 48M 205M 19% /boot
/dev/zram1 227M 193M 16M 93% /opt/zram/zram1
overlay1 227M 193M 16M 93% /var/log
tmpfs 388M 0 388M 0% /run/user/1000

[ndye (Ethan Dye)]

@michael_p

How can I verify i have the correct version?
It seems to still limit the disk size by the mem_limit.

The output of zramctl should look something like this and if it does then it is fixed. Pay special attention to the DISKSIZE column as is should match yours exactly if it is fixed.

NAME       ALGORITHM DISKSIZE  DATA COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram2 zstd          600M 42.3M  2.6M 10.4M       4 /opt/zram/zram2
/dev/zram1 zstd          900M 46.8M  1.8M 21.9M       4 /opt/zram/zram1
/dev/zram0 lzo-rle       600M 29.7M   12M 15.4M       4 [SWAP]

[michael_p (Michael)]

I feel I don have to fix…

openhabian@openHAB:/var/log/openhab $ zramctl --version
zramctl from util-linux 2.33.1
openhabian@openHAB:/var/log/openhab $ zramctl
NAME ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram1 lzo-rle 250M 214.4M 38M 107M 4 /opt/zram/zram1
/dev/zram0 lzo-rle 200M 4K 86B 4K 4 [SWAP]
openhabian@openHAB:/var/log/openhab $ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 29G 6.8G 22G 25% /
devtmpfs 1.8G 0 1.8G 0% /dev
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 1.9G 9.4M 1.9G 1% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/sda3 30G 45M 28G 1% /storage
/dev/mmcblk0p1 253M 48M 205M 19% /boot
/dev/zram1 227M 199M 11M 96% /opt/zram/zram1
overlay1 227M 199M 11M 96% /var/log
tmpfs 388M 0 388M 0% /run/user/1000

But if i read ztab right it should be 500MB not 250MB

# swap	alg		mem_limit	disk_size	swap_priority	page-cluster	swappiness
swap	lzo-rle		200M		600M		75		0		80

# dir	alg		mem_limit	disk_size	target_dir			bind_dir
dir	lzo-rle		150M		1000M		/var/lib/openhab2/persistence	/persistence.bind

# log	alg		mem_limit	disk_size	target_dir		bind_dir		oldlog_dir
log	lzo-rle		250M		500M		/var/log		/log.bind

I used openhabian-config to first uninstall and then reinstall zram. Is there a better method to ensure I have the correct version installed?

[ndye (Ethan Dye)]

No, you are not on the latest version. Try running it again make sure that you reboot in between uninstalling and reinstalling as a precaution. To be extra safe you could try running sudo rm -rf /opt/zram after uninstalling and rebooting to ensure that openhabian-config will redownload zram when installing.

[michael_p (Michael)]

@ndye Thanks! I think deleting /opt/zram did the trick!
I also noticed that the /etc/ztab got overwritten, but I guess that on purpose.

[ndye (Ethan Dye)]

@michael_p

I also noticed that the /etc/ztab got overwritten, but I guess that on purpose.

Yep, glad it worked!

[tdlr (Tony De La Rosa)]

Configuration

##   Release = Raspbian GNU/Linux 10 (buster)
##    Kernel = Linux 5.10.52-v7+
##  Platform = Raspberry Pi 3 Model B Rev 1.2
##    Uptime = 0 day(s). 0:19:19
## CPU Usage = 3.05% avg over 4 cpu(s) (4 core(s) x 1 socket(s))
##  CPU Load = 1m: 0.18, 5m: 0.25, 15m: 0.42
##    Memory = Free: 0.04GB (5%), Used: 0.90GB (95%), Total: 0.94GB
##      Swap = Free: 2.23GB (100%), Used: 0.00GB (0%), Total: 2.24GB
##      Root = Free: 21.52GB (79%), Used: 5.66GB (21%), Total: 28.38GB
##   Updates = 0 apt updates available.
##  Sessions = 2 session(s)
## Processes = 132 running processes of 32768 maximum processes
##Openhabian = 3.1.0 - Release Build

Summary
I’ve had a similar situation where frontail no longer shows the logs and df -h shows var/log at 100%. I’ve uninstall zram using openhabian-config, entered sudo rm -rf /opt/zram on the cli after I exited openhabian-config, and then rebooted. Afterwards, I went back into openhabian-config and installed zram. However, the version stays the same as well as the disksize. What am I’m doing wrong?

openhabian@casajuarez:~ $ zramctl --version
zramctl from util-linux 2.33.1
openhabian@casajuarez:~ $ zramctl
NAME       ALGORITHM DISKSIZE  DATA  COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram2 zstd          450M 17.2M 205.3K  564K       4 /opt/zram/zram2
/dev/zram1 zstd          350M 33.9M 654.1K  1.1M       4 /opt/zram/zram1
/dev/zram0 lzo-rle       450M  3.2M  59.7K  320K       4 [SWAP]
openhabian@casajuarez:~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        29G  5.7G   22G  21% /
devtmpfs        454M     0  454M   0% /dev
tmpfs           487M     0  487M   0% /dev/shm
tmpfs           487M  7.4M  479M   2% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           487M     0  487M   0% /sys/fs/cgroup
/dev/mmcblk0p1  253M   49M  204M  20% /boot
tmpfs            98M     0   98M   0% /run/user/1000
/dev/zram1      324M   18M  281M   7% /opt/zram/zram1
overlay1        324M   18M  281M   7% /var/lib/openhab/persistence
/dev/zram2      420M  1.6M  387M   1% /opt/zram/zram2
overlay2        420M  1.6M  387M   1% /var/log

Btw, I can see the logs when I open the Karaf console and type log:tail.

[ndye (Ethan Dye)]

zramctl is not controlled by the zram-config package, it is installed by the linux kernel. From what you have shared your zram is working correctly right now. Please provide logs relating to the topic at hand if you want me to be able to help you.

[narf27 (Peter)]

i have a similar problem.
system was set up a week ago (openhabian), zram was installed during unattended setup (no re-install):

###############################################################################
###############  ohab3  #######################################################
###############################################################################
##   Release = Raspbian GNU/Linux 10 (buster)
##    Kernel = Linux 5.10.52-v7l+
##  Platform = Raspberry Pi 4 Model B Rev 1.1
##    Uptime = 0 day(s). 16:38:6
## CPU Usage = 0% avg over 4 cpu(s) (4 core(s) x 1 socket(s))
##  CPU Load = 1m: 0.56, 5m: 0.24, 15m: 0.12
##    Memory = Free: 2.33GB (61%), Used: 1.46GB (39%), Total: 3.79GB
##      Swap = Free: 0.53GB (100%), Used: 0.00GB (0%), Total: 0.53GB
##      Root = Free: 2.90GB (43%), Used: 3.77GB (57%), Total: 7.00GB
##   Updates = 12 apt updates available.
##  Sessions = 2 session(s)
## Processes = 139 running processes of 32768 maximum processes
###############################################################################

                          _   _     _     ____   _
  ___   ___   ___   ___  | | | |   / \   | __ ) (_)  ____   ___
 / _ \ / _ \ / _ \ / _ \ | |_| |  / _ \  |  _ \ | | / _  \ / _ \
| (_) | (_) |  __/| | | ||  _  | / ___ \ | |_) )| || (_) || | | |
 \___/|  __/ \___/|_| |_||_| |_|/_/   \_\|____/ |_| \__|_||_| | |
      |_|                          3.2.0.M1 - Milestone Build

today the daily amanda backup mail was missing in my inbox so i checked frontail logs but it was empty. had the same problem on my old setup (2.5.x)…

i’m not sure if this is the same issue as it is mentioned here, because my system doesn’t say /var/log 100%:

openhabian@ohab3:~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       7.1G  3.8G  3.0G  57% /
devtmpfs        1.7G     0  1.7G   0% /dev
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           1.9G   26M  1.9G   2% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/sda3       7.3G  272M  6.7G   4% /storage
/dev/mmcblk0p1  253M   49M  204M  20% /boot
/dev/zram1      324M  177M  123M  60% /opt/zram/zram1
overlay1        324M  177M  123M  60% /var/lib/openhab/persistence
/dev/zram2      324M   28M  271M  10% /opt/zram/zram2
overlay2        324M   28M  271M  10% /var/lib/influxdb
/dev/zram3      420M   11M  379M   3% /opt/zram/zram3
overlay3        420M   11M  379M   3% /var/log
tmpfs           389M     0  389M   0% /run/user/1000

i guess this is normal with zram?

openhabian@ohab3:~ $ cat /var/log/openhab/openhab.log
cat: /var/log/openhab/openhab.log: No such file or directory

zram:

openhabian@ohab3:~ $ zramctl
NAME       ALGORITHM DISKSIZE   DATA COMPR  TOTAL STREAMS MOUNTPOINT
/dev/zram3 zstd          450M  25.9M  2.1M    13M       4 /opt/zram/zram3
/dev/zram2 zstd          350M  91.4M 56.7M  65.2M       4 /opt/zram/zram2
/dev/zram1 zstd          350M 212.2M  3.5M 105.8M       4 /opt/zram/zram1
/dev/zram0 lzo-rle       450M     4K   86B     4K       4 [SWAP]
openhabian@ohab3:~ $ zramctl --version
zramctl from util-linux 2.33.1

ztab:

# swap  alg             mem_limit       disk_size       swap_priority   page-cluster    swappiness
swap    lzo-rle         200M            450M            75              0               80

# dir   alg             mem_limit       disk_size       target_dir                      bind_dir
dir     zstd            150M            350M            /var/lib/openhab/persistence    /persistence.bind
dir     zstd            150M            350M            /var/lib/influxdb               /influxdb.bind

# log   alg             mem_limit       disk_size       target_dir              bind_dir                oldlog_dir
log     zstd            200M            450M            /var/log                /log.bind

while setting up the system i had a problem with influxdb (solved here), but i don’t know if this is relevant… journal -xe brings has many lines that are connected to influxdb…:

-- Logs begin at Sun 2021-08-08 20:34:36 CEST, end at Mon 2021-08-09 13:20:02 CEST. --
Aug 09 12:29:30 ohab3 influxd-systemd-start.sh[723]: [httpd] 127.0.0.1 - openhab [09/Aug/2021:12:29:30 +0200] "POST /write?db=openhab&rp=autogen&precision=n&consistency=one HTTP/1.1 " 204 0 "-" "okhttp/3.14.4" ae76cca9-f8fc-11eb-8098-dca6322f7bc3 1022
...
...
... (lots of entries)
Aug 09 13:19:18 ohab3 influxd-systemd-start.sh[723]: [httpd] 127.0.0.1 - openhab [09/Aug/2021:13:19:18 +0200] "POST /write?db=openhab&rp=autogen&precision=n&consistency=one HTTP/1.1 " 204 0 "-" "okhttp/3.14.4" a32c866d-f903-11eb-8426-dca6322f7bc3 10
Aug 09 13:19:19 ohab3 influxd-systemd-start.sh[723]: [httpd] 127.0.0.1 - openhab [09/Aug/2021:13:19:19 +0200] "POST /write?db=openhab&rp=autogen&precision=n&consistency=one HTTP/1.1 " 204 0 "-" "okhttp/3.14.4" a4030c77-f903-11eb-8427-dca6322f7bc3 12
Aug 09 13:19:32 ohab3 influxd-systemd-start.sh[723]: [httpd] 127.0.0.1 - openhab [09/Aug/2021:13:19:32 +0200] "POST /write?db=openhab&rp=autogen&precision=n&consistency=one HTTP/1.1 " 204 0 "-" "okhttp/3.14.4" aba7d64f-f903-11eb-8428-dca6322f7bc3 11
Aug 09 13:19:33 ohab3 influxd-systemd-start.sh[723]: [httpd] 127.0.0.1 - openhab [09/Aug/2021:13:19:33 +0200] "POST /write?db=openhab&rp=autogen&precision=n&consistency=one HTTP/1.1 " 204 0 "-" "okhttp/3.14.4" ac22bffa-f903-11eb-8429-dca6322f7bc3 10
Aug 09 13:19:34 ohab3 influxd-systemd-start.sh[723]: [httpd] 127.0.0.1 - openhab [09/Aug/2021:13:19:34 +0200] "POST /write?db=openhab&rp=autogen&precision=n&consistency=one HTTP/1.1 " 204 0 "-" "okhttp/3.14.4" ad17c095-f903-11eb-842a-dca6322f7bc3 11
Aug 09 13:19:38 ohab3 influxd-systemd-start.sh[723]: [httpd] 127.0.0.1 - openhab [09/Aug/2021:13:19:38 +0200] "POST /write?db=openhab&rp=autogen&precision=n&consistency=one HTTP/1.1 " 204 0 "-" "okhttp/3.14.4" af02cf8f-f903-11eb-842b-dca6322f7bc3 14
Aug 09 13:19:39 ohab3 influxd-systemd-start.sh[723]: [httpd] 127.0.0.1 - openhab [09/Aug/2021:13:19:39 +0200] "POST /write?db=openhab&rp=autogen&precision=n&consistency=one HTTP/1.1 " 204 0 "-" "okhttp/3.14.4" af9c438b-f903-11eb-842c-dca6322f7bc3 96
Aug 09 13:19:41 ohab3 influxd-systemd-start.sh[723]: [httpd] 127.0.0.1 - openhab [09/Aug/2021:13:19:41 +0200] "POST /write?db=openhab&rp=autogen&precision=n&consistency=one HTTP/1.1 " 204 0 "-" "okhttp/3.14.4" b0ceaa2c-f903-11eb-842d-dca6322f7bc3 25
Aug 09 13:19:43 ohab3 influxd-systemd-start.sh[723]: [httpd] 127.0.0.1 - openhab [09/Aug/2021:13:19:43 +0200] "POST /write?db=openhab&rp=autogen&precision=n&consistency=one HTTP/1.1 " 204 0 "-" "okhttp/3.14.4" b210bd4e-f903-11eb-842e-dca6322f7bc3 19
Aug 09 13:19:44 ohab3 influxd-systemd-start.sh[723]: [httpd] 127.0.0.1 - openhab [09/Aug/2021:13:19:44 +0200] "POST /write?db=openhab&rp=autogen&precision=n&consistency=one HTTP/1.1 " 204 0 "-" "okhttp/3.14.4" b2aa51bb-f903-11eb-842f-dca6322f7bc3 10
Aug 09 13:19:44 ohab3 influxd-systemd-start.sh[723]: [httpd] 127.0.0.1 - openhab [09/Aug/2021:13:19:44 +0200] "POST /write?db=openhab&rp=autogen&precision=n&consistency=one HTTP/1.1 " 204 0 "-" "okhttp/3.14.4" b2f75cda-f903-11eb-8430-dca6322f7bc3 19
Aug 09 13:19:47 ohab3 influxd-systemd-start.sh[723]: [httpd] 127.0.0.1 - openhab [09/Aug/2021:13:19:47 +0200] "POST /write?db=openhab&rp=autogen&precision=n&consistency=one HTTP/1.1 " 204 0 "-" "okhttp/3.14.4" b494f275-f903-11eb-8431-dca6322f7bc3 15
Aug 09 13:19:50 ohab3 influxd-systemd-start.sh[723]: [httpd] 127.0.0.1 - openhab [09/Aug/2021:13:19:50 +0200] "POST /write?db=openhab&rp=autogen&precision=n&consistency=one HTTP/1.1 " 204 0 "-" "okhttp/3.14.4" b67eaa39-f903-11eb-8432-dca6322f7bc3 19
Aug 09 13:19:57 ohab3 influxd-systemd-start.sh[723]: [httpd] 127.0.0.1 - openhab [09/Aug/2021:13:19:57 +0200] "POST /write?db=openhab&rp=autogen&precision=n&consistency=one HTTP/1.1 " 204 0 "-" "okhttp/3.14.4" ba429a5d-f903-11eb-8433-dca6322f7bc3 28
Aug 09 13:19:57 ohab3 influxd-systemd-start.sh[723]: [httpd] 127.0.0.1 - openhab [09/Aug/2021:13:19:57 +0200] "POST /write?db=openhab&rp=autogen&precision=n&consistency=one HTTP/1.1 " 204 0 "-" "okhttp/3.14.4" ba52f7f4-f903-11eb-8434-dca6322f7bc3 22
Aug 09 13:19:58 ohab3 influxd-systemd-start.sh[723]: [httpd] 127.0.0.1 - openhab [09/Aug/2021:13:19:58 +0200] "POST /write?db=openhab&rp=autogen&precision=n&consistency=one HTTP/1.1 " 204 0 "-" "okhttp/3.14.4" bafbd76e-f903-11eb-8435-dca6322f7bc3 10
Aug 09 13:19:59 ohab3 influxd-systemd-start.sh[723]: [httpd] 127.0.0.1 - openhab [09/Aug/2021:13:19:59 +0200] "POST /write?db=openhab&rp=autogen&precision=n&consistency=one HTTP/1.1 " 204 0 "-" "okhttp/3.14.4" bbf0fadf-f903-11eb-8436-dca6322f7bc3 15
Aug 09 13:20:00 ohab3 influxd-systemd-start.sh[723]: [httpd] 127.0.0.1 - openhab [09/Aug/2021:13:20:00 +0200] "POST /write?db=openhab&rp=autogen&precision=n&consistency=one HTTP/1.1 " 204 0 "-" "okhttp/3.14.4" bc5c9839-f903-11eb-8437-dca6322f7bc3 11
Aug 09 13:20:01 ohab3 influxd-systemd-start.sh[723]: [httpd] 127.0.0.1 - openhab [09/Aug/2021:13:20:01 +0200] "POST /write?db=openhab&rp=autogen&precision=n&consistency=one HTTP/1.1 " 204 0 "-" "okhttp/3.14.4" bcb8e0d9-f903-11eb-8438-dca6322f7bc3 17
Aug 09 13:20:01 ohab3 influxd-systemd-start.sh[723]: [httpd] 127.0.0.1 - openhab [09/Aug/2021:13:20:01 +0200] "POST /write?db=openhab&rp=autogen&precision=n&consistency=one HTTP/1.1 " 204 0 "-" "okhttp/3.14.4" bccc562b-f903-11eb-8439-dca6322f7bc3 20
Aug 09 13:20:02 ohab3 influxd-systemd-start.sh[723]: [httpd] 127.0.0.1 - openhab [09/Aug/2021:13:20:02 +0200] "POST /write?db=openhab&rp=autogen&precision=n&consistency=one HTTP/1.1 " 204 0 "-" "okhttp/3.14.4" bd56b40f-f903-11eb-843a-dca6322f7bc3 13

journal -xe says there are 1129 lines, most of these lines look like those above…

as i also had problems with amanda i also checked amanda:

[13:24:07] backup@ohab3:~$ amcheck openhab-dir
amcheck: critical (fatal): create debug directory "/var/log/amanda/server/": Permission denied
amcheck: create debug directory "/var/log/amanda/server/": Permission denied
[13:24:14] backup@ohab3:~$ amreport openhab-dir
amreport: critical (fatal): create debug directory "/var/log/amanda/server/": Permission denied
amreport: create debug directory "/var/log/amanda/server/": Permission denied

this is not my operative setup so i haven’t made a reboot, but if it’s the same as on my operative setup (2.5) a reboot would solve the problem for 2 or 3 days…

if my problem is not connected to this thread please tell me, so i can open a new one!

edit:
something’s wrong with mosquitto, too (i can tail logs via console):

20:03:26.015 [INFO ] [del.core.internal.ModelRepositoryImpl] - Loading model 'mqtt.things'
20:03:26.109 [INFO ] [hab.event.ThingStatusInfoChangedEvent] - Thing 'mqtt:broker:mosq' changed from OFFLINE (COMMUNICATION_ERROR): io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:1883 to OFFLINE
20:03:26.113 [INFO ] [hab.event.ThingStatusInfoChangedEvent] - Thing 'mqtt:broker:mosq' changed from OFFLINE (COMMUNICATION_ERROR): io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:1883 to OFFLINE
20:03:26.117 [INFO ] [o.transport.mqtt.MqttBrokerConnection] - Starting MQTT broker connection to 'localhost' with clientid openHAB3
20:03:26.132 [INFO ] [hab.event.ThingStatusInfoChangedEvent] - Thing 'mqtt:broker:mosq' changed from OFFLINE to OFFLINE (COMMUNICATION_ERROR): Timeout
20:03:26.146 [INFO ] [hab.event.ThingStatusInfoChangedEvent] - Thing 'mqtt:broker:mosq' changed from OFFLINE (COMMUNICATION_ERROR): Timeout to OFFLINE (COMMUNICATION_ERROR): io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:1883
20:03:36.132 [INFO ] [t.reconnect.PeriodicReconnectStrategy] - Try to restore connection to 'localhost'. Next attempt in 60000ms
20:03:36.140 [INFO ] [hab.event.ThingStatusInfoChangedEvent] - Thing 'mqtt:broker:mosq' changed from OFFLINE (COMMUNICATION_ERROR): io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:1883 to OFFLINE
20:03:36.145 [INFO ] [o.transport.mqtt.MqttBrokerConnection] - Starting MQTT broker connection to 'localhost' with clientid openHAB3
20:03:36.155 [INFO ] [hab.event.ThingStatusInfoChangedEvent] - Thing 'mqtt:broker:mosq' changed from OFFLINE to OFFLINE (COMMUNICATION_ERROR): io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:1883
20:04:36.151 [INFO ] [t.reconnect.PeriodicReconnectStrategy] - Try to restore connection to 'localhost'. Next attempt in 60000ms
20:04:36.157 [INFO ] [hab.event.ThingStatusInfoChangedEvent] - Thing 'mqtt:broker:mosq' changed from OFFLINE (COMMUNICATION_ERROR): io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:1883 to OFFLINE
20:04:36.161 [INFO ] [o.transport.mqtt.MqttBrokerConnection] - Starting MQTT broker connection to 'localhost' with clientid openHAB3
20:04:36.178 [INFO ] [hab.event.ThingStatusInfoChangedEvent] - Thing 'mqtt:broker:mosq' changed from OFFLINE to OFFLINE (COMMUNICATION_ERROR): io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:1883
.
.
.
etc.

[tdlr (Tony De La Rosa)]

My apologies but it appears that my events.log and openhab.log files were deleted when I uninstalled and reinstalled zram. I had done a clean install several months back and enabled zram to avoid sd corruption. Logs stopped displaying on frontail recently so I followed the instructions on this thread because the zram1 and overlay1 (/var/log) was at 100% and there was an error message saying it could not write. I’ve done the reinstall but the /var/log files are not populating even though I can see them when I do a log:display on karaf. I guess I’m trying to find out how to get /var/log files to populate and how to ensure they’re being done on zram. So which logs would you need?

[shutterfreak (Olivier Biot)]

@narf27

Aug 09 13:19:18 ohab3 influxd-systemd-start.sh[723]: [httpd] 127.0.0.1 - openhab [09/Aug/2021:13:19:18 +0200] "POST /write?db=openhab&rp=autogen&precision=n&consistency=one HTTP/1.1 " 204 0 "-" "okhttp/3.14.4" a32c866d-f903-11eb-8426-dca6322f7bc3 10

There was a file permission bug in InfluxDB 1.8.7. Maybe you have been affected by it. Here’s the link with the problem description and the fix:

InfluxDB 1.8.7 on Debian/Raspbian not starting (bug + fix)

Maybe your Influx data folders also have permission problems. Have a look at the file permissions of /var/lib/influxdb/data/

[narf27 (Peter)]

no, that has already been solved:

@narf27

while setting up the system i had a problem with influxdb (solved here ), but i don’t know if this is relevant… journal -xe brings has many lines that are connected to influxdb…:

must be something different

[Wolfgang_S]

I don’t see problems with these entries:

@narf27

Aug 09 13:19:18 ohab3 influxd-systemd-start.sh[723]: [httpd] 127.0.0.1 - openhab [09/Aug/2021:13:19:18 +0200] "POST /write?db=openhab&rp=autogen&precision=n&consistency=one HTTP/1.1 " 204 0 "-" "okhttp/3.14.4" a32c866d-f903-11eb-8426-dca6322f7bc3 10

Just shows that an entry was stored in influxdb and no value was returned ( 204 ).

With regard to

@narf27

[13:24:07] backup@ohab3:~$ amcheck openhab-dir
amcheck: critical (fatal): create debug directory "/var/log/amanda/server/": Permission denied

you ran the command as user backup. What is the permisson of directory /var/log/amanda/server ? who is the owner. I would assume that either it is not correct to run the command as user backup or the permissions do not allow user backup to create a directory or file in that directory structure.

[narf27 (Peter)]

@Wolfgang_S

Just shows that an entry was stored in influxdb and no value was returned ( 204 ).

but i didn’t configure persistence at all (just installed binding and influxdb/grafana via openhabian).

openhabian@ohab3:/etc/openhab/persistence $ ls -l
total 4
-rw-rw-r-- 1 openhab openhab 256 Jun 27 22:57 readme.txt

and i haven’t configured any rules for now.

@Wolfgang_S

you ran the command as user backup. What is the permisson of directory /var/log/amanda/server ? who is the owner. I would assume that either it is not correct to run the command as user backup or the permissions do not allow user backup to create a directory or file in that directory structure.

openhabian@ohab3:/etc/openhab/persistence $ ls -l /var/log/amanda/server/
ls: cannot access '/var/log/amanda/server/': No such file or directory

i’m pretty sure the directoy will be there if i reboot…
on my other system (2.5) i have the same problems every 2-3 days. i rebootet yesterday, atm i see the logs on frontail and the amanda directoy is there. next time the system has problems i can check if the directoy disappears…

[tdlr (Tony De La Rosa)]

Found the solution in the post oh-3-1-update-ui-dashboard-not-loading

[narf27 (Peter)]

tried to fix permissions via (updated) openhabian menu:

+ choice2='14 | Fix Permissions'
+ '[' 0 -eq 1 ']'
+ '[' 1 -eq 255 ']'
+ wait_for_apt_to_finish_update
++ timestamp
++ date +%F_%T_%Z
+ echo -n '2021-08-13_09:39:07_CEST [openHABian] Updating Linux package information... '
2021-08-13_09:39:07_CEST [openHABian] Updating Linux package information... + [[ -z 8218 ]]
+ tail --pid=8218 -f /dev/null
+ echo OK
OK
+ case "$choice2" in
+ permissions_corrections
+ local gpioDir=/sys/devices/platform/soc
+ groups=("audio" "bluetooth" "dialout" "gpio" "tty")
+ local groups
+ openhabFolders=("/etc/openhab" "/var/lib/openhab" "/var/log/openhab" "/usr/share/openhab")
+ local openhabFolders
+ local openhabHome=/var/lib/openhab
+ local backupsFolder=/var/lib/openhab/backups
+ local retval=0
++ timestamp
++ date +%F_%T_%Z
+ echo -n '2021-08-13_09:39:07_CEST [openHABian] Applying file permissions recommendations... '
2021-08-13_09:39:07_CEST [openHABian] Applying file permissions recommendations... + openhab_is_installed
+ openhab2_is_installed
++ dpkg -s openhab2
++ grep Status
++ cut '-d ' -f2
+ [[ '' == \i\n\s\t\a\l\l ]]
+ return 1
+ openhab3_is_installed
++ dpkg -s openhab
++ grep Status
++ cut '-d ' -f2
+ [[ install == \i\n\s\t\a\l\l ]]
+ return 0
+ return 0
+ cond_echo 'Setting Java and arping file capabilites'
+ [[ -z '' ]]
+ echo -e '\033[33;01mSetting Java and arping file capabilites\033[39;49;00m'
Setting Java and arping file capabilites
+++ command -v java
++ realpath /usr/bin/java
+ cond_redirect setcap 'cap_net_raw,cap_net_admin=+eip cap_net_bind_service=+ep' /opt/jdk/zulu11.50.19-ca-jdk11.0.12-linux_aarch32hf/bin/java
+ [[ -n '' ]]
+ echo -e '\n\033[90;01m$ setcap cap_net_raw,cap_net_admin=+eip cap_net_bind_service=+ep /opt/jdk/zulu11.50.19-ca-jdk11.0.12-linux_aarch32hf/bin/java \033[39;49;00m'

$ setcap cap_net_raw,cap_net_admin=+eip cap_net_bind_service=+ep /opt/jdk/zulu11.50.19-ca-jdk11.0.12-linux_aarch32hf/bin/java
+ setcap 'cap_net_raw,cap_net_admin=+eip cap_net_bind_service=+ep' /opt/jdk/zulu11.50.19-ca-jdk11.0.12-linux_aarch32hf/bin/java
+ return 0
+ cond_redirect setcap 'cap_net_raw,cap_net_admin=+eip cap_net_bind_service=+ep' /usr/sbin/arping
+ [[ -n '' ]]
+ echo -e '\n\033[90;01m$ setcap cap_net_raw,cap_net_admin=+eip cap_net_bind_service=+ep /usr/sbin/arping \033[39;49;00m'

$ setcap cap_net_raw,cap_net_admin=+eip cap_net_bind_service=+ep /usr/sbin/arping
+ setcap 'cap_net_raw,cap_net_admin=+eip cap_net_bind_service=+ep' /usr/sbin/arping
+ return 0
+ for pGroup in "${groups[@]}"
+ grep -qs '^[[:space:]]*audio:' /etc/group
+ cond_redirect usermod --append --groups audio openhab
+ [[ -n '' ]]
+ echo -e '\n\033[90;01m$ usermod --append --groups audio openhab \033[39;49;00m'

$ usermod --append --groups audio openhab
+ usermod --append --groups audio openhab
+ return 0
+ cond_redirect usermod --append --groups audio openhabian
+ [[ -n '' ]]
+ echo -e '\n\033[90;01m$ usermod --append --groups audio openhabian \033[39;49;00m'

$ usermod --append --groups audio openhabian
+ usermod --append --groups audio openhabian
+ return 0
+ for pGroup in "${groups[@]}"
+ grep -qs '^[[:space:]]*bluetooth:' /etc/group
+ cond_redirect usermod --append --groups bluetooth openhab
+ [[ -n '' ]]
+ echo -e '\n\033[90;01m$ usermod --append --groups bluetooth openhab \033[39;49;00m'

$ usermod --append --groups bluetooth openhab
+ usermod --append --groups bluetooth openhab
+ return 0
+ cond_redirect usermod --append --groups bluetooth openhabian
+ [[ -n '' ]]
+ echo -e '\n\033[90;01m$ usermod --append --groups bluetooth openhabian \033[39;49;00m'

$ usermod --append --groups bluetooth openhabian
+ usermod --append --groups bluetooth openhabian
+ return 0
+ for pGroup in "${groups[@]}"
+ grep -qs '^[[:space:]]*dialout:' /etc/group
+ cond_redirect usermod --append --groups dialout openhab
+ [[ -n '' ]]
+ echo -e '\n\033[90;01m$ usermod --append --groups dialout openhab \033[39;49;00m'

$ usermod --append --groups dialout openhab
+ usermod --append --groups dialout openhab
+ return 0
+ cond_redirect usermod --append --groups dialout openhabian
+ [[ -n '' ]]
+ echo -e '\n\033[90;01m$ usermod --append --groups dialout openhabian \033[39;49;00m'

$ usermod --append --groups dialout openhabian
+ usermod --append --groups dialout openhabian
+ return 0
+ for pGroup in "${groups[@]}"
+ grep -qs '^[[:space:]]*gpio:' /etc/group
+ cond_redirect usermod --append --groups gpio openhab
+ [[ -n '' ]]
+ echo -e '\n\033[90;01m$ usermod --append --groups gpio openhab \033[39;49;00m'

$ usermod --append --groups gpio openhab
+ usermod --append --groups gpio openhab
+ return 0
+ cond_redirect usermod --append --groups gpio openhabian
+ [[ -n '' ]]
+ echo -e '\n\033[90;01m$ usermod --append --groups gpio openhabian \033[39;49;00m'

$ usermod --append --groups gpio openhabian
+ usermod --append --groups gpio openhabian
+ return 0
+ for pGroup in "${groups[@]}"
+ grep -qs '^[[:space:]]*tty:' /etc/group
+ cond_redirect usermod --append --groups tty openhab
+ [[ -n '' ]]
+ echo -e '\n\033[90;01m$ usermod --append --groups tty openhab \033[39;49;00m'

$ usermod --append --groups tty openhab
+ usermod --append --groups tty openhab
+ return 0
+ cond_redirect usermod --append --groups tty openhabian
+ [[ -n '' ]]
+ echo -e '\n\033[90;01m$ usermod --append --groups tty openhabian \033[39;49;00m'

$ usermod --append --groups tty openhabian
+ usermod --append --groups tty openhabian
+ return 0
+ cond_redirect usermod --append --groups openhab openhabian
+ [[ -n '' ]]
+ echo -e '\n\033[90;01m$ usermod --append --groups openhab openhabian \033[39;49;00m'

$ usermod --append --groups openhab openhabian
+ usermod --append --groups openhab openhabian
+ return 0
+ cond_redirect chown --silent openhab:openhab /srv /opt
+ [[ -n '' ]]
+ echo -e '\n\033[90;01m$ chown --silent openhab:openhab /srv /opt \033[39;49;00m'

$ chown --silent openhab:openhab /srv /opt
+ chown --silent openhab:openhab /srv /opt
+ return 0
+ cond_redirect chmod --silent ugo+w /srv
+ [[ -n '' ]]
+ echo -e '\n\033[90;01m$ chmod --silent ugo+w /srv \033[39;49;00m'

$ chmod --silent ugo+w /srv
+ chmod --silent ugo+w /srv
+ return 0
+ cond_redirect chown --recursive openhab:openhab /etc/openhab /var/lib/openhab /var/log/openhab /usr/share/openhab
+ [[ -n '' ]]
+ echo -e '\n\033[90;01m$ chown --recursive openhab:openhab /etc/openhab /var/lib/openhab /var/log/openhab /usr/share/openhab \033[39;49;00m'

$ chown --recursive openhab:openhab /etc/openhab /var/lib/openhab /var/log/openhab /usr/share/openhab
+ chown --recursive openhab:openhab /etc/openhab /var/lib/openhab /var/log/openhab /usr/share/openhab
chown: cannot access '/var/log/openhab': No such file or directory
+ return 1
+ echo 'FAILED (openhab folders)'
FAILED (openhab folders)
+ retval=1
+ cond_redirect chmod --recursive ug+wX /opt /etc/openhab /var/lib/openhab /var/log/openhab /usr/share/openhab
+ [[ -n '' ]]
+ echo -e '\n\033[90;01m$ chmod --recursive ug+wX /opt /etc/openhab /var/lib/openhab /var/log/openhab /usr/share/openhab \033[39;49;00m'

$ chmod --recursive ug+wX /opt /etc/openhab /var/lib/openhab /var/log/openhab /usr/share/openhab
+ chmod --recursive ug+wX /opt /etc/openhab /var/lib/openhab /var/log/openhab /usr/share/openhab
chmod: cannot access '/var/log/openhab': No such file or directory
+ return 1
+ echo 'FAILED (folders)'
FAILED (folders)
+ retval=1
+ [[ -d /var/lib/openhab/.ssh ]]
+ [[ -d /var/lib/openhab/backups ]]
+ cond_redirect chown openhab:openhab /var/lib/openhab/backups
+ [[ -n '' ]]
+ echo -e '\n\033[90;01m$ chown openhab:openhab /var/lib/openhab/backups \033[39;49;00m'

$ chown openhab:openhab /var/lib/openhab/backups
+ chown openhab:openhab /var/lib/openhab/backups
+ return 0
+ cond_redirect chmod g+s /var/lib/openhab/backups
+ [[ -n '' ]]
+ echo -e '\n\033[90;01m$ chmod g+s /var/lib/openhab/backups \033[39;49;00m'

$ chmod g+s /var/lib/openhab/backups
+ chmod g+s /var/lib/openhab/backups
+ return 0
+ cond_redirect fix_permissions /home/openhabian openhabian:openhabian
+ [[ -n '' ]]
+ echo -e '\n\033[90;01m$ fix_permissions /home/openhabian openhabian:openhabian \033[39;49;00m'

$ fix_permissions /home/openhabian openhabian:openhabian
+ fix_permissions /home/openhabian openhabian:openhabian
+ [[ -e /home/openhabian ]]
+ chown openhabian:openhabian /home/openhabian
+ [[ -n '' ]]
+ return 0
+ return 0
+ [[ -f /etc/mosquitto/mosquitto.conf ]]
+ cond_redirect fix_permissions /etc/mosquitto/passwd mosquitto:openhabian 640 750
+ [[ -n '' ]]
+ echo -e '\n\033[90;01m$ fix_permissions /etc/mosquitto/passwd mosquitto:openhabian 640 750 \033[39;49;00m'

$ fix_permissions /etc/mosquitto/passwd mosquitto:openhabian 640 750
+ fix_permissions /etc/mosquitto/passwd mosquitto:openhabian 640 750
+ [[ -e /etc/mosquitto/passwd ]]
+ return 0
+ return 0
+ cond_redirect fix_permissions /var/log/mosquitto mosquitto:openhabian 644 755
+ [[ -n '' ]]
+ echo -e '\n\033[90;01m$ fix_permissions /var/log/mosquitto mosquitto:openhabian 644 755 \033[39;49;00m'

$ fix_permissions /var/log/mosquitto mosquitto:openhabian 644 755
+ fix_permissions /var/log/mosquitto mosquitto:openhabian 644 755
+ [[ -e /var/log/mosquitto ]]
+ return 0
+ return 0
+ [[ -f /etc/ztab ]]
+ cond_redirect fix_permissions /opt/zram/log.bind/mosquitto mosquitto:openhabian 644 755
+ [[ -n '' ]]
+ echo -e '\n\033[90;01m$ fix_permissions /opt/zram/log.bind/mosquitto mosquitto:openhabian 644 755 \033[39;49;00m'

$ fix_permissions /opt/zram/log.bind/mosquitto mosquitto:openhabian 644 755
+ fix_permissions /opt/zram/log.bind/mosquitto mosquitto:openhabian 644 755
+ [[ -e /opt/zram/log.bind/mosquitto ]]
+ return 0
+ return 0
+ cond_redirect setfacl --recursive --remove-all /etc/openhab /var/lib/openhab /var/log/openhab /usr/share/openhab
+ [[ -n '' ]]
+ echo -e '\n\033[90;01m$ setfacl --recursive --remove-all /etc/openhab /var/lib/openhab /var/log/openhab /usr/share/openhab \033[39;49;00m'

$ setfacl --recursive --remove-all /etc/openhab /var/lib/openhab /var/log/openhab /usr/share/openhab
+ setfacl --recursive --remove-all /etc/openhab /var/lib/openhab /var/log/openhab /usr/share/openhab
setfacl: /var/log/openhab: No such file or directory
+ return 0
+ cond_redirect fix_permissions /var/log/unattended-upgrades root:root 644 755
+ [[ -n '' ]]
+ echo -e '\n\033[90;01m$ fix_permissions /var/log/unattended-upgrades root:root 644 755 \033[39;49;00m'

$ fix_permissions /var/log/unattended-upgrades root:root 644 755
+ fix_permissions /var/log/unattended-upgrades root:root 644 755
+ [[ -e /var/log/unattended-upgrades ]]
+ chown root:root /var/log/unattended-upgrades
+ [[ -n 644 ]]
+ [[ -f /var/log/unattended-upgrades ]]
+ [[ -n 755 ]]
+ [[ -d /var/log/unattended-upgrades ]]
+ find /var/log/unattended-upgrades -type d -print0
+ xargs -0 chmod 755
+ return 0
+ return 0
+ cond_redirect fix_permissions /var/log/samba root:root 640 750
+ [[ -n '' ]]
+ echo -e '\n\033[90;01m$ fix_permissions /var/log/samba root:root 640 750 \033[39;49;00m'

$ fix_permissions /var/log/samba root:root 640 750
+ fix_permissions /var/log/samba root:root 640 750
+ [[ -e /var/log/samba ]]
+ return 0
+ return 0
+ cond_redirect fix_permissions /var/log/openhab openhab:openhabian 664 775
+ [[ -n '' ]]
+ echo -e '\n\033[90;01m$ fix_permissions /var/log/openhab openhab:openhabian 664 775 \033[39;49;00m'

$ fix_permissions /var/log/openhab openhab:openhabian 664 775
+ fix_permissions /var/log/openhab openhab:openhabian 664 775
+ [[ -e /var/log/openhab ]]
+ return 0
+ return 0
+ influxdb_is_installed
+ [[ -d /var/lib/influxdb ]]
+ return 0
+ chmod +x /usr/lib/influxdb/scripts/influxd-systemd-start.sh
+ [[ -f /etc/ztab ]]
+ [[ -d /var/log/grafana ]]
+ cond_redirect fix_permissions /opt/zram/log.bind/samba root:root 640 750
+ [[ -n '' ]]
+ echo -e '\n\033[90;01m$ fix_permissions /opt/zram/log.bind/samba root:root 640 750 \033[39;49;00m'

$ fix_permissions /opt/zram/log.bind/samba root:root 640 750
+ fix_permissions /opt/zram/log.bind/samba root:root 640 750
+ [[ -e /opt/zram/log.bind/samba ]]
+ return 0
+ return 0
+ cond_redirect fix_permissions /opt/zram/persistence.bind openhab:openhabian 664 775
+ [[ -n '' ]]
+ echo -e '\n\033[90;01m$ fix_permissions /opt/zram/persistence.bind openhab:openhabian 664 775 \033[39;49;00m'

$ fix_permissions /opt/zram/persistence.bind openhab:openhabian 664 775
+ fix_permissions /opt/zram/persistence.bind openhab:openhabian 664 775
+ [[ -e /opt/zram/persistence.bind ]]
+ chown openhab:openhabian /opt/zram/persistence.bind
+ [[ -n 664 ]]
+ [[ -f /opt/zram/persistence.bind ]]
+ [[ -n 775 ]]
+ [[ -d /opt/zram/persistence.bind ]]
+ find /opt/zram/persistence.bind -type d -print0
+ xargs -0 chmod 775
+ return 0
+ return 0
+ cond_redirect fix_permissions /opt/zram/log.bind/openhab openhab:openhabian 664 775
+ [[ -n '' ]]
+ echo -e '\n\033[90;01m$ fix_permissions /opt/zram/log.bind/openhab openhab:openhabian 664 775 \033[39;49;00m'

$ fix_permissions /opt/zram/log.bind/openhab openhab:openhabian 664 775
+ fix_permissions /opt/zram/log.bind/openhab openhab:openhabian 664 775
+ [[ -e /opt/zram/log.bind/openhab ]]
+ return 0
+ return 0
+ echo OK
OK
+ [[ -d /etc/homegear ]]
+ [[ -d /opt/habapp ]]
+ return 1
+ '[' 1 -ne 0 ']'
+ whiptail --msgbox 'There was an error or interruption during the execution of:\n  "10 | Apply Improvements"\n\nPlease try again. If the error persists, please read /opt/openhabian/docs/openhabian-DEBUG.md or https://github.com/openhab/openhabian/blob/main/docs/openhabian-DEBUG.md how to proceed.' 14 80

this is pretty depressing, i’ve been on openhab for 4 years now but suddenly i’m not able to setup a stable system.

should i open a new thread? i still don’t understand if this is even related to this thread…

[mstormi (Markus Storm)]

@narf27

should i open a new thread? i still don’t understand if this is even related to this thread…

Yes please. I don’t even understand what your issue is. Provide comprehensive status information but omit all history and guessing please. As your log complains about /var/log/openhab missing, show df, zramctl, ls /var/log/openhab.