Openhab Log Viewer (frontail) empty after some time

mdnx · December 21, 2020, 2:43pm

Hej all
my openhabian installation (v. 2.5.10 with openHABian Configuration Tool [stable]v1.6.1-963(1fa8959) ) on a rpi 4 - shows the same behaviour:
After a while of running (2 days maybe) frontail displays solely:

==> /var/log/openhab2/events.log <==
==> /var/log/openhab2/openhab.log <==

This, while openhab is still happily logging all events (vidible in the console log:tail)

A deletion of the logs in /var/log/openhab2/ helped for a while (2 days). However frontail stopped again.
The new logfiles are not big:

-rw-rw-r-- 1 openhab openhab 813986 Dec 20 21:40 events.log
-rw-rw-r-- 1 openhab openhab 970094 Dec 20 21:40 openhab.log

Suggestions? deinstall and install frontail outside openhabian?
Install an alternative to frontail? (I like it though, as it is a nice realtime way to debug code for openhab…)

Zitelli · December 21, 2020, 3:35pm

I don’t think reinstalling makes any difference. I had to reinstall a couple of times for other reasons. This issue appeared sooner or later in every installed version so far.

mdnx · December 21, 2020, 3:37pm

upps, thats disturbing - do you know whether this is allready reported as issue in the openhabian github? i couldnt find any entry

Zitelli · December 21, 2020, 3:48pm

I opened this topic very late and I could not find anything similar in the web. So I thought it had to something with a wrong configuration (items, rules, etc.) on my side only that caused this sudden and not reproducible (for me) issue. I haven’t reported it yet apart from this thread here.

matmai · December 28, 2020, 3:11pm

I do not think that the problem is related to log rotation since it occurs after a few hours on my installation.

Currently I think it is more related to ZRAM because this ist the major difference between my old and current installation. Also there seems to be no space left to log (even for other processes):

df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        59G  8.5G   48G  16% /
devtmpfs        1.8G     0  1.8G   0% /dev
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           1.9G   34M  1.9G   2% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/mmcblk0p1  253M   54M  199M  22% /boot
/dev/zram1      469M  459M     0 100% /opt/zram/zram1
overlay1        469M  459M     0 100% /var/log
/dev/zram2      469M  780K  433M   1% /opt/zram/zram2
overlay2        469M  780K  433M   1% /var/lib/openhab2/persistence
tmpfs           388M     0  388M   0% /run/user/1000

Also the installation of packages throws an error now:
dpkg: could not open log ‘/var/log/dpkg.log’: No space left on device

mdnx · December 29, 2020, 6:48am

Agree with you. I didn’t find a solution either to the problem, that frontail stops. A restart of the Rpi is the only workaround to solve the problem temporarily.

Several leads into the problem - no finite solution:

maybe it is a ZRAM related issue, which might be linked to a too large journal size, as discussed here. A change of parameters to SystemMaxUse=30M, RuntimeMaxUse=30M didn’t help me.
I found, that my openhab is after ca 50min no longer writing into the openhab2 logs in /var/log/openhab2/events.log & /var/log/openhab2/openhab.log . log:tail in the console shows all events though. As nothing is written into the *.log files it seems, that Frontail simply doesn’t get “new lines to display”.

@matmai I had the problem, that apt upgrade wouldn’t work anymore as well. The system feedback was in my case:

dpkg: could not open log ‘/var/log/dpkg.log’: Read-only file system

The reason, was that a lock file was set by the system in the package folder /var/lib/dpkg/ . Removing this file solved this issue

the tip came from here:
https://www.kubuntuforums.net/showthread.php/75067-How-to-make-Mongo-DB-work-on-18-04/page2

Zitelli · December 31, 2020, 10:11am

I can confirm from my side that I have always used ZRAM (enabled in openhabian-config). In my new OH3 installation I have not enabled it yet. Will wait some time to see if it still happens in my installation.
Thanks for the hint!

matmai · December 31, 2020, 11:38am

I disabled ZRAM via openhabian config and logging is fine since then.

Zitelli · December 31, 2020, 11:44am

Thanks for confirming. That is probably not the solution for ZRAM but surely for this issue here

mdnx · January 5, 2021, 9:39pm

Disabled ZRAM in OpenHabian 2.5 installation and the logging into the files runs smoothly since more than 24hrs
Very weird - I wonder what the cause of this problem is and how many are affected

Wolfgang_S · January 5, 2021, 10:18pm

there were some updates in openhab-config with regard to ZRAM ( Add service start dependencies on ZRAM up by mstormi · Pull Request #1367 · openhab/openhabian · GitHub ). Did you try to reinstall ZRAM after getting fresh openhabian-config updates ?

stritti · January 20, 2021, 2:46pm

I tried it today. I removed ZRAM and installed it again using latest version of openHabian on branch 3 ([openHAB3]patchday-20210119-1144(4d8e4c0)), but still does not receive any logs in frontail.

Edit: After complete reboot the logs are available now.

chrisinatl · March 23, 2021, 11:52am

Similar problem here using main version of OpenHAB 3 on 4 GB Pi4 with ZRAM enabled. After some hours, events.log stops being written to.

df -h shows:

Filesystem      Size  Used Avail Use% Mounted on
/dev/root        29G  5.1G   23G  19% /
devtmpfs        323M     0  323M   0% /dev
tmpfs           452M     0  452M   0% /dev/shm
tmpfs           452M  2.6M  450M   1% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           452M     0  452M   0% /sys/fs/cgroup
/dev/mmcblk0p1  253M   48M  205M  19% /boot
/dev/zram1      870M  2.8M  807M   1% /opt/zram/zram1
overlay1        870M  2.8M  807M   1% /var/lib/openhab/persistence
/dev/zram2      575M  282M  251M  53% /opt/zram/zram2
overlay2        575M  282M  251M  53% /var/log
tmpfs            91M     0   91M   0% /run/user/1000

lsblk -a -oNAME,SIZE,FSUSED,FSAVAIL,FSUSE%,TYPE,MOUNTPOINT shows:

NAME         SIZE FSUSED FSAVAIL FSUSE% TYPE MOUNTPOINT
ram0           4M                       disk
ram1           4M                       disk
ram2           4M                       disk
ram3           4M                       disk
ram4           4M                       disk
ram5           4M                       disk
ram6           4M                       disk
ram7           4M                       disk
ram8           4M                       disk
ram9           4M                       disk
ram10          4M                       disk
ram11          4M                       disk
ram12          4M                       disk
ram13          4M                       disk
ram14          4M                       disk
ram15          4M                       disk
loop0                                   loop
loop1                                   loop
loop2                                   loop
loop3                                   loop
loop4                                   loop
loop5                                   loop
loop6                                   loop
loop7                                   loop
sda         29.8G                       disk
├─sda1       256M                       part
└─sda2      29.5G                       part
mmcblk0     29.7G                       disk
├─mmcblk0p1  256M  47.7M  204.4M    19% part /boot
└─mmcblk0p2 29.5G     5G   22.8G    17% part /
zram0        600M                       disk [SWAP]
zram1        900M   2.8M  806.1M     0% disk /opt/zram/zram1
zram2        600M   282M  250.6M    49% disk /opt/zram/zram2

free shows:

              total        used        free      shared  buff/cache   available
Mem:         925612      505720       99604        2300      320288      357800
Swap:       2365432       64512     2300920

sda is a USB SSD used for backups. It appears I am not running out of space anywhere.

Only a reboot will cause logging to return.

Ideas?

Ez2517 · April 21, 2021, 10:37am

Hello,

I have the same issue.

EDIT: also here, the issue was solved by deactivating ZRAM.

Brandy

moonraker · May 23, 2021, 5:44am

Same issue here. Had to disable zram.

Seems to be a bug (OH3.1 M4). RPi 4.

jnherm · July 25, 2021, 1:02pm

Disabling ZRAM may shorten the life of your SD card. ZRAM was used to prevent/lessen the writing on the SD card.

HenrikF · September 3, 2021, 10:38am

I made a fresh install of openhabian and OH3.1 on rpi4 over the last few weeks.

I also experience this problem - and i’m not keen on just disabling Zram.

Does anyone know of other solutions ?

ddwdiot · September 11, 2021, 10:21am

I’d like to know too. I have same symptoms on my previous installation and recently on a fresh installation running OH 3.1.0 on a RPi 4. Seems like it’s not a good idea to disable zram and rebooting openhab every 3 days isn’t ideal.

HenrikF · September 17, 2021, 5:50pm

@mstormi mstormi
Would you have any input on this ?
I guess it seem like a systematic issue related to Openhabian and Zram.

I’m a linux/raspberry novice - but fairly ok in IT stuff generally.
Anything I can do to help debug this problem

mstormi · September 18, 2021, 7:31am

No. And to ping people is unwanted here.