Openhab Log Viewer (frontail) empty after some time

Zitelli · October 17, 2020, 9:54am

I installed the latest version of the master branch. Now the logs are gone and new files are created. I will have to wait again until they reach a certain size, I guess, to see if the issue still exists.

But as said, this is not new behavior in my openhabian installation. At the beginning logs are recorded properly, after a couple of days (or more often after a few weeks) the log contains only what is shown in the screenshot. This has happened to me with every installation of openhabian (different stable versions) probably since the very first version I had installed. I thought this was caused by an item or any internal error but I was not able to reproduce it. The only thing I can say is while the log file size is small there is no such issue.

denominator · October 17, 2020, 11:51am

It sounds like an SD card about to die.

Is your logging working?

Zitelli · October 17, 2020, 12:10pm

Yes, now it is working again. Every time the log files are deleted/created again, logging works fine. Now I have to wait again for the logging to stop working again. I believe this will happen in a couple of days/weeks.

lloydatron · October 26, 2020, 3:09pm

Card checks out as good + using ZRAM

matmai · December 9, 2020, 8:36pm

After a complete new installation of openHABian v1.6.0 on a new SD card and restoring my backup openHAB 2.5.10 worked without problems. After around three days the same happened to me and frontail does not display more than:
==> /var/log/openhab2/openhab.log <==
==> /var/log/openhab2/events.log <==

@Zitelli: Was the problem fixed in the end, what helped?

Zitelli · December 11, 2020, 9:32am

After the last time it happened I completely cleaned up the logs. The new files haven’t reached yet the size of the previous ones. Right now everything is fine (again). I will report as soon as the issue occurs again.

mdnx · December 21, 2020, 2:43pm

Hej all
my openhabian installation (v. 2.5.10 with openHABian Configuration Tool [stable]v1.6.1-963(1fa8959) ) on a rpi 4 - shows the same behaviour:
After a while of running (2 days maybe) frontail displays solely:

==> /var/log/openhab2/events.log <==
==> /var/log/openhab2/openhab.log <==

This, while openhab is still happily logging all events (vidible in the console log:tail)

A deletion of the logs in /var/log/openhab2/ helped for a while (2 days). However frontail stopped again.
The new logfiles are not big:

-rw-rw-r-- 1 openhab openhab 813986 Dec 20 21:40 events.log
-rw-rw-r-- 1 openhab openhab 970094 Dec 20 21:40 openhab.log

Suggestions? deinstall and install frontail outside openhabian?
Install an alternative to frontail? (I like it though, as it is a nice realtime way to debug code for openhab…)

Zitelli · December 21, 2020, 3:35pm

I don’t think reinstalling makes any difference. I had to reinstall a couple of times for other reasons. This issue appeared sooner or later in every installed version so far.

mdnx · December 21, 2020, 3:37pm

upps, thats disturbing - do you know whether this is allready reported as issue in the openhabian github? i couldnt find any entry

Zitelli · December 21, 2020, 3:48pm

I opened this topic very late and I could not find anything similar in the web. So I thought it had to something with a wrong configuration (items, rules, etc.) on my side only that caused this sudden and not reproducible (for me) issue. I haven’t reported it yet apart from this thread here.

matmai · December 28, 2020, 3:11pm

I do not think that the problem is related to log rotation since it occurs after a few hours on my installation.

Currently I think it is more related to ZRAM because this ist the major difference between my old and current installation. Also there seems to be no space left to log (even for other processes):

df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        59G  8.5G   48G  16% /
devtmpfs        1.8G     0  1.8G   0% /dev
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           1.9G   34M  1.9G   2% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/mmcblk0p1  253M   54M  199M  22% /boot
/dev/zram1      469M  459M     0 100% /opt/zram/zram1
overlay1        469M  459M     0 100% /var/log
/dev/zram2      469M  780K  433M   1% /opt/zram/zram2
overlay2        469M  780K  433M   1% /var/lib/openhab2/persistence
tmpfs           388M     0  388M   0% /run/user/1000

Also the installation of packages throws an error now:
dpkg: could not open log ‘/var/log/dpkg.log’: No space left on device

mdnx · December 29, 2020, 6:48am

Agree with you. I didn’t find a solution either to the problem, that frontail stops. A restart of the Rpi is the only workaround to solve the problem temporarily.

Several leads into the problem - no finite solution:

maybe it is a ZRAM related issue, which might be linked to a too large journal size, as discussed here. A change of parameters to SystemMaxUse=30M, RuntimeMaxUse=30M didn’t help me.
I found, that my openhab is after ca 50min no longer writing into the openhab2 logs in /var/log/openhab2/events.log & /var/log/openhab2/openhab.log . log:tail in the console shows all events though. As nothing is written into the *.log files it seems, that Frontail simply doesn’t get “new lines to display”.

@matmai I had the problem, that apt upgrade wouldn’t work anymore as well. The system feedback was in my case:

dpkg: could not open log ‘/var/log/dpkg.log’: Read-only file system

The reason, was that a lock file was set by the system in the package folder /var/lib/dpkg/ . Removing this file solved this issue

the tip came from here:

Zitelli · December 31, 2020, 10:11am

I can confirm from my side that I have always used ZRAM (enabled in openhabian-config). In my new OH3 installation I have not enabled it yet. Will wait some time to see if it still happens in my installation.
Thanks for the hint!

matmai · December 31, 2020, 11:38am

I disabled ZRAM via openhabian config and logging is fine since then.

Zitelli · December 31, 2020, 11:44am

Thanks for confirming. That is probably not the solution for ZRAM but surely for this issue here

mdnx · January 5, 2021, 9:39pm

Disabled ZRAM in OpenHabian 2.5 installation and the logging into the files runs smoothly since more than 24hrs
Very weird - I wonder what the cause of this problem is and how many are affected

Wolfgang_S · January 5, 2021, 10:18pm

there were some updates in openhab-config with regard to ZRAM ( Add service start dependencies on ZRAM up by mstormi · Pull Request #1367 · openhab/openhabian · GitHub ). Did you try to reinstall ZRAM after getting fresh openhabian-config updates ?

stritti · January 20, 2021, 2:46pm

I tried it today. I removed ZRAM and installed it again using latest version of openHabian on branch 3 ([openHAB3]patchday-20210119-1144(4d8e4c0)), but still does not receive any logs in frontail.

Edit: After complete reboot the logs are available now.

chrisinatl · March 23, 2021, 11:52am

Similar problem here using main version of OpenHAB 3 on 4 GB Pi4 with ZRAM enabled. After some hours, events.log stops being written to.

df -h shows:

Filesystem      Size  Used Avail Use% Mounted on
/dev/root        29G  5.1G   23G  19% /
devtmpfs        323M     0  323M   0% /dev
tmpfs           452M     0  452M   0% /dev/shm
tmpfs           452M  2.6M  450M   1% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           452M     0  452M   0% /sys/fs/cgroup
/dev/mmcblk0p1  253M   48M  205M  19% /boot
/dev/zram1      870M  2.8M  807M   1% /opt/zram/zram1
overlay1        870M  2.8M  807M   1% /var/lib/openhab/persistence
/dev/zram2      575M  282M  251M  53% /opt/zram/zram2
overlay2        575M  282M  251M  53% /var/log
tmpfs            91M     0   91M   0% /run/user/1000

lsblk -a -oNAME,SIZE,FSUSED,FSAVAIL,FSUSE%,TYPE,MOUNTPOINT shows:

NAME         SIZE FSUSED FSAVAIL FSUSE% TYPE MOUNTPOINT
ram0           4M                       disk
ram1           4M                       disk
ram2           4M                       disk
ram3           4M                       disk
ram4           4M                       disk
ram5           4M                       disk
ram6           4M                       disk
ram7           4M                       disk
ram8           4M                       disk
ram9           4M                       disk
ram10          4M                       disk
ram11          4M                       disk
ram12          4M                       disk
ram13          4M                       disk
ram14          4M                       disk
ram15          4M                       disk
loop0                                   loop
loop1                                   loop
loop2                                   loop
loop3                                   loop
loop4                                   loop
loop5                                   loop
loop6                                   loop
loop7                                   loop
sda         29.8G                       disk
├─sda1       256M                       part
└─sda2      29.5G                       part
mmcblk0     29.7G                       disk
├─mmcblk0p1  256M  47.7M  204.4M    19% part /boot
└─mmcblk0p2 29.5G     5G   22.8G    17% part /
zram0        600M                       disk [SWAP]
zram1        900M   2.8M  806.1M     0% disk /opt/zram/zram1
zram2        600M   282M  250.6M    49% disk /opt/zram/zram2

free shows:

              total        used        free      shared  buff/cache   available
Mem:         925612      505720       99604        2300      320288      357800
Swap:       2365432       64512     2300920

sda is a USB SSD used for backups. It appears I am not running out of space anywhere.

Only a reboot will cause logging to return.

Ideas?

Ez2517 · April 21, 2021, 10:37am

Hello,

I have the same issue.

EDIT: also here, the issue was solved by deactivating ZRAM.

Brandy