I do not understand that question.
Issues with SD card, filesystem, openHABian and OS settings etc are totally unrelated to the OH version you are running.
You advised to reinstall OH, I assumed that newest version of Openhabian that comes with OH3 installs some updated version of ZRAM that might have fixed the problem I see of ZRAM switching to Read Only.
Freshly after reboot, I deleted all the old rotated logs and now wait to see when it comes back.
My OH setup is very clean. Just one binding MQTT, persistance to Influx on the separate VM on other hardware.
Well, what ever is making it read only is the reason why the logs stop being written. You can’t write to a read only file system either. It might let you continue writing to a file that you opened before the file system went to read only, but as soon as OH decides to rotate the log files it’ll fail.
So this moves the investigation some. Something is causing the file system to be remounted as read only. Often that happens when the file system fills up but not always. Other errors can cause this to happen too. You might need to do some research and looking through the syslogs and zram logs (if it has them).
Unfortunately, per my discussion above, you can’t use the timestamp of the error from OH to narrow your search in the other logs. There could be quite some lag there.
Not always and this is a ZRAM file system which isn’t on the SD card, it’s in RAM. While a worn out SD card could possibly cause a problem like this, it’s not the only thing that fits the symptoms.
I don’t think this is true at all. For the most part, neither the OS nor the SD card itself knows that it’s failing. Weird stuff just starts happening (e.g. a deleted file comes back on reboot, strange characters in weird files, random crashes, changes to files not saving, etc.).
It’s right that in 99% of cases, too much data in /var/log (note with ZRAM there’s more limits to apply than just the sum of file sizes). zramctl --output-all will tell.
Just to update you on the issue, I deleted all rotated logs from the /var/log/openhab2 and the problem went away, ex. the system did not crash after 12-18h as previously and is now running for 2 days.
I do not need as much as 7 of the old log files. 1-2 will do. Where can I limit the number of log rotates that OH2 does?
Maciej
This same thing happened to me on OH3.3 (openHABian with ZRAM). My events.log and openhab.log files stopped updating yesterday afternoon, and I didn’t notice until today. In the Karaf console, I could see that log entries were still being generated. Restarting Frontail and rebooting didn’t fix the problem.
I’ve had a ton of log entries over the past week while trying to figure out an issue with the IPP binding, and then I generated more while trying to get the LG ThinQ binding to work yesterday.
After reading this thread, I thought it likely that I overloaded the ZRAM log (if I’m interpreting this discussion correctly). I went back into the console and reset the log with log:clear, then rebooted again. That seemed to do the trick, and my log is now working.
1 Like
Following your solution post I tried to limit number of the events log logfiles to just 2.
But adding a line like:
log4j2.appender.event.strategy.max = 2
To the section that I belive is setup for the event log.
# Event log appender
log4j2.appender.event.type = RollingRandomAccessFile
log4j2.appender.event.name = EVENT
log4j2.appender.event.fileName = ${openhab.logdir}/events.log
log4j2.appender.event.filePattern = ${openhab.logdir}/events.log.%i
log4j2.appender.event.immediateFlush = true
log4j2.appender.event.append = true
log4j2.appender.event.layout.type = PatternLayout
log4j2.appender.event.layout.pattern = %d{yyyy-MM-dd HH:mm:ss.SSS} [%-26.26c] - %m%n
log4j2.appender.event.policies.type = Policies
log4j2.appender.event.policies.size.type = SizeBasedTriggeringPolicy
log4j2.appender.event.policies.size.size = 16MB
Just stops the logging for both events.log and openhab.log immediately.
Your suggestion was to add that into appender.out section but that seen only related to openhab.log file and also resulsts in logging to stop.
So that is not working as a limitation to the number of log files created.
If I wipe logs dir just after reboot, it will hold about 4-5 days before ZRAM overload.
I am having the same issue and have had it for years over multiple versions of OH. I am currently on 3.4.1. When my system quits logging, I remove all the log files and reboot. The logs are good for about a week and then lock up again.
==> /var/log/openhab/openhab.log <==
==> /var/log/openhab/events.log <==
==> /var/log/openhab/openhab.log <==
==> /var/log/openhab/events.log <==
==> /var/log/openhab/openhab.log <==
==> /var/log/openhab/events.log <==
==> /var/log/openhab/openhab.log <==
==> /var/log/openhab/events.log <==
==> /var/log/openhab/openhab.log <==
==> /var/log/openhab/events.log <==
==> /var/log/openhab/openhab.log <==
###############################################################################
############### openhab8 ####################################################
###############################################################################
Ip = 192.168.1.8
Release = Raspbian GNU/Linux 10 (buster)
Kernel = Linux 5.10.103-v7l+
Platform = none
Uptime = 5 day(s). 0:16:26
CPU Usage = 3.94% avg over 4 cpu(s) (4 core(s) x 1 socket(s))
CPU Load = 1m: 0.49, 5m: 0.33, 15m: 0.30
Memory = Free: 2.34GB (62%), Used: 1.45GB (38%), Total: 3.79GB
Swap = Free: 2.19GB (100%), Used: 0.00GB (0%), Total: 2.19GB
Root = Free: 5.02GB (36%), Used: 8.68GB (64%), Total: 14.32GB
Updates = 3 apt updates available.
Sessions = 1 session(s)
Processes = 135 running processes of 32768 maximum processes
###############################################################################
openhabian@openhab8:/var/log/openhab $ ls
audit.log events.log.2.gz events.log.5.gz openhab.log openhab.log.3.gz openhab.log.6.gz
events.log events.log.3.gz events.log.6.gz openhab.log.1.gz openhab.log.4.gz openhab.log.7.gz
events.log.1.gz events.log.4.gz events.log.7.gz openhab.log.2.gz openhab.log.5.gz Readme.txt
openhabian@openhab8:~ $ zramctl --output-all
NAME DISKSIZE DATA COMPR ALGORITHM STREAMS ZERO-PAGES TOTAL MEM-LIMIT MEM-USED MIGRATED MOUNTPOINT
/dev/zram1 150M 142.3M 20.4M lzo-rle 4 140 71.3M 150M 71.3M 7B /opt/zram/zram1
/dev/zram0 200M 4K 86B lzo-rle 4 0 4K 200M 4K 0B [SWAP]
I went into the console and ran log:clear and rebooted. It is logging again, but all the files are still there in the log directory. I will see what happens this week.
I didn’t manage to solve the issue, i moved away from the RPI sbc and took the system with the nvme drive and installed the OH from apt instead of openhabian. That was a PITA as then all the litte tools your used to has to be done manually, like smb shares and frontail service.
So if you manage to fix it let us know.
The issue is not known to happen on latest openHABian.
But in the end it’s pretty simple, simply reduce the number and/or size of logfiles in /var/lib/openhab/etc/log4j2.xml.
This file is part of the openHAB distribution and as such may get replaced on upgrades, so eventually you have to repeat this after an upgrade.
As I wrote 3 posts before, touching log config breaks logging in my case.
Tried croning log delete, does not help either eve if I only have 2 files it will die after the week
Will check latest version of the openhabian.
Maciej
It wouldn’t if you don’t make a mistake in editing.
I set the events.log and openhab.log to 4. I’ll see what happens. Thanks
That was my syntax and it brakes logging. is that wrong?
As it broke, obviously so.
As the name says already it needs to be XML format .
Hm, according to first post we talk about OH 2.5.X.
As I am on OH 3.4.X I cannot check in my system but according to openhab-distro/distributions/openhab/src/main/resources/userdata/etc/org.ops4j.pax.logging.cfg at 2.5.x · openhab/openhab-distro · GitHub the configuration file format for logging that was used for OH 2.5 is a text file.
log4j2.appender.out.strategy.max = 2
log4j2.appender.event.strategy.max = 2
log4j2.appender.audit.strategy.max = 2
then should be correct.
1 Like
Show the full file and since we are many months from your previous attempt please state/restate which version of OH we are referring to. Between OH 2 and OH 3 there was a move from the properties file to an XML file and the syntax is radically different (and the actual config is in a new file).
Still on OH2.5 on that particular install.
I add one single line as detailed in the post #14