Is there a way to change the log level for specific bindings? I am running into an issue caused by the Shelly binding and the Systeminfo Binding. Due to extensive logging of these two bindings, the filesystem switches to read-only after some time, resulting in nothing being logged at all. The only solution is to reboot the system periodically.
First, is it really logs from the binding (i.e. logs in openhab.log) or is it events.log? If the former, then the advice given here is good plus, in OH 4 you can navigate to the binding in Settings → Binding → <name of binding> there’s a tiny gear icon in the upper right. Click on that and one of the options is “Add-on log settings” where you can change the level.
If the latter, you’ll need to create a Regex filter for the events.log logger as those logs come from the event bus, not from the binding so changing the logging level of the binding will have no effect. See openHAB - Filtering event logs.
Alternatively, you could change the max size and total number of files to save for both loggers to reduce the max size that the log files can grow. Under <Appenders> you’ll find a pair of RollingFile entries. Between those you’ll find SizeBasedTriggeringPolicy which is set to “16 MB”. You can change that to a lower value. You’ll also find DefaultRolloverStrategy with a max attribute. That attribute controls how many old log files to keep, deleting those that are older than that.
So by default the max space for each log should be 16 MB * 8 = 128 MB. Reducing those numbers will reduce the max space required.
Reducing the logging levels will really only change how often you need to reboot, not address the root running out of space problem.
@Benjy can we reduce number and size of default log setting in OH4 ?
It’s too much for a typical Raspi so the default should be less.
(not 100% sure if there’s an open issue already I thought I once had opened one but don’t find it any more)
Given that openHABian is going to be the most resource constrained in terms of disk space for logs I wonder it if makes sense for openHABian to have their own config. Though dealing with upgrades would be a pain.
I wish we could come up with a way to separate custom logging configs from the default one. I’ve tried but there really isn’t a way to do it reliably using XML includes. That would make it pretty easy though.
It’s a hassle to setup in a scripted manner. Would need to parse-modify xml config after openhab installation. Would need to work with various versions throughout time.
No it’s way easier to use the default and make it small enough so it also works with openHABian on Pi.
I do the opposite and turn on some additional events like StartlevelEvent and sometimes RuleStatusInfoEvent as those often help me debug problems the ItemChangedEvent and ItemCommandEvent alone are not sufficient.
Ansible has an series of XML tasks for that use case (also YAML, INI, JSON, etc.).
But I was thinking simpler than that with just a couple of regex find and replace using sed or awk since all we are needing to do is replace a couple of attributes which have pretty clear markers. But there are more XML aware command line tools available like xmlstarlet which could be used as well.
I wonder if there is a way to do this from the karaf console…doesn’t appear so.exit
sed -i -e 's/<SizeBasedTriggeringPolicy size=".*"\/>/<SizeBasedTriggeringPolicy size="7 MB"\/>/g' /srv/openhab/userdata/etc/log4j2.xml
sed -i -e 's/<DefaultRolloverStrategy max=".*"/>/<DefaultRolloverStrategy max="2"/>/g' /srv/openhab/userdata/etc/log4j2.xml
These elements only occur three times in the file, once for openhab.log, once for events.log, and once for audit.log (which I’m not sure is really being used, mine is always empty. Even if one completely replaces the XML config, these elements will either be present or they won’t. If they are not present there’s nothing to do. If they are present, with the above, it doesn’t really matter where they appear in the file with the commands above. And I checked and these two elements only exist for the Rolling series of appenders so even if a user drastically changes the config file.
Of course there are some edge cases where this won’t work but in those cases it means the end user knows enough to understand how to modify this file themselves and have done so with full knowledge of what they are doing. If it doesn’t work in those cases I’m inclined to say “it’s your own fault.” Though, as mentioned, it’s slight more complicated but xmlstarlet or other similar command line tools that know XML could be installed and we could handle those cases.
My thought there would be that when doing the upgrade from the openahbian-config tool it would be just another call it adds in addition to the apt and npm steps. Or it even could be an optional config menu item and when users have a running out of space problem we tell them to go turn down the logging from openhabian-config. This doesn’t seem to be a problem faced by most users or we would hear about it more often I think.
I’m not so sure. I do agree that the current settings are excessive for a default and definitely support changing them but it doesn’t feel right to base the default on the most limited environment of them all (zram).
I also wonder if the zipping of the old files is working against ZRAM. Because it’s already compressed it won’t take up any less space in RAM through ZRAM’s compression and perhaps ZRAM’s compression is more efficient. But that’s an aside.
The default should be based on how much logging a typical OH install generates per day coupled with how many days worth of logging is required to diagnose and debug typical problems. It should not be based on the default ZRAM config in openHABian (at least not in total).
I don’t know that anyone has given it much thought from this perspective because the default is so large.
Tl;dr: I support changing the default logs size but it should be based on what we think makes sense from a utility perspective, not based on the amount of space ZRAM in openHABian has available.
If we do some analysis (WAGs are OK) and determine that we still need more space than openHABian can support, I think it is up to openHABian to reduce the size even further. I don’t think that will be the case though as 16 M is a lot of logging and based on what I’ve seen it would take a month or more for OH to generate that much for the average user. But that’s just a feeling. It would be better to have something more data based to set the defaults.
I must admit I’ve been suffering from fs full myself because in the config of my production machine old logs ain’t zipped. Whyever that is, the default config will zip so consume much less space.
That mislead me somehow to think we need another round of shrinking but as it would only apply to the current logfiles that’ll probably not have so much effect.
@ltty do your logs get compressed ? look at your log4j2.xml the filename patterns would need to end with “.gz” for that to happen
Probably not typical openhabian user and certainly not a forum/OH heavy weight, but unsolicited feedback that might help .
Situation: Running OH4M3 on a 6+ year old Rpi3b with zram and openhabian and occasionally struggle with memory/logging. Just trying to make it last . First 16MB is an absurd (IMO) amount of logging. From google 1 MB is 500 pages. My current OH log file is 2KB after 17 days. Most OPs with problems maybe post 20-100KB at most. And the frontail is a page or two? Anyway I don’t know if reducing is going to help. What I have tried to do was reduce production.
First I’m firmly in the WARN for events camp. That shuts that log off. If there is a concern or you are looking for something you can use the developer tools-side bar and stream events right from the UI.
I have made liberal use of the Regex filter. Some bindings are overly verbose and repetitive on messages that really do not indicate a problem. I also use the startup trigger policy so get a new log on startup and usually delete the old one unless I’m saving it for something.
Moved all binding except Zwave and the Systeminfo over to my Rpi4 (Use remoteOH to get the info off the Rpi3.)
Since I have never unpacked a .gz log in 6 years, I dug into the log rotate routines to save just the current log and one uncompressed backup on all the logs (syslog, auth, etc,)
My most recent effort was reduce mapdb to just the items that were not in rrd4j. It had somehow grown to 27MB, but is now 33KB after 10 days.
My goal is to restart OH with the monthly (ish) milestone releases. Hopefully with the latest changes I’ll make it. Last restart was June 6.