Is there a way to change the log level for specific bindings? I am running into an issue caused by the Shelly binding and the Systeminfo Binding. Due to extensive logging of these two bindings, the filesystem switches to read-only after some time, resulting in nothing being logged at all. The only solution is to reboot the system periodically.
First, is it really logs from the binding (i.e. logs in openhab.log) or is it events.log? If the former, then the advice given here is good plus, in OH 4 you can navigate to the binding in Settings ā Binding ā <name of binding> thereās a tiny gear icon in the upper right. Click on that and one of the options is āAdd-on log settingsā where you can change the level.
If the latter, youāll need to create a Regex filter for the events.log logger as those logs come from the event bus, not from the binding so changing the logging level of the binding will have no effect. See openHAB - Filtering event logs.
Alternatively, you could change the max size and total number of files to save for both loggers to reduce the max size that the log files can grow. Under <Appenders> youāll find a pair of RollingFile entries. Between those youāll find SizeBasedTriggeringPolicy which is set to ā16 MBā. You can change that to a lower value. Youāll also find DefaultRolloverStrategy with a max attribute. That attribute controls how many old log files to keep, deleting those that are older than that.
So by default the max space for each log should be 16 MB * 8 = 128 MB. Reducing those numbers will reduce the max space required.
Reducing the logging levels will really only change how often you need to reboot, not address the root running out of space problem.
@Benjy can we reduce number and size of default log setting in OH4 ?
Itās too much for a typical Raspi so the default should be less.
(not 100% sure if thereās an open issue already I thought I once had opened one but donāt find it any more)
Given that openHABian is going to be the most resource constrained in terms of disk space for logs I wonder it if makes sense for openHABian to have their own config. Though dealing with upgrades would be a pain.
I wish we could come up with a way to separate custom logging configs from the default one. Iāve tried but there really isnāt a way to do it reliably using XML includes. That would make it pretty easy though.
Itās a hassle to setup in a scripted manner. Would need to parse-modify xml config after openhab installation. Would need to work with various versions throughout time.
No itās way easier to use the default and make it small enough so it also works with openHABian on Pi.
This is something thatās controlled by the distro, I think. Not specifically through the Linux packages. Iād agree with Rich that it makes sense for a custom logging configuration.
Personally I have always changed openhab.event from INFO to WARN after I have a stable setup, it reduces my own setupās filesize over time considerably.
Scripting to change a fixed-format xml isnāt too hard, Ansible or bash or whatever.
Scripting to change a non-fixed full-scale XML format potentially modified file is, thatās our problem here.
As you hinted yourself itās about dealing with upgrades.
Plus with any type of manual intervention users might or might not have made as part of their operations.
@benjy as Rich correctly noted it isnāt about logging config, itās about logfilesize config.
Less logging postpones the problem to happen, but it does not solve it.
Itās clearly the more clever idea to just start any new OH install with a mini-filesize config.
I do the opposite and turn on some additional events like StartlevelEvent and sometimes RuleStatusInfoEvent as those often help me debug problems the ItemChangedEvent and ItemCommandEvent alone are not sufficient.
Ansible has an series of XML tasks for that use case (also YAML, INI, JSON, etc.).
But I was thinking simpler than that with just a couple of regex find and replace using sed or awk since all we are needing to do is replace a couple of attributes which have pretty clear markers. But there are more XML aware command line tools available like xmlstarlet which could be used as well.
I wonder if there is a way to do this from the karaf consoleā¦doesnāt appear so.exit
sed -i -e 's/<SizeBasedTriggeringPolicy size=".*"\/>/<SizeBasedTriggeringPolicy size="7 MB"\/>/g' /srv/openhab/userdata/etc/log4j2.xml
sed -i -e 's/<DefaultRolloverStrategy max=".*"/>/<DefaultRolloverStrategy max="2"/>/g' /srv/openhab/userdata/etc/log4j2.xml
These elements only occur three times in the file, once for openhab.log, once for events.log, and once for audit.log (which Iām not sure is really being used, mine is always empty. Even if one completely replaces the XML config, these elements will either be present or they wonāt. If they are not present thereās nothing to do. If they are present, with the above, it doesnāt really matter where they appear in the file with the commands above. And I checked and these two elements only exist for the Rolling series of appenders so even if a user drastically changes the config file.
Of course there are some edge cases where this wonāt work but in those cases it means the end user knows enough to understand how to modify this file themselves and have done so with full knowledge of what they are doing. If it doesnāt work in those cases Iām inclined to say āitās your own fault.ā Though, as mentioned, itās slight more complicated but xmlstarlet or other similar command line tools that know XML could be installed and we could handle those cases.
My thought there would be that when doing the upgrade from the openahbian-config tool it would be just another call it adds in addition to the apt and npm steps. Or it even could be an optional config menu item and when users have a running out of space problem we tell them to go turn down the logging from openhabian-config. This doesnāt seem to be a problem faced by most users or we would hear about it more often I think.
Iām not so sure. I do agree that the current settings are excessive for a default and definitely support changing them but it doesnāt feel right to base the default on the most limited environment of them all (zram).
I also wonder if the zipping of the old files is working against ZRAM. Because itās already compressed it wonāt take up any less space in RAM through ZRAMās compression and perhaps ZRAMās compression is more efficient. But thatās an aside.
The default should be based on how much logging a typical OH install generates per day coupled with how many days worth of logging is required to diagnose and debug typical problems. It should not be based on the default ZRAM config in openHABian (at least not in total).
I donāt know that anyone has given it much thought from this perspective because the default is so large.
Tl;dr: I support changing the default logs size but it should be based on what we think makes sense from a utility perspective, not based on the amount of space ZRAM in openHABian has available.
If we do some analysis (WAGs are OK) and determine that we still need more space than openHABian can support, I think it is up to openHABian to reduce the size even further. I donāt think that will be the case though as 16 M is a lot of logging and based on what Iāve seen it would take a month or more for OH to generate that much for the average user. But thatās just a feeling. It would be better to have something more data based to set the defaults.
I must admit Iāve been suffering from fs full myself because in the config of my production machine old logs aināt zipped. Whyever that is, the default config will zip so consume much less space.
That mislead me somehow to think we need another round of shrinking but as it would only apply to the current logfiles thatāll probably not have so much effect.
@ltty do your logs get compressed ? look at your log4j2.xml the filename patterns would need to end with ā.gzā for that to happen
Probably not typical openhabian user and certainly not a forum/OH heavy weight, but unsolicited feedback that might help .
Situation: Running OH4M3 on a 6+ year old Rpi3b with zram and openhabian and occasionally struggle with memory/logging. Just trying to make it last . First 16MB is an absurd (IMO) amount of logging. From google 1 MB is 500 pages. My current OH log file is 2KB after 17 days. Most OPs with problems maybe post 20-100KB at most. And the frontail is a page or two? Anyway I donāt know if reducing is going to help. What I have tried to do was reduce production.
First Iām firmly in the WARN for events camp. That shuts that log off. If there is a concern or you are looking for something you can use the developer tools-side bar and stream events right from the UI.
I have made liberal use of the Regex filter. Some bindings are overly verbose and repetitive on messages that really do not indicate a problem. I also use the startup trigger policy so get a new log on startup and usually delete the old one unless Iām saving it for something.
Moved all binding except Zwave and the Systeminfo over to my Rpi4 (Use remoteOH to get the info off the Rpi3.)
Since I have never unpacked a .gz log in 6 years, I dug into the log rotate routines to save just the current log and one uncompressed backup on all the logs (syslog, auth, etc,)
My most recent effort was reduce mapdb to just the items that were not in rrd4j. It had somehow grown to 27MB, but is now 33KB after 10 days.
My goal is to restart OH with the monthly (ish) milestone releases. Hopefully with the latest changes Iāll make it. Last restart was June 6.