Issue with growing logs in openhabian

  • Platform information:
    • Hardware: Raspberry4
    • OS: openhabian latest release
    • Java Runtime Environment: openhabian
    • openHAB version: latest 4.1.1.
  • Issue of the topic: suddenly I have an endless growing syslog and daemon log that kills my openhab instance
  • Please post configurations (if applicable):
    I am not aware of any changes done in the past but suddenly I have an endless growing syslog and daemon log that kills my openhab instance. currently they are both at 7.1 GB size.
    I haven’t changed anything in the openhabian setup, instead for updating to the newest release lately which might be a coincidence? I’m not very familiar in logrotation and what I found is below.

I wouldn’t want to play with my openhabian configuration as it works for me since years without adaptation.

I only adapted the xml4j to use a remote syslog appender to write to my remote Graylog instance.

cat /etc/logrotate.d/rsyslog
/var/log/syslog
/var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
/var/log/messages
{
rotate 4
weekly
missingok
notifempty
compress
delaycompress
sharedscripts
postrotate
/usr/lib/rsyslog/rsyslog-rotate
endscript
}

Thank you for any help on how to investigate further without mixing up my openhabian instance.

Well you should inspect syslog messages to get a hint what’s going on but it’s usually no openHAB or openHABian issue if syslogs suddenly start exploding.
Taming them isn’t either. Log rotation is daily but if you have too much a day to fit there’s something severely wrong.

Quickest and best solution I suggest if you don’t find anything is to reinstall to another SD.

What is logging to syslog so much? Since you are sending syslog to Graylog surely you’ve checked?

Do you see evidence of log rotation happening at all? There a cron job in /etc/cron.daily that does the rotation.

you need to figure out what is filling up your logs and fix that so it’s no longer filling up your syslog.

I had no time for investigation and just deleted them which didn’t solve the issue.
So my daemon.log and and syslog. are the problematic ones with sizes > 4GB… I found out that the daemon log has about 740 entries per minute (!) of this type…

May 17 19:02:31 openhabian influxd-systemd-start.sh[666]: ts=2024-05-17T17:02:31.385662Z lvl=info msg=“Executing query” log_id=0p8QJc3l000 service=query query=“SELECT value::field, item::tag FROM openhab.autogen.MA_sensor7_humid ORDER BY time DESC LIMIT 1”

and my syslog has the exact same entries. Any idea from where this could come?

It’s right there in the log.

May 17 19:02:31 openhabian influxd-systemd-start.sh…

Those come from I fluxDB. It looks like it’s logging every query ("msg=“Executing query” ") that it receives to syslog. I’m not sure why there would be so many queries, maybe you have a bunch of charts open in UIs? I also don’t know why InfluxDB logs these but suspect if you look in their docs you can figure out how to configure InfluxDB to log at the WARN level instead of INFO or otherwise exclude queries from the logs.

With so many queries going on though I would expect your machine to be quite heavily loaded.

Thank you @rlkoshak - This was a quite tricky one. I switched all loggers to ERROR which didn’t help. But in the /etc/influxdb/influxdb.conf I had to switch off the http request logging with
log-enabled=false
to finally succeed… I don’t know what is behind it, as I have never changed anything in the influx setup on my openhabian - something must have changed with some release…