[SOLVED] Log file ring buffer

Hello good folk,
My OpenHAB deployment is coming on and great performance, albeit the log files are now quite large approaching GB size - which is ok because I have been debugging and adding Info logging. I do not want to lose the Info logs but perhaps it would be advantageous if I could implement inside OpenHAB configuration - somehow - using the systemd/journal log system.
I would like all logs to be managed systemctl/journalctl.
My openhab2.service file looks like this:

[Unit]
Description=openHAB 2 - empowering the smart home
Documentation=https://www.openhab.org/docs/
Documentation=https://community.openhab.org
Wants=network-online.target
Requires=EnergenieMQTT.service
After=network-online.target

[Service]
Environment=OPENHAB_HOME=/usr/share/openhab2
Environment=OPENHAB_CONF=/etc/openhab2
Environment=OPENHAB_RUNTIME=/usr/share/openhab2/runtime
Environment=OPENHAB_USERDATA=/var/lib/openhab2
Environment=OPENHAB_STARTMODE=daemon
EnvironmentFile=-/etc/default/openhab2

User=openhab
Group=openhab

WorkingDirectory=/usr/share/openhab2
ExecStart=/usr/share/openhab2/runtime/bin/karaf ${OPENHAB_STARTMODE}
ExecStop=/usr/share/openhab2/runtime/bin/karaf stop


# Uncomment to have openhab control logging. Karaf may also need turing??
#Environment=OPENHAB_LOGDIR=/var/log/openhab2
#StandardOuput=/var/log/openhab2/openhab.log
#StandardError=inherit

SuccessExitStatus=0 143
RestartSec=5
Restart=on-failure
TimeoutStopSec=120

LimitNOFILE=102642

[Install]
WantedBy=multi-user.target

I have commented out the OPENHAB_LOGDIR and instructed systemd to log to /var/log/openhab2/openhab.log

However, karaf is also logging now to /var/lib/openhab2/logs

I’d prefer all logs to goto standard output / standard error that way I can use journalctl and fully manage as I want to.

Seen this https://www.openhab.org/docs/administration/logging.html but it is not clear to me how I would achieve what I want. Maybe I need to use karaf to change settings but not sure what to do.

Help!!?

I don’t know if you can have all logs managed by systemctl. You can have all logs go to the syslog (see https://stackoverflow.com/questions/17947611/log4j2-syslog-appender-and-patternlayout) but it’s going to be a bit of work on your part and you’ll have to learn log4j2 and according to that thread there are bugs that need to be worked around. I know of no one who has attempted to do this so you will largely be on your own.

Log4j2 Sample Config shows how to move all the logging from a specific binding to it’s own log file and change the logging level of just that one binding. It should be an example to get you started. But to expand beyond that example and what’s in the docs you will have to go to the Log4j docs.

But if you want full control over the logs you will need to muck around with the log4j2 configuration anyway so you may as well just do it all there. You have full control over what get’s logged and at what level to what files and how those files are managed and rotated there.

Have you done any changes to your log configuration? By default, the logger is configured to rotate the files at 17 mb and only keeps the last 10 files to two different log files (events.log and openhab.log). So either you changed something or there is something wrong because you should never see more than 340 MB of log files from openHAB at any given time, assuming the default configuration.

1 Like

Mmmm
interesting about rotation, that’s essentially what I’m after and systemd/systemctl was one way i had envisaged. I’ve got a lot of services running, converted many applications on my setup to run and log that way.
Right, i shall take a more in-depth look at karaf and “have a play”

I’m with you ritch, log4j2 will be a pain, much like log4c was for me with other apps.

thanks for the education around OH logging, i clearly have “done / broke something”

For reference, here is my near stock log4j config. Pay attention to the appenders. That is where the log rotation config resides.

# Common pattern layout for appenders
#log4j2.pattern = %d{ISO8601} | %-5p | %-16t | %-32c{1} | %X{bundle.id} - %X{bundle.name} - %X{bundle.version} | %m%n

# Root logger
log4j2.rootLogger.level = WARN
log4j2.rootLogger.appenderRefs = out, osgi
log4j2.rootLogger.appenderRef.out.ref = LOGFILE
log4j2.rootLogger.appenderRef.osgi.ref = OSGI

# Karaf Shell logger
log4j2.logger.shell.name = org.apache.karaf.shell.support
log4j2.logger.shell.level = OFF
log4j2.logger.shell.appenderRefs = stdout
log4j2.logger.shell.appenderRef.stdout.ref = STDOUT

# Security audit logger
log4j2.logger.audit.name = org.apache.karaf.jaas.modules.audit
log4j2.logger.audit.level = INFO
log4j2.logger.audit.additivity = false
log4j2.logger.audit.appenderRefs = audit
log4j2.logger.audit.appenderRef.audit.ref = AUDIT

# openHAB specific logger configuration

log4j2.logger.openhab.name = org.openhab
log4j2.logger.openhab.level = INFO

log4j2.logger.smarthome.name = org.eclipse.smarthome
log4j2.logger.smarthome.level = INFO

log4j2.logger.smarthomeItemStateEvent.name = smarthome.event.ItemStateEvent
log4j2.logger.smarthomeItemStateEvent.level = ERROR
log4j2.logger.smarthomeItemAddedEvent.name = smarthome.event.ItemAddedEvent
log4j2.logger.smarthomeItemAddedEvent.level = ERROR
log4j2.logger.smarthomeItemRemovedEvent.name = smarthome.event.ItemRemovedEvent
log4j2.logger.smarthomeItemRemovedEvent.level = ERROR
log4j2.logger.smarthomeThingStatusInfoEvent.name = smarthome.event.ThingStatusInfoEvent
log4j2.logger.smarthomeThingStatusInfoEvent.level = ERROR
log4j2.logger.smarthomeThingAddedEvent.name = smarthome.event.ThingAddedEvent
log4j2.logger.smarthomeThingAddedEvent.level = ERROR
log4j2.logger.smarthomeThingRemovedEvent.name = smarthome.event.ThingRemovedEvent
log4j2.logger.smarthomeThingRemovedEvent.level = ERROR
log4j2.logger.smarthomeInboxUpdatedEvent.name = smarthome.event.InboxUpdatedEvent
log4j2.logger.smarthomeInboxUpdatedEvent.level = ERROR

log4j2.logger.events.name = smarthome.event
log4j2.logger.events.level = INFO
log4j2.logger.events.additivity = false
log4j2.logger.events.appenderRefs = event
log4j2.logger.events.appenderRef.event.ref = EVENT
log4j2.logger.events.appenderRef.osgi.ref = OSGI

log4j2.logger.jupnp.name = org.jupnp
log4j2.logger.jupnp.level = ERROR

log4j2.logger.jmdns.name = javax.jmdns
log4j2.logger.jmdns.level = ERROR

log4j2.logger.paperui.name = org.openhab.ui.paper
log4j2.logger.paperui.level = WARN
log4j2.logger.paperuiint.name = org.openhab.ui.paper.internal
log4j2.logger.paperuiint.level = INFO

# This suppresses all Maven download issues from the log when doing feature installations
# as we are logging errors ourselves in a nicer way anyhow.
log4j2.logger.paxurl.name = org.ops4j.pax.url.mvn.internal.AetherBasedResolver
log4j2.logger.paxurl.level = ERROR

# Filters known issues of pax-web (issue link to be added here).
# Can be removed once the issues are resolved in an upcoming version.
log4j2.logger.paxweb.name = org.ops4j.pax.web.pax-web-runtime
log4j2.logger.paxweb.level = OFF

# Filters known issues of lsp4j, see
# https://github.com/eclipse/smarthome/issues/4639
# https://github.com/eclipse/smarthome/issues/4629
# https://github.com/eclipse/smarthome/issues/4643
# Can be removed once the issues are resolved in an upcoming version.
log4j2.logger.lsp4j.name = org.eclipse.lsp4j
log4j2.logger.lsp4j.level = OFF

# Filters known issues of KarServiceImpl, see
# https://github.com/openhab/openhab-distro/issues/519#issuecomment-351944506
# Can be removed once the issues are resolved in an upcoming version.
log4j2.logger.karservice.name = org.apache.karaf.kar.internal.KarServiceImpl
log4j2.logger.karservice.level = ERROR

# Filters known issues of javax.mail, see
# https://github.com/openhab/openhab2-addons/issues/5530
log4j2.logger.javaxmail.name = javax.mail
log4j2.logger.javaxmail.level = ERROR


# Appenders configuration

# Console appender not used by default (see log4j2.rootLogger.appenderRefs)
log4j2.appender.console.type = Console
log4j2.appender.console.name = STDOUT
log4j2.appender.console.layout.type = PatternLayout
log4j2.appender.console.layout.pattern = %d{HH:mm:ss.SSS} [%-5.5p] [%-36.36c] - %m%n

# Rolling file appender
log4j2.appender.out.type = RollingRandomAccessFile
log4j2.appender.out.name = LOGFILE
log4j2.appender.out.fileName = ${openhab.logdir}/openhab.log
log4j2.appender.out.filePattern = ${openhab.logdir}/openhab.log.%i
log4j2.appender.out.immediateFlush = true
log4j2.appender.out.append = true
log4j2.appender.out.layout.type = PatternLayout
log4j2.appender.out.layout.pattern = %d{yyyy-MM-dd HH:mm:ss.SSS} [%-5.5p] [%-36.36c] - %m%n
log4j2.appender.out.policies.type = Policies
log4j2.appender.out.policies.size.type = SizeBasedTriggeringPolicy
log4j2.appender.out.policies.size.size = 16MB

# Event log appender
log4j2.appender.event.type = RollingRandomAccessFile
log4j2.appender.event.name = EVENT
log4j2.appender.event.fileName = ${openhab.logdir}/events.log
log4j2.appender.event.filePattern = ${openhab.logdir}/events.log.%i
log4j2.appender.event.immediateFlush = true
log4j2.appender.event.append = true
log4j2.appender.event.layout.type = PatternLayout
log4j2.appender.event.layout.pattern = %d{yyyy-MM-dd HH:mm:ss.SSS} [%-26.26c] - %m%n
log4j2.appender.event.policies.type = Policies
log4j2.appender.event.policies.size.type = SizeBasedTriggeringPolicy
log4j2.appender.event.policies.size.size = 16MB

# Audit file appender
log4j2.appender.audit.type = RollingRandomAccessFile
log4j2.appender.audit.name = AUDIT
log4j2.appender.audit.fileName = ${openhab.logdir}/audit.log
log4j2.appender.audit.filePattern = ${openhab.logdir}/audit.log.%i
log4j2.appender.audit.append = true
log4j2.appender.audit.layout.type = PatternLayout
log4j2.appender.audit.layout.pattern = %d{yyyy-MM-dd HH:mm:ss.SSS} [%-5.5p] [%-36.36c] - %m%n
log4j2.appender.audit.policies.type = Policies
log4j2.appender.audit.policies.size.type = SizeBasedTriggeringPolicy
log4j2.appender.audit.policies.size.size = 8MB

# OSGi appender
log4j2.appender.osgi.type = PaxOsgi
log4j2.appender.osgi.name = OSGI
log4j2.appender.osgi.filter = *


#### CUSTOM CONFIG ###
## JSR223 ##
log4j2.logger.rules.level = INFO
log4j2.logger.rules.name = jsr223
log4j2.logger.rules.additivity = false
log4j2.logger.rules.appenderRefs = rules
log4j2.logger.rules.appenderRef.rules.ref = RULES

# appender
log4j2.appender.rules.type = RollingRandomAccessFile
log4j2.appender.rules.name = RULES
log4j2.appender.rules.fileName = ${openhab.logdir}/rules.log
log4j2.appender.rules.filePattern = ${openhab.logdir}/rules.log.%i
log4j2.appender.rules.immediateFlush = true
log4j2.appender.rules.append = true
log4j2.appender.rules.layout.type = PatternLayout
log4j2.appender.rules.layout.pattern = %d{yyyy-MM-dd HH:mm:ss.SSS} [%-26.26c] - %m%n
log4j2.appender.rules.policies.type = Policies
log4j2.appender.rules.policies.size.type = SizeBasedTriggeringPolicy
log4j2.appender.rules.policies.size.size = 16MB

# SSH
log4j2.logger.org_apache_sshd.level = ERROR
log4j2.logger.org_apache_sshd.name = org.apache.sshd

# Rule Status
log4j2.logger.smarthomeRuleStatusInfoEvent.name = smarthome.event.RuleStatusInfoEvent
log4j2.logger.smarthomeRuleStatusInfoEvent.level = ERROR

# NUT
log4j2.logger.nut.name = org.openhab.binding.networkupstools
log4j2.logger.nut.level = INFO

## ZWave
#log4j2.logger.zwave.name = org.openhab.binding.zwave
#log4j2.logger.zwave.level = DEBUG
#log4j2.logger.zwave.additivity = false
#log4j2.logger.zwave.appenderRefs = ZWAVE
#log4j2.logger.zwave.appenderRef.zwave.ref = ZWAVE
#
#log4j2.appender.zwave.name = ZWAVE
#log4j2.appender.zwave.type = RollingRandomAccessFile
#log4j2.appender.zwave.fileName = ${openhab.logdir}/zwave.log
#log4j2.appender.zwave.filePattern = ${openhab.logdir}/zwave.log.%i
#log4j2.appender.zwave.immediateFlush = true
#log4j2.appender.zwave.append = true
#log4j2.appender.zwave.layout.type = PatternLayout
#log4j2.appender.zwave.layout.pattern = %d{dd-MMM-yyyy HH:mm:ss.SSS} [%-5.5p] [%-50.50c] - %m%n
#log4j2.appender.zwave.policies.type = Policies
#log4j2.appender.zwave.policies.size.type = SizeBasedTriggeringPolicy
#log4j2.appender.zwave.policies.size.size = 10MB
#log4j2.appender.zwave.strategy.type = DefaultRolloverStrategy
#log4j2.appender.zwave.strategy.max = 10

## Zigbee
#log4j2.logger.zigbee.name = org.openhab.binding.zigbee
#log4j2.logger.zigbee.level = DEBUG
#log4j2.logger.zigbee.additivity = false
#log4j2.logger.zigbee.appenderRefs = ZIGBEE
#log4j2.logger.zigbee.appenderRef.zigbee.ref = ZIGBEE
#
#log4j2.appender.zigbee.name = ZIGBEE
#log4j2.appender.zigbee.type = RollingRandomAccessFile
#log4j2.appender.zigbee.fileName = ${openhab.logdir}/zigbee.log
#log4j2.appender.zigbee.filePattern = ${openhab.logdir}/zigbee.log.%i
#log4j2.appender.zigbee.immediateFlush = true
#log4j2.appender.zigbee.append = true
#log4j2.appender.zigbee.layout.type = PatternLayout
#log4j2.appender.zigbee.layout.pattern = %d{dd-MMM-yyyy HH:mm:ss.SSS} [%-5.5p] [%-50.50c] - %m%n
#log4j2.appender.zigbee.policies.type = Policies
#log4j2.appender.zigbee.policies.size.type = SizeBasedTriggeringPolicy
#log4j2.appender.zigbee.policies.size.size = 10MB
#log4j2.appender.zigbee.strategy.type = DefaultRolloverStrategy
#log4j2.appender.zigbee.strategy.max = 10

## MQTT
#log4j2.logger.mqtt.name = org.eclipse.smarthome.binding.mqtt
#log4j2.logger.mqtt.level = TRACE
#log4j2.logger.mqtt.additivity = false
#log4j2.logger.mqtt.appenderRefs = MQTT
#log4j2.logger.mqtt.appenderRef.mqtt.ref = MQTT

#log4j2.appender.mqtt.name = MQTT
#log4j2.appender.mqtt.type = RollingRandomAccessFile
#log4j2.appender.mqtt.fileName = ${openhab.logdir}/mqtt.log
#log4j2.appender.mqtt.filePattern = ${openhab.logdir}/mqtt.log.%i
#log4j2.appender.mqtt.immediateFlush = true
#log4j2.appender.mqtt.append = true
#log4j2.appender.mqtt.layout.type = PatternLayout
#log4j2.appender.mqtt.layout.pattern = %d{dd-MMM-yyyy HH:mm:ss.SSS} [%-5.5p] [%-50.50c] - %m%n
#log4j2.appender.mqtt.policies.type = Policies
#log4j2.appender.mqtt.policies.size.type = SizeBasedTriggeringPolicy
#log4j2.appender.mqtt.policies.size.size = 10MB
#log4j2.appender.mqtt.strategy.type = DefaultRolloverStrategy
#log4j2.appender.mqtt.strategy.max = 10
1 Like

Excellent, great start for me👍