Logging suddenly stops working

RichardWegier · September 3, 2020, 4:30pm

Hello

Since a few days we face issues with the log files. The logging suddenly stops, often within the log entry the logger was just writing. For example we have a log file (rwga.log) where we write some debug information for our rules (but we also have it within openhab.log and event.log). Today the last logentries in this file are:


2020-09-03 11:46:58.210 [DEBUG] [.eclipse.smarthome.model.script.rwga] - Secomat OFF
2020-09-03 11:47:03.456 [DEBUG] [.eclipse.smarthome.model.script.rwga] - Secomat ON
2020-09-03 11:47:03.573 [DEBUG] [.eclipse.smarthome.model.scri

Here’s the content of our org.ops4j.pax.logging.cfg:

# Common pattern layout for appenders
#log4j2.pattern = %d{ISO8601} | %-5p | %-16t | %-32c{1} | %X{bundle.id} - %X{bundle.name} - %X{bundle.version} | %m%n

# Root logger
log4j2.rootLogger.level = WARN
log4j2.rootLogger.appenderRefs = out, osgi
log4j2.rootLogger.appenderRef.out.ref = LOGFILE
log4j2.rootLogger.appenderRef.osgi.ref = OSGI

# Karaf Shell logger
log4j2.logger.shell.name = org.apache.karaf.shell.support
log4j2.logger.shell.level = OFF
log4j2.logger.shell.appenderRefs = stdout
log4j2.logger.shell.appenderRef.stdout.ref = STDOUT

# Security audit logger
log4j2.logger.audit.name = org.apache.karaf.jaas.modules.audit
log4j2.logger.audit.level = INFO
log4j2.logger.audit.additivity = false
log4j2.logger.audit.appenderRefs = audit
log4j2.logger.audit.appenderRef.audit.ref = AUDIT

# openHAB specific logger configuration

log4j2.logger.openhab.name = org.openhab
log4j2.logger.openhab.level = INFO

log4j2.logger.smarthome.name = org.eclipse.smarthome
log4j2.logger.smarthome.level = INFO

log4j2.logger.smarthomeItemStateEvent.name = smarthome.event.ItemStateEvent
log4j2.logger.smarthomeItemStateEvent.level = ERROR
log4j2.logger.smarthomeItemAddedEvent.name = smarthome.event.ItemAddedEvent
log4j2.logger.smarthomeItemAddedEvent.level = ERROR
log4j2.logger.smarthomeItemRemovedEvent.name = smarthome.event.ItemRemovedEvent
log4j2.logger.smarthomeItemRemovedEvent.level = ERROR
log4j2.logger.smarthomeThingStatusInfoEvent.name = smarthome.event.ThingStatusInfoEvent
log4j2.logger.smarthomeThingStatusInfoEvent.level = ERROR
log4j2.logger.smarthomeThingAddedEvent.name = smarthome.event.ThingAddedEvent
log4j2.logger.smarthomeThingAddedEvent.level = ERROR
log4j2.logger.smarthomeThingRemovedEvent.name = smarthome.event.ThingRemovedEvent
log4j2.logger.smarthomeThingRemovedEvent.level = ERROR
log4j2.logger.smarthomeInboxUpdatedEvent.name = smarthome.event.InboxUpdatedEvent
log4j2.logger.smarthomeInboxUpdatedEvent.level = ERROR

log4j2.logger.events.name = smarthome.event
log4j2.logger.events.level = INFO
log4j2.logger.events.additivity = false
log4j2.logger.events.appenderRefs = event
log4j2.logger.events.appenderRef.event.ref = EVENT
log4j2.logger.events.appenderRef.osgi.ref = OSGI

log4j2.logger.jupnp.name = org.jupnp
log4j2.logger.jupnp.level = ERROR

log4j2.logger.jmdns.name = javax.jmdns
log4j2.logger.jmdns.level = ERROR

log4j2.logger.paperui.name = org.openhab.ui.paper
log4j2.logger.paperui.level = WARN
log4j2.logger.paperuiint.name = org.openhab.ui.paper.internal
log4j2.logger.paperuiint.level = INFO

# This suppresses all Maven download issues from the log when doing feature installations
# as we are logging errors ourselves in a nicer way anyhow.
log4j2.logger.paxurl.name = org.ops4j.pax.url.mvn.internal.AetherBasedResolver
log4j2.logger.paxurl.level = ERROR

# Filters known issues of pax-web (issue link to be added here).
# Can be removed once the issues are resolved in an upcoming version.
log4j2.logger.paxweb.name = org.ops4j.pax.web.pax-web-runtime
log4j2.logger.paxweb.level = OFF

# Filters known issues of lsp4j, see
# https://github.com/eclipse/smarthome/issues/4639
# https://github.com/eclipse/smarthome/issues/4629
# https://github.com/eclipse/smarthome/issues/4643
# Can be removed once the issues are resolved in an upcoming version.
log4j2.logger.lsp4j.name = org.eclipse.lsp4j
log4j2.logger.lsp4j.level = OFF

# Filters known issues of KarServiceImpl, see
# https://github.com/openhab/openhab-distro/issues/519#issuecomment-351944506
# Can be removed once the issues are resolved in an upcoming version.
log4j2.logger.karservice.name = org.apache.karaf.kar.internal.KarServiceImpl
log4j2.logger.karservice.level = ERROR

# Filters warnings about unavailable ciphers when JCE is not installed, see
# https://github.com/openhab/openhab-distro/issues/999
log4j2.logger.sshutils.name = org.apache.karaf.shell.ssh.SshUtils
log4j2.logger.sshutils.level = ERROR

# Filters known issues of javax.mail, see
# https://github.com/openhab/openhab2-addons/issues/5530
log4j2.logger.javaxmail.name = javax.mail
log4j2.logger.javaxmail.level = ERROR

# Appenders configuration

# Console appender not used by default (see log4j2.rootLogger.appenderRefs)
log4j2.appender.console.type = Console
log4j2.appender.console.name = STDOUT
log4j2.appender.console.layout.type = PatternLayout
log4j2.appender.console.layout.pattern = %d{HH:mm:ss.SSS} [%-5.5p] [%-36.36c] - %m%n

# Rolling file appender
log4j2.appender.out.type = RollingRandomAccessFile
log4j2.appender.out.name = LOGFILE
log4j2.appender.out.fileName = ${openhab.logdir}/openhab.log
log4j2.appender.out.filePattern = ${openhab.logdir}/openhab.log.%i
log4j2.appender.out.immediateFlush = true
log4j2.appender.out.append = true
log4j2.appender.out.layout.type = PatternLayout
log4j2.appender.out.layout.pattern = %d{yyyy-MM-dd HH:mm:ss.SSS} [%-5.5p] [%-36.36c] - %m%n
log4j2.appender.out.policies.type = Policies
log4j2.appender.out.policies.size.type = SizeBasedTriggeringPolicy
log4j2.appender.out.policies.size.size = 16MB

# Event log appender
log4j2.appender.event.type = RollingRandomAccessFile
log4j2.appender.event.name = EVENT
log4j2.appender.event.fileName = ${openhab.logdir}/events.log
log4j2.appender.event.filePattern = ${openhab.logdir}/events.log.%i
log4j2.appender.event.immediateFlush = true
log4j2.appender.event.append = true
log4j2.appender.event.layout.type = PatternLayout
log4j2.appender.event.layout.pattern = %d{yyyy-MM-dd HH:mm:ss.SSS} [%-26.26c] - %m%n
log4j2.appender.event.policies.type = Policies
log4j2.appender.event.policies.size.type = SizeBasedTriggeringPolicy
log4j2.appender.event.policies.size.size = 16MB

# Audit file appender
log4j2.appender.audit.type = RollingRandomAccessFile
log4j2.appender.audit.name = AUDIT
log4j2.appender.audit.fileName = ${openhab.logdir}/audit.log
log4j2.appender.audit.filePattern = ${openhab.logdir}/audit.log.%i
log4j2.appender.audit.append = true
log4j2.appender.audit.layout.type = PatternLayout
log4j2.appender.audit.layout.pattern = %d{yyyy-MM-dd HH:mm:ss.SSS} [%-5.5p] [%-36.36c] - %m%n
log4j2.appender.audit.policies.type = Policies
log4j2.appender.audit.policies.size.type = SizeBasedTriggeringPolicy
log4j2.appender.audit.policies.size.size = 8MB

# OSGi appender
log4j2.appender.osgi.type = PaxOsgi
log4j2.appender.osgi.name = OSGI
log4j2.appender.osgi.filter = *

### Custom Loggers ###
# RWGA
log4j2.logger.rwga.name = org.eclipse.smarthome.model.script
log4j2.logger.rwga.level = DEBUG
log4j2.logger.rwga.additivity = false
log4j2.logger.rwga.appenderRefs = rwga
log4j2.logger.rwga.appenderRef.Demo.ref = rwga

### Custom Appenders ###
# RWGA
log4j2.appender.rwga.name = rwga
log4j2.appender.rwga.type = RollingRandomAccessFile
log4j2.appender.rwga.fileName = ${openhab.logdir}/rwga.log
log4j2.appender.rwga.filePattern = ${openhab.logdir}/rwga.log.%i
log4j2.appender.rwga.immediateFlush = true
log4j2.appender.rwga.append = true
log4j2.appender.rwga.layout.type = PatternLayout
log4j2.appender.rwga.layout.pattern = %d{yyyy-MM-dd HH:mm:ss.SSS} [%-5.5p] [%-36.36c] - %m%n
log4j2.appender.rwga.policies.type = Policies
log4j2.appender.rwga.policies.size.type = SizeBasedTriggeringPolicy
log4j2.appender.rwga.policies.size.size = 10MB
log4j2.appender.rwga.strategy.type = DefaultRolloverStrategy
log4j2.appender.rwga.strategy.max = 10

Our setup

Hardware: Raspberry PI 4 4GB
OS: Linux openhab 5.4.51-v7l+, openHABian v1.6
Java Runtime Environment: openjdk 1.8.0_265
openHAB version: 2.5.7
Storage: SanDisk 240GB SSD

Short backstory
Last week we switched our openHABian installation on a Raspberry Pi 4 6GB from SD-Card to SSD. We run our house automation over openhab (e.g. car charging based on the available solar power, many Shellys to optimize solar power usage). Therefore we did not want to run the risk of suddenly loosing our automation because of a failing SD card. In the same time when switching to the SSD we also installed the very latest versions of each component (openHAB openHABian, Grafana, InfluxDB).

We already searched the forum but unfortunately did not see any solution for this. This is my first post here, so if anything is missing please let me know Your help is very appreciated!

Best wishes
Richie

RichardWegier · September 4, 2020, 7:28pm

Hello

I have found some information which might lead to the solution

Today I tried to run the openhabian function for fixing the access rights. The result was:

chown: changing ownership of '/var/log/openhab2/events.log.1': No such file or directory
chown: changing ownership of '/var/log/openhab2/openhab.log': No space left on device
chown: changing ownership of '/var/log/openhab2/events.log': No space left on device
chown: changing ownership of '/var/log/openhab2/rwga.log': No space left on device
chown: changing ownership of '/var/log/openhab2/audit.log': No space left on device
chown: changing ownership of '/var/log/openhab2': No space left on device

I was very surprised by this information as we have a lot of disk space left on our SSD. Therefore I ran df -h and got the following information back:

Filesystem      Size  Used Avail Use% Mounted on
/dev/root       230G  7.0G  213G   4% /
devtmpfs        1.8G     0  1.8G   0% /dev
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           1.9G  1.5M  1.9G   1% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/sda1       253M   55M  198M  22% /boot
/dev/zram1      469M  459M     0 100% /opt/zram/zram1
overlay1        469M  459M     0 100% /var/log
/dev/zram2      469M  832K  433M   1% /opt/zram/zram2
overlay2        469M  832K  433M   1% /var/lib/openhab2/persistence
tmpfs           388M     0  388M   0% /run/user/1000

/var/log and dev/zram1 seem to be full or/and have any quota applied.

Unfortunately I’m not very fit with Linux commands therefore I would like to ask, if someone could could give me some tips on how to give more disks pace to the directories in question?

Best wishes
Richard