Logs not getting written any more

Hi, I’m going nuts here… any pointers appreciated… not sure what I have done…

Upgraded Openhabian and Openhab to the latest 3.0.2 and hit a load of problems… but let’s park that for the moment… While debugging that this morning my log files have stopped being written to… nothing in them after 10.43…

Looking in syslog shows me a bunch of errors related to file permissions (events.log permission denied along with log4j2 errors not being able to create olugins or invoke factory method). I reset the file permissions via the main setup console and checked - they were all openhab. Re-running the openHAB server and the files are all owned by root again.

What am I missing? I can then get back to finding out what’s killed my setup…


how do you do this ?

Just by rebooting the Raspberry Pi


What happens if you just would restart the openhab service ?

Still nothing in the logs since this morning… Any way to reset logging?

  • did you do any modifications to logging configuration?
  • when was it last working ? before the upgrade ? which version did you use before the upgrade ?

I’m not sure I’d know how to modify the logging config @Wolfgang_S :slight_smile:

Logs were working post upgrade… after upgrade I noticed that z-wave lights were simply not working right so I was trying to solve that. I turned on maximum logging. I was then looking into some Amazonechocontrol warnings and the fact I was getting and found reference to restdocs, paperUI and ruleengine in my addons.config, which I tried removing, based on a note by @Bruce_Osborne. I have reverted all that back though…

I installed the Log Viewer from the config tool (frontail) - that just shows me the logs up to 10.43 yesterday morning.

z-wave is still not working but felt getting the logs back was a good first step.

If I go into the developer tools, and turn on streaming in the debug tab I can see events when turning lights on and off.

/var/log/syslog is still complaining about permissions… Might be unrelated but logrotate is complaining about "error skipping “/var/log/uinattended-upgrades/unattended-upgrades.log” because parent directory has insecure permissions (It’s world writable or writable by group which is not “root” Set “su” directive in config file to tell logrotate which user/group should be used for rotation. Same for a couple of other files in that directory.

What permissions should the log files & folder have?

root@cubietruck:/var/log# ls -ld /var/log
drwxrwxr-x 20 root syslog 4096 May 16 07:36 /var/log
root@cubietruck:/var/log# ls -ld openhab/ openhab/*
drwxr-xr-x 2 openhab openhab     4096 May  6 08:31 openhab/
-rw-r--r-- 1 openhab openhab        0 Apr 29 06:15 openhab/audit.log
-rw-r--r-- 1 openhab openhab 11401501 May 16 14:58 openhab/events.log
-rw-r--r-- 1 openhab openhab  1511781 May 16 14:00 openhab/openhab.log

So, I’m…

openhabian@openhab:/etc $ ls -ld /var/log
drwxr-xr-x 10 root root 4096 May 16 00:01 /var/log


openhabian@openhab:/var/log $ ls -ld openhab/ openhab/*
drwxrwxr-x 2 openhab    openhabian   4096 May 15 12:30 openhab/
-rw-rw-r-- 1 openhab    openhab         0 Dec 29 09:32 openhab/audit.log
-rw-rw-r-- 1 root       root        52153 May 15 10:43 openhab/events.log
-rw-rw-r-- 1 openhab    openhab     41038 May 14 15:52 openhab/events.log.1
-rw-rw-r-- 1 openhab    openhab      7005 May 14 16:10 openhab/events.log.2
-rw-rw-r-- 1 openhab    openhab    103703 May 14 16:26 openhab/events.log.3
-rw-rw-r-- 1 openhab    openhab    259491 May 14 23:14 openhab/events.log.4
-rw-rw-r-- 1 openhab    openhab     96010 May 15 10:07 openhab/events.log.5
-rw-rw-r-- 1 openhab    openhab    121878 May 15 10:11 openhab/events.log.6
-rw-rw-r-- 1 root       root        47862 May 15 10:13 openhab/events.log.7
-rw-rw-r-- 1 openhab    openhabian      0 May 15 09:36 openhab/HABApp_events.log
-rw-rw-r-- 1 openhab    openhabian   3656 May 15 09:44 openhab/HABApp.log
-rw-rw-r-- 1 root       root       115932 May 15 10:43 openhab/openhab.log
-rw-rw-r-- 1 openhab    openhab     32823 May 14 15:52 openhab/openhab.log.1
-rw-rw-r-- 1 openhab    openhab     32126 May 14 15:54 openhab/openhab.log.2
-rw-rw-r-- 1 openhab    openhab     39515 May 14 16:26 openhab/openhab.log.3
-rw-rw-r-- 1 openhab    openhab     47523 May 14 23:12 openhab/openhab.log.4
-rw-rw-r-- 1 openhab    openhab     33509 May 15 10:07 openhab/openhab.log.5
-rw-rw-r-- 1 openhab    openhab     25871 May 15 10:11 openhab/openhab.log.6
-rw-rw-r-- 1 root       root        23949 May 15 10:13 openhab/openhab.log.7
-rwxrwxr-x 1 openhab    openhab         0 Apr 21 20:49 openhab/Readme.txt
-rw-r----- 1 openhabian openhab    635692 May 15 12:30 openhab/syslog

Just re-run the reset permissions tool from openhabian-config - that’s reset everything to:

-rw-rw-r-- 1 openhab openhab     52153 May 15 10:43 openhab/events.log
-rw-rw-r-- 1 openhab openhab    115932 May 15 10:43 openhab/openhab.log

But still nothing being written since 10:43 yesterday…

Arggh… what have I managed to do here? Appreciate the help so far @Wolfgang_S

What to try next?

What is returned by:
ps -ef | grep openhab

What is the content of these files:

ps -ef | grep openhab

openhabian@openhab:~ $ ps -ef | grep openhab
avahi      321     1  0 May15 ?        00:01:35 avahi-daemon: running [openhab.local]
openhab    567     1  2 May15 ?        00:35:04 /usr/bin/java -XX:-UsePerfData -Dopenhab.home=/usr/share/openhab -Dopenhab.conf=/etc/openhab -Dopenhab.runtime=/usr/share/openhab/runtime -Dopenhab.userdata=/var/lib/openhab -Dopenhab.logdir=/var/log/openhab -Dfelix.cm.dir=/var/lib/openhab/config -Djava.library.path=/var/lib/openhab/tmp/lib -Djetty.host= -Djetty.http.compliance=RFC2616 -Dnashorn.args=--no-deprecation-warning -Dorg.ops4j.pax.web.listening.addresses= -Dorg.osgi.service.http.port=8080 -Dorg.osgi.service.http.port.secure=8443 -Djava.awt.headless=true -Xms192m -Xmx320m --add-reads=java.xml=java.logging --add-exports=java.base/org.apache.karaf.specs.locator=java.xml,ALL-UNNAMED --patch-module java.base=/usr/share/openhab/runtime/lib/endorsed/org.apache.karaf.specs.locator-4.2.7.jar --patch-module java.xml=/usr/share/openhab/runtime/lib/endorsed/org.apache.karaf.specs.java.xml-4.2.7.jar --add-opens java.base/java.security=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.naming/javax.naming.spi=ALL-UNNAMED --add-opens java.rmi/sun.rmi.transport.tcp=ALL-UNNAMED --add-opens java.base/java.io=ALL-UNNAMED --add-opens java.base/java.lang.reflect=ALL-UNNAMED --add-opens java.base/java.text=ALL-UNNAMED --add-opens java.desktop/java.awt.font=ALL-UNNAMED --add-exports=java.base/sun.net.www.protocol.http=ALL-UNNAMED --add-exports=java.base/sun.net.www.protocol.https=ALL-UNNAMED --add-exports=java.base/sun.net.www.protocol.jar=ALL-UNNAMED --add-exports=jdk.xml.dom/org.w3c.dom.html=ALL-UNNAMED --add-exports=jdk.naming.rmi/com.sun.jndi.url.rmi=ALL-UNNAMED -Dkaraf.instances=/var/lib/openhab/tmp/instances -Dkaraf.home=/usr/share/openhab/runtime -Dkaraf.base=/var/lib/openhab -Dkaraf.data=/var/lib/openhab -Dkaraf.etc=/var/lib/openhab/etc -Dkaraf.log=/var/log/openhab -Dkaraf.restart.jvm.supported=true -Djava.io.tmpdir=/var/lib/openhab/tmp -Djava.util.logging.config.file=/var/lib/openhab/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -classpath /usr/share/openhab/runtime/lib/boot/org.apache.karaf.diagnostic.boot-4.2.7.jar:/usr/share/openhab/runtime/lib/boot/org.apache.karaf.jaas.boot-4.2.7.jar:/usr/share/openhab/runtime/lib/boot/org.apache.karaf.main-4.2.7.jar:/usr/share/openhab/runtime/lib/boot/org.apache.karaf.specs.activator-4.2.7.jar:/usr/share/openhab/runtime/lib/boot/osgi.core-6.0.0.jar:/usr/share/openhab/runtime/lib/jdk9plus/istack-commons-runtime-3.0.8.jar:/usr/share/openhab/runtime/lib/jdk9plus/jakarta.xml.bind-api-2.3.2.jar:/usr/share/openhab/runtime/lib/jdk9plus/javax.activation-1.2.0.jar:/usr/share/openhab/runtime/lib/jdk9plus/javax.annotation-api-1.3.1.jar:/usr/share/openhab/runtime/lib/jdk9plus/jaxb-runtime-2.3.2.jar:/usr/share/openhab/runtime/lib/jdk9plus/txw2-2.3.2.jar org.apache.karaf.main.Main
frontail   568     1  0 May15 ?        00:00:02 node /usr/lib/node_modules/frontail/bin/frontail --disable-usage-stats --ui-highlight --ui-highlight-preset /usr/lib/node_modules/frontail/preset/openhab_AEM.json --theme openhab_AEM --lines 2000 --number 200 /var/log/openhab/openhab.log /var/log/openhab/events.log
frontail   695   568  0 May15 ?        00:00:00 tail -n 200 -F /var/log/openhab/openhab.log /var/log/openhab/events.log
root     29228   627  0 16:12 ?        00:00:00 sshd: openhabian [priv]
openhab+ 29237     1  0 16:12 ?        00:00:00 /lib/systemd/systemd --user
openhab+ 29238 29237  0 16:12 ?        00:00:00 (sd-pam)
openhab+ 29251 29228  0 16:12 ?        00:00:00 sshd: openhabian@pts/0
openhab+ 29252 29251  0 16:12 pts/0    00:00:00 -bash
openhab+ 29657 29252  0 16:13 pts/0    00:00:00 ps -ef
openhab+ 29658 29252  0 16:13 pts/0    00:00:00 grep --color=auto openhab


<?xml version="1.0" encoding="UTF-8" standalone="no"?><Configuration monitorInterval="10">

                <!-- Console appender not used by default (see Root logger AppenderRefs) -->
                <Console name="STDOUT">
                        <PatternLayout pattern="%d{HH:mm:ss.SSS} [%-5.5p] [%-36.36c] - %m%n"/>

                <!-- Rolling file appender -->
                <RollingFile fileName="${sys:openhab.logdir}/openhab.log" filePattern="${sys:openhab.logdir}/openhab.log.%i" name="LOGFILE">
                        <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} [%-5.5p] [%-36.36c] - %m%n"/>
                                <SizeBasedTriggeringPolicy size="16 MB"/>

                <!-- Event log appender -->
                <RollingRandomAccessFile fileName="${sys:openhab.logdir}/events.log" filePattern="${sys:openhab.logdir}/events.log.%i" name="EVENT">
                        <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} [%-5.5p] [%-36.36c] - %m%n"/>
                                <SizeBasedTriggeringPolicy size="16 MB"/>

                <!-- Audit file appender -->
                <RollingRandomAccessFile fileName="${sys:openhab.logdir}/audit.log" filePattern="${sys:openhab.logdir}/audit.log.%i" name="AUDIT">
                        <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} [%-5.5p] [%-36.36c] - %m%n"/>
                                <SizeBasedTriggeringPolicy size="8 MB"/>

                <!-- OSGi appender -->
                <PaxOsgi filter="*" name="OSGI"/>

                <!-- Root logger configuration -->
                <Root level="WARN">
                        <AppenderRef ref="LOGFILE"/>
                        <AppenderRef ref="OSGI"/>

                <!-- Karaf Shell logger -->
                <Logger level="OFF" name="org.apache.karaf.shell.support">
                        <AppenderRef ref="STDOUT"/>

                <!-- Security audit logger -->
                <Logger additivity="false" level="INFO" name="org.apache.karaf.jaas.modules.audit">
                        <AppenderRef ref="AUDIT"/>

                <!-- openHAB specific logger configuration -->

                <Logger level="INFO" name="org.openhab"/>

                <Logger level="ERROR" name="openhab.event.ItemStateEvent"/>
                <Logger level="ERROR" name="openhab.event.ItemAddedEvent"/>
                <Logger level="ERROR" name="openhab.event.ItemRemovedEvent"/>
                <Logger level="ERROR" name="openhab.event.ItemChannelLinkAddedEvent"/>
                <Logger level="ERROR" name="openhab.event.ItemChannelLinkRemovedEvent"/>
                <Logger level="ERROR" name="openhab.event.ThingStatusInfoEvent"/>
                <Logger level="ERROR" name="openhab.event.ThingAddedEvent"/>
                <Logger level="ERROR" name="openhab.event.ThingUpdatedEvent"/>
                <Logger level="ERROR" name="openhab.event.ThingRemovedEvent"/>
                <Logger level="ERROR" name="openhab.event.InboxUpdatedEvent"/>
                <Logger level="ERROR" name="openhab.event.RuleStatusInfoEvent"/>
                <Logger level="ERROR" name="openhab.event.RuleAddedEvent"/>
                <Logger level="ERROR" name="openhab.event.RuleRemovedEvent"/>
                <Logger level="ERROR" name="openhab.event.StartlevelEvent"/>
                <Logger level="ERROR" name="openhab.event.AddonEvent"/>

                <Logger additivity="false" level="INFO" name="openhab.event">
                        <AppenderRef ref="EVENT"/>
                        <AppenderRef ref="OSGI"/>

                <Logger level="ERROR" name="javax.jmdns"/>
                <Logger level="ERROR" name="org.jupnp"/>

                <!-- This suppresses all Maven download issues from the log when doing feature installations -->
                <!-- as we are logging errors ourselves in a nicer way anyhow. -->
                <Logger level="ERROR" name="org.ops4j.pax.url.mvn.internal.AetherBasedResolver"/>

                <!-- Filters known issues of pax-web (issue link to be added here). -->
                <!-- Can be removed once the issues are resolved in an upcoming version. -->
                <Logger level="OFF" name="org.ops4j.pax.web.pax-web-runtime"/>

                <!-- Filters known issues of lsp4j, see -->
                <!-- https://github.com/eclipse/smarthome/issues/4639 -->
                <!-- https://github.com/eclipse/smarthome/issues/4629 -->
                <!-- https://github.com/eclipse/smarthome/issues/4643 -->
                <!-- Can be removed once the issues are resolved in an upcoming version. -->
                <Logger level="OFF" name="org.eclipse.lsp4j"/>

                <!-- Filters warnings for events that could not be delivered to a disconnected client. -->
                <Logger level="ERROR" name="org.apache.cxf.jaxrs.sse.SseEventSinkImpl"/>

                <!-- Filters known issues of KarServiceImpl, see -->
                <!-- https://github.com/openhab/openhab-distro/issues/519#issuecomment-351944506 -->
                <!-- Can be removed once the issues are resolved in an upcoming version. -->
                <Logger level="ERROR" name="org.apache.karaf.kar.internal.KarServiceImpl"/>

                <!-- Filters warnings about unavailable ciphers when JCE is not installed, see -->
                <!-- https://github.com/openhab/openhab-distro/issues/999 -->
                <Logger level="ERROR" name="org.apache.karaf.shell.ssh.SshUtils"/>

                <!-- Filters known issues of javax.mail, see -->
                <!-- https://github.com/openhab/openhab-addons/issues/5530 -->
                <Logger level="ERROR" name="javax.mail"/>

                <!-- Filters disconnection warnings of the ChromeCast Java API, see -->
                <!-- https://github.com/openhab/openhab-addons/issues/3770 -->
                <Logger level="ERROR" name="su.litvak.chromecast.api.v2.Channel"/>

                <!-- Added by Karaf to prevent debug logging loops, see -->
                <!-- https://issues.apache.org/jira/browse/KARAF-5559 -->
                <Logger level="WARN" name="org.apache.sshd"/>






As far as I can see that is ok.

Your system has enough free diskspace to do logging ?
I asume you use zram ? You may try to disable zram; check the permissions and then enable zram again in case you see the problem with the ownership again.

That was my wife’s first question :slight_smile:

openhabian@openhab:~ $ df
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/root       61133444 8519420  50098652  15% /
devtmpfs         1852756       0   1852756   0% /dev
tmpfs            1984852       0   1984852   0% /dev/shm
tmpfs            1984852    1496   1983356   1% /run
tmpfs               5120       8      5112   1% /run/lock
tmpfs            1984852       0   1984852   0% /sys/fs/cgroup
/dev/mmcblk0p1    258096   48799    209297  19% /boot
tmpfs             396968       0    396968   0% /run/user/1000

Should be enough space…

Was not aware I was using zram but just reading up on it I see that it’s enabled by default with the latest openhabian. How do I disable/enable that? Permissions on the log file are still openhab, no changes… but still no updates to it. But then, I’m getting nothing happening on my z-wave network either… The dev debug still shows when I select things on the iOS app but no real actions… Let’s get the logs going first :slight_smile:

This is driving me nuts… so for some unknown reason the logs have now started to work again…

But… now all my z-wave is offline… Z-wave things in OH3 are showing as ERROR:BRIDGE and “Controller is offline”. I’m using a z-stick via a USB hub into the Pi4. It’s still connected and the LED is cycling colour… Any thoughts?

At least the logs are back… wish I knew what that was about?

First the disclaimers; not an expert: have had a beer (or two):

Write down the exact ID of the zstick that is not working. Delete thing; Go to inbox: Zwave binding: manually add bridge; Change the ID to exactly what you had before (or everything will be messed up-seriously): click add; See what happens, should come back: If not check log: Repeat with Zwave in Debug mode.

One reality check. Before you do any of that get out of OH UI and go back. Sometimes the bridge error is transitory and you just need to refresh the page refresh.