Log filtering in OH 3

As I learned the log configuration format has been changed between version 2.5 and 3.0

How could I adapt the following configuration to the new XML-based format?

# custom filtering rules
log4j2.appender.event.filter.uselessevents.type = RegexFilter
log4j2.appender.event.filter.uselessevents.regex = .*(heartBeat|LastUpdate|lastUpdate|LetzteAktualisierung|Uptime|Laufzeit|ZuletztGesehen).*
log4j2.appender.event.filter.uselessevents.onMatch = DENY
log4j2.appender.event.filter.uselessevents.onMisMatch = NEUTRAL
2 Likes

It’s pretty easy to adapt using the Log4j2 configuration documentation:

that brings us to

  • changing userdata/etc/log4j2.xml
  • insert “…” under RollingFile openhab.log and/or events.log

correct?

...
		<!-- Rolling file appender -->
		<RollingFile fileName="${sys:openhab.logdir}/openhab.log" filePattern="${sys:openhab.logdir}/openhab.log.%i" name="LOGFILE">
			<PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} [%-5.5p] [%-36.36c] - %m%n"/>
            <RegexFilter regex=".*(heartBeat|LastUpdate|lastUpdate|LetzteAktualisierung|Uptime|Laufzeit|ZuletztGesehen).*" onMatch="DENY" onMismatch="NEUTRAL"/>
			<Policies>
				<SizeBasedTriggeringPolicy size="16 MB"/>
			</Policies>
		</RollingFile>
...
5 Likes

Yes that’s it! :slight_smile:

Hello I tried below script , but the items
UG_Waschkueche_Trockner_CurrTime and EG_Kueche_Spuelm_CurrTime
still in the event log
Have i made a new appendar ?

<!-- Event log appender -->
		<RollingRandomAccessFile fileName="${sys:openhab.logdir}/events.log" filePattern="${sys:openhab.logdir}/events.log.%i" name="EVENT">
			<PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} [%-5.5p] [%-36.36c] - %m%n"/>
			<RegexFilter regex=".*(UG_Waschkueche_Trockner_CurrTime|EG_Kueche_Spuelm_CurrTime).*" onMatch="ACCEPT" onMismatch="NEUTRAL"/>
			<Policies>
				<OnStartupTriggeringPolicy/>
				<SizeBasedTriggeringPolicy size="16 MB"/>
			</Policies>
		</RollingRandomAccessFile>

try onMatch=“DENY”

Hello Markus,
i’m absolutly sure does have i tried this :thinking:, but never mind now it works

The logic is:

  • if the pattern matches DENY writing the log
  • onMismatch=NEUTRAL means continue processing rules, if there are no more output to logfile
  • vs. onMismatch=NEUTRAL means: don’t check for other rules in the chain, just write to log

I had some trouble getting this to work, even after touching the org.ops4j.pax.logging.cfg file to have OH pick up the changes. It’ll only update when I fully restart the openHAB service. Not great, but once I have all of my filters in it’s not a big deal.

Hmm, is this really correctly explained? A bit confusing, at least for me.

Hi,
trying to set a filter in the new openHAB3 filter config file but I simply cannot get it to filter anything. The entries in the log below is what I am trying to filter out.

System:
openHAB 3.01 official release
OS: Windows 10

From what I understand I should add something along this line in the log4j2.xml file after PatternLayout:

<RegexFilter onMatch="DENY" onMismatch="ACCEPT" regex=".*(Curtains).*"/>

I have also tried NEUTRAL instead of ACCEPT - no change. I have also tried to put the regex="… parameter just after RegexFilter. Same result.

I have restarted OH several times. I put the line in several appenders: Console appender, Rolling file appender and Event log appender, just to try and get it to work as expected. But I fail and now I don’t really know how to go on. I have read the posts about how to append and for me it looks right. But OH probably only does what I tell it to do, it is me not knowing how to define it correctly.

Am I doing something wrong that possibly is obvious to someone else? Some input would be appreciated.

The complete config file (my changes are in the beginning):

<?xml version="1.0" encoding="UTF-8" standalone="no"?><Configuration monitorInterval="30">

	<Appenders>
		<!-- Console appender not used by default (see Root logger AppenderRefs) -->
		<Console name="STDOUT">
			<PatternLayout pattern="%d{HH:mm:ss.SSS} [%-5.5p] [%-36.36c] - %m%n"/>
			<RegexFilter onMatch="DENY" onMismatch="ACCEPT" regex=".*(Curtains).*"/>
		</Console>

		<!-- Rolling file appender -->
		<RollingFile fileName="${sys:openhab.logdir}/openhab.log" filePattern="${sys:openhab.logdir}/openhab.log.%i" name="LOGFILE">
			<PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} [%-5.5p] [%-36.36c] - %m%n"/>
			<RegexFilter onMatch="DENY" onMismatch="ACCEPT" regex=".*(Curtains).*"/>
			<Policies>
				<OnStartupTriggeringPolicy/>
				<SizeBasedTriggeringPolicy size="16 MB"/>
			</Policies>
		</RollingFile>

		<!-- Event log appender -->
		<RollingRandomAccessFile fileName="${sys:openhab.logdir}/events.log" filePattern="${sys:openhab.logdir}/events.log.%i" name="EVENT">
			<PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} [%-5.5p] [%-36.36c] - %m%n"/>
			<RegexFilter onMatch="DENY" onMismatch="ACCEPT" regex=".*(Curtains).*"/>
			<Policies>
				<OnStartupTriggeringPolicy/>
				<SizeBasedTriggeringPolicy size="16 MB"/>
			</Policies>
		</RollingRandomAccessFile>

		<!-- Audit file appender -->
		<RollingRandomAccessFile fileName="${sys:openhab.logdir}/audit.log" filePattern="${sys:openhab.logdir}/audit.log.%i" name="AUDIT">
			<PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} [%-5.5p] [%-36.36c] - %m%n"/>
			<Policies>
				<OnStartupTriggeringPolicy/>
				<SizeBasedTriggeringPolicy size="8 MB"/>
			</Policies>
		</RollingRandomAccessFile>

		<!-- OSGi appender -->
		<PaxOsgi filter="*" name="OSGI"/>
	</Appenders>

	<Loggers>
		<!-- Root logger configuration -->
		<Root level="WARN">
			<AppenderRef ref="LOGFILE"/>
			<AppenderRef ref="OSGI"/>
		</Root>

		<!-- Karaf Shell logger -->
		<Logger level="OFF" name="org.apache.karaf.shell.support">
			<AppenderRef ref="STDOUT"/>
		</Logger>

		<!-- Security audit logger -->
		<Logger additivity="false" level="INFO" name="org.apache.karaf.jaas.modules.audit">
			<AppenderRef ref="AUDIT"/>
		</Logger>

		<!-- openHAB specific logger configuration -->

		<Logger level="INFO" name="org.openhab"/>

		<Logger level="ERROR" name="openhab.event.ItemStateEvent"/>
		<Logger level="ERROR" name="openhab.event.ItemAddedEvent"/>
		<Logger level="ERROR" name="openhab.event.ItemRemovedEvent"/>
		<Logger level="ERROR" name="openhab.event.ItemChannelLinkAddedEvent"/>
		<Logger level="ERROR" name="openhab.event.ItemChannelLinkRemovedEvent"/>
		<Logger level="ERROR" name="openhab.event.ThingStatusInfoEvent"/>
		<Logger level="ERROR" name="openhab.event.ThingAddedEvent"/>
		<Logger level="ERROR" name="openhab.event.ThingUpdatedEvent"/>
		<Logger level="ERROR" name="openhab.event.ThingRemovedEvent"/>
		<Logger level="ERROR" name="openhab.event.InboxUpdatedEvent"/>
		<Logger level="ERROR" name="openhab.event.RuleStatusInfoEvent"/>
		<Logger level="ERROR" name="openhab.event.RuleAddedEvent"/>
		<Logger level="ERROR" name="openhab.event.RuleRemovedEvent"/>
		<Logger level="ERROR" name="openhab.event.StartlevelEvent"/>
		<Logger level="ERROR" name="openhab.event.AddonEvent"/>

		<Logger additivity="false" level="INFO" name="openhab.event">
			<AppenderRef ref="EVENT"/>
			<AppenderRef ref="OSGI"/>
		</Logger>

		<Logger level="ERROR" name="javax.jmdns"/>
		<Logger level="ERROR" name="org.jupnp"/>

		<!-- This suppresses all Maven download issues from the log when doing feature installations -->
		<!-- as we are logging errors ourselves in a nicer way anyhow. -->
		<Logger level="ERROR" name="org.ops4j.pax.url.mvn.internal.AetherBasedResolver"/>

		<!-- Filters known issues of pax-web (issue link to be added here). -->
		<!-- Can be removed once the issues are resolved in an upcoming version. -->
		<Logger level="OFF" name="org.ops4j.pax.web.pax-web-runtime"/>

		<!-- Filters known issues of lsp4j, see -->
		<!-- https://github.com/eclipse/smarthome/issues/4639 -->
		<!-- https://github.com/eclipse/smarthome/issues/4629 -->
		<!-- https://github.com/eclipse/smarthome/issues/4643 -->
		<!-- Can be removed once the issues are resolved in an upcoming version. -->
		<Logger level="OFF" name="org.eclipse.lsp4j"/>

		<!-- Filters warnings for events that could not be delivered to a disconnected client. -->
		<Logger level="ERROR" name="org.apache.cxf.jaxrs.sse.SseEventSinkImpl"/>

		<!-- Filters known issues of KarServiceImpl, see -->
		<!-- https://github.com/openhab/openhab-distro/issues/519#issuecomment-351944506 -->
		<!-- Can be removed once the issues are resolved in an upcoming version. -->
		<Logger level="ERROR" name="org.apache.karaf.kar.internal.KarServiceImpl"/>

		<!-- Filters warnings about unavailable ciphers when JCE is not installed, see -->
		<!-- https://github.com/openhab/openhab-distro/issues/999 -->
		<Logger level="ERROR" name="org.apache.karaf.shell.ssh.SshUtils"/>

		<!-- Filters known issues of javax.mail, see -->
		<!-- https://github.com/openhab/openhab-addons/issues/5530 -->
		<Logger level="ERROR" name="javax.mail"/>

		<!-- Filters disconnection warnings of the ChromeCast Java API, see -->
		<!-- https://github.com/openhab/openhab-addons/issues/3770 -->
		<Logger level="ERROR" name="su.litvak.chromecast.api.v2.Channel"/>

		<!-- Added by Karaf to prevent debug logging loops, see -->
		<!-- https://issues.apache.org/jira/browse/KARAF-5559 -->
		<Logger level="ERROR" name="org.apache.sshd"/>
		<Logger level="INFO" name="org.openhab.binding.shelly"/>
	</Loggers>

</Configuration>

These are the events I try to filter out when I have log:tail running in the cmd-window of Karaf:

16:25:46.190 [INFO ] [openhab.event.ItemStateChangedEvent  ] - Item 'Curtains' changed from 0.0 to 100.0
16:25:48.518 [INFO ] [openhab.event.ItemStateChangedEvent  ] - Item 'Curtains' changed from 100.0 to 0.0
16:26:01.196 [INFO ] [openhab.event.ItemStateChangedEvent  ] - Item 'Curtains' changed from 0.0 to 100.0
16:26:04.045 [INFO ] [openhab.event.ItemStateChangedEvent  ] - Item 'Curtains' changed from 100.0 to 0.0
16:26:17.548 [INFO ] [openhab.event.ItemStateChangedEvent  ] - Item 'Curtains' changed from 0.0 to 100.0
16:26:19.355 [INFO ] [openhab.event.ItemStateChangedEvent  ] - Item 'Curtains' changed from 100.0 to 0.0
16:27:02.285 [INFO ] [openhab.event.ItemStateChangedEvent  ] - Item 'Curtains' changed from 0.0 to 100.0

This one is actually the one used in karaf console. See http://karaf.922171.n3.nabble.com/PaxOsgi-Appender-td4048647.html

Since that one does not have the filter in place, you still get the log entries in karaf console.

However, the log files should not have the lines anymore.

You only need the RegexFilter to be defined for the file you’re trying to modify. So for events.log, it’s the <!-- Event log appender -->. Remove it everyywhere else you have it.

You should also change onMismatch="ACCEPT" to onMismatch="NEUTRAL".

If you have entries in openhab.log that you also want to filter, add a similar filter line to <!-- Rolling file appender -->

I’m not sure if the order matters in a RegexFilter, but in mine I have them ordered as regex, onMatch, and onMismatch.

Than you both of you for your replies! Unfortunately the OSGI log does not accept a regex filter but I think this is up to me to research. The parameters seems to be different in this case. I will look into it - but big thanks for helping me a bit on the way, much appreciated!

If that is the case, you can also add the filter to corresponding Logger, I.e adding RegexFilter to

<Logger additivity="false" level="INFO" name="openhab.event">
			<AppenderRef ref="EVENT"/>
			<AppenderRef ref="OSGI"/>
                        <RegexFilter... />
		</Logger>

Wow, that worked like a charm - thank you very much, much appreciated! My knowledge about the logging is obviously far from sufficient.

Thanks again :slight_smile:

I want to filter certain events from the openHAB log, but can’t make it work

I see the following entries in events.log

2021-04-24 11:15:05.670 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'PondPump1_HeartBeat' changed from 2021-04-24T11:14:51.000+0200 to 2021-04-24T11:15:05.000+0200
2021-04-24 11:15:05.679 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'PondPump1_PumpWatts' changed from 97.11 W to 97.06 W
2021-04-24 11:15:06.399 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'PondPump1_HeartBeat' changed from 2021-04-24T11:15:05.000+0200 to 2021-04-24T11:15:06.000+0200
2021-04-24 11:15:07.751 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'GasSensorJonietz_HeartBeat' changed from 2021-04-24T11:14:57.000+0200 to 2021-04-24T11:15:07.000+0200
2021-04-24 11:15:08.075 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'PondLights_HeartBeat' changed from 2021-04-24T11:14:54.000+0200 to 2021-04-24T11:15:08.000+0200
2021-04-24 11:15:09.919 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'PondLights_HeartBeat' changed from 2021-04-24T11:15:08.000+0200 to 2021-04-24T11:15:09.000+0200
2021-04-24 11:15:10.390 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'Teich4Terrassensteckdose1_HeartBeat' changed from 2021-04-24T11:14:55.000+0200 to 2021-04-24T11:15:10.000+0200
2021-04-24 11:15:11.073 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'PondControl_Watts' changed from 192.75 W to 192.58 W
2021-04-24 11:15:11.075 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'PondControl_TotalWatts' changed from 2.650 kWh to 2.653 kWh

and want to filter those.

I changed event log appender in log2j2.xml

		<!-- Event log appender -->
		<RollingRandomAccessFile fileName="${sys:openhab.logdir}/events.log" filePattern="${sys:openhab.logdir}/events.log.%i" name="EVENT">
			<RegexFilter regex=".*_HeartBeat|LastEvent|PumpWatts|_Watts|_TotalWatts|_Signal|KitchenRadio_InfoText.*" onMatch="DENY" onMismatch="NEUTRAL"/>
			<PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} [%-5.5p] [%-36.36c] - %m%n"/>
			<Policies>
				<OnStartupTriggeringPolicy/>
				<SizeBasedTriggeringPolicy size="4 MB"/>
			</Policies>
		</RollingRandomAccessFile>

butt still see those events.

I used https://regex101.com to verify the regex pattern and it brings up 9 matches as expected

I also tried

		<Logger additivity="false" level="INFO" name="openhab.event">
			<AppenderRef ref="EVENT"/>
			<AppenderRef ref="OSGI"/>
			<RegexFilter onMatch="DENY" onMismatch="ACCEPT" regex=".*_HeartBeat|LastEvent|PumpWatts|_Watts|_TotalWatts|_Signal|KitchenRadio_InfoText.*"/>
		</Logger>

I also restarted OH to make sure that changes become active.

What’s wrong?

I’m confused. Didn’t you start this thread and have it working at one point? Mine is based on what you previously posted, and it works.

It looks like you’ve made a lot of changes, and somewhere along the way you removed the parentheses from your RegexFilter. That’s most likely the reason why it’s not doing anything.

<RegexFilter regex=".*_Heartbeat|etc.*"

Should be:

<RegexFilter regex=".*(_Heartbeat|etc).*"

Beyond the regex, I note that:

  1. The RegexFilter line has been moved ahead of PatternLayout.
  2. The SizeBasedTriggeringPolicy has changed from 16 MB to 4 MB.

I don’t know if these are problems, but I’m curious as to why you made these edits.

hehe, me too, I had it working, but for some reason…

I noticed that the first example above defines the filter for LOGFILE. Now my understanding is that this doesn’t make sense, because those events are written to events.log/EVENT so I moved the line (after it didn’t worked as expected). Then I tried @ssalonen’s approach, but this also doesn’t work.

I tried with and without brackets. In fact () puts the result in group 1. Log4j – Log4j Filters shows an example without (), but I tried both.

This drives nuts :slight_smile:

@ssalonen @wborn Do you have an idea what’s wrong with my config above?

So even if you revert to what was working, it won’t work any more?